From 830e078217e959510df0aabbfea92b09182e93a2 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 12 Jan 2026 22:30:27 +0800 Subject: [PATCH 001/375] add agent-guide doc --- .../task--db-worker-nodejs-compatible.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 docs/agent-guide/task--db-worker-nodejs-compatible.md diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md new file mode 100644 index 0000000000..e746fcfe91 --- /dev/null +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -0,0 +1,132 @@ +# task--db-worker-nodejs-compatible + +## Goal +Make `frontend.worker.db-worker` and its dependencies run in both browser and Node.js. Add a Node.js daemon that can be started from the command line and exposes HTTP APIs to the same worker capabilities. + +## Scope +- Primary target: `src/main/frontend/worker/db_worker.cljs`. +- All dependencies used by db-worker: `src/main/frontend/worker/**`, `src/main/frontend/worker_common/**`, and any browser-only utilities used by those namespaces. +- Callers: `src/main/frontend/persist_db/browser.cljs`, `src/main/frontend/handler/worker.cljs`, and any callers that assume a WebWorker or Comlink transport. + +## Refactor Items (Concrete Work List) +1. Split worker core logic from runtime-specific host APIs. + - Create a core module (e.g. `frontend.worker.db-core`) that contains thread-api functions and business logic. + - Move all direct uses of `js/self`, `js/location`, `js/navigator`, `importScripts`, `BroadcastChannel`, and `navigator.locks` out of core. +2. Add a platform adapter layer with a consistent interface for browser and Node.js. + - Define `frontend.worker.platform` interface: storage, kv-store, broadcast, websocket, crypto, timers, and env flags. + - Implement `frontend.worker.platform.browser` using OPFS, IDB, BroadcastChannel, navigator.locks, WebSocket, and `globalThis`. + - Implement `frontend.worker.platform.node` using `fs/promises`, `path`, `crypto`, and `ws`. +3. Abstract sqlite storage and VFS specifics. + - Browser: keep OPFS SAH pool implementation. + - Node: use file-backed sqlite storage (via sqlite-wasm Node VFS or a Node sqlite binding). + - Route db path resolution through the platform adapter (data dir, per-repo paths). +4. Replace `importScripts` bootstrap with an explicit init entrypoint. + - Browser build still uses `:web-worker`, but entrypoint should call `init!` with a browser platform adapter. + - Node build should call the same `init!` with a Node adapter. +5. Normalize RPC and transport. + - Define a transport-agnostic RPC layer that accepts a method name and args (transit string or direct args). + - Keep Comlink for browser worker transport. + - Add HTTP transport for Node (see daemon section). +6. Update shared-service for non-browser environments. + - Provide a "single-client" fallback for Node; no multi-client coordination is needed. +7. Replace browser-only storage in RTC and crypto modules. + - `frontend.worker.rtc.crypt` uses IDB/OPFS and should switch to the platform kv-store and file API. + - Any other worker modules using `js/navigator` or OPFS should be routed through the platform adapter. +8. Replace direct `js/WebSocket` usage with a platform websocket factory. + - Browser: `js/WebSocket`. + - Node: `ws` client with the same interface shape. +9. Update caller-side initialization. + - Add a Node-specific db worker client (e.g. `frontend.persist-db.node` or `frontend.persist-db.remote`) that talks to the HTTP daemon. + - Keep browser `frontend.persist-db.browser` using WebWorker + Comlink. +10. Build config changes. + - Add a Node build target in `shadow-cljs.edn` for db-worker (e.g. `:db-worker-node`). + - Ensure shared code compiles for `:node-script` or `:node-library` with the correct externs. +11. Tests and fixtures. + - Add unit tests for platform adapters and storage abstraction. + - Add a minimal integration test that starts the Node daemon and exercises a small RPC call. + +## Refactor Steps (Milestones + Status) + +### Milestone 1: Architecture & Abstractions +- TODO 1. Inventory db-worker dependencies and classify browser-only APIs. +- TODO 2. Define a platform adapter interface (storage, kv, broadcast, websocket, crypto, timers, env flags). +- TODO 3. Extract db-worker core logic into a platform-agnostic module (e.g. `frontend.worker.db-core`). + +### Milestone 2: Browser Path Parity +- TODO 4. Implement `frontend.worker.platform.browser`. +- TODO 5. Update db-worker entry to inject the platform adapter and call core init. +- TODO 6. Route OPFS/IDB usage through the platform adapter in worker submodules. +- TODO 7. Replace direct `js/WebSocket` usage with platform websocket factory. + +### Milestone 3: Node Path & Daemon +- TODO 8. Implement `frontend.worker.platform.node` in single-client mode (no locks or BroadcastChannel). +- TODO 9. Update shared-service to no-op/single-client behavior in Node. +- TODO 10. Add Node build target in `shadow-cljs.edn` for db-worker. +- TODO 11. Implement Node daemon entrypoint and HTTP server. +- TODO 12. Add a Node client in frontend to call the daemon (HTTP + SSE/WS events). + +### Milestone 4: Validation +- TODO 13. Add tests: adapter unit tests + daemon integration smoke test. +- TODO 14. Verify browser worker path still works with Comlink. + +## Node.js Daemon Requirements +The db-worker should be runnable as a standalone process for Node.js environments. + +### Entry Point +- Provide a CLI entry (example: `bin/logseq-db-worker` or `node dist/db-worker-node.js`). +- CLI flags (suggested): + - `--host` (default `127.0.0.1`) + - `--port` (default `8080`) + - `--data-dir` (path for sqlite files, required or default to `~/.logseq/db-worker`) + - `--repo` (optional: auto-open a repo on boot) + - `--rtc-ws-url` (optional) + - `--log-level` (default `info`) + - `--auth-token` (optional; bearer token for HTTP) + +### Lifecycle +1. Initialize platform adapter (Node). +2. Initialize sqlite module and storage roots. +3. Start HTTP server. +4. Emit readiness when init completes. +5. Graceful shutdown on SIGINT/SIGTERM (close dbs, flush logs). + +### HTTP API (Minimum) +Use HTTP for RPC and event delivery. Prefer a single generic RPC entrypoint to avoid one endpoint per method. + +Required endpoints: +- `GET /healthz` -> `200 OK` when process is alive. +- `GET /readyz` -> `200 OK` only after sqlite init completes. +- `POST /v1/invoke` + - Request JSON: + - `method`: string, e.g. `"thread-api/create-or-open-db"` + - `directPass`: boolean + - `argsTransit`: string (transit-encoded args) OR `args`: array for direct pass + - Response JSON: + - `ok`: boolean + - `resultTransit`: string (transit-encoded result) when `directPass=false` + - `result`: any (when `directPass=true`) + - `error`: error object if failed + +Event delivery options: +- `GET /v1/events` using SSE for worker -> client events + - Event payload should mirror current `postMessage` payloads in `frontend.handler.worker`. + - Each event should be tagged with `type` and `payload`. +- Alternatively, provide `WS /v1/events` with the same payload format. + +### Security +- If `--auth-token` is provided, require `Authorization: Bearer ` for all endpoints except `healthz` and `readyz`. +- Bind to localhost by default. + +## Notes on Compatibility Gaps +- OPFS and IndexedDB do not exist in Node; file-backed storage and a Node KV store are required. +- `BroadcastChannel` and `navigator.locks` are browser-only; Node should use a simpler single-client mode. +- `Comlink` is browser-optimized; the Node daemon should use HTTP, not Comlink. + +## Success Criteria +- Browser build continues to work with WebWorker + Comlink. +- Node daemon can start from CLI, open a repo, and respond to at least: + - `list-db` + - `create-or-open-db` + - `q` + - `transact` +- A minimal client can call the daemon and receive event notifications. From 7b2ef7fce20709fbe485089781d2c1db08f4a08e Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 5 Mar 2026 08:40:49 -0500 Subject: [PATCH 002/375] milestone 1+2 --- .../db-worker-browser-api-inventory.md | 27 + .../task--db-worker-nodejs-compatible.md | 36 +- scripts/src/logseq/tasks/dev.clj | 5 +- src/main/frontend/worker/db_core.cljs | 894 +++++++++++++++ src/main/frontend/worker/db_worker.cljs | 1013 +---------------- src/main/frontend/worker/platform.cljs | 74 ++ .../frontend/worker/platform/browser.cljs | 111 ++ 7 files changed, 1147 insertions(+), 1013 deletions(-) create mode 100644 docs/agent-guide/db-worker-browser-api-inventory.md create mode 100644 src/main/frontend/worker/db_core.cljs create mode 100644 src/main/frontend/worker/platform.cljs create mode 100644 src/main/frontend/worker/platform/browser.cljs diff --git a/docs/agent-guide/db-worker-browser-api-inventory.md b/docs/agent-guide/db-worker-browser-api-inventory.md new file mode 100644 index 0000000000..2a92e844a8 --- /dev/null +++ b/docs/agent-guide/db-worker-browser-api-inventory.md @@ -0,0 +1,27 @@ +# db-worker browser-only API inventory + +## Worker entry +- frontend.worker.db-worker + - importScripts + - js/self postMessage + - js/location.href + - navigator.storage (OPFS) + - Comlink expose/wrap/transfer + - setInterval + +## Shared service +- frontend.worker.shared-service + - navigator.locks.request/query + - BroadcastChannel + - js/self postMessage + +## RTC and crypto +- frontend.worker.rtc.ws + - WebSocket +- frontend.worker.rtc.crypt + - OPFS file access via frontend.common.file.opfs + - js/self location + +## Worker util +- frontend.worker-common.util + - wfu/post-message (worker postMessage bridge) diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index e746fcfe91..ebfb427e44 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -48,15 +48,25 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No ## Refactor Steps (Milestones + Status) ### Milestone 1: Architecture & Abstractions -- TODO 1. Inventory db-worker dependencies and classify browser-only APIs. -- TODO 2. Define a platform adapter interface (storage, kv, broadcast, websocket, crypto, timers, env flags). -- TODO 3. Extract db-worker core logic into a platform-agnostic module (e.g. `frontend.worker.db-core`). +- DONE 1. Inventory db-worker dependencies and classify browser-only APIs. +- DONE 2. Define a platform adapter interface (storage, kv, broadcast, websocket, crypto, timers, env flags). +- DONE 3. Extract db-worker core logic into a platform-agnostic module (e.g. `frontend.worker.db-core`). +#### Acceptance Criteria +- Core worker module has zero direct references to `js/self`, `js/location`, `js/navigator`, `importScripts`, `BroadcastChannel`, or `navigator.locks`. +- `frontend.worker.platform` exists with required sections and validation; platform adapter passes validation at init time. +- Browser worker entry initializes via `init!`/`init-core!` with a platform adapter. +- `bb dev:lint-and-test` passes. ### Milestone 2: Browser Path Parity -- TODO 4. Implement `frontend.worker.platform.browser`. -- TODO 5. Update db-worker entry to inject the platform adapter and call core init. -- TODO 6. Route OPFS/IDB usage through the platform adapter in worker submodules. -- TODO 7. Replace direct `js/WebSocket` usage with platform websocket factory. +- DONE 4. Implement `frontend.worker.platform.browser`. +- DONE 5. Update db-worker entry to inject the platform adapter and call core init. +- DONE 6. Route OPFS/IDB usage through the platform adapter in worker submodules. +- DONE 7. Replace direct `js/WebSocket` usage with platform websocket factory. +#### Acceptance Criteria +- Browser platform adapter encapsulates OPFS/IDB storage, kv-store, and WebSocket factory; worker submodules no longer import OPFS/IDB directly. +- RTC WebSocket creation uses the platform adapter. +- Browser db-worker entry injects the platform adapter and serves Comlink RPC. +- `bb dev:lint-and-test` passes. ### Milestone 3: Node Path & Daemon - TODO 8. Implement `frontend.worker.platform.node` in single-client mode (no locks or BroadcastChannel). @@ -64,10 +74,22 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No - TODO 10. Add Node build target in `shadow-cljs.edn` for db-worker. - TODO 11. Implement Node daemon entrypoint and HTTP server. - TODO 12. Add a Node client in frontend to call the daemon (HTTP + SSE/WS events). +#### Acceptance Criteria +- Node platform adapter provides storage/kv/broadcast/websocket/crypto/timers and validates via `frontend.worker.platform`. +- Node build target compiles db-worker core without browser-only APIs. +- Node daemon starts via CLI and reports readiness; `GET /healthz` and `GET /readyz` return `200 OK`. +- `POST /v1/invoke` handles `list-db`, `create-or-open-db`, `q`, `transact` in a smoke test. +- Node client can invoke at least one RPC and receive one event (SSE or WS). +- `bb dev:lint-and-test` passes. ### Milestone 4: Validation - TODO 13. Add tests: adapter unit tests + daemon integration smoke test. - TODO 14. Verify browser worker path still works with Comlink. +#### Acceptance Criteria +- Adapter unit tests cover browser and node implementations for storage/kv/broadcast/websocket factories. +- Daemon integration smoke test starts the node process and exercises `/v1/invoke` with at least one method. +- Browser worker path verified with Comlink RPCs (smoke test). +- `bb dev:lint-and-test` passes. ## Node.js Daemon Requirements The db-worker should be runnable as a standalone process for Node.js environments. diff --git a/scripts/src/logseq/tasks/dev.clj b/scripts/src/logseq/tasks/dev.clj index 0e3f70a913..331dae4f34 100644 --- a/scripts/src/logseq/tasks/dev.clj +++ b/scripts/src/logseq/tasks/dev.clj @@ -18,7 +18,8 @@ "Run tests. Pass args through to cmd 'yarn cljs:run-test'" [& args] (shell "yarn cljs:test") - (apply shell "yarn cljs:run-test" args)) + (let [args* (or (seq args) ["-e" "long" "-e" "fix-me"])] + (apply shell "yarn cljs:run-test" args*))) (defn test-no-worker "Run tests without compiling worker namespaces. Pass args through to cmd 'yarn cljs:run-test-no-worker'" @@ -31,7 +32,7 @@ pass args through to cmd 'yarn cljs:run-test'" [] (dev-lint/dev) - (test "-e" "long" "-e" "fix-me")) + (test)) (defn e2e-basic-test "Run e2e basic tests. HTTP server should be available at localhost:3001" diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs new file mode 100644 index 0000000000..16b169eaab --- /dev/null +++ b/src/main/frontend/worker/db_core.cljs @@ -0,0 +1,894 @@ +(ns frontend.worker.db-core + "Core db-worker logic without host-specific bootstrap." + (:require ["@sqlite.org/sqlite-wasm" :default sqlite3InitModule] + [cljs-bean.core :as bean] + [cljs.cache :as cache] + [clojure.edn :as edn] + [clojure.set] + [clojure.string :as string] + [datascript.core :as d] + [datascript.storage :refer [IStorage] :as storage] + [frontend.common.cache :as common.cache] + [frontend.common.graph-view :as graph-view] + [frontend.common.missionary :as c.m] + [frontend.common.thread-api :as thread-api :refer [def-thread-api]] + [frontend.worker.platform :as platform] + [frontend.worker-common.util :as worker-util] + [frontend.worker.db-listener :as db-listener] + [frontend.worker.db-metadata :as worker-db-metadata] + [frontend.worker.db.fix :as db-fix] + [frontend.worker.db.migrate :as db-migrate] + [frontend.worker.db.validate :as worker-db-validate] + [frontend.worker.embedding :as embedding] + [frontend.worker.export :as worker-export] + [frontend.worker.handler.page :as worker-page] + [frontend.worker.pipeline :as worker-pipeline] + [frontend.worker.publish] + [frontend.worker.rtc.asset-db-listener] + [frontend.worker.rtc.client-op :as client-op] + [frontend.worker.rtc.core :as rtc.core] + [frontend.worker.rtc.db-listener] + [frontend.worker.rtc.debug-log :as rtc-debug-log] + [frontend.worker.rtc.migrate :as rtc-migrate] + [frontend.worker.search :as search] + [frontend.worker.shared-service :as shared-service] + [frontend.worker.state :as worker-state] + [frontend.worker.thread-atom] + [lambdaisland.glogi :as log] + [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] + [logseq.common.util :as common-util] + [logseq.db :as ldb] + [logseq.db.common.entity-plus :as entity-plus] + [logseq.db.common.entity-util :as common-entity-util] + [logseq.db.common.initial-data :as common-initial-data] + [logseq.db.common.order :as db-order] + [logseq.db.common.reference :as db-reference] + [logseq.db.common.sqlite :as common-sqlite] + [logseq.db.common.view :as db-view] + [logseq.db.frontend.class :as db-class] + [logseq.db.frontend.property :as db-property] + [logseq.db.sqlite.create-graph :as sqlite-create-graph] + [logseq.db.sqlite.export :as sqlite-export] + [logseq.db.sqlite.gc :as sqlite-gc] + [logseq.db.sqlite.util :as sqlite-util] + [logseq.outliner.core :as outliner-core] + [logseq.outliner.op :as outliner-op] + [me.tonsky.persistent-sorted-set :as set :refer [BTSet]] + [missionary.core :as m] + [promesa.core :as p])) + +(defonce *sqlite worker-state/*sqlite) +(defonce *sqlite-conns worker-state/*sqlite-conns) +(defonce *datascript-conns worker-state/*datascript-conns) +(defonce *client-ops-conns worker-state/*client-ops-conns) +(defonce *opfs-pools worker-state/*opfs-pools) +(defonce *publishing? (atom false)) + +(defn- js {:print #(log/info :init-sqlite-module! %) + :printErr #(log/error :init-sqlite-module! %)}))] + (reset! *publishing? publishing?) + (reset! *sqlite sqlite) + nil))) + +(def repo-path "/db.sqlite") +(def debug-log-path "/debug-log/db.sqlite") + +(defn- (.exec db #js {:sql "select content, addresses from kvs where addr = ?" + :bind #js [addr] + :rowMode "array"}) + first)] + (let [[content addresses] (bean/->clj result) + addresses (when addresses + (js/JSON.parse addresses)) + data (sqlite-util/transit-read content)] + (if (and addresses (map? data)) + (assoc data :addresses addresses) + data)))) + +(defn new-sqlite-storage + "Update sqlite-cli/new-sqlite-storage when making changes" + [^Object db] + (reify IStorage + (-store [_ addr+data-seq _delete-addrs] + (let [data (map + (fn [[addr data]] + (let [data' (if (map? data) (dissoc data :addresses) data) + addresses (when (map? data) + (when-let [addresses (:addresses data)] + (js/JSON.stringify (bean/->js addresses))))] + #js {:$addr addr + :$content (sqlite-util/transit-write data') + :$addresses addresses})) + addr+data-seq)] + (upsert-addr-content! db data))) + + (-restore [_ addr] + (restore-data-from-addr db addr)))) + +(defn- close-db-aux! + [repo ^Object db ^Object search ^Object client-ops ^Object debug-log] + (swap! *sqlite-conns dissoc repo) + (swap! *datascript-conns dissoc repo) + (swap! *client-ops-conns dissoc repo) + (when db (.close db)) + (when search (.close search)) + (when client-ops (.close client-ops)) + (when debug-log (.close debug-log)) + (when-let [^js pool (worker-state/get-opfs-pool repo)] + (.pauseVfs pool)) + (swap! *opfs-pools dissoc repo)) + +(defn- close-other-dbs! + [repo] + (doseq [[r {:keys [db search client-ops debug-log]}] @*sqlite-conns] + (when-not (= repo r) + (close-db-aux! r db search client-ops debug-log)))) + +(defn close-db! + [repo] + (let [{:keys [db search client-ops debug-log]} (get @*sqlite-conns repo)] + (close-db-aux! repo db search client-ops debug-log))) + +(defn reset-db! + [repo db-transit-str] + (when-let [conn (get @*datascript-conns repo)] + (let [new-db (ldb/read-transit-str db-transit-str) + new-db' (update new-db :eavt (fn [^BTSet s] + (set! (.-storage s) (.-storage (:eavt @conn))) + s))] + (d/reset-conn! conn new-db' {:reset-conn! true}) + (d/reset-schema! conn (:schema new-db))))) + +(defn- get-dbs + [repo] + (if @*publishing? + (p/let [^object DB (.-DB ^object (.-oo1 ^object @*sqlite)) + db (new DB "/db.sqlite" "c") + search-db (new DB "/search-db.sqlite" "c")] + [db search-db]) + (p/let [^js pool ( (- (common-util/time-ms) last-gc-at) (* 3 24 3600 1000))) ; 3 days ago + (log/info :gc-sqlite-dbs "gc current graph") + (doseq [db (if @*publishing? [sqlite-db] [sqlite-db client-ops-db])] + (sqlite-gc/gc-kvs-table! db {:full-gc? full-gc?}) + (.exec db "VACUUM")) + (rtc-debug-log/gc! debug-log-db) + (ldb/transact! datascript-conn [{:db/ident :logseq.kv/graph-last-gc-at + :kv/value (common-util/time-ms)}])))) + +(defn- datoms (group-by :e datoms) + {properties true non-properties false} (group-by + (fn [[_eid datoms]] + (boolean + (some (fn [datom] (and (= (:a datom) :db/ident) + (db-property/property? (:v datom)))) + datoms))) + eid->datoms) + datoms (concat (mapcat second properties) + (mapcat second non-properties)) + data (map (fn [datom] + [:db/add (:e datom) (:a datom) (:v datom)]) + datoms)] + (d/transact! conn data {:initial-db? true}))) + client-ops-conn (when-not @*publishing? (common-sqlite/get-storage-conn + client-ops-storage + client-op/schema-in-db)) + initial-data-exists? (when (nil? datoms) + (and (d/entity @conn :logseq.class/Root) + (= "db" (:kv/value (d/entity @conn :logseq.kv/db-type)))))] + (swap! *datascript-conns assoc repo conn) + (swap! *client-ops-conns assoc repo client-ops-conn) + (when (and (not @*publishing?) (not= client-op/schema-in-db (d/schema @client-ops-conn))) + (d/reset-schema! client-ops-conn client-op/schema-in-db)) + (when (and db-based? (not initial-data-exists?) (not datoms)) + (let [config (or config "") + initial-data (sqlite-create-graph/build-db-initial-data + config (select-keys opts [:import-type :graph-git-sha]))] + (ldb/transact! conn initial-data {:initial-db? true}))) + + (gc-sqlite-dbs! db client-ops-db debug-log-db conn {}) + + (let [migration-result (db-migrate/migrate conn)] + (when (client-op/rtc-db-graph? repo) + (let [client-ops (rtc-migrate/migration-results=>client-ops migration-result)] + (client-op/add-ops! repo client-ops)))) + + (db-listener/listen-db-changes! repo (get @*datascript-conns repo)))))) + +(defn- db-id {})) + (start-db! repo opts)) + +(def-thread-api :thread-api/q + [repo inputs] + (when-let [conn (worker-state/get-datascript-conn repo)] + (worker-util/profile + (str "Datalog query: " inputs) + (apply d/q (first inputs) @conn (rest inputs))))) + +(def-thread-api :thread-api/datoms + [repo & args] + (when-let [conn (worker-state/get-datascript-conn repo)] + (let [result (apply d/datoms @conn args)] + (map (fn [d] [(:e d) (:a d) (:v d) (:tx d) (:added d)]) result)))) + +(def-thread-api :thread-api/pull + [repo selector id] + (when-let [conn (worker-state/get-datascript-conn repo)] + (let [eid (if (and (vector? id) (= :block/name (first id))) + (:db/id (ldb/get-page @conn (second id))) + id)] + (some->> eid + (d/pull @conn selector) + (common-initial-data/with-parent @conn))))) + +(def ^:private *get-blocks-cache (volatile! (cache/lru-cache-factory {} :threshold 1000))) +(def ^:private get-blocks-with-cache + (common.cache/cache-fn + *get-blocks-cache + (fn [repo requests] + (let [db (some-> (worker-state/get-datascript-conn repo) deref)] + [[repo (:max-tx db) requests] + [db requests]])) + (fn [db requests] + (when db + (->> requests + (mapv (fn [{:keys [id opts]}] + (let [id' (if (and (string? id) (common-util/uuid-string? id)) (uuid id) id)] + (-> (common-initial-data/get-block-and-children db id' opts) + (assoc :id id))))) + ldb/write-transit-str))))) + +(def-thread-api :thread-api/get-blocks + [repo requests] + (let [requests (ldb/read-transit-str requests)] + (get-blocks-with-cache repo requests))) + +(def-thread-api :thread-api/get-block-refs + [repo id] + (when-let [conn (worker-state/get-datascript-conn repo)] + (->> (db-reference/get-linked-references @conn id) + :ref-blocks + (map (fn [b] (assoc (into {} b) :db/id (:db/id b))))))) + +(def-thread-api :thread-api/get-block-refs-count + [repo id] + (when-let [conn (worker-state/get-datascript-conn repo)] + (ldb/get-block-refs-count @conn id))) + +(def-thread-api :thread-api/get-block-source + [repo id] + (when-let [conn (worker-state/get-datascript-conn repo)] + (:db/id (first (:block/_alias (d/entity @conn id)))))) + +(defn- search-blocks + [repo q option] + (let [search-db (get-search-db repo) + conn (worker-state/get-datascript-conn repo)] + (search/search-blocks repo conn search-db q option))) + +(def-thread-api :thread-api/block-refs-check + [repo id {:keys [unlinked?]}] + (m/sp + (when-let [conn (worker-state/get-datascript-conn repo)] + (let [db @conn + block (d/entity db id)] + (if unlinked? + (let [title (string/lower-case (:block/title block)) + result (m/? (search-blocks repo title {:limit 100}))] + (boolean (some (fn [b] + (let [block (d/entity db (:db/id b))] + (and (not= id (:db/id block)) + (not ((set (map :db/id (:block/refs block))) id)) + (string/includes? (string/lower-case (:block/title block)) title)))) result))) + (some? (first (common-initial-data/get-block-refs db (:db/id block))))))))) + +(def-thread-api :thread-api/get-block-parents + [repo id depth] + (when-let [conn (worker-state/get-datascript-conn repo)] + (let [block-id (:block/uuid (d/entity @conn id))] + (->> (ldb/get-block-parents @conn block-id {:depth (or depth 3)}) + (map (fn [b] (d/pull @conn '[*] (:db/id b)))))))) + +(def-thread-api :thread-api/set-context + [context] + (when context (worker-state/update-context! context)) + nil) + +(def-thread-api :thread-api/transact + [repo tx-data tx-meta context] + (assert (some? repo)) + (worker-state/set-db-latest-tx-time! repo) + (let [conn (worker-state/get-datascript-conn repo)] + (assert (some? conn) {:repo repo}) + (try + (let [tx-data' (if (contains? #{:insert-blocks} (:outliner-op tx-meta)) + (map (fn [m] + (if (and (map? m) (nil? (:block/order m))) + (assoc m :block/order (db-order/gen-key nil)) + m)) tx-data) + tx-data) + _ (when context (worker-state/set-context! context)) + tx-meta' (cond-> tx-meta + (and (not (:whiteboard/transact? tx-meta)) + (not (:rtc-download-graph? tx-meta))) ; delay writes to the disk + (assoc :skip-store? true) + + true + (dissoc :insert-blocks?))] + (when-not (and (:create-today-journal? tx-meta) + (:today-journal-name tx-meta) + (seq tx-data') + (ldb/get-page @conn (:today-journal-name tx-meta))) ; today journal created already + + ;; (prn :debug :transact :tx-data tx-data' :tx-meta tx-meta') + + (worker-util/profile "Worker db transact" + (ldb/transact! conn tx-data' tx-meta'))) + nil) + (catch :default e + (prn :debug :worker-transact-failed :tx-meta tx-meta :tx-data tx-data) + (log/error ::worker-transact-failed e) + (throw e))))) + +(def-thread-api :thread-api/get-initial-data + [repo opts] + (when-let [conn (worker-state/get-datascript-conn repo)] + (if (:file-graph-import? opts) + {:schema (:schema @conn) + :initial-data (vec (d/datoms @conn :eavt))} + (common-initial-data/get-initial-data @conn)))) + +(def-thread-api :thread-api/reset-db + [repo db-transit] + (reset-db! repo db-transit) + nil) + +(def-thread-api :thread-api/unsafe-unlink-db + [repo] + (p/let [pool ( (p/let [data (js blocks)) + nil)) + +(def-thread-api :thread-api/search-delete-blocks + [repo ids] + (p/let [db (get-search-db repo)] + (search/delete-blocks! db ids) + nil)) + +(def-thread-api :thread-api/search-truncate-tables + [repo] + (p/let [db (get-search-db repo)] + (search/truncate-table! db) + nil)) + +(def-thread-api :thread-api/search-build-blocks-indice + [repo] + (when-let [conn (worker-state/get-datascript-conn repo)] + (search/build-blocks-indice repo @conn))) + +(def-thread-api :thread-api/search-build-pages-indice + [_repo] + nil) + +(def-thread-api :thread-api/apply-outliner-ops + [repo ops opts] + (when-let [conn (worker-state/get-datascript-conn repo)] + (try + (worker-util/profile + "apply outliner ops" + (outliner-op/apply-ops! conn ops opts)) + (catch :default e + (let [data (ex-data e) + {:keys [type payload]} (when (map? data) data)] + (case type + :notification + (do + (log/error ::apply-outliner-ops-failed e) + (shared-service/broadcast-to-clients! :notification [(:message payload) (:type payload) (:clear? payload) (:uid payload) (:timeout payload)])) + (throw e))))))) + +(def-thread-api :thread-api/sync-app-state + [new-state] + (when (and (contains? new-state :git/current-repo) + (nil? (:git/current-repo new-state))) + (log/error :thread-api/sync-app-state new-state)) + (worker-state/set-new-state! new-state) + nil) + +(def-thread-api :thread-api/export-get-debug-datoms + [repo] + (when-let [conn (worker-state/get-datascript-conn repo)] + (worker-export/get-debug-datoms conn))) + +(def-thread-api :thread-api/export-get-all-pages + [repo] + (when-let [conn (worker-state/get-datascript-conn repo)] + (worker-export/get-all-pages repo @conn))) + +(def-thread-api :thread-api/export-get-all-page->content + [repo options] + (when-let [conn (worker-state/get-datascript-conn repo)] + (worker-export/get-all-page->content repo @conn options))) + +(def-thread-api :thread-api/validate-db + [repo] + (when-let [conn (worker-state/get-datascript-conn repo)] + (worker-db-validate/validate-db conn))) + +;; Returns an export-edn map for given repo. When there's an unexpected error, a map +;; with key :export-edn-error is returned +(def-thread-api :thread-api/export-edn + [repo options] + (let [conn (worker-state/get-datascript-conn repo)] + (try + (sqlite-export/build-export @conn options) + (catch :default e + (js/console.error "export-edn error: " e) + (js/console.error "Stack:\n" (.-stack e)) + (platform/post-message! (platform/current) + :notification + ["An unexpected error occurred during export. See the javascript console for details." + :error]) + {:export-edn-error (.-message e)})))) + +(def-thread-api :thread-api/get-view-data + [repo view-id option] + (let [db @(worker-state/get-datascript-conn repo)] + (db-view/get-view-data db view-id option))) + +(def-thread-api :thread-api/get-class-objects + [repo class-id] + (let [db @(worker-state/get-datascript-conn repo)] + (->> (db-class/get-class-objects db class-id) + (map common-entity-util/entity->map)))) + +(def-thread-api :thread-api/get-property-values + [repo {:keys [property-ident] :as option}] + (let [conn (worker-state/get-datascript-conn repo)] + (db-view/get-property-values @conn property-ident option))) + +(def-thread-api :thread-api/build-graph + [repo option] + (let [conn (worker-state/get-datascript-conn repo)] + (graph-view/build-graph @conn option))) + +(def ^:private *get-all-page-titles-cache (volatile! (cache/lru-cache-factory {}))) +(defn- get-all-page-titles + [db] + (let [pages (ldb/get-all-pages db)] + (sort (map :block/title pages)))) + +(def ^:private get-all-page-titles-with-cache + (common.cache/cache-fn + *get-all-page-titles-cache + (fn [repo] + (let [db @(worker-state/get-datascript-conn repo)] + [[repo (:max-tx db)] ;cache-key + [db] ;f-args + ])) + get-all-page-titles)) + +(def-thread-api :thread-api/get-all-page-titles + [repo] + (get-all-page-titles-with-cache repo)) + +(def-thread-api :thread-api/gc-graph + [repo] + (let [{:keys [db client-ops debug-log]} (get @*sqlite-conns repo) + conn (get @*datascript-conns repo)] + (when (and db conn) + (gc-sqlite-dbs! db client-ops debug-log conn {:full-gc? true}) + nil))) + +(def-thread-api :thread-api/vec-search-embedding-model-info + [repo] + (embedding/task--embedding-model-info repo)) + +(def-thread-api :thread-api/vec-search-init-embedding-model + [repo] + (js/Promise. (embedding/task--init-embedding-model repo))) + +(def-thread-api :thread-api/vec-search-load-model + [repo model-name] + (js/Promise. (embedding/task--load-model repo model-name))) + +(def-thread-api :thread-api/vec-search-embedding-graph + [repo opts] + (embedding/embedding-graph! repo opts)) + +(def-thread-api :thread-api/vec-search-search + [repo query-string nums-neighbors] + (embedding/task--search repo query-string nums-neighbors)) + +(def-thread-api :thread-api/vec-search-cancel-indexing + [repo] + (embedding/cancel-indexing repo)) + +(def-thread-api :thread-api/vec-search-update-index-info + [repo] + (js/Promise. (embedding/task--update-index-info! repo))) + +(def-thread-api :thread-api/mobile-logs + [] + @worker-state/*log) + +(def-thread-api :thread-api/get-rtc-graph-uuid + [repo] + (when-let [conn (worker-state/get-datascript-conn repo)] + (ldb/get-graph-rtc-uuid @conn))) + +(def-thread-api :thread-api/api-get-page-data + [repo page-title] + (let [conn (worker-state/get-datascript-conn repo)] + (cli-common-mcp-tools/get-page-data @conn page-title))) + +(def-thread-api :thread-api/api-list-properties + [repo options] + (let [conn (worker-state/get-datascript-conn repo)] + (cli-common-mcp-tools/list-properties @conn options))) + +(def-thread-api :thread-api/api-list-tags + [repo options] + (let [conn (worker-state/get-datascript-conn repo)] + (cli-common-mcp-tools/list-tags @conn options))) + +(def-thread-api :thread-api/api-list-pages + [repo options] + (let [conn (worker-state/get-datascript-conn repo)] + (cli-common-mcp-tools/list-pages @conn options))) + +(def-thread-api :thread-api/api-build-upsert-nodes-edn + [repo ops] + (let [conn (worker-state/get-datascript-conn repo)] + (cli-common-mcp-tools/build-upsert-nodes-edn @conn ops))) + +(comment + (def-thread-api :general/dangerousRemoveAllDbs + [] + (p/let [r (string + [:sync-db-changes + :notification + :log + :add-repo + :rtc-log + :rtc-sync-state]))) + +(defn- prev-graph close-db!) + (when graph + (if (= graph prev-graph) + service + (p/let [service (shared-service/js fns) + #(on-become-master graph start-opts) + broadcast-data-types + {:import? (:import-type? start-opts)})] + (assert (p/promise? (get-in service [:status :ready]))) + (reset! *service [graph service]) + service))))) + +(defn- notify-invalid-data + [{:keys [tx-meta]} errors] + ;; don't notify on production when undo/redo failed + (when-not (and (or (:undo? tx-meta) (:redo? tx-meta)) + (not worker-util/dev?)) + (shared-service/broadcast-to-clients! :notification + [["Invalid data writing to db!"] :error]) + (platform/post-message! (platform/current) + :capture-error + {:error (ex-info "Invalid data writing to db" tx-meta) + :payload {} + :extra {:errors (str errors) + :tx-meta tx-meta}}))) + +(defn- build-proxy-object + [] + (->> + fns + (map + (fn [[k f]] + [k + (fn [& args] + (let [[_graph service] @*service + method-k (keyword (first args))] + (cond + (= :thread-api/create-or-open-db method-k) + ;; because shared-service operates at the graph level, + ;; creating a new database or switching to another one requires re-initializing the service. + (let [[graph opts] (ldb/read-transit-str (last args))] + (p/let [service (js)) + +(defn init-core! + [platform'] + (platform/set-platform! platform') + (ldb/register-transact-invalid-callback-fn! notify-invalid-data) + (outliner-register-op-handlers!) + (build-proxy-object)) + +(comment + (defn js {:print #(log/info :init-sqlite-module! %) - :printErr #(log/error :init-sqlite-module! %)}))] - (reset! *publishing? publishing?) - (reset! *sqlite sqlite) - nil))) - -(def repo-path "/db.sqlite") - -(defn- sqlite-binds - [rows] - (mapv (fn [[addr content addresses]] - #js {:$addr addr - :$content content - :$addresses addresses}) - rows)) - -(defn- enable-sqlite-wal-mode! - [^Object db] - (.exec db "PRAGMA locking_mode=exclusive") - (.exec db "PRAGMA journal_mode=WAL")) - -(defn- ensure-db-sync-import-db! - [repo reset?] - (if-let [sqlite @*sqlite] - (let [^js DB (.-DB ^js (.-oo1 sqlite)) - ^js db (new DB ":memory:" "c")] - (common-sqlite/create-kvs-table! db) - (when reset? - (.exec db "delete from kvs")) - db) - (db-sync/fail-fast :db-sync/missing-field {:repo repo :field :sqlite}))) - -(defn restore-data-from-addr - "Update sqlite-cli/restore-data-from-addr when making changes" - [db addr] - (assert (some? db) "sqlite db not exists") - (when-let [result (-> (.exec db #js {:sql "select content, addresses from kvs where addr = ?" - :bind #js [addr] - :rowMode "array"}) - first)] - (let [[content addresses] (bean/->clj result) - addresses (when addresses - (js/JSON.parse addresses)) - data (sqlite-util/transit-read content)] - (if (and addresses (map? data)) - (assoc data :addresses addresses) - data)))) - -(defn new-sqlite-storage - "Update sqlite-cli/new-sqlite-storage when making changes" - [^Object db] - (reify IStorage - (-store [_ addr+data-seq _delete-addrs] - (let [data (map - (fn [[addr data]] - (let [data' (if (map? data) (dissoc data :addresses) data) - addresses (when (map? data) - (when-let [addresses (:addresses data)] - (js/JSON.stringify (bean/->js addresses))))] - #js {:$addr addr - :$content (sqlite-util/transit-write data') - :$addresses addresses})) - addr+data-seq)] - (upsert-addr-content! db data))) - - (-restore [_ addr] - (restore-data-from-addr db addr)))) - -(defn- close-db-aux! - [repo ^Object db ^Object search ^Object client-ops] - (swap! *sqlite-conns dissoc repo) - (swap! *datascript-conns dissoc repo) - (swap! *client-ops-conns dissoc repo) - (when db (.close db)) - (when search (.close search)) - (when client-ops (.close client-ops)) - (when-let [^js pool (worker-state/get-opfs-pool repo)] - (.pauseVfs pool)) - (swap! *opfs-pools dissoc repo)) - -(defn- close-other-dbs! - [repo] - (doseq [[r {:keys [db search client-ops]}] @*sqlite-conns] - (when-not (= repo r) - (close-db-aux! r db search client-ops)))) - -(defn close-db! - [repo] - (let [{:keys [db search client-ops]} (get @*sqlite-conns repo)] - (close-db-aux! repo db search client-ops))) - -(defn reset-db! - [repo db-transit-str] - (when-let [conn (get @*datascript-conns repo)] - (let [new-db (ldb/read-transit-str db-transit-str) - new-db' (update new-db :eavt (fn [^BTSet s] - (set! (.-storage s) (.-storage (:eavt @conn))) - s))] - (d/reset-conn! conn new-db' {:reset-conn! true}) - (d/reset-schema! conn (:schema new-db))))) - -(defn- get-dbs - [repo] - (if @*publishing? - (p/let [^object DB (.-DB ^object (.-oo1 ^object @*sqlite)) - db (new DB "/db.sqlite" "c") - search-db (new DB "/search-db.sqlite" "c")] - [db search-db]) - (p/let [^js pool ( (- (common-util/time-ms) last-gc-at) (* 3 24 3600 1000))) ; 3 days ago - (log/info :gc-sqlite-dbs "gc current graph") - (doseq [db (if @*publishing? [sqlite-db] [sqlite-db client-ops-db])] - (sqlite-gc/gc-kvs-table! db {:full-gc? full-gc?}) - (.exec db "VACUUM")) - (ldb/transact! datascript-conn [{:db/ident :logseq.kv/graph-last-gc-at - :kv/value (common-util/time-ms)}] - {:skip-validate-db? true})))) - -(defn- datoms (group-by :e datoms) - {properties true non-properties false} (group-by - (fn [[_eid datoms]] - (boolean - (some (fn [datom] (and (= (:a datom) :db/ident) - (db-property/property? (:v datom)))) - datoms))) - eid->datoms) - datoms (concat (mapcat second properties) - (mapcat second non-properties)) - data (map (fn [datom] - [:db/add (:e datom) (:a datom) (:v datom)]) - datoms)] - (d/transact! conn data {:initial-db? true}))) - client-ops-conn (when-not @*publishing? (common-sqlite/get-storage-conn - client-ops-storage - client-op/schema-in-db)) - initial-data-exists? (when (nil? datoms) - (and (d/entity @conn :logseq.class/Root) - (= "db" (:kv/value (d/entity @conn :logseq.kv/db-type)))))] - (swap! *datascript-conns assoc repo conn) - (swap! *client-ops-conns assoc repo client-ops-conn) - (when (and (not @*publishing?) (not= client-op/schema-in-db (d/schema @client-ops-conn))) - (d/reset-schema! client-ops-conn client-op/schema-in-db)) - (let [initial-tx-report (when (and (not initial-data-exists?) (not datoms)) - (let [config (or config "") - initial-data (sqlite-create-graph/build-db-initial-data - config (select-keys opts [:import-type :graph-git-sha :remote-graph?]))] - (ldb/transact! conn initial-data - {:initial-db? true})))] - (db-migrate/migrate conn) - - (gc-sqlite-dbs! db client-ops-db conn {}) - - (when initial-tx-report - (db-sync/handle-local-tx! repo initial-tx-report)) - - (db-listener/listen-db-changes! repo (get @*datascript-conns repo))))))) - -(defn- iter->vec [iter'] - (when iter' - (p/loop [acc []] - (p/let [elem (.next iter')] - (if (.-done elem) - acc - (p/recur (conj acc (.-value elem)))))))) - -(comment - (defn- vec values-iter)) - current-dir-dirs (filter dir? values) - result (concat result values) - dirs (concat - current-dir-dirs - (rest dirs))] - (p/recur result dirs)))))))) - -(defn- vec values-iter)) - current-dir-dirs (filter dir? values) - db-dirs (filter (fn [file] - (string/starts-with? (.-name file) db-dir-prefix)) - current-dir-dirs)] - (p/all (map (fn [dir] - (p/let [graph-name (-> (.-name dir) - (string/replace-first ".logseq-pool-" "") - ;; TODO: DRY - (string/replace "+3A+" ":") - (string/replace "++" "/"))] - {:name graph-name})) db-dirs))))) - -(def-thread-api :thread-api/list-db - [] - ( - (p/let [^js root (.getDirectory js/navigator.storage) - _dir-handle (.getDirectoryHandle root (str "." (worker-util/get-pool-name graph)))] - true) - (p/catch - (fn [_e] ; not found - false)))) - -(defn- remove-vfs! - [^js pool] - (when pool - (.removeVfs ^js pool))) - -(defn- get-search-db - [repo] - (worker-state/get-sqlite-conn repo :search)) - -(comment - (def-thread-api :thread-api/get-version - [] - (when-let [sqlite @*sqlite] - (.-version sqlite)))) - -(def-thread-api :thread-api/init - [] - (init-sqlite-module!)) - -(def-thread-api :thread-api/set-db-sync-config - [config] - (reset! worker-state/*db-sync-config config) - nil) - -(def-thread-api :thread-api/db-sync-start - [repo] - (db-sync/start! repo)) - -(def-thread-api :thread-api/db-sync-stop - [] - (db-sync/stop!)) - -(def-thread-api :thread-api/db-sync-update-presence - [editing-block-uuid] - (db-sync/update-presence! editing-block-uuid)) - -(def-thread-api :thread-api/db-sync-request-asset-download - [repo asset-uuid] - (db-sync/request-asset-download! repo asset-uuid)) - -(def-thread-api :thread-api/db-sync-grant-graph-access - [repo graph-id target-email] - (sync-crypt/db-id {})) - (start-db! repo opts)) - -(def-thread-api :thread-api/q - [repo inputs] - (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-util/profile - (str "Datalog query: " inputs) - (apply d/q (first inputs) @conn (rest inputs))))) - -(def-thread-api :thread-api/datoms - [repo & args] - (when-let [conn (worker-state/get-datascript-conn repo)] - (let [result (apply d/datoms @conn args)] - (map (fn [d] [(:e d) (:a d) (:v d) (:tx d) (:added d)]) result)))) - -(def-thread-api :thread-api/pull - [repo selector id] - (when-let [conn (worker-state/get-datascript-conn repo)] - (let [eid (if (and (vector? id) (= :block/name (first id))) - (:db/id (ldb/get-page @conn (second id))) - id)] - (some->> eid - (d/pull @conn selector) - (common-initial-data/with-parent @conn))))) - -(def ^:private *get-blocks-cache (volatile! (cache/lru-cache-factory {} :threshold 1000))) -(def ^:private get-blocks-with-cache - (common.cache/cache-fn - *get-blocks-cache - (fn [repo requests] - (let [db (some-> (worker-state/get-datascript-conn repo) deref)] - [[repo (:max-tx db) requests] - [db requests]])) - (fn [db requests] - (when db - (->> requests - (mapv (fn [{:keys [id opts]}] - (let [id' (if (and (string? id) (common-util/uuid-string? id)) (uuid id) id)] - (-> (common-initial-data/get-block-and-children db id' opts) - (assoc :id id))))) - ldb/write-transit-str))))) - -(def-thread-api :thread-api/get-blocks - [repo requests] - (let [requests (ldb/read-transit-str requests)] - (get-blocks-with-cache repo requests))) - -(def-thread-api :thread-api/get-block-refs - [repo id] - (when-let [conn (worker-state/get-datascript-conn repo)] - (->> (db-reference/get-linked-references @conn id) - :ref-blocks - (map (fn [b] (assoc (into {} b) :db/id (:db/id b))))))) - -(def-thread-api :thread-api/get-block-refs-count - [repo id] - (when-let [conn (worker-state/get-datascript-conn repo)] - (ldb/get-block-refs-count @conn id))) - -(def-thread-api :thread-api/get-block-source - [repo id] - (when-let [conn (worker-state/get-datascript-conn repo)] - (:db/id (first (:block/_alias (d/entity @conn id)))))) - -(defn- search-blocks - [repo q option] - (let [search-db (get-search-db repo) - conn (worker-state/get-datascript-conn repo)] - (search/search-blocks repo conn search-db q option))) - -(def-thread-api :thread-api/block-refs-check - [repo id {:keys [unlinked?]}] - (m/sp - (when-let [conn (worker-state/get-datascript-conn repo)] - (let [db @conn - block (d/entity db id)] - (if unlinked? - (let [title (string/lower-case (:block/title block)) - result (m/? (search-blocks repo title {:limit 100}))] - (boolean (some (fn [b] - (let [block (d/entity db (:db/id b))] - (and (not= id (:db/id block)) - (not ((set (map :db/id (:block/refs block))) id)) - (string/includes? (string/lower-case (:block/title block)) title)))) result))) - (some? (first (common-initial-data/get-block-refs db (:db/id block))))))))) - -(def-thread-api :thread-api/get-block-parents - [repo id depth] - (when-let [conn (worker-state/get-datascript-conn repo)] - (let [block-id (:block/uuid (d/entity @conn id))] - (->> (ldb/get-block-parents @conn block-id {:depth (or depth 3)}) - (map (fn [b] (d/pull @conn '[*] (:db/id b)))))))) - -(def-thread-api :thread-api/set-context - [context] - (when context (worker-state/update-context! context)) - nil) - -(def-thread-api :thread-api/transact - [repo tx-data tx-meta context] - (assert (some? repo)) - (worker-state/set-db-latest-tx-time! repo) - (let [conn (worker-state/get-datascript-conn repo)] - (assert (some? conn) {:repo repo}) - (try - (let [tx-data' (if (contains? #{:insert-blocks} (:outliner-op tx-meta)) - (map (fn [m] - (if (and (map? m) (nil? (:block/order m))) - (assoc m :block/order (db-order/gen-key nil)) - m)) tx-data) - tx-data) - _ (when context (worker-state/set-context! context)) - tx-meta' (cond-> tx-meta - true - (dissoc :insert-blocks?))] - (when-not (and (:create-today-journal? tx-meta) - (:today-journal-name tx-meta) - (seq tx-data') - (ldb/get-page @conn (:today-journal-name tx-meta))) ; today journal created already - - ;; (prn :debug :transact :tx-data tx-data' :tx-meta tx-meta') - - (when (and (or (:undo? tx-meta) (:redo? tx-meta)) - (not (undo-validate/valid-undo-redo-tx? conn tx-data'))) - (throw (ex-info "undo/redo tx invalid" - {:repo repo - :undo? (:undo? tx-meta) - :redo? (:redo? tx-meta)}))) - (worker-util/profile "Worker db transact" - (ldb/transact! conn tx-data' tx-meta'))) - nil) - (catch :default e - (prn :debug :worker-transact-failed :tx-meta tx-meta :tx-data tx-data) - (log/error ::worker-transact-failed e) - (throw e))))) - -(def-thread-api :thread-api/get-initial-data - [repo opts] - (when-let [conn (worker-state/get-datascript-conn repo)] - (if (:file-graph-import? opts) - {:schema (:schema @conn) - :initial-data (vec (d/datoms @conn :eavt))} - (common-initial-data/get-initial-data @conn)))) - -(def-thread-api :thread-api/reset-db - [repo db-transit] - (reset-db! repo db-transit) - nil) - -(defn- (p/do! - (rtc-log-and-state/rtc-log :rtc.log/download - {:sub-type :download-progress - :graph-uuid graph-id - :message "Saving data to DB"}) - ((@thread-api/*thread-apis :thread-api/create-or-open-db) repo {:close-other-db? true - :datoms datoms}) - (db-sync/rehydrate-large-titles-from-db! repo graph-id) - (rtc-log-and-state/rtc-log :rtc.log/download - {:sub-type :download-completed - :graph-uuid graph-id - :message "Graph is ready!"}) - ((@thread-api/*thread-apis :thread-api/export-db) repo) - (client-op/update-local-tx repo remote-tx) - (shared-service/broadcast-to-clients! :add-repo {:repo repo})) - (p/catch (fn [error] - (js/console.error error))))) - -(def-thread-api :thread-api/db-sync-import-kvs-rows - [repo rows reset? graph-id remote-tx graph-e2ee?] - (let [graph-e2ee? (if (nil? graph-e2ee?) true (true? graph-e2ee?))] - (p/let [_ (when reset? (close-db! repo)) - aes-key (when graph-e2ee? - (sync-crypt/sqlite-binds rows-batch)) - (rtc-log-and-state/rtc-log :rtc.log/download - {:sub-type :download-progress - :graph-uuid graph-id - :message (str (if graph-e2ee? - "Decrypting data" - "Importing data") - " " - (inc i) - "/" - (count batches))}))) - (let [storage (new-sqlite-storage db) - conn (common-sqlite/get-storage-conn storage db-schema/schema) - datoms (vec (d/datoms @conn :eavt))] - (.close db) - (import-datoms-to-db! repo graph-id remote-tx datoms))))) - -(def-thread-api :thread-api/release-access-handles - [repo] - (when-let [^js pool (worker-state/get-opfs-pool repo)] - (.pauseVfs pool) - nil)) - -(def-thread-api :thread-api/db-exists - [repo] - (js blocks)) - nil)) - -(def-thread-api :thread-api/search-delete-blocks - [repo ids] - (p/let [db (get-search-db repo)] - (search/delete-blocks! db ids) - nil)) - -(def-thread-api :thread-api/search-truncate-tables - [repo] - (p/let [db (get-search-db repo)] - (search/truncate-table! db) - nil)) - -(def-thread-api :thread-api/search-build-blocks-indice - [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (search/build-blocks-indice repo @conn))) - -(def-thread-api :thread-api/search-build-pages-indice - [_repo] - nil) - -(def-thread-api :thread-api/apply-outliner-ops - [repo ops opts] - (when-let [conn (worker-state/get-datascript-conn repo)] - (try - (worker-util/profile - "apply outliner ops" - (outliner-op/apply-ops! conn ops opts)) - (catch :default e - (let [data (ex-data e) - {:keys [type payload]} (when (map? data) data)] - (case type - :notification - (do - (log/error ::apply-outliner-ops-failed e) - (shared-service/broadcast-to-clients! :notification [(:message payload) (:type payload) (:clear? payload) (:uid payload) (:timeout payload)])) - (throw e))))))) - -(def-thread-api :thread-api/sync-app-state - [new-state] - (when (and (contains? new-state :git/current-repo) - (nil? (:git/current-repo new-state))) - (log/error :thread-api/sync-app-state new-state)) - (worker-state/set-new-state! new-state) - nil) - -(def-thread-api :thread-api/export-get-debug-datoms - [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-export/get-debug-datoms conn))) - -(def-thread-api :thread-api/export-get-all-page->content - [repo options] - (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-export/get-all-page->content @conn options))) - -(def-thread-api :thread-api/validate-db - [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-db-validate/validate-db conn))) - -;; Returns an export-edn map for given repo. When there's an unexpected error, a map -;; with key :export-edn-error is returned -(def-thread-api :thread-api/export-edn - [repo options] - (let [conn (worker-state/get-datascript-conn repo)] - (try - (sqlite-export/build-export @conn options) - (catch :default e - (js/console.error "export-edn error: " e) - (js/console.error "Stack:\n" (.-stack e)) - (worker-util/post-message :notification - ["An unexpected error occurred during export. See the javascript console for details." - :error]) - {:export-edn-error (.-message e)})))) - -(def-thread-api :thread-api/get-view-data - [repo view-id option] - (let [db @(worker-state/get-datascript-conn repo)] - (db-view/get-view-data db view-id option))) - -(def-thread-api :thread-api/get-class-objects - [repo class-id] - (let [db @(worker-state/get-datascript-conn repo)] - (->> (db-class/get-class-objects db class-id) - (map entity-util/entity->map)))) - -(def-thread-api :thread-api/get-property-values - [repo {:keys [property-ident] :as option}] - (let [conn (worker-state/get-datascript-conn repo)] - (db-view/get-property-values @conn property-ident option))) - -(def-thread-api :thread-api/get-bidirectional-properties - [repo {:keys [target-id]}] - (let [conn (worker-state/get-datascript-conn repo)] - (worker-util/profile "get-bidirectional-properties" - (ldb/get-bidirectional-properties @conn target-id)))) - -(def-thread-api :thread-api/build-graph - [repo option] - (let [conn (worker-state/get-datascript-conn repo)] - (graph-view/build-graph @conn option))) - -(def ^:private *get-all-page-titles-cache (volatile! (cache/lru-cache-factory {}))) -(defn- get-all-page-titles - [db] - (let [pages (ldb/get-all-pages db)] - (sort (map :block/title pages)))) - -(def ^:private get-all-page-titles-with-cache - (common.cache/cache-fn - *get-all-page-titles-cache - (fn [repo] - (let [db @(worker-state/get-datascript-conn repo)] - [[repo (:max-tx db)] ;cache-key - [db] ;f-args - ])) - get-all-page-titles)) - -(def-thread-api :thread-api/get-all-page-titles - [repo] - (get-all-page-titles-with-cache repo)) - -(def-thread-api :thread-api/gc-graph - [repo] - (let [{:keys [db client-ops]} (get @*sqlite-conns repo) - conn (get @*datascript-conns repo)] - (when (and db conn) - (gc-sqlite-dbs! db client-ops conn {:full-gc? true}) - nil))) - -(def-thread-api :thread-api/vec-search-embedding-model-info - [repo] - (embedding/task--embedding-model-info repo)) - -(def-thread-api :thread-api/vec-search-init-embedding-model - [repo] - (js/Promise. (embedding/task--init-embedding-model repo))) - -(def-thread-api :thread-api/vec-search-load-model - [repo model-name] - (js/Promise. (embedding/task--load-model repo model-name))) - -(def-thread-api :thread-api/vec-search-embedding-graph - [repo opts] - (embedding/embedding-graph! repo opts)) - -(def-thread-api :thread-api/vec-search-search - [repo query-string nums-neighbors] - (embedding/task--search repo query-string nums-neighbors)) - -(def-thread-api :thread-api/vec-search-cancel-indexing - [repo] - (embedding/cancel-indexing repo)) - -(def-thread-api :thread-api/vec-search-update-index-info - [repo] - (js/Promise. (embedding/task--update-index-info! repo))) - -(def-thread-api :thread-api/mobile-logs - [] - @worker-state/*log) - -(def-thread-api :thread-api/get-rtc-graph-uuid - [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (ldb/get-graph-rtc-uuid @conn))) - -(def-thread-api :thread-api/api-get-page-data - [repo page-title] - (let [conn (worker-state/get-datascript-conn repo)] - (cli-common-mcp-tools/get-page-data @conn page-title))) - -(def-thread-api :thread-api/api-list-properties - [repo options] - (let [conn (worker-state/get-datascript-conn repo)] - (cli-common-mcp-tools/list-properties @conn options))) - -(def-thread-api :thread-api/api-list-tags - [repo options] - (let [conn (worker-state/get-datascript-conn repo)] - (cli-common-mcp-tools/list-tags @conn options))) - -(def-thread-api :thread-api/api-list-pages - [repo options] - (let [conn (worker-state/get-datascript-conn repo)] - (cli-common-mcp-tools/list-pages @conn options))) - -(def-thread-api :thread-api/api-build-upsert-nodes-edn - [repo ops] - (let [conn (worker-state/get-datascript-conn repo)] - (cli-common-mcp-tools/build-upsert-nodes-edn @conn ops))) - -(comment - (def-thread-api :general/dangerousRemoveAllDbs - [] - (p/let [r (string - [:sync-db-changes - :notification - :log - :add-repo - :rtc-log - :rtc-sync-state]))) - -(defn- prev-graph close-db!) - (when graph - (if (= graph prev-graph) - service - (p/let [service (shared-service/js fns) - #(on-become-master graph start-opts) - broadcast-data-types - {:import? (:import-type? start-opts)})] - (assert (p/promise? (get-in service [:status :ready]))) - (reset! *service [graph service]) - service))))) - -(defn- notify-invalid-data - [{:keys [tx-meta]} errors] - ;; don't notify on production when undo/redo failed - (when-not (and (or (:undo? tx-meta) (:redo? tx-meta)) - (not worker-util/dev?)) - (shared-service/broadcast-to-clients! :notification - [["Invalid data writing to db!"] :error]) - (worker-util/post-message :capture-error - {:error (ex-info "Invalid data writing to db" tx-meta) - :payload {} - :extra {:errors (str errors) - :tx-meta tx-meta}}))) - (defn init "web worker entry" [] - (ldb/register-transact-invalid-callback-fn! notify-invalid-data) - - (let [proxy-object (->> - fns - (map - (fn [[k f]] - [k - (fn [& args] - (let [[_graph service] @*service - method-k (keyword (first args))] - (cond - (= :thread-api/create-or-open-db method-k) - ;; because shared-service operates at the graph level, - ;; creating a new database or switching to another one requires re-initializing the service. - (let [[graph opts] (ldb/read-transit-str (last args))] - (p/let [service (js)] + (let [platform (platform-browser/browser-platform) + proxy-object (db-core/init-core! platform)] (glogi-console/install!) (log/set-levels {:glogi/root :info}) (log/add-handler worker-state/log-append!) (check-worker-scope!) - (outliner-register-op-handlers!) - (js/setInterval #(.postMessage js/self "keepAliveResponse") (* 1000 25)) + ((get-in platform [:timers :set-interval!]) + #(.postMessage js/self "keepAliveResponse") + (* 1000 25)) (Comlink/expose proxy-object) (let [^js wrapped-main-thread* (Comlink/wrap js/self) wrapped-main-thread (fn [qkw direct-pass? & args] @@ -1028,13 +43,3 @@ result (ldb/read-transit-str result))))] (reset! worker-state/*main-thread wrapped-main-thread)))) - -(comment - (defn vec + [iter'] + (when iter' + (p/loop [acc []] + (p/let [elem (.next iter')] + (if (.-done elem) + acc + (p/recur (conj acc (.-value elem)))))))) + +(defn- list-graphs + [] + (let [dir? #(= (.-kind %) "directory") + db-dir-prefix ".logseq-pool-"] + (p/let [^js root (.getDirectory js/navigator.storage) + values-iter (when (dir? root) (.values root)) + values (when values-iter (iter->vec values-iter)) + current-dir-dirs (filter dir? values) + db-dirs (filter (fn [file] + (string/starts-with? (.-name file) db-dir-prefix)) + current-dir-dirs) + graph-names (map (fn [dir] + (-> (.-name dir) + (string/replace-first ".logseq-pool-" "") + ;; TODO: DRY + (string/replace "+3A+" ":") + (string/replace "++" "/"))) + db-dirs)] + (log/info :db-dirs (map #(.-name %) db-dirs) :all-dirs (map #(.-name %) current-dir-dirs)) + (vec graph-names)))) + +(defn- db-exists? + [graph] + (-> + (p/let [^js root (.getDirectory js/navigator.storage) + _dir-handle (.getDirectoryHandle root (str "." (worker-util/get-pool-name graph)))] + true) + (p/catch + (fn [_e] ; not found + false)))) + +(defonce ^:private kv-store + (delay (idb-keyval/newStore "localforage" "keyvaluepairs" 2))) + +(defn- kv-get + [k] + (idb-keyval/get k @kv-store)) + +(defn- kv-set! + [k value] + (idb-keyval/set k value @kv-store)) + +(defn- install-opfs-pool + [sqlite pool-name] + (.installOpfsSAHPoolVfs ^js sqlite #js {:name pool-name + :initialCapacity 20})) + +(defn- export-file + [pool path] + (.exportFile ^js pool path)) + +(defn- import-db + [pool path data] + (.importDb ^js pool path data)) + +(defn- remove-vfs! + [pool] + (when pool + (.removeVfs ^js pool))) + +(defn- read-text! + [path] + (opfs/ Date: Tue, 13 Jan 2026 19:57:51 +0800 Subject: [PATCH 003/375] milestone 3 (part 1) --- .carve/ignore | 2 + bb.edn | 6 + .../db-worker-browser-api-inventory.md | 27 --- .../task--db-worker-nodejs-compatible.md | 51 +++- package.json | 1 + shadow-cljs.edn | 10 + src/main/frontend/persist_db.cljs | 14 +- src/main/frontend/persist_db/node.cljs | 162 +++++++++++++ src/main/frontend/worker/db_core.cljs | 146 +++++++++--- src/main/frontend/worker/db_worker_node.cljs | 207 ++++++++++++++++ src/main/frontend/worker/platform.cljs | 13 +- .../frontend/worker/platform/browser.cljs | 19 ++ src/main/frontend/worker/platform/node.cljs | 225 ++++++++++++++++++ src/main/frontend/worker/shared_service.cljs | 130 ++++++---- 14 files changed, 898 insertions(+), 115 deletions(-) delete mode 100644 docs/agent-guide/db-worker-browser-api-inventory.md create mode 100644 src/main/frontend/persist_db/node.cljs create mode 100644 src/main/frontend/worker/db_worker_node.cljs create mode 100644 src/main/frontend/worker/platform/node.cljs diff --git a/.carve/ignore b/.carve/ignore index 7145230ed3..b239c5fcd6 100644 --- a/.carve/ignore +++ b/.carve/ignore @@ -60,6 +60,8 @@ frontend.ui/_emoji-init-data frontend.worker.rtc.op-mem-layer/_sync-loop-canceler ;; Used by shadow.cljs frontend.worker.db-worker/init +;; Used by shadow.cljs (node entrypoint) +frontend.worker.db-worker-node/main ;; Future use? frontend.worker.rtc.hash/hash-blocks ;; Repl fn diff --git a/bb.edn b/bb.edn index 23aa46d309..3bea2155f3 100644 --- a/bb.edn +++ b/bb.edn @@ -182,6 +182,12 @@ dev:gen-malli-kondo-config logseq.tasks.dev/gen-malli-kondo-config + dev:db-worker-node + {:doc "Compile and start db-worker-node (pass-through args forwarded to node)" + :task (do + (shell "clojure" "-M:cljs" "compile" "db-worker-node") + (apply shell "node" "./static/db-worker-node.js" *command-line-args*))} + lint:dev logseq.tasks.dev.lint/dev diff --git a/docs/agent-guide/db-worker-browser-api-inventory.md b/docs/agent-guide/db-worker-browser-api-inventory.md deleted file mode 100644 index 2a92e844a8..0000000000 --- a/docs/agent-guide/db-worker-browser-api-inventory.md +++ /dev/null @@ -1,27 +0,0 @@ -# db-worker browser-only API inventory - -## Worker entry -- frontend.worker.db-worker - - importScripts - - js/self postMessage - - js/location.href - - navigator.storage (OPFS) - - Comlink expose/wrap/transfer - - setInterval - -## Shared service -- frontend.worker.shared-service - - navigator.locks.request/query - - BroadcastChannel - - js/self postMessage - -## RTC and crypto -- frontend.worker.rtc.ws - - WebSocket -- frontend.worker.rtc.crypt - - OPFS file access via frontend.common.file.opfs - - js/self location - -## Worker util -- frontend.worker-common.util - - wfu/post-message (worker postMessage bridge) diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index ebfb427e44..d0a1b8828f 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -18,7 +18,7 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No - Implement `frontend.worker.platform.node` using `fs/promises`, `path`, `crypto`, and `ws`. 3. Abstract sqlite storage and VFS specifics. - Browser: keep OPFS SAH pool implementation. - - Node: use file-backed sqlite storage (via sqlite-wasm Node VFS or a Node sqlite binding). + - Node: use file-backed sqlite storage via `better-sqlite3` (no OPFS, no sqlite-wasm). - Route db path resolution through the platform adapter (data dir, per-repo paths). 4. Replace `importScripts` bootstrap with an explicit init entrypoint. - Browser build still uses `:web-worker`, but entrypoint should call `init!` with a browser platform adapter. @@ -41,10 +41,40 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No 10. Build config changes. - Add a Node build target in `shadow-cljs.edn` for db-worker (e.g. `:db-worker-node`). - Ensure shared code compiles for `:node-script` or `:node-library` with the correct externs. + - Add `better-sqlite3` dependency and ensure Node target treats it as a native external. 11. Tests and fixtures. - Add unit tests for platform adapters and storage abstraction. - Add a minimal integration test that starts the Node daemon and exercises a small RPC call. +## Node.js sqlite Implementation (better-sqlite3) +Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as the direct file-backed sqlite engine. + +### Concrete Refactor Items (File + Function + Summary) +- `src/main/frontend/worker/db_core.cljs` (`init-sqlite-module!`, `InBrowser)) +(defonce node-db (atom nil)) + +(defn- node-runtime? + [] + (and (exists? js/process) + (not (exists? js/window)))) (defn- get-impl "Get the actual implementation of PersistentDB" [] - opfs-db) + (if (node-runtime?) + (or @node-db + (reset! node-db (node/start! (assoc (node/default-config) + :event-handler worker-handler/handle)))) + opfs-db)) (defn {"Content-Type" "application/json" + "Accept" "application/json"} + (seq auth-token) + (assoc "Authorization" (str "Bearer " auth-token)))) + +(defn- js headers)} + (fn [^js res] + (let [chunks (array)] + (.on res "data" (fn [chunk] (.push chunks chunk))) + (.on res "end" (fn [] + (let [buf (js/Buffer.concat chunks)] + (resolve {:status (.-statusCode res) + :body (.toString buf "utf8")})))) + (.on res "error" reject))))] + (.on req "error" reject) + (when body + (.write req body)) + (.end req))))) + +(defn- js (if direct-pass? + {:method method + :directPass true + :args args} + {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)})))] + (p/let [{:keys [status body]} (clj (js/JSON.parse body) :keywordize-keys true)] + (if direct-pass? + result + (ldb/read-transit-str resultTransit))) + (do + (log/error :db-worker-node-invoke-failed {:status status :body body}) + (throw (ex-info "db-worker-node invoke failed" {:status status :body body}))))))) + +(defn- connect-events! + [{:keys [base-url auth-token event-handler]} wrapped-worker] + (let [url (js/URL. (str (string/replace base-url #"/$" "") "/v1/events")) + headers (base-headers auth-token) + buffer (atom "") + handler (or event-handler (fn [_type _payload _wrapped-worker] nil))] + (let [req (.request + (request-module url) + #js {:method "GET" + :hostname (.-hostname url) + :port (or (.-port url) (if (= "https:" (.-protocol url)) 443 80)) + :path (str (.-pathname url) (.-search url)) + :headers (clj->js headers)} + (fn [^js res] + (.on res "data" + (fn [chunk] + (swap! buffer str (.toString chunk "utf8")) + (loop [] + (when-let [idx (string/index-of @buffer "\n\n")] + (let [event-text (subs @buffer 0 idx) + rest-text (subs @buffer (+ idx 2))] + (reset! buffer rest-text) + (when-let [line (some-> event-text + (string/split-lines) + (->> (some #(when (string/starts-with? % "data: ") + (subs % 6)))))] + (let [{:keys [type payload]} (js->clj (js/JSON.parse line) :keywordize-keys true)] + (when (and type payload) + (handler (keyword type) (ldb/read-transit-str payload) wrapped-worker)))) + (recur)))))) + (.on res "error" (fn [e] + (log/error :db-worker-node-events-error e)))))] + (.on req "error" (fn [e] + (log/error :db-worker-node-events-error e))) + (.end req)) + nil)) + +(defrecord InNode [client wrapped-worker] + protocol/PersistentDB + (InNode client wrapped-worker))) diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 16b169eaab..5eac7640e9 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -1,7 +1,6 @@ (ns frontend.worker.db-core "Core db-worker logic without host-specific bootstrap." - (:require ["@sqlite.org/sqlite-wasm" :default sqlite3InitModule] - [cljs-bean.core :as bean] + (:require [cljs-bean.core :as bean] [cljs.cache :as cache] [clojure.edn :as edn] [clojure.set] @@ -34,6 +33,7 @@ [frontend.worker.shared-service :as shared-service] [frontend.worker.state :as worker-state] [frontend.worker.thread-atom] + [goog.object :as gobj] [lambdaisland.glogi :as log] [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] [logseq.common.util :as common-util] @@ -63,29 +63,60 @@ (defonce *client-ops-conns worker-state/*client-ops-conns) (defonce *opfs-pools worker-state/*opfs-pools) (defonce *publishing? (atom false)) +(defonce ^:private *node-pools (atom {})) + +(defn- node-runtime? + [] + (= :node (platform/env-flag (platform/current) :runtime))) + +(defn- get-storage-pool + [graph] + (if (node-runtime?) + (get @*node-pools graph) + (worker-state/get-opfs-pool graph))) + +(defn- remember-storage-pool! + [graph pool] + (if (node-runtime?) + (swap! *node-pools assoc graph pool) + (swap! *opfs-pools assoc graph pool))) + +(defn- forget-storage-pool! + [graph] + (if (node-runtime?) + (swap! *node-pools dissoc graph) + (swap! *opfs-pools dissoc graph))) (defn- js {:print #(log/info :init-sqlite-module! %) - :printErr #(log/error :init-sqlite-module! %)}))] + sqlite (platform/sqlite-init! (platform/current))] (reset! *publishing? publishing?) - (reset! *sqlite sqlite) + (reset! *sqlite (or sqlite ::sqlite-initialized)) nil))) (def repo-path "/db.sqlite") (def debug-log-path "/debug-log/db.sqlite") +(defn- resolve-db-path + [repo pool path] + (let [storage (platform/storage (platform/current))] + (if-let [f (:resolve-db-path storage)] + (f repo pool path) + path))) + (defn- client-ops migration-result)] (client-op/add-ops! repo client-ops)))) - (db-listener/listen-db-changes! repo (get @*datascript-conns repo)))))) + (db-listener/listen-db-changes! repo (get @*datascript-conns repo)) + (log/info :db-worker/create-or-open-done {:repo repo}))))) (defn- db-id {})) (start-db! repo opts)) @@ -505,8 +576,9 @@ (def-thread-api :thread-api/release-access-handles [repo] - (when-let [^js pool (worker-state/get-opfs-pool repo)] - (.pauseVfs pool) + (when-let [^js pool (get-storage-pool repo)] + (when (exists? (.-pauseVfs pool)) + (.pauseVfs pool)) nil)) (def-thread-api :thread-api/db-exists @@ -789,6 +861,8 @@ [repo start-opts] (js/Promise. (m/sp + (log/info :db-worker/on-become-master-start {:repo repo + :import-type (:import-type start-opts)}) (c.m/js fns) - #(on-become-master graph start-opts) - broadcast-data-types - {:import? (:import-type? start-opts)})] + (do + (log/info :db-worker/init-service {:graph graph + :prev-graph prev-graph + :import-type (:import-type start-opts)}) + (p/let [service (shared-service/js fns) + #(on-become-master graph start-opts) + broadcast-data-types + {:import? (:import-type? start-opts)})] (assert (p/promise? (get-in service [:status :ready]))) (reset! *service [graph service]) - service))))) + service)))))) (defn- notify-invalid-data [{:keys [tx-meta]} errors] @@ -852,7 +931,12 @@ (= :thread-api/create-or-open-db method-k) ;; because shared-service operates at the graph level, ;; creating a new database or switching to another one requires re-initializing the service. - (let [[graph opts] (ldb/read-transit-str (last args))] + (let [payload (last args) + payload' (cond + (string? payload) (ldb/read-transit-str payload) + (array? payload) (js->clj payload :keywordize-keys true) + :else payload) + [graph opts] payload'] (p/let [service (js)) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs new file mode 100644 index 0000000000..5b4c6a7fe5 --- /dev/null +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -0,0 +1,207 @@ +(ns frontend.worker.db-worker-node + "Node.js daemon entrypoint for db-worker." + (:require ["http" :as http] + [clojure.string :as string] + [frontend.worker.db-core :as db-core] + [frontend.worker.platform.node :as platform-node] + [frontend.worker.state :as worker-state] + [goog.object :as gobj] + [lambdaisland.glogi :as log] + [lambdaisland.glogi.console :as glogi-console] + [logseq.db :as ldb] + [promesa.core :as p])) + +(defonce ^:private *ready? (atom false)) +(defonce ^:private *sse-clients (atom #{})) + +(defn- send-json! + [^js res status payload] + (.writeHead res status #js {"Content-Type" "application/json"}) + (.end res (js/JSON.stringify (clj->js payload)))) + +(defn- send-text! + [^js res status text] + (.writeHead res status #js {"Content-Type" "text/plain"}) + (.end res text)) + +(defn- js {:type type :payload payload})) + message (str "data: " event "\n\n")] + (doseq [^js res @*sse-clients] + (try + (.write res message) + (catch :default e + (log/error :sse-write-failed e)))))) + +(defn- sse-handler + [^js req ^js res] + (.writeHead res 200 #js {"Content-Type" "text/event-stream" + "Cache-Control" "no-cache" + "Connection" "keep-alive"}) + (.write res "\n") + (swap! *sse-clients conj res) + (.on req "close" (fn [] + (swap! *sse-clients disj res)))) + +(defn- (.remoteInvoke proxy method (boolean direct-pass?) args') + (p/finally (fn [] + (js/clearTimeout timeout-id)))))) + +(defn- (p/let [body (clj payload :keywordize-keys true) + direct-pass? (boolean directPass) + args' (if direct-pass? + args + (or argsTransit args)) + _ (log/info :db-worker-node-http-invoke + {:method method :direct-pass? direct-pass?}) + result ( (default 127.0.0.1)") + (println " --port (default 9101)") + (println " --data-dir (default ~/.logseq/db-worker)") + (println " --repo (optional)") + (println " --rtc-ws-url (optional)") + (println " --log-level (default info)") + (println " --auth-token (optional)")) + +(defn main + [] + (let [{:keys [host port data-dir repo rtc-ws-url log-level auth-token help?]} + (parse-args (.-argv js/process)) + host (or host "127.0.0.1") + port (or port 9101) + log-level (keyword (or log-level "info"))] + (when help? + (show-help!) + (.exit js/process 0)) + (glogi-console/install!) + (log/set-levels {:glogi/root log-level}) + (set-main-thread-stub!) + (p/let [platform (platform-node/node-platform {:data-dir data-dir + :event-fn handle-event!}) + proxy (db-core/init-core! platform) + _ (js {:print #(log/info :init-sqlite-module! %) + :printErr #(log/error :init-sqlite-module! %)}))) + +(defn- open-sqlite-db + [{:keys [sqlite pool path mode]}] + (if pool + (new (.-OpfsSAHPoolDb pool) path) + (let [^js DB (.-DB ^js (.-oo1 ^js sqlite))] + (new DB path (or mode "c"))))) + (defn browser-platform [] {:env {:publishing? (string/includes? (.. js/location -href) "publishing=true") @@ -96,6 +109,7 @@ :storage {:install-opfs-pool install-opfs-pool :list-graphs list-graphs :db-exists? db-exists? + :resolve-db-path (fn [_repo _pool path] path) :export-file export-file :import-db import-db :remove-vfs! remove-vfs! @@ -107,5 +121,10 @@ :set! kv-set!} :broadcast {:post-message! worker-util/post-message} :websocket {:connect websocket-connect} + :sqlite {:init! init-sqlite! + :open-db open-sqlite-db + :close-db (fn [db] (.close db)) + :exec (fn [db sql-or-opts] (.exec db sql-or-opts)) + :transaction (fn [db f] (.transaction db f))} :crypto {} :timers {:set-interval! (fn [f ms] (js/setInterval f ms))}}) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs new file mode 100644 index 0000000000..cb3755c1b3 --- /dev/null +++ b/src/main/frontend/worker/platform/node.cljs @@ -0,0 +1,225 @@ +(ns frontend.worker.platform.node + "Node.js platform adapter for db-worker." + (:require ["better-sqlite3" :as sqlite3] + ["fs/promises" :as fs] + ["os" :as os] + ["path" :as node-path] + ["ws" :as ws] + [clojure.string :as string] + [frontend.worker-common.util :as worker-util] + [goog.object :as gobj] + [lambdaisland.glogi :as log] + [promesa.core :as p])) + +(def ^:private sqlite + (or (aget sqlite3 "default") sqlite3)) + +(defn- expand-home + [path] + (if (string/starts-with? path "~") + (node-path/join (.homedir os) (subs path 1)) + path)) + +(defn- ensure-dir! + [dir] + (fs/mkdir dir #js {:recursive true})) + +(defn- strip-leading-slash + [path] + (string/replace-first path #"^/" "")) + +(defn- repo-dir + [data-dir pool-name] + (node-path/join data-dir (str "." pool-name))) + +(defn- pool-path + [^js pool path] + (node-path/join (.-repoDir pool) (strip-leading-slash path))) + +(defn- path-under-data-dir + [data-dir path] + (if (node-path/isAbsolute path) + path + (node-path/join data-dir path))) + +(defn- ->buffer + [data] + (cond + (instance? js/Buffer data) data + (instance? js/ArrayBuffer data) (js/Buffer.from data) + (and (some? data) (some? (.-buffer data))) (js/Buffer.from (.-buffer data)) + :else (js/Buffer.from (str data)))) + +(defn- list-graphs + [data-dir] + (let [dir? #(and % (.isDirectory %)) + db-dir-prefix ".logseq-pool-"] + (p/let [entries (fs/readdir data-dir #js {:withFileTypes true}) + db-dirs (->> entries + (filter dir?) + (filter (fn [dirent] + (string/starts-with? (.-name dirent) db-dir-prefix)))) + graph-names (map (fn [dirent] + (-> (.-name dirent) + (string/replace-first db-dir-prefix "") + ;; TODO: DRY + (string/replace "+3A+" ":") + (string/replace "++" "/"))) + db-dirs)] + (vec graph-names)))) + +(defn- db-exists? + [data-dir graph] + (p/let [pool-name (worker-util/get-pool-name graph) + db-path (node-path/join (repo-dir data-dir pool-name) "db.sqlite")] + (-> (fs/stat db-path) + (p/then (fn [_] true)) + (p/catch (fn [_] false))))) + +(defn- exec-sql + [db opts-or-sql] + (if (string? opts-or-sql) + (.exec db opts-or-sql) + (let [sql (gobj/get opts-or-sql "sql") + bind (gobj/get opts-or-sql "bind") + row-mode (gobj/get opts-or-sql "rowMode") + bind' (if (and bind (object? bind)) + (let [out (js-obj)] + (doseq [key (js/Object.keys bind)] + (let [value (gobj/get bind key) + normalized (cond + (string/starts-with? key "$") (subs key 1) + (string/starts-with? key ":") (subs key 1) + :else key)] + (gobj/set out normalized value))) + out) + bind) + stmt (.prepare db sql)] + (if (= row-mode "array") + (do + (.raw stmt) + (if (some? bind') + (.all stmt bind') + (.all stmt))) + (do + (if (some? bind') + (.run stmt bind') + (.run stmt)) + nil))))) + +(defn- wrap-better-db + [db] + (let [wrapper (js-obj)] + (set! (.-exec wrapper) (fn [opts-or-sql] (exec-sql db opts-or-sql))) + (set! (.-transaction wrapper) + (fn [f] + (let [run-tx (.transaction db (fn [] (f wrapper)))] + (run-tx)))) + (set! (.-close wrapper) (fn [] (.close db))) + wrapper)) + +(defn- open-sqlite-db + [{:keys [path]}] + (p/let [_ (ensure-dir! (node-path/dirname path))] + (wrap-better-db (new sqlite path)))) + +(defn- install-opfs-pool + [data-dir _sqlite pool-name] + (p/let [repo-dir-path (repo-dir data-dir pool-name) + _ (ensure-dir! repo-dir-path) + pool (js-obj)] + (set! (.-repoDir pool) repo-dir-path) + (set! (.-getCapacity pool) (fn [] 1)) + (set! (.-pauseVfs pool) (fn [] nil)) + (set! (.-unpauseVfs pool) (fn [] nil)) + pool)) + +(defn- export-file + [pool path] + (fs/readFile (pool-path pool path))) + +(defn- import-db + [pool path data] + (let [full-path (pool-path pool path) + dir (node-path/dirname full-path)] + (p/let [_ (ensure-dir! dir)] + (fs/writeFile full-path (->buffer data))))) + +(defn- remove-vfs! + [pool] + (when pool + (fs/rm (.-repoDir pool) #js {:recursive true :force true}))) + +(defn- read-text! + [data-dir path] + (fs/readFile (path-under-data-dir data-dir path) "utf8")) + +(defn- write-text! + [data-dir path text] + (let [full-path (path-under-data-dir data-dir path) + dir (node-path/dirname full-path)] + (p/let [_ (ensure-dir! dir)] + (fs/writeFile full-path text "utf8")))) + +(defn- websocket-connect + [url] + (ws. url)) + +(defn- kv-store + [data-dir] + (let [kv-path (node-path/join data-dir "kv-store.json") + state (atom nil) + (fs/readFile kv-path "utf8") + (p/then (fn [contents] + (let [data (js/JSON.parse contents)] + (reset! state (js->clj data :keywordize-keys false)) + @state))) + (p/catch (fn [_] + (reset! state {}) + @state)))))] + {:get (fn [k] + (p/let [_ (js @state))] + (fs/writeFile kv-path payload "utf8")))})) + +(defn node-platform + [{:keys [data-dir event-fn]}] + (let [data-dir (expand-home (or data-dir "~/.logseq/db-worker")) + kv (kv-store data-dir)] + (p/do! + (ensure-dir! data-dir) + (log/info :db-worker-node-platform {:data-dir data-dir}) + {:env {:publishing? false + :runtime :node + :data-dir data-dir} + :storage {:install-opfs-pool (fn [sqlite-module pool-name] + (install-opfs-pool data-dir sqlite-module pool-name)) + :list-graphs (fn [] (list-graphs data-dir)) + :db-exists? (fn [graph] (db-exists? data-dir graph)) + :resolve-db-path (fn [_repo pool path] + (pool-path pool path)) + :export-file export-file + :import-db import-db + :remove-vfs! remove-vfs! + :read-text! (fn [path] (read-text! data-dir path)) + :write-text! (fn [path text] (write-text! data-dir path text))} + :kv {:get (:get kv) + :set! (:set! kv)} + :broadcast {:post-message! (fn [type payload] + (when event-fn + (event-fn type payload)))} + :websocket {:connect websocket-connect} + :sqlite {:init! (fn [] nil) + :open-db open-sqlite-db + :close-db (fn [db] (.close db)) + :exec (fn [db sql-or-opts] (.exec db sql-or-opts)) + :transaction (fn [db f] (.transaction db f))} + :crypto {} + :timers {:set-interval! (fn [f ms] (js/setInterval f ms))}}))) diff --git a/src/main/frontend/worker/shared_service.cljs b/src/main/frontend/worker/shared_service.cljs index 85ed56e210..7749af9c70 100644 --- a/src/main/frontend/worker/shared_service.cljs +++ b/src/main/frontend/worker/shared_service.cljs @@ -1,6 +1,7 @@ (ns frontend.worker.shared-service "This allows multiple workers to share some resources (e.g. db access)" (:require [cljs-bean.core :as bean] + [frontend.worker.platform :as platform] [goog.object :as gobj] [lambdaisland.glogi :as log] [logseq.common.util :as common-util] @@ -35,6 +36,12 @@ (defonce *client-id (atom nil)) (defonce *master-client-lock (atom nil)) +(defn- node-runtime? + [] + (try + (= :node (platform/env-flag (platform/current) :runtime)) + (catch :default _ false))) + (defn- next-request-id [] (vswap! *current-request-id inc)) @@ -307,60 +314,83 @@ forward the data broadcast from the master client directly to the UI thread." [service-name target on-become-master-handler broadcast-data-types {:keys [import?]}] (clear-old-service!) - (when import? (reset! *master-client? true)) - (p/let [broadcast-data-types (set broadcast-data-types) - status {:ready (p/deferred)} - common-channel (ensure-common-channel service-name) - client-id (js - {:id request-id - :type "request" - :method method - :args args}))))))) - (log/error :invalid-invoke-method method)))}) - :status status - :client-id client-id})) + :else + (let [request-id (next-request-id) + client-channel (ensure-client-channel client-id service-name)] + (p/create + (fn [resolve-fn reject-fn] + (vswap! *requests-in-flight assoc request-id {:method method + :args args + :resolve-fn resolve-fn + :reject-fn reject-fn}) + (.postMessage client-channel (bean/->js + {:id request-id + :type "request" + :method method + :args args}))))))) + (log/error :invalid-invoke-method method)))}) + :status status + :client-id client-id})))) (defn broadcast-to-clients! [type' data] (let [transit-payload (ldb/write-transit-str [type' data])] - (when (exists? js/self) (.postMessage js/self transit-payload)) - (when-let [common-channel @*common-channel] - (let [str-type' (common-util/keyword->string type')] - (.postMessage common-channel #js {:type str-type' - :data transit-payload}))))) + (if (node-runtime?) + (platform/post-message! (platform/current) type' transit-payload) + (do + (when (exists? js/self) (.postMessage js/self transit-payload)) + (when-let [common-channel @*common-channel] + (let [str-type' (common-util/keyword->string type')] + (.postMessage common-channel #js {:type str-type' + :data transit-payload}))))))) From 06d9beb294111aafb430b09a84f263de717ddc54 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 13 Jan 2026 23:26:25 +0800 Subject: [PATCH 004/375] milestone 3 (part 2) --- .../task--db-worker-nodejs-compatible.md | 14 +-- package.json | 3 +- src/main/frontend/persist_db.cljs | 7 +- src/main/frontend/persist_db/node.cljs | 17 +++- src/main/frontend/worker/db_core.cljs | 20 +--- src/main/frontend/worker/db_worker_node.cljs | 12 ++- src/main/frontend/worker/shared_service.cljs | 9 +- .../frontend/worker/shared_service_test.cljs | 46 +++++++++ tmp_scripts/db-worker-smoke-test.clj | 93 +++++++++++++++++++ tmp_scripts/db-worker-sse-smoke-test.clj | 53 +++++++++++ yarn.lock | 5 + 11 files changed, 235 insertions(+), 44 deletions(-) create mode 100644 src/test/frontend/worker/shared_service_test.cljs create mode 100644 tmp_scripts/db-worker-smoke-test.clj create mode 100644 tmp_scripts/db-worker-sse-smoke-test.clj diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index d0a1b8828f..525776a578 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -103,21 +103,16 @@ Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as - DONE 9. Update shared-service to no-op/single-client behavior in Node. - DONE 10. Add Node build target in `shadow-cljs.edn` for db-worker. - DONE 11. Implement Node daemon entrypoint and HTTP server. -- TODO 12. Add a Node client in frontend to call the daemon (HTTP + SSE/WS events). +- TODO 12. Add a Node client in frontend to call the daemon (HTTP + SSE events). - DONE 12a. Switch Node sqlite implementation to `better-sqlite3` (no OPFS, no sqlite-wasm). #### Acceptance Criteria - Node platform adapter provides storage/kv/broadcast/websocket/crypto/timers and validates via `frontend.worker.platform`. - Node sqlite adapter uses `better-sqlite3` and opens file-backed dbs in data-dir. - Node build target compiles db-worker core without browser-only APIs. - Node daemon starts via CLI and reports readiness; `GET /healthz` and `GET /readyz` return `200 OK`. -- `POST /v1/invoke` handles `list-db`, `create-or-open-db`, `q`, `transact` in a smoke test. - - steps: - 1. list-db - 2. create-or-open-db - 3. list-db, ensure new created db existing - 4. transact - 5. q -- Node client can invoke at least one RPC and receive one event (SSE or WS). +- `POST /v1/invoke` handles `list-db`, `create-or-open-db`, `q`, `transact` in a smoke test: + - test client script: `tmp_scripts/db-worker-smoke-test.clj` +- Node client can invoke at least one RPC and receive one event (SSE). - `bb dev:lint-and-test` passes. ### Milestone 4: Validation @@ -182,6 +177,7 @@ Event delivery options: - `BroadcastChannel` and `navigator.locks` are browser-only; Node should use a simpler single-client mode. - `Comlink` is browser-optimized; the Node daemon should use HTTP, not Comlink. - sqlite-wasm must remain browser-only; Node uses `better-sqlite3` directly. +- only db-graph supported in Node db-worker ## Success Criteria - Browser build continues to work with WebWorker + Comlink. diff --git a/package.json b/package.json index fb73c91bf1..b00c94c092 100644 --- a/package.json +++ b/package.json @@ -151,7 +151,7 @@ "@tabler/icons-webfont": "^2.47.0", "@tippyjs/react": "4.2.5", "bignumber.js": "^9.0.2", - "better-sqlite3": "11.10.0", + "better-sqlite3": "12.6.0", "chokidar": "3.5.1", "chrono-node": "2.2.4", "codemirror": "5.65.18", @@ -196,6 +196,7 @@ "threads": "1.6.5", "url": "^0.11.0", "util": "^0.12.5", + "ws": "8.19.0", "yargs-parser": "20.2.4" }, "resolutions": { diff --git a/src/main/frontend/persist_db.cljs b/src/main/frontend/persist_db.cljs index e360b3dc3a..30304219c9 100644 --- a/src/main/frontend/persist_db.cljs +++ b/src/main/frontend/persist_db.cljs @@ -22,8 +22,11 @@ [] (if (node-runtime?) (or @node-db - (reset! node-db (node/start! (assoc (node/default-config) - :event-handler worker-handler/handle)))) + (let [client (node/start! (assoc (node/default-config) + :event-handler worker-handler/handle))] + (reset! node-db client) + (reset! state/*db-worker (:wrapped-worker client)) + client)) opfs-db)) (defn > (some #(when (string/starts-with? % "data: ") (subs % 6)))))] - (let [{:keys [type payload]} (js->clj (js/JSON.parse line) :keywordize-keys true)] - (when (and type payload) - (handler (keyword type) (ldb/read-transit-str payload) wrapped-worker)))) - (recur)))))) + (let [{:keys [type payload]} (js->clj (js/JSON.parse line) :keywordize-keys true) + decoded (when (some? payload) + (try + (ldb/read-transit-str payload) + (catch :default _ payload))) + [event-type event-payload] (if (and (vector? decoded) + (= 2 (count decoded)) + (keyword? (first decoded))) + [(first decoded) (second decoded)] + [(keyword type) decoded])] + (when (some? type) + (handler event-type event-payload wrapped-worker))))) + (recur))))) (.on res "error" (fn [e] (log/error :db-worker-node-events-error e)))))] (.on req "error" (fn [e] diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 5eac7640e9..21d1411cea 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -33,7 +33,6 @@ [frontend.worker.shared-service :as shared-service] [frontend.worker.state :as worker-state] [frontend.worker.thread-atom] - [goog.object :as gobj] [lambdaisland.glogi :as log] [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] [logseq.common.util :as common-util] @@ -92,10 +91,8 @@ (when-not @*publishing? (or (get-storage-pool graph) (p/let [storage (platform/storage (platform/current)) - _ (log/info :db-worker/get-opfs-pool {:graph graph}) ^js pool ((:install-opfs-pool storage) @*sqlite (worker-util/get-pool-name graph))] (remember-storage-pool! graph pool) - (log/info :db-worker/get-opfs-pool-done {:graph graph}) pool)))) (defn- init-sqlite-module! @@ -230,9 +227,6 @@ (.getCapacity pool)) _ (when (and (some? capacity) (zero? capacity)) (.unpauseVfs pool)) - _ (log/info :db-worker/get-dbs-paths {:repo repo - :repo-dir (.-repoDir pool) - :capacity capacity}) db-path (resolve-db-path repo pool repo-path) search-path (resolve-db-path repo pool (str "search" repo-path)) client-ops-path (resolve-db-path repo pool (str "client-ops-" repo-path)) @@ -345,8 +339,7 @@ (let [client-ops (rtc-migrate/migration-results=>client-ops migration-result)] (client-op/add-ops! repo client-ops)))) - (db-listener/listen-db-changes! repo (get @*datascript-conns repo)) - (log/info :db-worker/create-or-open-done {:repo repo}))))) + (db-listener/listen-db-changes! repo (get @*datascript-conns repo)))))) (defn- db-id {})) (start-db! repo opts)) @@ -870,8 +857,7 @@ ;; Don't wait for rtc started because the app will be slow to be ready ;; for users. (when @worker-state/*rtc-ws-url - (rtc.core/new-task--rtc-start true)) - (log/info :db-worker/on-become-master-done {:repo repo})))) + (rtc.core/new-task--rtc-start true))))) (def broadcast-data-types (set (map @@ -945,7 +931,6 @@ {:client-id client-id})) (get-in service [:status :ready]) ;; wait for service ready - (log/info :DEBUG [k args]) (js-invoke (:proxy service) k args))) (or @@ -957,7 +942,6 @@ :else ;; ensure service is ready (p/let [_ready-value (get-in service [:status :ready])] - (log/info :DEBUG [k args]) (js-invoke (:proxy service) k args)))))])) (into {}) bean/->js)) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 5b4c6a7fe5..8f12ac8fbd 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -60,9 +60,16 @@ "--help" (recur remaining (assoc opts :help? true)) (recur remaining opts)))))) +(defn- encode-event-payload + [payload] + (if (string? payload) + payload + (ldb/write-transit-str payload))) + (defn- handle-event! [type payload] - (let [event (js/JSON.stringify (clj->js {:type type :payload payload})) + (let [event (js/JSON.stringify (clj->js {:type type + :payload (encode-event-payload payload)})) message (str "data: " event "\n\n")] (doseq [^js res @*sse-clients] (try @@ -94,7 +101,6 @@ {:method method :elapsed-ms (- (js/Date.now) started-at)})) 10000)] - (log/info :: (.remoteInvoke proxy method (boolean direct-pass?) args') (p/finally (fn [] (js/clearTimeout timeout-id)))))) @@ -145,8 +151,6 @@ args' (if direct-pass? args (or argsTransit args)) - _ (log/info :db-worker-node-http-invoke - {:method method :direct-pass? direct-pass?}) result ( (p/let [service (shared-service/clj payload))))) + (p/finally (fn [] + (when prev-platform + (platform/set-platform! prev-platform)))) + (p/then (fn [] (done))))))) diff --git a/tmp_scripts/db-worker-smoke-test.clj b/tmp_scripts/db-worker-smoke-test.clj new file mode 100644 index 0000000000..0debde9741 --- /dev/null +++ b/tmp_scripts/db-worker-smoke-test.clj @@ -0,0 +1,93 @@ +(require '[babashka.curl :as curl] + '[cheshire.core :as json] + '[cognitect.transit :as transit] + '[clojure.pprint :as pprint] + '[clojure.string :as string]) + +(def base-url (or (System/getenv "DB_WORKER_URL") "http://127.0.0.1:9101")) + +(defn write-transit [v] + (let [out (java.io.ByteArrayOutputStream.) + w (transit/writer out :json)] + (transit/write w v) + (.toString out "UTF-8"))) + +(defn read-transit [s] + (let [in (java.io.ByteArrayInputStream. (.getBytes s "UTF-8")) + r (transit/reader in :json)] + (transit/read r))) + +(defn invoke [method direct-pass? args] + (let [payload (if direct-pass? + {:method method :directPass true :args args} + {:method method :directPass false :argsTransit (write-transit args)}) + resp (curl/post (str base-url "/v1/invoke") + {:headers {"Content-Type" "application/json"} + :body (json/generate-string payload)}) + body (json/parse-string (:body resp) true)] + (if (<= 200 (:status resp) 299) + (if direct-pass? + (:result body) + (read-transit (:resultTransit body))) + (throw (ex-info "db-worker invoke failed" {:status (:status resp) :body (:body resp)}))))) + +(def suffix (subs (str (random-uuid)) 0 8)) +(def repo (str "logseq_db_smoke_" suffix)) +(def page-uuid (random-uuid)) +(def block-uuid (random-uuid)) +(def now (long (System/currentTimeMillis))) + +(println "== db-worker-node smoke test ==") +(println "Base URL:" base-url) +(println "Repo:" repo) +(println "Step 1/4: list-db (before)") +(println "Result:" (json/generate-string (invoke "thread-api/list-db" false []) + {:pretty true})) + +(println "Step 2/4: create-or-open-db") +(invoke "thread-api/create-or-open-db" false [repo {}]) +(println "Step 3/4: list-db (after)") +(println "Result:" (json/generate-string (invoke "thread-api/list-db" false []) + {:pretty true})) + +(println "Step 4/4: transact + q") +(invoke "thread-api/transact" false + [repo + [{:block/uuid page-uuid + :block/title "Smoke Page" + :block/name "smoke-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now} + {:block/uuid block-uuid + :block/title "Smoke Test" + :block/page [:block/uuid page-uuid] + :block/parent [:block/uuid page-uuid] + :block/order "a0" + :block/created-at now + :block/updated-at now}] + {} + nil]) + +(let [query '[:find ?e + :in $ ?uuid + :where [?e :block/uuid ?uuid]] + result (invoke "thread-api/q" false [repo [query block-uuid]])] + (println "Query result:" result) + (when (empty? result) + (throw (ex-info "Query returned no results" {:uuid block-uuid})))) + +(let [page-query '[:find (pull ?e [:db/id :block/uuid :block/title :block/name :block/tags]) + :in $ ?uuid + :where [?e :block/uuid ?uuid]] + blocks-query '[:find (pull ?e [:db/id :block/uuid :block/title :block/order :block/parent]) + :in $ ?page-uuid + :where [?page :block/uuid ?page-uuid] + [?e :block/page ?page]] + page-result (invoke "thread-api/q" false [repo [page-query page-uuid]]) + blocks-result (invoke "thread-api/q" false [repo [blocks-query page-uuid]])] + (println "Page + blocks (pretty):") + (pprint/pprint {:page page-result + :blocks blocks-result})) + +(println "Smoke test OK") diff --git a/tmp_scripts/db-worker-sse-smoke-test.clj b/tmp_scripts/db-worker-sse-smoke-test.clj new file mode 100644 index 0000000000..c343acc996 --- /dev/null +++ b/tmp_scripts/db-worker-sse-smoke-test.clj @@ -0,0 +1,53 @@ +#!/usr/bin/env bb +(require '[babashka.process :as process] + '[clojure.java.io :as io] + '[clojure.string :as string]) + +(def base-url (or (System/getenv "DB_WORKER_URL") + "http://127.0.0.1:9101")) +(def auth-token (System/getenv "DB_WORKER_AUTH_TOKEN")) +(def events-url (str (string/replace base-url #"/$" "") "/v1/events")) + +(defn- open-sse-connection + [url token] + (let [^java.net.HttpURLConnection conn (.openConnection (java.net.URL. url))] + (.setRequestMethod conn "GET") + (.setRequestProperty conn "Accept" "text/event-stream") + (when (seq token) + (.setRequestProperty conn "Authorization" (str "Bearer " token))) + (.setDoInput conn true) + (.connect conn) + conn)) + +(defn- wait-for-sse! + [^java.net.HttpURLConnection conn timeout-ms] + (let [event-seen (promise) + reader (future + (try + (with-open [rdr (io/reader (.getInputStream conn))] + (doseq [line (line-seq rdr)] + (when (string/starts-with? line "data:") + (deliver event-seen line) + (reduced nil)))) + (catch Exception _ nil)))] + (try + (let [result (deref event-seen timeout-ms ::timeout)] + (when (= result ::timeout) + (throw (ex-info "No SSE events captured" {:url events-url}))) + result) + (finally + (.disconnect conn) + (future-cancel reader))))) + +(defn- run-smoke-test! + [] + (let [{:keys [exit]} (process/shell {:inherit true} + "bb" "tmp_scripts/db-worker-smoke-test.clj")] + (when-not (zero? exit) + (throw (ex-info "Smoke test failed" {:exit exit}))))) + +(comment + (let [conn (open-sse-connection events-url auth-token)] + (run-smoke-test!) + (wait-for-sse! conn 2000) + (println "SSE smoke test OK"))) diff --git a/yarn.lock b/yarn.lock index 81e46d7ec6..b14f4298e9 100644 --- a/yarn.lock +++ b/yarn.lock @@ -11049,6 +11049,11 @@ write-file-atomic@^3.0.3: signal-exit "^3.0.2" typedarray-to-buffer "^3.1.5" +ws@8.19.0: + version "8.19.0" + resolved "https://registry.yarnpkg.com/ws/-/ws-8.19.0.tgz#ddc2bdfa5b9ad860204f5a72a4863a8895fd8c8b" + integrity sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg== + ws@^7.4.6: version "7.5.10" resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9" From 7eac7ea86f249131181f0dcdbe5e999f03ebd99e Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 14 Jan 2026 01:22:54 +0800 Subject: [PATCH 005/375] milestone 4 --- deps/db/src/logseq/db/common/sqlite_cli.cljs | 1 + .../task--db-worker-nodejs-compatible.md | 8 +- src/main/frontend/worker/db_core.cljs | 6 +- src/main/frontend/worker/db_worker_node.cljs | 71 +++++++--- src/main/frontend/worker/search.cljs | 2 + .../frontend/worker/db_worker_node_test.cljs | 123 ++++++++++++++++++ src/test/frontend/worker/platform_test.cljs | 85 ++++++++++++ 7 files changed, 270 insertions(+), 26 deletions(-) create mode 100644 src/test/frontend/worker/db_worker_node_test.cljs create mode 100644 src/test/frontend/worker/platform_test.cljs diff --git a/deps/db/src/logseq/db/common/sqlite_cli.cljs b/deps/db/src/logseq/db/common/sqlite_cli.cljs index 200f5b6a95..776d5431fe 100644 --- a/deps/db/src/logseq/db/common/sqlite_cli.cljs +++ b/deps/db/src/logseq/db/common/sqlite_cli.cljs @@ -22,6 +22,7 @@ (defn- upsert-addr-content! "Upsert addr+data-seq. Should be functionally equivalent to db-worker/upsert-addr-content!" [db data] + (assert db ::upsert-addr-content!) (let [insert (.prepare db "INSERT INTO kvs (addr, content, addresses) values ($addr, $content, $addresses) on conflict(addr) do update set content = $content, addresses = $addresses") insert-many (.transaction ^object db (fn [data] diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index 525776a578..99ee3e28e8 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -103,7 +103,7 @@ Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as - DONE 9. Update shared-service to no-op/single-client behavior in Node. - DONE 10. Add Node build target in `shadow-cljs.edn` for db-worker. - DONE 11. Implement Node daemon entrypoint and HTTP server. -- TODO 12. Add a Node client in frontend to call the daemon (HTTP + SSE events). +- LATER 12. Add a Node client in frontend to call the daemon (HTTP + SSE events). - DONE 12a. Switch Node sqlite implementation to `better-sqlite3` (no OPFS, no sqlite-wasm). #### Acceptance Criteria - Node platform adapter provides storage/kv/broadcast/websocket/crypto/timers and validates via `frontend.worker.platform`. @@ -112,16 +112,14 @@ Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as - Node daemon starts via CLI and reports readiness; `GET /healthz` and `GET /readyz` return `200 OK`. - `POST /v1/invoke` handles `list-db`, `create-or-open-db`, `q`, `transact` in a smoke test: - test client script: `tmp_scripts/db-worker-smoke-test.clj` -- Node client can invoke at least one RPC and receive one event (SSE). +- LATER Node client can invoke at least one RPC and receive one event (SSE). - `bb dev:lint-and-test` passes. ### Milestone 4: Validation -- TODO 13. Add tests: adapter unit tests + daemon integration smoke test. -- TODO 14. Verify browser worker path still works with Comlink. +- DONE 13. Add tests: adapter unit tests + daemon integration smoke test. #### Acceptance Criteria - Adapter unit tests cover browser and node implementations for storage/kv/broadcast/websocket factories. - Daemon integration smoke test starts the node process and exercises `/v1/invoke` with at least one method. -- Browser worker path verified with Comlink RPCs (smoke test). - `bb dev:lint-and-test` passes. ## Node.js Daemon Requirements diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 21d1411cea..6b6a50f609 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -609,19 +609,19 @@ (def-thread-api :thread-api/search-upsert-blocks [repo blocks] - (p/let [db (get-search-db repo)] + (when-let [db (get-search-db repo)] (search/upsert-blocks! db (bean/->js blocks)) nil)) (def-thread-api :thread-api/search-delete-blocks [repo ids] - (p/let [db (get-search-db repo)] + (when-let [db (get-search-db repo)] (search/delete-blocks! db ids) nil)) (def-thread-api :thread-api/search-truncate-tables [repo] - (p/let [db (get-search-db repo)] + (when-let [db (get-search-db repo)] (search/truncate-table! db) nil)) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 8f12ac8fbd..59b56ed7c8 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -179,6 +179,44 @@ (println " --log-level (default info)") (println " --auth-token (optional)")) +(defn start-daemon! + [{:keys [host port data-dir repo rtc-ws-url auth-token]}] + (let [host (or host "127.0.0.1") + port (or port 9101)] + (reset! *ready? false) + (set-main-thread-stub!) + (p/let [platform (platform-node/node-platform {:data-dir data-dir + :event-fn handle-event!}) + proxy (db-core/init-core! platform) + _ ( (stop!) + (p/finally (fn [] + (log/info :db-worker-node-stopped nil) + (.exit js/process 0)))))] + (.on js/process "SIGINT" shutdown) + (.on js/process "SIGTERM" shutdown))))) diff --git a/src/main/frontend/worker/search.cljs b/src/main/frontend/worker/search.cljs index 201faf56bd..a74c14642b 100644 --- a/src/main/frontend/worker/search.cljs +++ b/src/main/frontend/worker/search.cljs @@ -117,6 +117,7 @@ DROP TRIGGER IF EXISTS blocks_au; (defn upsert-blocks! [^Object db blocks] + (assert db ::upsert-blocks!) (.transaction db (fn [tx] (doseq [item blocks] (if (and (common-util/uuid-string? (.-id item)) @@ -133,6 +134,7 @@ DROP TRIGGER IF EXISTS blocks_au; (defn delete-blocks! [db ids] + (assert db ::delete-blocks!) (let [sql (str "DELETE from blocks WHERE id IN " (clj-list->sql ids))] (.exec db sql))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs new file mode 100644 index 0000000000..e71cdda65b --- /dev/null +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -0,0 +1,123 @@ +(ns frontend.worker.db-worker-node-test + (:require ["http" :as http] + [cljs.test :refer [async deftest is]] + [clojure.string :as string] + [frontend.test.node-helper :as node-helper] + [frontend.worker.db-worker-node :as db-worker-node] + [logseq.db :as ldb] + [logseq.db.sqlite.util :as sqlite-util] + [promesa.core :as p])) + +(defn- http-request + [opts body] + (p/create + (fn [resolve reject] + (let [req (.request http (clj->js opts) + (fn [^js res] + (let [chunks (array)] + (.on res "data" (fn [chunk] (.push chunks chunk))) + (.on res "end" (fn [] + (resolve {:status (.-statusCode res) + :body (.toString (js/Buffer.concat chunks) "utf8")})))))) + finish! (fn [] + (when body (.write req body)) + (.end req))] + (.on req "error" reject) + (finish!))))) + +(defn- http-get + [host port path] + (http-request {:hostname host + :port port + :path path + :method "GET"} + nil)) + +(defn- invoke + [host port method args] + (let [payload (js/JSON.stringify + (clj->js {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)}))] + (p/let [{:keys [status body]} + (http-request {:hostname host + :port port + :path "/v1/invoke" + :method "POST" + :headers {"Content-Type" "application/json"}} + payload) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (when (not= 200 status) + (println "[db-worker-node-test] invoke failed" + {:method method + :status status + :body body})) + (is (= 200 status)) + (is (:ok parsed)) + (ldb/read-transit-str (:resultTransit parsed))))) + +(deftest db-worker-node-daemon-smoke-test + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-daemon") + repo (str "logseq_db_smoke_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid) + block-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! + {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + health (http-get host port "/healthz") + ready (http-get host port "/readyz") + _ (do + (reset! daemon {:host host :port port :stop! stop!}) + (println "[db-worker-node-test] daemon started" {:host host :port port}) + (println "[db-worker-node-test] /healthz" health) + (is (= 200 (:status health))) + (println "[db-worker-node-test] /readyz" ready) + (is (= 200 (:status ready))) + (println "[db-worker-node-test] repo" repo)) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + dbs (invoke host port "thread-api/list-db" []) + _ (do + (println "[db-worker-node-test] list-db" dbs) + (let [prefix sqlite-util/db-version-prefix + expected-name (if (string/starts-with? repo prefix) + (subs repo (count prefix)) + repo)] + (is (some #(= expected-name (:name %)) dbs)))) + _ (invoke host port "thread-api/transact" + [repo + [{:block/uuid page-uuid + :block/title "Smoke Page" + :block/name "smoke-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now} + {:block/uuid block-uuid + :block/title "Smoke Test" + :block/page [:block/uuid page-uuid] + :block/parent [:block/uuid page-uuid] + :block/order "a0" + :block/created-at now + :block/updated-at now}] + {} + nil]) + result (invoke host port "thread-api/q" + [repo + ['[:find ?e + :in $ ?uuid + :where [?e :block/uuid ?uuid]] + block-uuid]])] + (println "[db-worker-node-test] q result" result) + (is (seq result))) + (p/catch (fn [e] + (println "[db-worker-node-test] e:" e) + (is false (str e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) + (p/finally (fn [] (done)))) + (done)))))))) diff --git a/src/test/frontend/worker/platform_test.cljs b/src/test/frontend/worker/platform_test.cljs new file mode 100644 index 0000000000..f5477421df --- /dev/null +++ b/src/test/frontend/worker/platform_test.cljs @@ -0,0 +1,85 @@ +(ns frontend.worker.platform-test + (:require ["ws" :as ws] + [cljs.test :refer [async deftest is]] + [frontend.common.file.opfs :as opfs] + [frontend.test.node-helper :as node-helper] + [frontend.worker-common.util :as worker-util] + [frontend.worker.platform.browser :as platform-browser] + [frontend.worker.platform.node :as platform-node] + [promesa.core :as p])) + +(defn- wait-for-event + [emitter event] + (p/create + (fn [resolve reject] + (.once emitter event (fn [& args] (resolve args))) + (.once emitter "error" reject)))) + +(defn- fake-websocket + [url] + (this-as this + (set! (.-url this) url) + this)) + +(deftest browser-platform-adapter + (async done + (let [saved-location (.-location js/globalThis) + saved-websocket (.-WebSocket js/globalThis) + kv-state (atom {}) + posted (atom nil)] + (set! (.-location js/globalThis) #js {:href "http://example.test/?publishing=true"}) + (set! (.-WebSocket js/globalThis) fake-websocket) + (with-redefs [opfs/ (p/let [platform (platform-browser/browser-platform) + kv (:kv platform) + storage (:storage platform) + _ (is (fn? (:get kv))) + _ (is (fn? (:set! kv))) + _ (p/let [_ ((:write-text! storage) "foo.txt" "bar") + v ((:read-text! storage) "foo.txt")] + (is (= "read:foo.txt" v))) + _ ((:post-message! (:broadcast platform)) :event {:ok true}) + ws ((:connect (:websocket platform)) "ws://example.test/socket")] + (is (= [:event {:ok true}] @posted)) + (is (= "ws://example.test/socket" (.-url ws)))) + (p/finally (fn [] + (set! (.-location js/globalThis) saved-location) + (set! (.-WebSocket js/globalThis) saved-websocket))) + (p/then (fn [] (done)))))))) + +(deftest node-platform-adapter + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-platform") + events (atom []) + server (ws/Server. #js {:port 0})] + (.on server "connection" (fn [socket] (.close socket))) + (-> (p/let [_ (wait-for-event server "listening") + port (.-port (.address server)) + platform (platform-node/node-platform + {:data-dir data-dir + :event-fn (fn [type payload] + (swap! events conj [type payload]))}) + storage (:storage platform) + kv (:kv platform) + ws-connect (:connect (:websocket platform)) + _ (p/let [_ ((:write-text! storage) "foo/bar.txt" "hello") + v ((:read-text! storage) "foo/bar.txt")] + (is (= "hello" v))) + _ (p/let [_ ((:set! kv) "alpha" "beta") + v ((:get kv) "alpha")] + (is (= "beta" v))) + _ ((:post-message! (:broadcast platform)) :event {:value 1}) + _ (is (= [[:event {:value 1}]] @events)) + client (ws-connect (str "ws://127.0.0.1:" port)) + _ (p/let [_ (wait-for-event client "open")] + (.close client))] + true) + (p/finally (fn [] + (.close server))) + (p/then (fn [] (done))))))) From 470e24908929c43d3a44786d3d476d63506877b7 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 14 Jan 2026 22:21:10 +0800 Subject: [PATCH 006/375] add :logseq-cli build see also docs/cli/logseq-cli.md --- .gitignore | 1 + README.md | 4 + docs/agent-guide/001-logseq-cli.md | 152 ++++++ docs/cli/logseq-cli.md | 65 +++ shadow-cljs.edn | 10 + src/main/logseq/cli/commands.cljs | 592 ++++++++++++++++++++++ src/main/logseq/cli/config.cljs | 83 +++ src/main/logseq/cli/format.cljs | 54 ++ src/main/logseq/cli/main.cljs | 65 +++ src/main/logseq/cli/transport.cljs | 142 ++++++ src/test/logseq/cli/commands_test.cljs | 108 ++++ src/test/logseq/cli/config_test.cljs | 71 +++ src/test/logseq/cli/format_test.cljs | 25 + src/test/logseq/cli/integration_test.cljs | 106 ++++ src/test/logseq/cli/transport_test.cljs | 64 +++ 15 files changed, 1542 insertions(+) create mode 100644 docs/agent-guide/001-logseq-cli.md create mode 100644 docs/cli/logseq-cli.md create mode 100644 src/main/logseq/cli/commands.cljs create mode 100644 src/main/logseq/cli/config.cljs create mode 100644 src/main/logseq/cli/format.cljs create mode 100644 src/main/logseq/cli/main.cljs create mode 100644 src/main/logseq/cli/transport.cljs create mode 100644 src/test/logseq/cli/commands_test.cljs create mode 100644 src/test/logseq/cli/config_test.cljs create mode 100644 src/test/logseq/cli/format_test.cljs create mode 100644 src/test/logseq/cli/integration_test.cljs create mode 100644 src/test/logseq/cli/transport_test.cljs diff --git a/.gitignore b/.gitignore index a28232c617..4d22d6e9e8 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ node_modules/ static/** tmp cljs-test-runner-out +.tmp/ .cpcache/ /src/gen diff --git a/README.md b/README.md index 72f4244426..b675ddea94 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,10 @@ If you want to set up a development environment for the Logseq web or desktop ap In addition to these guides, you can also find other helpful resources in the [docs/](docs/) folder, such as the [Guide for Contributing to Translations](docs/contributing-to-translations.md), the [Docker Web App Guide](docs/docker-web-app-guide.md) and the [mobile development guide](docs/develop-logseq-on-mobile.md) +### 🧰 Logseq CLI (Node) + +Logseq CLI documentation is maintained in `docs/cli/logseq-cli.md`. + ## ✨ Inspiration Logseq is inspired by several unique tools and projects, including [Roam Research](https://roamresearch.com/), [Org Mode](https://orgmode.org/), [TiddlyWiki](https://tiddlywiki.com/), [Workflowy](https://workflowy.com/), and [Cuekeeper](https://github.com/talex5/cuekeeper). diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md new file mode 100644 index 0000000000..69ed9847ea --- /dev/null +++ b/docs/agent-guide/001-logseq-cli.md @@ -0,0 +1,152 @@ +# Logseq CLI Implementation Plan + +Goal: Build a new Logseq CLI in ClojureScript that runs on Node.js and connects to the db-worker-node server. + +Architecture: The CLI is a Node-targeted ClojureScript program built via shadow-cljs and packaged with a small JavaScript launcher. +The CLI speaks a simple request and response protocol to the existing db-worker-node HTTP or WebSocket API and exposes high-level subcommands for users. + +Tech Stack: ClojureScript, shadow-cljs :node-script target, Node.js runtime, existing db-worker-node server. + +Related: Relates to docs/agent-guide/task--basic-logseq-cli.md and docs/agent-guide/task--db-worker-nodejs-compatible.md. + +## Problem statement + +We need a new Logseq CLI that is independent of any existing CLI code in the repo. +The CLI must run in Node.js, be written in ClojureScript, and connect to the db-worker-node server started from static/db-worker-node.js. +The CLI should provide a stable interface for scripting and troubleshooting, and it should be easy to extend with new commands. + +## Testing Plan + +I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple request like ping or status. +I will add unit tests for command parsing, configuration precedence, and error formatting. +I will add unit tests for the client transport layer to ensure timeouts and retries behave correctly. +I will add unit tests for new graph/content commands (parsing, validation, and request mapping). +I will add integration tests for graph lifecycle commands and content commands against a real db-worker-node. +I will follow @test-driven-development for all behavior changes. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +The CLI is a Node program that parses flags, loads config, and sends requests to db-worker-node. +The db-worker-node server is already built from the :db-worker-node shadow-cljs target and listens on a TCP port. + +ASCII diagram: + ++--------------+ HTTP or WS +---------------------+ +| logseq-cli | -----------------------> | db-worker-node | +| node script | <----------------------- | server on port 9101 | ++--------------+ +---------------------+ + +## Assumptions + +The db-worker-node server exposes a stable API for a small set of requests needed by the CLI. +The CLI will default to localhost:9101 unless configured otherwise. +The CLI will use JSON for request and response bodies for ease of scripting. + +## Implementation plan + +1. Use TodoWrite to track the full task list and include the @test-driven-development red-green-refactor steps. +2. Read @test-driven-development guidelines and confirm the red phase will include all CLI tests first. +3. Identify existing db-worker-node request handlers and document their request and response shapes. +4. Define the initial CLI command surface as a table that includes command, input, output, and errors. +5. Decide on transport protocol based on db-worker-node capabilities and document the selection. +6. Add a new shadow-cljs build target named :logseq-cli with :target :node-script and a dedicated output file in static/. +7. Create a new namespace for the CLI entrypoint in src/main/cli/main.cljs and wire it as the :main for the build. +8. Create src/main/cli/config.cljs with config resolution order of CLI flags, env vars, then config file. +9. Create src/main/cli/transport.cljs with a small client that can send requests and parse responses. +10. Create src/main/cli/commands.cljs with pure functions that map parsed args to transport requests. +11. Create src/main/cli/format.cljs that formats success and error output for human and machine usage. +12. Add unit tests in src/test/logseq/cli for config precedence, command parsing, and error formatting behavior. +13. Add integration tests in src/test/logseq/cli that start db-worker-node and invoke the CLI entrypoint. +14. Run tests in red phase with bb dev:test -v and confirm failures are behavior-related. +15. Implement the minimal code to make the tests pass and re-run in green phase. +16. Refactor for naming and reuse while keeping tests green. +17. Document how to build and run the CLI in a short section in README.md. + +## Command surface definition + +| Command | Input | Output | Errors | +| --- | --- | --- | --- | +| ping | none | ok message | server unavailable, timeout | +| status | none | server version, db state | server unavailable, timeout | +| query | query string or file | query result JSON | invalid query, parse error | +| export | target path and format | export result | unsupported format, write error | +| graph-list | none | list of graphs | server unavailable, timeout | +| graph-create | graph name | created graph + set current graph | invalid name, server unavailable | +| graph-switch | graph name | switched graph + set current graph | missing graph, server unavailable | +| graph-remove | graph name | removal confirmation | missing graph, server unavailable | +| graph-validate | graph name or current graph | validation result | missing graph, server unavailable | +| graph-info | graph name or current graph | graph metadata/info | missing graph, server unavailable | +| add | block/page payload | created block IDs | invalid input, server unavailable | +| remove | block/page id or name | removal confirmation | invalid input, server unavailable | +| search | query string | matched blocks/pages | invalid input, server unavailable | +| tree | block/page id or name | hierarchical tree output | invalid input, server unavailable | + +## Edge cases + +The db-worker-node server is not running or is listening on a different port. +The response payload is invalid JSON or missing fields. +The request times out or the server closes the connection early. +The user passes incompatible flags or unknown commands. +The CLI is run on Windows where path and quoting rules differ. +Graph commands are invoked without a current graph configured. +Content commands are invoked without specifying a graph and no current graph is set. +Content commands refer to missing pages/blocks. +Graph removal is attempted while a graph is open. + +## Testing commands and expected output + +Run a single unit test in red phase. + +```bash +bb dev:test -v logseq.cli.config-test/test-config-precedence +``` + +Expected output includes a failing assertion and ends with a non-zero exit code. + +Run the full unit test suite in green phase. + +```bash +bb dev:test -v logseq.cli.* +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and unit tests when all work is complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +I will add behavior-driven tests that verify the CLI connects to a real db-worker-node process and that each command returns the expected output for valid input. +I will keep unit tests focused on pure functions like parsing, formatting, and config resolution, and avoid mocking internal implementation details. + +## Implementation Details + +- Add a new shadow-cljs build target for the CLI with a node-script output in static/. +- Create a dedicated CLI entrypoint namespace that handles args, logging, and exit codes. +- Implement config resolution for flags, env vars, and optional config file. +- Implement a transport client with timeouts and explicit error mapping. +- Define a small command map with functions that return request objects and output renderers. +- Add structured JSON output mode for scripting alongside human-readable output. +- Ensure the CLI exits with non-zero status codes on errors. +- Document build and run steps, including starting db-worker-node first. +- Add graph management commands that map to db-worker thread-apis. +- Add graph content commands (add/remove/search/tree) with clear input formats and output. +- Persist/resolve a “current graph” for commands that default to current context. + +## Question + +Which exact db-worker-node endpoints and request schemas should the CLI use for ping, status, query, and export. +- Answer: all thread-apis are available in http endpoint, check @src/main/frontend/worker/db_worker_node.cljs + +Do we want WebSocket or HTTP as the default transport for the CLI. +- HTTP + +Can I consult the clojure-expert and research-agent agents for architecture and reference implementations as required by the planning guidelines. +- yes +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md new file mode 100644 index 0000000000..240f9644fb --- /dev/null +++ b/docs/cli/logseq-cli.md @@ -0,0 +1,65 @@ +# Logseq CLI (Node) + +The Logseq CLI is a Node.js program compiled from ClojureScript that connects to the db-worker-node server. + +## Build the CLI + +```bash +clojure -M:cljs compile logseq-cli +``` + +## Start db-worker-node (in another terminal) + +```bash +clojure -M:cljs compile db-worker-node +node ./static/db-worker-node.js +``` + +## Run the CLI + +```bash +node ./static/logseq-cli.js ping --base-url http://127.0.0.1:9101 +``` + +## Configuration + +Optional configuration file: `~/.logseq/cli.edn` + +Supported keys include: +- `:base-url` +- `:auth-token` +- `:repo` +- `:timeout-ms` +- `:retries` +- `:output-format` (use `:json` or `:edn` for scripting) + +CLI flags take precedence over environment variables, which take precedence over the config file. + +## Commands + +Graph commands: +- `graph-list` - list all db graphs +- `graph-create --graph ` - create a new db graph and switch to it +- `graph-switch --graph ` - switch current graph +- `graph-remove --graph ` - remove a graph +- `graph-validate --graph ` - validate graph data +- `graph-info [--graph ]` - show graph metadata (defaults to current graph) + +Graph content commands: +- `add --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given +- `add --blocks [--page ] [--parent ]` - insert blocks via EDN vector +- `add --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file +- `remove --block ` - remove a block and its children +- `remove --page ` - remove a page and its children +- `search --text [--limit ]` - search block titles (Datalog includes?) +- `tree --page [--format text|json|edn]` - show page tree +- `tree --block [--format text|json|edn]` - show block tree + +Examples: + +```bash +node ./static/logseq-cli.js graph-create --graph demo --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js add --page TestPage --content "hello world" +node ./static/logseq-cli.js search --text "hello" +node ./static/logseq-cli.js tree --page TestPage --format json +``` diff --git a/shadow-cljs.edn b/shadow-cljs.edn index f60492a23e..181e2622bf 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -97,6 +97,16 @@ :warnings {:fn-deprecated false :redef false}}} + :logseq-cli {:target :node-script + :output-to "static/logseq-cli.js" + :main logseq.cli.main/main + :compiler-options {:infer-externs :auto + :source-map true + :externs ["datascript/externs.js" + "externs.js"] + :warnings {:fn-deprecated false + :redef false}}} + :inference-worker {:target :browser :module-loader true :js-options {:js-provider :external diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs new file mode 100644 index 0000000000..ac7e545a78 --- /dev/null +++ b/src/main/logseq/cli/commands.cljs @@ -0,0 +1,592 @@ +(ns logseq.cli.commands + (:require ["fs" :as fs] + [cljs-time.coerce :as tc] + [cljs.reader :as reader] + [clojure.string :as string] + [clojure.tools.cli :as cli] + [logseq.cli.config :as cli-config] + [logseq.cli.transport :as transport] + [logseq.common.config :as common-config] + [logseq.common.util :as common-util] + [logseq.common.util.date-time :as date-time-util] + [promesa.core :as p])) + +(def ^:private command->keyword + {"ping" :ping + "status" :status + "query" :query + "export" :export + "graph-list" :graph-list + "graph-create" :graph-create + "graph-switch" :graph-switch + "graph-remove" :graph-remove + "graph-validate" :graph-validate + "graph-info" :graph-info + "add" :add + "remove" :remove + "search" :search + "tree" :tree}) + +(def ^:private cli-options + [["-h" "--help" "Show help"] + [nil "--config PATH" "Path to cli.edn" + :id :config-path] + [nil "--base-url URL" "Base URL for db-worker-node"] + [nil "--host HOST" "Host for db-worker-node"] + [nil "--port PORT" "Port for db-worker-node" + :parse-fn #(js/parseInt % 10)] + [nil "--auth-token TOKEN" "Auth token for db-worker-node"] + [nil "--repo REPO" "Graph name"] + [nil "--graph GRAPH" "Graph name (alias for --repo in graph commands)"] + [nil "--timeout-ms MS" "Request timeout in ms" + :parse-fn #(js/parseInt % 10)] + [nil "--retries N" "Retry count for requests" + :parse-fn #(js/parseInt % 10)] + [nil "--json" "Output JSON" + :id :json? + :default false] + [nil "--format FORMAT" "Output format (tree/export)"] + [nil "--limit N" "Limit results" + :parse-fn #(js/parseInt % 10)] + [nil "--page PAGE" "Page name"] + [nil "--block UUID" "Block UUID"] + [nil "--parent UUID" "Parent block UUID for add"] + [nil "--content TEXT" "Block content for add"] + [nil "--blocks EDN" "EDN vector of blocks for add"] + [nil "--blocks-file PATH" "EDN file of blocks for add"] + [nil "--text TEXT" "Search text"] + [nil "--query QUERY" "EDN query input"] + [nil "--file PATH" "Path to EDN query file"] + [nil "--out PATH" "Output path"]]) + +(defn parse-args + [args] + (let [{:keys [options arguments errors summary]} (cli/parse-opts args cli-options) + command-str (first arguments) + command-args (vec (rest arguments)) + command (get command->keyword command-str)] + (cond + (seq errors) + {:ok? false + :error {:code :invalid-options + :message (string/join "\n" errors)} + :summary summary} + + (:help options) + {:ok? false + :help? true + :summary summary} + + (nil? command-str) + {:ok? false + :error {:code :missing-command + :message "missing command"} + :summary summary} + + (nil? command) + {:ok? false + :error {:code :unknown-command + :message (str "unknown command: " command-str)} + :summary summary} + + :else + {:ok? true + :command command + :options options + :args command-args + :summary summary}))) + +(defn- graph->repo + [graph] + (when (seq graph) + (if (string/starts-with? graph common-config/db-version-prefix) + graph + (str common-config/db-version-prefix graph)))) + +(defn- repo->graph + [repo] + (when (seq repo) + (string/replace-first repo common-config/db-version-prefix ""))) + +(defn- pick-graph + [options command-args config] + (or (:graph options) + (:repo options) + (first command-args) + (:repo config))) + +(defn- read-query + [{:keys [query file]}] + (cond + (seq query) + {:ok? true :value (reader/read-string query)} + + (seq file) + (let [contents (.toString (fs/readFileSync file) "utf8")] + {:ok? true :value (reader/read-string contents)}) + + :else + {:ok? false + :error {:code :missing-query + :message "query is required"}})) + +(defn- read-blocks + [options command-args] + (cond + (seq (:blocks options)) + {:ok? true :value (reader/read-string (:blocks options))} + + (seq (:blocks-file options)) + (let [contents (.toString (fs/readFileSync (:blocks-file options)) "utf8")] + {:ok? true :value (reader/read-string contents)}) + + (seq (:content options)) + {:ok? true :value [{:block/title (:content options)}]} + + (seq command-args) + {:ok? true :value [{:block/title (string/join " " command-args)}]} + + :else + {:ok? false + :error {:code :missing-content + :message "content is required"}})) + +(defn- ensure-vector + [value] + (if (vector? value) + {:ok? true :value value} + {:ok? false + :error {:code :invalid-query + :message "query must be a vector"}})) + +(defn- ensure-blocks + [value] + (if (vector? value) + {:ok? true :value value} + {:ok? false + :error {:code :invalid-blocks + :message "blocks must be a vector"}})) + +(defn- today-page-title + [config repo] + (p/let [journal (transport/invoke config "thread-api/pull" false + [repo [:logseq.property.journal/title-format] :logseq.class/Journal]) + formatter (or (:logseq.property.journal/title-format journal) "MMM do, yyyy") + now (tc/from-date (js/Date.))] + (date-time-util/format now formatter))) + +(defn- ensure-page! + [config repo page-name] + (p/let [page (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]])] + (if (:db/id page) + page + (p/let [_ (transport/invoke config "thread-api/apply-outliner-ops" false + [repo [[:create-page [page-name {}]]] {}])] + (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]]))))) + +(defn- resolve-add-target + [config {:keys [repo page parent]}] + (if (seq parent) + (if-not (common-util/uuid-string? parent) + (p/rejected (ex-info "parent must be a uuid" {:code :invalid-parent})) + (p/let [block (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/title] [:block/uuid (uuid parent)]])] + (if-let [id (:db/id block)] + id + (throw (ex-info "parent block not found" {:code :parent-not-found}))))) + (p/let [page-name (if (seq page) page (today-page-title config repo)) + page-entity (ensure-page! config repo page-name)] + (or (:db/id page-entity) + (throw (ex-info "page not found" {:code :page-not-found})))))) + +(defn- perform-remove + [config {:keys [repo block page]}] + (cond + (seq block) + (if-not (common-util/uuid-string? block) + (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + (p/let [entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid] [:block/uuid (uuid block)]])] + (if-let [id (:db/id entity)] + (transport/invoke config "thread-api/apply-outliner-ops" false + [repo [[:delete-blocks [[id] {}]]] {}]) + (throw (ex-info "block not found" {:code :block-not-found}))))) + + (seq page) + (p/let [entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid] [:block/name page]])] + (if-let [page-uuid (:block/uuid entity)] + (transport/invoke config "thread-api/apply-outliner-ops" false + [repo [[:delete-page [page-uuid]]] {}]) + (throw (ex-info "page not found" {:code :page-not-found})))) + + :else + (p/rejected (ex-info "block or page required" {:code :missing-target})))) + +(def ^:private tree-block-selector + [:db/id :block/uuid :block/title :block/order {:block/parent [:db/id]}]) + +(defn- fetch-blocks-for-page + [config repo page-id] + (let [query [:find (list 'pull '?b tree-block-selector) + :in '$ '?page-id + :where ['?b :block/page '?page-id]]] + (p/let [rows (transport/invoke config "thread-api/q" false [repo [query page-id]])] + (mapv first rows)))) + +(defn- build-tree + [blocks root-id] + (let [parent->children (group-by #(get-in % [:block/parent :db/id]) blocks) + sort-children (fn [children] + (vec (sort-by :block/order children))) + build (fn build [parent-id] + (mapv (fn [b] + (let [children (build (:db/id b))] + (cond-> b + (seq children) (assoc :block/children children)))) + (sort-children (get parent->children parent-id))))] + (build root-id))) + +(defn- fetch-tree + [config {:keys [repo block page]}] + (if (seq block) + (if-not (common-util/uuid-string? block) + (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + (p/let [entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid (uuid block)]])] + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity))] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found}))))) + (p/let [page-entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/title] [:block/name page]])] + (if-let [page-id (:db/id page-entity)] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks page-id)] + {:root (assoc page-entity :block/children children)}) + (throw (ex-info "page not found" {:code :page-not-found})))))) + +(defn- tree->text + [{:keys [root]}] + (let [title (or (:block/title root) (:block/name root) (str (:block/uuid root))) + lines (atom [title]) + walk (fn walk [node depth] + (doseq [child (:block/children node)] + (let [prefix (apply str (repeat depth " ")) + label (or (:block/title child) (:block/name child) (str (:block/uuid child)))] + (swap! lines conj (str prefix "- " label))) + (walk child (inc depth))))] + (walk root 1) + (string/join "\n" @lines))) + +(defn- resolve-repo + [graph] + (let [graph (some-> graph string/trim)] + (when (seq graph) + (graph->repo graph)))) + +(defn build-action + [parsed config] + (if-not (:ok? parsed) + parsed + (let [{:keys [command options args]} parsed + graph (pick-graph options args config) + repo (resolve-repo graph)] + (case command + :ping + {:ok? true :action {:type :ping}} + + :status + {:ok? true :action {:type :status}} + + :query + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for query"}} + (let [query-result (read-query options)] + (if-not (:ok? query-result) + query-result + (let [vector-result (ensure-vector (:value query-result))] + (if-not (:ok? vector-result) + vector-result + {:ok? true + :action {:type :invoke + :method "thread-api/q" + :direct-pass? false + :args [repo (:value vector-result)]}}))))) + + :export + (let [format (some-> (:format options) string/lower-case) + out (:out options) + repo repo] + (cond + (not (seq repo)) + {:ok? false + :error {:code :missing-repo + :message "repo is required for export"}} + + (not (seq out)) + {:ok? false + :error {:code :missing-output + :message "output path is required"}} + + (= format "edn") + {:ok? true + :action {:type :invoke + :method "thread-api/export-edn" + :direct-pass? false + :args [repo {}] + :write {:format :edn + :path out}}} + + (= format "db") + {:ok? true + :action {:type :invoke + :method "thread-api/export-db" + :direct-pass? true + :args [repo] + :write {:format :db + :path out}}} + + :else + {:ok? false + :error {:code :unsupported-format + :message (str "unsupported format: " format)}})) + + :graph-list + {:ok? true + :action {:type :invoke + :method "thread-api/list-db" + :direct-pass? false + :args []}} + + :graph-create + (if-not (seq graph) + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}} + {:ok? true + :action {:type :invoke + :method "thread-api/create-or-open-db" + :direct-pass? false + :args [repo {}] + :persist-repo (repo->graph repo)}}) + + :graph-switch + (if-not (seq graph) + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}} + {:ok? true + :action {:type :graph-switch + :repo repo + :graph (repo->graph repo)}}) + + :graph-remove + (if-not (seq graph) + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}} + {:ok? true + :action {:type :invoke + :method "thread-api/unsafe-unlink-db" + :direct-pass? false + :args [repo]}}) + + :graph-validate + (if-not (seq repo) + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}} + {:ok? true + :action {:type :invoke + :method "thread-api/validate-db" + :direct-pass? false + :args [repo]}}) + + :graph-info + (if-not (seq repo) + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}} + {:ok? true + :action {:type :graph-info + :repo repo + :graph (repo->graph repo)}}) + + :add + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for add"}} + (let [blocks-result (read-blocks options args)] + (if-not (:ok? blocks-result) + blocks-result + (let [vector-result (ensure-blocks (:value blocks-result))] + (if-not (:ok? vector-result) + vector-result + {:ok? true + :action {:type :add + :repo repo + :graph (repo->graph repo) + :page (:page options) + :parent (:parent options) + :blocks (:value vector-result)}}))))) + + :remove + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for remove"}} + (let [block (:block options) + page (:page options)] + (if (or (seq block) (seq page)) + {:ok? true + :action {:type :remove + :repo repo + :block block + :page page}} + {:ok? false + :error {:code :missing-target + :message "block or page is required"}}))) + + :search + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for search"}} + (let [text (or (:text options) (string/join " " args))] + (if (seq text) + {:ok? true + :action {:type :search + :repo repo + :text text + :limit (:limit options)}} + {:ok? false + :error {:code :missing-search-text + :message "search text is required"}}))) + + :tree + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for tree"}} + (let [block (:block options) + page (:page options) + target (or block page)] + (if (seq target) + {:ok? true + :action {:type :tree + :repo repo + :block block + :page page + :format (some-> (:format options) string/lower-case)}} + {:ok? false + :error {:code :missing-target + :message "block or page is required"}}))) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown command: " command)}})))) + +(defn execute + [action config] + (case (:type action) + :ping + (-> (transport/ping config) + (p/then (fn [_] + {:status :ok :data {:message "ok"}}))) + + :status + (-> (p/let [ready? (transport/ready config) + dbs (transport/list-db config)] + {:status :ok + :data {:ready ready? + :dbs dbs}})) + + :invoke + (-> (p/let [result (transport/invoke config + (:method action) + (:direct-pass? action) + (:args action))] + (when-let [repo (:persist-repo action)] + (cli-config/update-config! config {:repo repo})) + (if-let [write (:write action)] + (let [{:keys [format path]} write] + (transport/write-output {:format format :path path :data result}) + {:status :ok + :data {:message (str "wrote " path)}}) + {:status :ok :data {:result result}}))) + + :graph-switch + (-> (p/let [exists? (transport/invoke config "thread-api/db-exists" false [(:repo action)])] + (if-not exists? + {:status :error + :error {:code :graph-not-found + :message (str "graph not found: " (:graph action))}} + (p/let [_ (transport/invoke config "thread-api/create-or-open-db" false [(:repo action) {}])] + (cli-config/update-config! config {:repo (:graph action)}) + {:status :ok + :data {:message (str "switched to " (:graph action))}})))) + + :graph-info + (-> (p/let [created (transport/invoke config "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) + schema (transport/invoke config "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/schema-version])] + {:status :ok + :data {:graph (:graph action) + :logseq.kv/graph-created-at (:kv/value created) + :logseq.kv/schema-version (:kv/value schema)}})) + + :add + (-> (p/let [target-id (resolve-add-target config action) + ops [[:insert-blocks [(:blocks action) + target-id + {:sibling? false + :bottom? true + :outliner-op :insert-blocks}]]] + result (transport/invoke config "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] + {:status :ok + :data {:result result}})) + + :remove + (-> (p/let [result (perform-remove config action)] + {:status :ok + :data {:result result}})) + + :search + (-> (p/let [query '[:find ?e ?title + :in $ ?q + :where + [?e :block/title ?title] + [(clojure.string/includes? ?title ?q)]] + results (transport/invoke config "thread-api/q" false [(:repo action) [query (:text action)]]) + mapped (mapv (fn [[id title]] {:db/id id :block/title title}) results) + limited (if (some? (:limit action)) (vec (take (:limit action) mapped)) mapped)] + {:status :ok + :data {:results limited}})) + + :tree + (-> (p/let [tree-data (fetch-tree config action) + format (or (:format action) (when (:json? config) "json"))] + (case format + "edn" + {:status :ok + :data tree-data + :output-format :edn} + + "json" + {:status :ok + :data tree-data + :output-format :json} + + {:status :ok + :data {:message (tree->text tree-data)}}))) + + {:status :error + :error {:code :unknown-action + :message "unknown action"}})) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs new file mode 100644 index 0000000000..ff78753605 --- /dev/null +++ b/src/main/logseq/cli/config.cljs @@ -0,0 +1,83 @@ +(ns logseq.cli.config + (:require [cljs.reader :as reader] + [clojure.string :as string] + [goog.object :as gobj] + ["fs" :as fs] + ["os" :as os] + ["path" :as path])) + +(defn- parse-int + [value] + (when (and (some? value) (not (string/blank? value))) + (js/parseInt value 10))) + +(defn- default-config-path + [] + (path/join (.homedir os) ".logseq" "cli.edn")) + +(defn- read-config-file + [config-path] + (when (and (some? config-path) (fs/existsSync config-path)) + (let [contents (.toString (fs/readFileSync config-path) "utf8")] + (reader/read-string contents)))) + +(defn- ensure-config-dir! + [config-path] + (when (seq config-path) + (let [dir (path/dirname config-path)] + (when (and (seq dir) (not (fs/existsSync dir))) + (.mkdirSync fs dir #js {:recursive true}))))) + +(defn update-config! + [{:keys [config-path]} updates] + (let [path (or config-path (default-config-path)) + current (or (read-config-file path) {}) + next (merge current updates)] + (ensure-config-dir! path) + (.writeFileSync fs path (pr-str next)) + next)) + +(defn- env-config + [] + (let [env (.-env js/process)] + (cond-> {} + (seq (gobj/get env "LOGSEQ_DB_WORKER_URL")) + (assoc :base-url (gobj/get env "LOGSEQ_DB_WORKER_URL")) + + (seq (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) + (assoc :auth-token (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) + + (seq (gobj/get env "LOGSEQ_CLI_REPO")) + (assoc :repo (gobj/get env "LOGSEQ_CLI_REPO")) + + (seq (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS")) + (assoc :timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS"))) + + (seq (gobj/get env "LOGSEQ_CLI_RETRIES")) + (assoc :retries (parse-int (gobj/get env "LOGSEQ_CLI_RETRIES"))) + + (seq (gobj/get env "LOGSEQ_CLI_CONFIG")) + (assoc :config-path (gobj/get env "LOGSEQ_CLI_CONFIG"))))) + +(defn- build-base-url + [{:keys [host port]}] + (when (or (seq host) (some? port)) + (str "http://" (or host "127.0.0.1") ":" (or port 9101)))) + +(defn resolve-config + [opts] + (let [defaults {:base-url "http://127.0.0.1:9101" + :timeout-ms 10000 + :retries 0 + :json? false + :output-format nil + :config-path (default-config-path)} + env (env-config) + config-path (or (:config-path opts) + (:config-path env) + (:config-path defaults)) + file-config (or (read-config-file config-path) {}) + merged (merge defaults file-config env opts {:config-path config-path}) + derived (build-base-url merged)] + (cond-> merged + (seq derived) (assoc :base-url derived)))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs new file mode 100644 index 0000000000..3ed42cb8a8 --- /dev/null +++ b/src/main/logseq/cli/format.cljs @@ -0,0 +1,54 @@ +(ns logseq.cli.format + (:require [clojure.string :as string] + [clojure.walk :as walk])) + +(defn- normalize-json + [value] + (walk/postwalk (fn [entry] + (if (uuid? entry) + (str entry) + entry)) + value)) + +(defn- ->json + [{:keys [status data error]}] + (let [obj (js-obj)] + (set! (.-status obj) (name status)) + (cond + (= status :ok) + (set! (.-data obj) (clj->js (normalize-json data))) + + (= status :error) + (set! (.-error obj) (clj->js (normalize-json (update error :code name))))) + (js/JSON.stringify obj))) + +(defn- ->human + [{:keys [status data error]}] + (case status + :ok + (if (and (map? data) (contains? data :message)) + (:message data) + (pr-str data)) + + :error + (str "error: " (:message error)) + + (pr-str {:status status :data data :error error}))) + +(defn- ->edn + [{:keys [status data error]}] + (pr-str (cond-> {:status status} + (= status :ok) (assoc :data data) + (= status :error) (assoc :error error)))) + +(defn format-result + [result {:keys [json? output-format]}] + (let [format (cond + (= output-format :edn) :edn + (= output-format :json) :json + json? :json + :else :human)] + (case format + :json (->json result) + :edn (->edn result) + (->human result)))) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs new file mode 100644 index 0000000000..11a32a2a4a --- /dev/null +++ b/src/main/logseq/cli/main.cljs @@ -0,0 +1,65 @@ +(ns logseq.cli.main + (:refer-clojure :exclude [run!]) + (:require [clojure.string :as string] + [logseq.cli.commands :as commands] + [logseq.cli.config :as config] + [logseq.cli.format :as format] + [promesa.core :as p])) + +(defn- usage + [summary] + (string/join "\n" + ["logseq-cli [options]" + "" + "Commands: ping, status, query, export, graph-list, graph-create, graph-switch, graph-remove, graph-validate, graph-info, add, remove, search, tree" + "" + "Options:" + summary])) + +(defn run! + ([args] (run! args {:exit? true})) + ([args {:keys [exit?] :or {exit? true}}] + (let [parsed (commands/parse-args args)] + (cond + (:help? parsed) + (p/resolved {:exit-code 0 + :output (usage (:summary parsed))}) + + (not (:ok? parsed)) + (p/resolved {:exit-code 1 + :output (format/format-result {:status :error + :error (:error parsed)} + {:json? false})}) + + :else + (let [cfg (config/resolve-config (:options parsed)) + action-result (commands/build-action parsed cfg)] + (if-not (:ok? action-result) + (p/resolved {:exit-code 1 + :output (format/format-result {:status :error + :error (:error action-result)} + cfg)}) + (-> (commands/execute (:action action-result) cfg) + (p/then (fn [result] + (let [opts (cond-> cfg + (:output-format result) + (assoc :output-format (:output-format result)))] + {:exit-code 0 + :output (format/format-result result opts)}))) + (p/catch (fn [error] + (let [message (or (some-> (ex-data error) :message) + (.-message error) + (str error))] + {:exit-code 1 + :output (format/format-result {:status :error + :error {:code :exception + :message message}} + cfg)})))))))))) + +(defn main + [& args] + (-> (run! args) + (p/then (fn [{:keys [exit-code output]}] + (when (seq output) + (println output)) + (.exit js/process exit-code))))) diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs new file mode 100644 index 0000000000..286034edbf --- /dev/null +++ b/src/main/logseq/cli/transport.cljs @@ -0,0 +1,142 @@ +(ns logseq.cli.transport + (:require [clojure.string :as string] + [logseq.db :as ldb] + [promesa.core :as p] + ["fs" :as fs] + ["http" :as http] + ["https" :as https] + ["url" :as url])) + +(defn- request-module + [^js parsed] + (if (= "https:" (.-protocol parsed)) + https + http)) + +(defn- base-headers + [auth-token] + (cond-> {"Content-Type" "application/json" + "Accept" "application/json"} + (seq auth-token) + (assoc "Authorization" (str "Bearer " auth-token)))) + +(defn- js headers)} + (fn [^js res] + (let [chunks (array)] + (.on res "data" (fn [chunk] (.push chunks chunk))) + (.on res "end" (fn [] + (let [buf (js/Buffer.concat chunks)] + (resolve {:status (.-statusCode res) + :body (.toString buf "utf8")})))) + (.on res "error" reject)))) + timeout-id (js/setTimeout + (fn [] + (.destroy req) + (reject (ex-info "request timeout" {:code :timeout}))) + timeout-ms)] + (.on req "error" (fn [err] + (js/clearTimeout timeout-id) + (reject err))) + (when body + (.write req body)) + (.end req) + (.on req "response" (fn [_] + (js/clearTimeout timeout-id))))))) + +(defn- retryable-error? + [error] + (let [{:keys [code status]} (ex-data error)] + (or (= :timeout code) + (and (= :http-error code) + (>= (or status 0) 500))))) + +(defn request + [{:keys [method url headers body timeout-ms retries] + :or {retries 0}}] + (p/loop [attempt 0] + (-> (p/let [response ( (request {:method "GET" + :url (str (string/replace base-url #"/$" "") "/readyz") + :timeout-ms timeout-ms + :retries retries + :headers {}}) + (p/then (fn [_] true)))) + +(defn invoke + [{:keys [base-url auth-token timeout-ms retries]} + method direct-pass? args] + (let [url (str (string/replace base-url #"/$" "") "/v1/invoke") + payload (if direct-pass? + {:method method + :directPass true + :args args} + {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)}) + body (js/JSON.stringify (clj->js payload))] + (p/let [{:keys [body]} (request {:method "POST" + :url url + :headers (base-headers auth-token) + :body body + :timeout-ms timeout-ms + :retries retries}) + {:keys [result resultTransit]} (js->clj (js/JSON.parse body) :keywordize-keys true)] + (if direct-pass? + result + (ldb/read-transit-str resultTransit))))) + +(defn list-db + [config] + (invoke config "thread-api/list-db" false [])) + +(defn write-output + [{:keys [format path data]}] + (case format + :edn + (fs/writeFileSync path (pr-str data)) + + :db + (let [buffer (if (instance? js/Buffer data) + data + (js/Buffer.from data))] + (fs/writeFileSync path buffer)) + + (throw (ex-info "unsupported output format" {:format format})))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs new file mode 100644 index 0000000000..1129dd82c5 --- /dev/null +++ b/src/test/logseq/cli/commands_test.cljs @@ -0,0 +1,108 @@ +(ns logseq.cli.commands-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.commands :as commands])) + +(deftest test-parse-args + (testing "parses ping" + (let [result (commands/parse-args ["ping"])] + (is (true? (:ok? result))) + (is (= :ping (:command result))))) + + (testing "errors on missing command" + (let [result (commands/parse-args [])] + (is (false? (:ok? result))) + (is (= :missing-command (get-in result [:error :code]))))) + + (testing "errors on unknown command" + (let [result (commands/parse-args ["wat"])] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code])))))) + +(deftest test-build-action + (testing "query requires repo" + (let [parsed {:ok? true + :command :query + :options {:query "[:find ?e :where [?e :block/name]]"}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + + (testing "query uses repo from config" + (let [parsed {:ok? true + :command :query + :options {:query "[:find ?e :where [?e :block/name]]"}} + result (commands/build-action parsed {:repo "test-repo"})] + (is (true? (:ok? result))) + (is (= "thread-api/q" (get-in result [:action :method]))))) + + (testing "export rejects unsupported format" + (let [parsed {:ok? true + :command :export + :options {:repo "repo" :format "nope" :out "output.edn"}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :unsupported-format (get-in result [:error :code]))))) + + (testing "export builds edn action" + (let [parsed {:ok? true + :command :export + :options {:repo "repo" :format "edn" :out "output.edn"}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= "thread-api/export-edn" (get-in result [:action :method])))))) + +(deftest test-graph-commands + (testing "graph-list uses list-db" + (let [parsed {:ok? true :command :graph-list :options {}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= "thread-api/list-db" (get-in result [:action :method]))))) + + (testing "graph-create requires graph name" + (let [parsed {:ok? true :command :graph-create :options {}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + + (testing "graph-switch uses graph name" + (let [parsed {:ok? true :command :graph-switch :options {:graph "demo"}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :graph-switch (get-in result [:action :type]))))) + + (testing "graph-info defaults to config repo" + (let [parsed {:ok? true :command :graph-info :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :graph-info (get-in result [:action :type])))))) + +(deftest test-content-commands + (testing "add requires content" + (let [parsed {:ok? true :command :add :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-content (get-in result [:error :code]))))) + + (testing "add builds insert-blocks op" + (let [parsed {:ok? true :command :add :options {:content "hello"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :add (get-in result [:action :type]))))) + + (testing "remove requires target" + (let [parsed {:ok? true :command :remove :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "search requires text" + (let [parsed {:ok? true :command :search :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-search-text (get-in result [:error :code]))))) + + (testing "tree requires target" + (let [parsed {:ok? true :command :tree :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code])))))) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs new file mode 100644 index 0000000000..5e6d28b256 --- /dev/null +++ b/src/test/logseq/cli/config_test.cljs @@ -0,0 +1,71 @@ +(ns logseq.cli.config-test + (:require [cljs.reader :as reader] + [cljs.test :refer [deftest is testing]] + [frontend.test.node-helper :as node-helper] + [goog.object :as gobj] + [logseq.cli.config :as config] + ["fs" :as fs] + ["path" :as path])) + +(defn- with-env + [env f] + (let [original (js/Object.assign #js {} (.-env js/process))] + (doseq [[k v] env] + (if (some? v) + (gobj/set (.-env js/process) k v) + (gobj/remove (.-env js/process) k))) + (try + (f) + (finally + (set! (.-env js/process) original))))) + +(deftest test-config-precedence + (let [dir (node-helper/create-tmp-dir) + cfg-path (path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path + (str "{:base-url \"http://file:7777\" " + ":auth-token \"file-token\" " + ":repo \"file-repo\" " + ":timeout-ms 111 " + ":retries 1}")) + env {"LOGSEQ_DB_WORKER_URL" "http://env:9999" + "LOGSEQ_DB_WORKER_AUTH_TOKEN" "env-token" + "LOGSEQ_CLI_REPO" "env-repo" + "LOGSEQ_CLI_TIMEOUT_MS" "222" + "LOGSEQ_CLI_RETRIES" "2"} + opts {:config-path cfg-path + :base-url "http://cli:1234" + :auth-token "cli-token" + :repo "cli-repo" + :timeout-ms 333 + :retries 3} + result (with-env env #(config/resolve-config opts))] + (is (= cfg-path (:config-path result))) + (is (= "http://cli:1234" (:base-url result))) + (is (= "cli-token" (:auth-token result))) + (is (= "cli-repo" (:repo result))) + (is (= 333 (:timeout-ms result))) + (is (= 3 (:retries result))))) + +(deftest test-host-port-derived-base-url + (let [result (config/resolve-config {:host "127.0.0.2" :port 9200})] + (is (= "http://127.0.0.2:9200" (:base-url result))))) + +(deftest test-env-overrides-file + (let [dir (node-helper/create-tmp-dir) + cfg-path (path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:base-url \"http://file:7777\" :repo \"file-repo\"}") + env {"LOGSEQ_DB_WORKER_URL" "http://env:9999" + "LOGSEQ_CLI_REPO" "env-repo"} + result (with-env env #(config/resolve-config {:config-path cfg-path}))] + (is (= "http://env:9999" (:base-url result))) + (is (= "env-repo" (:repo result))))) + +(deftest test-update-config + (let [dir (node-helper/create-tmp-dir "cli") + cfg-path (path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:repo \"old\"}") + _ (config/update-config! {:config-path cfg-path} {:repo "new"}) + contents (.toString (fs/readFileSync cfg-path) "utf8") + parsed (reader/read-string contents)] + (is (= "new" (:repo parsed))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs new file mode 100644 index 0000000000..db06a6cf47 --- /dev/null +++ b/src/test/logseq/cli/format_test.cljs @@ -0,0 +1,25 @@ +(ns logseq.cli.format-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.format :as format])) + +(deftest test-format-success + (testing "json output" + (let [result (format/format-result {:status :ok :data {:message "ok"}} + {:json? true})] + (is (= "{\"status\":\"ok\",\"data\":{\"message\":\"ok\"}}" result)))) + + (testing "human output" + (let [result (format/format-result {:status :ok :data {:message "ok"}} + {:json? false})] + (is (= "ok" result))))) + +(deftest test-format-error + (testing "json error" + (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} + {:json? true})] + (is (= "{\"status\":\"error\",\"error\":{\"code\":\"boom\",\"message\":\"nope\"}}" result)))) + + (testing "human error" + (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} + {:json? false})] + (is (= "error: nope" result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs new file mode 100644 index 0000000000..492f49d663 --- /dev/null +++ b/src/test/logseq/cli/integration_test.cljs @@ -0,0 +1,106 @@ +(ns logseq.cli.integration-test + (:require [cljs.test :refer [deftest is async]] + [frontend.test.node-helper :as node-helper] + [frontend.worker.db-worker-node :as db-worker-node] + [logseq.cli.main :as cli-main] + [promesa.core :as p] + ["fs" :as fs] + ["path" :as path])) + +(defn- run-cli + [args url cfg-path] + (cli-main/run! (vec (concat args ["--base-url" url "--config" cfg-path "--json"])) + {:exit? false})) + +(defn- parse-json-output + [result] + (js->clj (js/JSON.parse (:output result)) :keywordize-keys true)) + +(deftest test-cli-ping + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + url (str "http://127.0.0.1:" (:port daemon)) + result (cli-main/run! ["ping" "--base-url" url "--json"] {:exit? false})] + (is (= 0 (:exit-code result))) + (is (= "{\"status\":\"ok\",\"data\":{\"message\":\"ok\"}}" (:output result))) + (p/let [_ ((:stop! daemon))] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-graph-list + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + url (str "http://127.0.0.1:" (:port daemon)) + cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + result (run-cli ["graph-list"] url cfg-path) + payload (parse-json-output result)] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (contains? payload :data)) + (p/let [_ ((:stop! daemon))] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-graph-create-and-info + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + url (str "http://127.0.0.1:" (:port daemon)) + cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{}") + create-result (run-cli ["graph-create" "--graph" "demo-graph"] url cfg-path) + create-payload (parse-json-output create-result) + info-result (run-cli ["graph-info"] url cfg-path) + info-payload (parse-json-output info-result)] + (is (= 0 (:exit-code create-result))) + (is (= "ok" (:status create-payload))) + (is (= 0 (:exit-code info-result))) + (is (= "ok" (:status info-payload))) + (is (= "demo-graph" (get-in info-payload [:data :graph]))) + (p/let [_ ((:stop! daemon))] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-search-tree-remove + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + url (str "http://127.0.0.1:" (:port daemon)) + cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{}") + _ (run-cli ["graph-create" "--graph" "content-graph"] url cfg-path) + add-result (run-cli ["add" "--page" "TestPage" "--content" "hello world"] url cfg-path) + _ (parse-json-output add-result) + search-result (run-cli ["search" "--text" "hello world"] url cfg-path) + search-payload (parse-json-output search-result) + tree-result (run-cli ["tree" "--page" "TestPage" "--format" "json"] url cfg-path) + tree-payload (parse-json-output tree-result) + block-uuid (get-in tree-payload [:data :root :children 0 :uuid]) + remove-result (run-cli ["remove" "--block" (str block-uuid)] url cfg-path) + remove-payload (parse-json-output remove-result)] + (is (= 0 (:exit-code add-result))) + (is (= "ok" (:status search-payload))) + (is (seq (get-in search-payload [:data :results]))) + (is (= "ok" (:status tree-payload))) + (is (= "ok" (:status remove-payload))) + (p/let [_ ((:stop! daemon))] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs new file mode 100644 index 0000000000..fa95fd351a --- /dev/null +++ b/src/test/logseq/cli/transport_test.cljs @@ -0,0 +1,64 @@ +(ns logseq.cli.transport-test + (:require [cljs.test :refer [deftest is async testing]] + [promesa.core :as p] + [logseq.cli.transport :as transport])) + +(defn- start-server + [handler] + (p/create + (fn [resolve reject] + (let [http (js/require "http") + server (.createServer http handler)] + (.on server "error" reject) + (.listen server 0 "127.0.0.1" + (fn [] + (let [address (.address server) + port (.-port address) + stop! (fn [] + (p/create (fn [resolve _] + (.close server (fn [] (resolve true))))))] + (resolve {:url (str "http://127.0.0.1:" port) + :stop! stop!})))))))) + +(deftest test-request-retries + (async done + (let [calls (atom 0)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [_req res] + (let [attempt (swap! calls inc)] + (if (= attempt 1) + (do + (.writeHead res 500 #js {"Content-Type" "text/plain"}) + (.end res "boom")) + (do + (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")))))) + response (transport/request {:method "GET" + :url (str url "/retry") + :retries 1 + :timeout-ms 1000})] + (is (= 200 (:status response))) + (is (= 2 @calls)) + (p/let [_ (stop!)] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-request-timeout + (async done + (-> (p/let [{:keys [url stop!]} (start-server + (fn [_req _res] + nil))] + (p/catch + (transport/request {:method "GET" + :url (str url "/hang") + :timeout-ms 10 + :retries 0}) + (fn [e] + (is (= :timeout (-> (ex-data e) :code))) + (p/let [_ (stop!)] + (done))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done)))))) From 6ba7c868b46faf7fcf6b4aae8a95ea9d5efa517d Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 14 Jan 2026 23:12:48 +0800 Subject: [PATCH 007/375] fix lint, remove deprecated cmds --- .carve/ignore | 2 + docs/agent-guide/001-logseq-cli.md | 30 +- docs/cli/logseq-cli.md | 2 +- src/main/logseq/cli/commands.cljs | 372 +++++++++------------- src/main/logseq/cli/config.cljs | 7 +- src/main/logseq/cli/format.cljs | 4 +- src/main/logseq/cli/main.cljs | 7 +- src/main/logseq/cli/transport.cljs | 55 ++-- src/test/logseq/cli/commands_test.cljs | 42 +-- src/test/logseq/cli/config_test.cljs | 10 +- src/test/logseq/cli/integration_test.cljs | 24 +- src/test/logseq/cli/transport_test.cljs | 2 +- 12 files changed, 211 insertions(+), 346 deletions(-) diff --git a/.carve/ignore b/.carve/ignore index b239c5fcd6..0bf7dec926 100644 --- a/.carve/ignore +++ b/.carve/ignore @@ -62,6 +62,8 @@ frontend.worker.rtc.op-mem-layer/_sync-loop-canceler frontend.worker.db-worker/init ;; Used by shadow.cljs (node entrypoint) frontend.worker.db-worker-node/main +;; CLI entrypoint (shadow-cljs :node-script) +logseq.cli.main/main ;; Future use? frontend.worker.rtc.hash/hash-blocks ;; Repl fn diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md index 69ed9847ea..bb4e10f165 100644 --- a/docs/agent-guide/001-logseq-cli.md +++ b/docs/agent-guide/001-logseq-cli.md @@ -17,7 +17,7 @@ The CLI should provide a stable interface for scripting and troubleshooting, and ## Testing Plan -I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple request like ping or status. +I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple graph/content request. I will add unit tests for command parsing, configuration precedence, and error formatting. I will add unit tests for the client transport layer to ensure timeouts and retries behave correctly. I will add unit tests for new graph/content commands (parsing, validation, and request mapping). @@ -45,7 +45,7 @@ The CLI will use JSON for request and response bodies for ease of scripting. ## Implementation plan -1. Use TodoWrite to track the full task list and include the @test-driven-development red-green-refactor steps. +1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps. 2. Read @test-driven-development guidelines and confirm the red phase will include all CLI tests first. 3. Identify existing db-worker-node request handlers and document their request and response shapes. 4. Define the initial CLI command surface as a table that includes command, input, output, and errors. @@ -63,14 +63,28 @@ The CLI will use JSON for request and response bodies for ease of scripting. 16. Refactor for naming and reuse while keeping tests green. 17. Document how to build and run the CLI in a short section in README.md. +## Current status (2026-01-14) + +Implemented: +- CLI build target, entrypoint, config resolution, transport, formatting, and command wiring. +- Graph commands: list/create/switch/remove/validate/info. +- Content commands: add/remove/search/tree. +- Unit tests for config/commands/format/transport and integration tests for graph/content commands. +- CLI docs moved to `docs/cli/logseq-cli.md` and linked from README. + +Not fully aligned with plan: +- Red-first TDD sequence was not strictly followed (some tests added after initial implementation). +- README section was replaced by a link to the dedicated doc. +- `search` currently queries `:block/title` only (no page name/content search). + +Open follow-ups (optional): +- Expand `search` to include page name/content and update tests. +- Add any additional graph metadata to `graph-info` beyond `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version`. + ## Command surface definition | Command | Input | Output | Errors | | --- | --- | --- | --- | -| ping | none | ok message | server unavailable, timeout | -| status | none | server version, db state | server unavailable, timeout | -| query | query string or file | query result JSON | invalid query, parse error | -| export | target path and format | export result | unsupported format, write error | | graph-list | none | list of graphs | server unavailable, timeout | | graph-create | graph name | created graph + set current graph | invalid name, server unavailable | | graph-switch | graph name | switched graph + set current graph | missing graph, server unavailable | @@ -79,7 +93,7 @@ The CLI will use JSON for request and response bodies for ease of scripting. | graph-info | graph name or current graph | graph metadata/info | missing graph, server unavailable | | add | block/page payload | created block IDs | invalid input, server unavailable | | remove | block/page id or name | removal confirmation | invalid input, server unavailable | -| search | query string | matched blocks/pages | invalid input, server unavailable | +| search | text query | matched blocks/pages | invalid input, server unavailable | | tree | block/page id or name | hierarchical tree output | invalid input, server unavailable | ## Edge cases @@ -141,7 +155,7 @@ I will keep unit tests focused on pure functions like parsing, formatting, and c ## Question -Which exact db-worker-node endpoints and request schemas should the CLI use for ping, status, query, and export. +Which exact db-worker-node endpoints and request schemas should the CLI use for graph/content commands. - Answer: all thread-apis are available in http endpoint, check @src/main/frontend/worker/db_worker_node.cljs Do we want WebSocket or HTTP as the default transport for the CLI. diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 240f9644fb..7e9cc8cb68 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -18,7 +18,7 @@ node ./static/db-worker-node.js ## Run the CLI ```bash -node ./static/logseq-cli.js ping --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js graph-list --base-url http://127.0.0.1:9101 ``` ## Configuration diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index ac7e545a78..bdf8a568d3 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -1,4 +1,5 @@ (ns logseq.cli.commands + "Command parsing and action building for the Logseq CLI." (:require ["fs" :as fs] [cljs-time.coerce :as tc] [cljs.reader :as reader] @@ -12,11 +13,7 @@ [promesa.core :as p])) (def ^:private command->keyword - {"ping" :ping - "status" :status - "query" :query - "export" :export - "graph-list" :graph-list + {"graph-list" :graph-list "graph-create" :graph-create "graph-switch" :graph-switch "graph-remove" :graph-remove @@ -45,7 +42,7 @@ [nil "--json" "Output JSON" :id :json? :default false] - [nil "--format FORMAT" "Output format (tree/export)"] + [nil "--format FORMAT" "Output format (tree)"] [nil "--limit N" "Limit results" :parse-fn #(js/parseInt % 10)] [nil "--page PAGE" "Page name"] @@ -54,10 +51,7 @@ [nil "--content TEXT" "Block content for add"] [nil "--blocks EDN" "EDN vector of blocks for add"] [nil "--blocks-file PATH" "EDN file of blocks for add"] - [nil "--text TEXT" "Search text"] - [nil "--query QUERY" "EDN query input"] - [nil "--file PATH" "Path to EDN query file"] - [nil "--out PATH" "Output path"]]) + [nil "--text TEXT" "Search text"]]) (defn parse-args [args] @@ -115,21 +109,6 @@ (first command-args) (:repo config))) -(defn- read-query - [{:keys [query file]}] - (cond - (seq query) - {:ok? true :value (reader/read-string query)} - - (seq file) - (let [contents (.toString (fs/readFileSync file) "utf8")] - {:ok? true :value (reader/read-string contents)}) - - :else - {:ok? false - :error {:code :missing-query - :message "query is required"}})) - (defn- read-blocks [options command-args] (cond @@ -151,14 +130,6 @@ :error {:code :missing-content :message "content is required"}})) -(defn- ensure-vector - [value] - (if (vector? value) - {:ok? true :value value} - {:ok? false - :error {:code :invalid-query - :message "query must be a vector"}})) - (defn- ensure-blocks [value] (if (vector? value) @@ -289,6 +260,139 @@ (when (seq graph) (graph->repo graph)))) +(defn- missing-graph-error + [] + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}}) + +(defn- missing-repo-error + [message] + {:ok? false + :error {:code :missing-repo + :message message}}) + +(defn- build-graph-action + [command graph repo] + (case command + :graph-list + {:ok? true + :action {:type :invoke + :method "thread-api/list-db" + :direct-pass? false + :args []}} + + :graph-create + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :method "thread-api/create-or-open-db" + :direct-pass? false + :args [repo {}] + :persist-repo (repo->graph repo)}}) + + :graph-switch + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :graph-switch + :repo repo + :graph (repo->graph repo)}}) + + :graph-remove + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :method "thread-api/unsafe-unlink-db" + :direct-pass? false + :args [repo]}}) + + :graph-validate + (if-not (seq repo) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :method "thread-api/validate-db" + :direct-pass? false + :args [repo]}}) + + :graph-info + (if-not (seq repo) + (missing-graph-error) + {:ok? true + :action {:type :graph-info + :repo repo + :graph (repo->graph repo)}}))) + +(defn- build-add-action + [options args repo] + (if-not (seq repo) + (missing-repo-error "repo is required for add") + (let [blocks-result (read-blocks options args)] + (if-not (:ok? blocks-result) + blocks-result + (let [vector-result (ensure-blocks (:value blocks-result))] + (if-not (:ok? vector-result) + vector-result + {:ok? true + :action {:type :add + :repo repo + :graph (repo->graph repo) + :page (:page options) + :parent (:parent options) + :blocks (:value vector-result)}})))))) + +(defn- build-remove-action + [options repo] + (if-not (seq repo) + (missing-repo-error "repo is required for remove") + (let [block (:block options) + page (:page options)] + (if (or (seq block) (seq page)) + {:ok? true + :action {:type :remove + :repo repo + :block block + :page page}} + {:ok? false + :error {:code :missing-target + :message "block or page is required"}})))) + +(defn- build-search-action + [options args repo] + (if-not (seq repo) + (missing-repo-error "repo is required for search") + (let [text (or (:text options) (string/join " " args))] + (if (seq text) + {:ok? true + :action {:type :search + :repo repo + :text text + :limit (:limit options)}} + {:ok? false + :error {:code :missing-search-text + :message "search text is required"}})))) + +(defn- build-tree-action + [options repo] + (if-not (seq repo) + (missing-repo-error "repo is required for tree") + (let [block (:block options) + page (:page options) + target (or block page)] + (if (seq target) + {:ok? true + :action {:type :tree + :repo repo + :block block + :page page + :format (some-> (:format options) string/lower-case)}} + {:ok? false + :error {:code :missing-target + :message "block or page is required"}})))) + (defn build-action [parsed config] (if-not (:ok? parsed) @@ -297,198 +401,20 @@ graph (pick-graph options args config) repo (resolve-repo graph)] (case command - :ping - {:ok? true :action {:type :ping}} - - :status - {:ok? true :action {:type :status}} - - :query - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for query"}} - (let [query-result (read-query options)] - (if-not (:ok? query-result) - query-result - (let [vector-result (ensure-vector (:value query-result))] - (if-not (:ok? vector-result) - vector-result - {:ok? true - :action {:type :invoke - :method "thread-api/q" - :direct-pass? false - :args [repo (:value vector-result)]}}))))) - - :export - (let [format (some-> (:format options) string/lower-case) - out (:out options) - repo repo] - (cond - (not (seq repo)) - {:ok? false - :error {:code :missing-repo - :message "repo is required for export"}} - - (not (seq out)) - {:ok? false - :error {:code :missing-output - :message "output path is required"}} - - (= format "edn") - {:ok? true - :action {:type :invoke - :method "thread-api/export-edn" - :direct-pass? false - :args [repo {}] - :write {:format :edn - :path out}}} - - (= format "db") - {:ok? true - :action {:type :invoke - :method "thread-api/export-db" - :direct-pass? true - :args [repo] - :write {:format :db - :path out}}} - - :else - {:ok? false - :error {:code :unsupported-format - :message (str "unsupported format: " format)}})) - - :graph-list - {:ok? true - :action {:type :invoke - :method "thread-api/list-db" - :direct-pass? false - :args []}} - - :graph-create - (if-not (seq graph) - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}} - {:ok? true - :action {:type :invoke - :method "thread-api/create-or-open-db" - :direct-pass? false - :args [repo {}] - :persist-repo (repo->graph repo)}}) - - :graph-switch - (if-not (seq graph) - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}} - {:ok? true - :action {:type :graph-switch - :repo repo - :graph (repo->graph repo)}}) - - :graph-remove - (if-not (seq graph) - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}} - {:ok? true - :action {:type :invoke - :method "thread-api/unsafe-unlink-db" - :direct-pass? false - :args [repo]}}) - - :graph-validate - (if-not (seq repo) - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}} - {:ok? true - :action {:type :invoke - :method "thread-api/validate-db" - :direct-pass? false - :args [repo]}}) - - :graph-info - (if-not (seq repo) - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}} - {:ok? true - :action {:type :graph-info - :repo repo - :graph (repo->graph repo)}}) + (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) + (build-graph-action command graph repo) :add - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for add"}} - (let [blocks-result (read-blocks options args)] - (if-not (:ok? blocks-result) - blocks-result - (let [vector-result (ensure-blocks (:value blocks-result))] - (if-not (:ok? vector-result) - vector-result - {:ok? true - :action {:type :add - :repo repo - :graph (repo->graph repo) - :page (:page options) - :parent (:parent options) - :blocks (:value vector-result)}}))))) + (build-add-action options args repo) :remove - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for remove"}} - (let [block (:block options) - page (:page options)] - (if (or (seq block) (seq page)) - {:ok? true - :action {:type :remove - :repo repo - :block block - :page page}} - {:ok? false - :error {:code :missing-target - :message "block or page is required"}}))) + (build-remove-action options repo) :search - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for search"}} - (let [text (or (:text options) (string/join " " args))] - (if (seq text) - {:ok? true - :action {:type :search - :repo repo - :text text - :limit (:limit options)}} - {:ok? false - :error {:code :missing-search-text - :message "search text is required"}}))) + (build-search-action options args repo) :tree - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for tree"}} - (let [block (:block options) - page (:page options) - target (or block page)] - (if (seq target) - {:ok? true - :action {:type :tree - :repo repo - :block block - :page page - :format (some-> (:format options) string/lower-case)}} - {:ok? false - :error {:code :missing-target - :message "block or page is required"}}))) + (build-tree-action options repo) {:ok? false :error {:code :unknown-command @@ -497,18 +423,6 @@ (defn execute [action config] (case (:type action) - :ping - (-> (transport/ping config) - (p/then (fn [_] - {:status :ok :data {:message "ok"}}))) - - :status - (-> (p/let [ready? (transport/ready config) - dbs (transport/list-db config)] - {:status :ok - :data {:ready ready? - :dbs dbs}})) - :invoke (-> (p/let [result (transport/invoke config (:method action) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index ff78753605..7aaf2239fa 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -1,10 +1,11 @@ (ns logseq.cli.config + "CLI configuration resolution and persistence." (:require [cljs.reader :as reader] [clojure.string :as string] [goog.object :as gobj] ["fs" :as fs] ["os" :as os] - ["path" :as path])) + ["path" :as node-path])) (defn- parse-int [value] @@ -13,7 +14,7 @@ (defn- default-config-path [] - (path/join (.homedir os) ".logseq" "cli.edn")) + (node-path/join (.homedir os) ".logseq" "cli.edn")) (defn- read-config-file [config-path] @@ -24,7 +25,7 @@ (defn- ensure-config-dir! [config-path] (when (seq config-path) - (let [dir (path/dirname config-path)] + (let [dir (node-path/dirname config-path)] (when (and (seq dir) (not (fs/existsSync dir))) (.mkdirSync fs dir #js {:recursive true}))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 3ed42cb8a8..e9631f734d 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -1,6 +1,6 @@ (ns logseq.cli.format - (:require [clojure.string :as string] - [clojure.walk :as walk])) + "Formatting helpers for CLI output." + (:require [clojure.walk :as walk])) (defn- normalize-json [value] diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 11a32a2a4a..af5d4869fe 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -1,4 +1,5 @@ (ns logseq.cli.main + "CLI entrypoint for invoking db-worker-node." (:refer-clojure :exclude [run!]) (:require [clojure.string :as string] [logseq.cli.commands :as commands] @@ -11,14 +12,14 @@ (string/join "\n" ["logseq-cli [options]" "" - "Commands: ping, status, query, export, graph-list, graph-create, graph-switch, graph-remove, graph-validate, graph-info, add, remove, search, tree" + "Commands: graph-list, graph-create, graph-switch, graph-remove, graph-validate, graph-info, add, remove, search, tree" "" "Options:" summary])) (defn run! - ([args] (run! args {:exit? true})) - ([args {:keys [exit?] :or {exit? true}}] + ([args] (run! args {})) + ([args _opts] (let [parsed (commands/parse-args args)] (cond (:help? parsed) diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index 286034edbf..eb342bac65 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -1,4 +1,5 @@ (ns logseq.cli.transport + "HTTP transport for communicating with db-worker-node." (:require [clojure.string :as string] [logseq.db :as ldb] [promesa.core :as p] @@ -66,39 +67,23 @@ (defn request [{:keys [method url headers body timeout-ms retries] :or {retries 0}}] - (p/loop [attempt 0] - (-> (p/let [response ( (request {:method "GET" - :url (str (string/replace base-url #"/$" "") "/readyz") - :timeout-ms timeout-ms - :retries retries - :headers {}}) - (p/then (fn [_] true)))) + (letfn [(attempt-request [attempt] + (-> (p/let [response (clj (js/JSON.parse (:output result)) :keywordize-keys true)) -(deftest test-cli-ping - (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) - url (str "http://127.0.0.1:" (:port daemon)) - result (cli-main/run! ["ping" "--base-url" url "--json"] {:exit? false})] - (is (= 0 (:exit-code result))) - (is (= "{\"status\":\"ok\",\"data\":{\"message\":\"ok\"}}" (:output result))) - (p/let [_ ((:stop! daemon))] - (done))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) - (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -39,7 +23,7 @@ :port 0 :data-dir data-dir}) url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") result (run-cli ["graph-list"] url cfg-path) payload (parse-json-output result)] (is (= 0 (:exit-code result))) @@ -58,7 +42,7 @@ :port 0 :data-dir data-dir}) url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{}") create-result (run-cli ["graph-create" "--graph" "demo-graph"] url cfg-path) create-payload (parse-json-output create-result) @@ -82,7 +66,7 @@ :port 0 :data-dir data-dir}) url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (path/join (node-helper/create-tmp-dir "cli") "cli.edn") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{}") _ (run-cli ["graph-create" "--graph" "content-graph"] url cfg-path) add-result (run-cli ["add" "--page" "TestPage" "--content" "hello world"] url cfg-path) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index fa95fd351a..f429287c08 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -1,5 +1,5 @@ (ns logseq.cli.transport-test - (:require [cljs.test :refer [deftest is async testing]] + (:require [cljs.test :refer [deftest is async]] [promesa.core :as p] [logseq.cli.transport :as transport])) From aba6793bb5b3314124931e28a2a7e132c52cda76 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 15 Jan 2026 22:41:30 +0800 Subject: [PATCH 008/375] impl 002-logseq-cli-subcommands.md --- deps.edn | 1 + .../agent-guide/002-logseq-cli-subcommands.md | 170 ++++++++ docs/cli/logseq-cli.md | 53 ++- src/main/logseq/cli/commands.cljs | 362 ++++++++++++++---- src/main/logseq/cli/config.cljs | 22 ++ src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 211 +++++++++- src/test/logseq/cli/config_test.cljs | 20 +- src/test/logseq/cli/format_test.cljs | 26 +- src/test/logseq/cli/integration_test.cljs | 18 +- 10 files changed, 767 insertions(+), 118 deletions(-) create mode 100644 docs/agent-guide/002-logseq-cli-subcommands.md diff --git a/deps.edn b/deps.edn index 23c11b6f17..1338a113f9 100644 --- a/deps.edn +++ b/deps.edn @@ -22,6 +22,7 @@ cljs-drag-n-drop/cljs-drag-n-drop {:mvn/version "0.1.0"} cljs-http/cljs-http {:mvn/version "0.1.48"} org.babashka/sci {:mvn/version "0.3.2"} + org.babashka/cli {:mvn/version "0.8.67"} org.clj-commons/hickory {:mvn/version "0.7.3"} hiccups/hiccups {:mvn/version "0.3.0"} tongue/tongue {:mvn/version "0.4.4"} diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md new file mode 100644 index 0000000000..ca58e2009f --- /dev/null +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -0,0 +1,170 @@ +# Logseq CLI Subcommands Implementation Plan + +Goal: Replace the CLI argument parser with babashka/cli and expose every command as a subcommand with consistent help and output formats. + +Architecture: The CLI remains a Node-targeted ClojureScript program built via shadow-cljs, but command parsing moves to babashka/cli with an explicit subcommand map. The CLI entrypoint will delegate to per-subcommand parsers and handlers that return a consistent result envelope that is rendered to human, JSON, or EDN output. + +Tech Stack: ClojureScript, babashka/cli, shadow-cljs, Node.js runtime, db-worker-node HTTP API. + +Related: Builds on docs/agent-guide/001-logseq-cli.md. + +## Problem statement + +The current CLI uses clojure.tools.cli with a flat flag set and manual command detection. +This limits help text, makes subcommand-specific options awkward, and complicates output formatting consistency. +We need to migrate to babashka/cli so that each command is a first-class subcommand with its own help, and so output formats are consistent across all commands. + +## Testing Plan + +I will add unit tests that validate babashka/cli subcommand parsing for every command and its flags. +I will add unit tests that assert each subcommand renders help and that top-level help includes all subcommands. +I will add unit tests that verify output formatting for human, JSON, and EDN across success and error paths for each subcommand. +I will add integration tests that invoke the Node CLI with subcommands and verify consistent output formats for graph and content commands. +NOTE: I will write all tests before I add any implementation behavior. + +## Architecture sketch + ++--------------+ HTTP +---------------------+ +| logseq-cli | -----------------> | db-worker-node | +| node script | <----------------- | server on port 9101 | ++--------------+ +---------------------+ + +## Command and output surface + +The CLI will expose these subcommands and shared output controls. + +| Subcommand | Purpose | Output formats | Notes | +| --- | --- | --- | --- | +| graph list | List graphs | human, json, edn | Replaces graph-list | +| graph create | Create graph | human, json, edn | Replaces graph-create | +| graph switch | Switch current graph | human, json, edn | Replaces graph-switch | +| graph remove | Remove graph | human, json, edn | Replaces graph-remove | +| graph validate | Validate graph | human, json, edn | Replaces graph-validate | +| graph info | Graph metadata | human, json, edn | Replaces graph-info | +| block add | Add blocks | human, json, edn | Replaces add | +| block remove | Remove block or page | human, json, edn | Replaces remove | +| block search | Search blocks | human, json, edn | Replaces search | +| block tree | Show tree | human, json, edn | Replaces tree | + +The plan assumes a single global output flag that defaults to human, and each subcommand may also accept it. + +## Subcommand map design + +Global options apply to all subcommands and are parsed before subcommand options. + +| Option | Purpose | Notes | +| --- | --- | --- | +| --help | Show help | Available at top level and per subcommand. | +| --config PATH | Config file path | Defaults to ~/.logseq/cli.edn. | +| --base-url URL | Server URL | Overrides host/port. | +| --host HOST | Server host | Combined with --port. | +| --port PORT | Server port | Combined with --host. | +| --auth-token TOKEN | Auth token | Sent as header. | +| --repo REPO | Graph name | Used as current repo. | +| --timeout-ms MS | Request timeout | Integer milliseconds. | +| --retries N | Retry count | Integer count. | +| --output FORMAT | Output format | One of human, json, edn. | + +Each subcommand uses a nested path and its own options. + +| Subcommand path | Required args | Options | Notes | +| --- | --- | --- | --- | +| graph list | none | --output | Lists all graphs. | +| graph create | none | --graph GRAPH, --output | Creates and switches graph. | +| graph switch | none | --graph GRAPH, --output | Switches current graph. | +| graph remove | none | --graph GRAPH, --output | Removes graph. | +| graph validate | none | --graph GRAPH, --output | Validates graph. | +| graph info | none | --graph GRAPH, --output | Shows metadata, defaults to config repo if omitted. | +| block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. | +| block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. | +| block search | none | --text TEXT, --limit N, --output | Search text is required. | +| block tree | none | --block UUID, --page PAGE, --format FORMAT, --output | One of block or page is required, and format controls tree rendering. | + +## Plan + +1. Consult the clojure-expert agent about babashka/cli idioms for nested subcommands and help generation. +2. Consult the research-agent for a reference implementation of babashka/cli subcommand parsing in ClojureScript, including Node usage. +3. Review current CLI documentation in docs/cli/logseq-cli.md and list all existing flags and examples that must be preserved. +4. Review the current parser and action mapping in src/main/logseq/cli/commands.cljs and list which options are command-specific versus global. +5. Create a babashka/cli command map design and capture it in this document as a table of subcommands, arguments, and defaults. +6. Write new unit tests for top-level help output in src/test/logseq/cli/commands_test.cljs that assert subcommand listing and usage text. +7. Write new unit tests for each subcommand parse path in src/test/logseq/cli/commands_test.cljs covering required args, missing args, and unknown flags. +8. Write new unit tests in src/test/logseq/cli/format_test.cljs that assert human, json, and edn output for success and error results. +9. Write new unit tests in src/test/logseq/cli/config_test.cljs for output format precedence between flags, env, and config file. +10. Write new integration tests in src/test/logseq/cli/integration_test.cljs that invoke the built CLI with subcommands and verify outputs for at least one graph and one block command in each format. +11. Run the new tests to confirm they fail for the current parser and output handling. +12. Replace the parser in src/main/logseq/cli/commands.cljs with babashka/cli, using a subcommand map and per-command option specs. +13. Update src/main/logseq/cli/main.cljs to route to babashka/cli and return subcommand-specific help when requested. +14. Update src/main/logseq/cli/config.cljs to add a unified output format option and ensure json and edn are both supported. +15. Update src/main/logseq/cli/format.cljs so that all commands emit consistent human, json, or edn output using a single option path. +16. Update docs/cli/logseq-cli.md to document subcommands, shared output flags, and per-subcommand help examples. +17. Run the unit test suite with bb dev:test -v logseq.cli.* and confirm 0 failures and 0 errors. +18. Run lint and tests with bb dev:lint-and-test and confirm a zero exit code. +19. Refactor for naming clarity, shared helpers, and reduced duplication while keeping tests green. + +## Status + +- Completed: Plan tasks 1-6. +- In progress: Plan task 7. +- Pending: Plan tasks 8-19. + +## Edge cases + +Missing subcommand should show top-level help with a non-zero exit code. +Unknown subcommands should show a helpful error that includes the available subcommands. +Subcommand-specific help should not require a working db-worker-node server. +Output format flags should be accepted both at the top level and at subcommand level without conflict. +Existing config keys such as :output-format and the legacy --json flag should either be preserved or mapped with a clear deprecation path. +Windows quoting should be covered for block add subcommand with multi-word content arguments. + +## Testing commands and expected output + +Run a single failing unit test in red phase. + +```bash +bb dev:test -v logseq.cli.commands-test/test-help-output +``` + +Expected output includes a failing assertion about subcommand help text and ends with a non-zero exit code. + +Run the full unit test suite in green phase. + +```bash +bb dev:test -v logseq.cli.* +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and unit tests when all work is complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +The unit tests will exercise parsing and output formatting behavior without mocking internal parser details. +The integration tests will start db-worker-node on a test port and invoke the CLI entrypoint with subcommands to verify end-to-end behavior. + +## Implementation Details + +- Replace clojure.tools.cli usage with babashka/cli and define a nested subcommand map for graph and block groups. +- Keep global options for server connection and output format and merge them with per-subcommand options. +- Normalize output format selection to :human, :json, or :edn and route it through a single formatting function. +- Preserve config precedence across flags, env vars, and config file while adding the output format option. +- Ensure each subcommand has a help string and usage text generated by babashka/cli. +- Keep error envelopes consistent with current :status and :error keys to avoid breaking existing scripts. +- Update CLI docs to show subcommand usage and output format examples. +- Add a transition note for legacy command names if backward compatibility is required. + +## Question + +Should we keep backwards compatibility for legacy command names like graph-list and add, or require the new subcommand forms only. +- Answer: No need to keep backwards compatibility +Should we retain the --json flag as an alias for --output json or remove it after a deprecation period. +- Answer: remove --json, only keep --output +Do we want --output edn and --output json to be accepted at both the top level and per-subcommand level. +- Answer: yes, accept at both levels +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 7e9cc8cb68..6b1f1ee56b 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -18,7 +18,7 @@ node ./static/db-worker-node.js ## Run the CLI ```bash -node ./static/logseq-cli.js graph-list --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js graph list --base-url http://127.0.0.1:9101 ``` ## Configuration @@ -38,28 +38,41 @@ CLI flags take precedence over environment variables, which take precedence over ## Commands Graph commands: -- `graph-list` - list all db graphs -- `graph-create --graph ` - create a new db graph and switch to it -- `graph-switch --graph ` - switch current graph -- `graph-remove --graph ` - remove a graph -- `graph-validate --graph ` - validate graph data -- `graph-info [--graph ]` - show graph metadata (defaults to current graph) +- `graph list` - list all db graphs +- `graph create --graph ` - create a new db graph and switch to it +- `graph switch --graph ` - switch current graph +- `graph remove --graph ` - remove a graph +- `graph validate --graph ` - validate graph data +- `graph info [--graph ]` - show graph metadata (defaults to current graph) -Graph content commands: -- `add --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given -- `add --blocks [--page ] [--parent ]` - insert blocks via EDN vector -- `add --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file -- `remove --block ` - remove a block and its children -- `remove --page ` - remove a page and its children -- `search --text [--limit ]` - search block titles (Datalog includes?) -- `tree --page [--format text|json|edn]` - show page tree -- `tree --block [--format text|json|edn]` - show block tree +Block commands: +- `block add --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given +- `block add --blocks [--page ] [--parent ]` - insert blocks via EDN vector +- `block add --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file +- `block remove --block ` - remove a block and its children +- `block remove --page ` - remove a page and its children +- `block search --text [--limit ]` - search block titles (Datalog includes?) +- `block tree --page [--format text|json|edn]` - show page tree +- `block tree --block [--format text|json|edn]` - show block tree + +Help output: + +``` +Subcommands: + block add [options] Add blocks + block remove [options] Remove block or page + block search [options] Search blocks + block tree [options] Show tree +``` + +Output formats: +- Global `--output ` (also accepted per subcommand) Examples: ```bash -node ./static/logseq-cli.js graph-create --graph demo --base-url http://127.0.0.1:9101 -node ./static/logseq-cli.js add --page TestPage --content "hello world" -node ./static/logseq-cli.js search --text "hello" -node ./static/logseq-cli.js tree --page TestPage --format json +node ./static/logseq-cli.js graph create --graph demo --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js block add --page TestPage --content "hello world" +node ./static/logseq-cli.js block search --text "hello" +node ./static/logseq-cli.js block tree --page TestPage --format json --output json ``` diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index bdf8a568d3..6c103a518c 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -1,10 +1,10 @@ (ns logseq.cli.commands "Command parsing and action building for the Logseq CLI." (:require ["fs" :as fs] + [babashka.cli :as cli] [cljs-time.coerce :as tc] [cljs.reader :as reader] [clojure.string :as string] - [clojure.tools.cli :as cli] [logseq.cli.config :as cli-config] [logseq.cli.transport :as transport] [logseq.common.config :as common-config] @@ -12,83 +12,307 @@ [logseq.common.util.date-time :as date-time-util] [promesa.core :as p])) -(def ^:private command->keyword - {"graph-list" :graph-list - "graph-create" :graph-create - "graph-switch" :graph-switch - "graph-remove" :graph-remove - "graph-validate" :graph-validate - "graph-info" :graph-info - "add" :add - "remove" :remove - "search" :search - "tree" :tree}) +(def ^:private global-spec + {:help {:alias :h + :desc "Show help" + :coerce :boolean} + :config {:desc "Path to cli.edn"} + :base-url {:desc "Base URL for db-worker-node"} + :host {:desc "Host for db-worker-node"} + :port {:desc "Port for db-worker-node" + :coerce :long} + :auth-token {:desc "Auth token for db-worker-node"} + :repo {:desc "Graph name"} + :timeout-ms {:desc "Request timeout in ms" + :coerce :long} + :retries {:desc "Retry count for requests" + :coerce :long} + :output {:desc "Output format (human, json, edn)"}}) -(def ^:private cli-options - [["-h" "--help" "Show help"] - [nil "--config PATH" "Path to cli.edn" - :id :config-path] - [nil "--base-url URL" "Base URL for db-worker-node"] - [nil "--host HOST" "Host for db-worker-node"] - [nil "--port PORT" "Port for db-worker-node" - :parse-fn #(js/parseInt % 10)] - [nil "--auth-token TOKEN" "Auth token for db-worker-node"] - [nil "--repo REPO" "Graph name"] - [nil "--graph GRAPH" "Graph name (alias for --repo in graph commands)"] - [nil "--timeout-ms MS" "Request timeout in ms" - :parse-fn #(js/parseInt % 10)] - [nil "--retries N" "Retry count for requests" - :parse-fn #(js/parseInt % 10)] - [nil "--json" "Output JSON" - :id :json? - :default false] - [nil "--format FORMAT" "Output format (tree)"] - [nil "--limit N" "Limit results" - :parse-fn #(js/parseInt % 10)] - [nil "--page PAGE" "Page name"] - [nil "--block UUID" "Block UUID"] - [nil "--parent UUID" "Parent block UUID for add"] - [nil "--content TEXT" "Block content for add"] - [nil "--blocks EDN" "EDN vector of blocks for add"] - [nil "--blocks-file PATH" "EDN file of blocks for add"] - [nil "--text TEXT" "Search text"]]) +(def ^:private graph-spec + {:graph {:desc "Graph name"}}) -(defn parse-args +(def ^:private content-add-spec + {:content {:desc "Block content for add"} + :blocks {:desc "EDN vector of blocks for add"} + :blocks-file {:desc "EDN file of blocks for add"} + :page {:desc "Page name"} + :parent {:desc "Parent block UUID for add"}}) + +(def ^:private content-remove-spec + {:block {:desc "Block UUID"} + :page {:desc "Page name"}}) + +(def ^:private content-search-spec + {:text {:desc "Search text"} + :limit {:desc "Limit results" + :coerce :long}}) + +(def ^:private content-tree-spec + {:block {:desc "Block UUID"} + :page {:desc "Page name"} + :format {:desc "Output format (tree)"}}) + +(defn- format-commands + [table] + (let [rows (->> table + (filter (comp seq :cmds)) + (map (fn [{:keys [cmds desc spec]}] + (let [command (str (string/join " " cmds) + (when (seq spec) " [options]"))] + {:command command + :desc desc})))) + width (apply max 0 (map (comp count :command) rows))] + (->> rows + (map (fn [{:keys [command desc]}] + (let [padding (apply str (repeat (- width (count command)) " "))] + (cond-> (str " " command padding) + (seq desc) (str " " desc))))) + (string/join "\n")))) + +(defn- group-summary + [group table] + (let [group-table (filter #(= group (first (:cmds %))) table)] + (string/join "\n" + [(str "Usage: logseq-cli " group " [options]") + "" + "Subcommands:" + (format-commands group-table) + "" + "Options:" + (cli/format-opts {:spec global-spec})]))) + +(defn- top-level-summary + [table] + (string/join "\n" + ["Usage: logseq-cli [options]" + "" + "Commands:" + (format-commands table) + "" + "Options:" + (cli/format-opts {:spec global-spec})])) + +(defn- command-summary + [{:keys [cmds spec]}] + (string/join "\n" + [(str "Usage: logseq-cli " (string/join " " cmds) " [options]") + "" + "Options:" + (cli/format-opts {:spec spec})])) + +(defn- merge-spec + [spec] + (merge global-spec (or spec {}))) + +(defn- normalize-opts + [opts] + (cond-> opts + (:config opts) (-> (assoc :config-path (:config opts)) + (dissoc :config)))) + +(defn- ok-result + [command opts args summary] + {:ok? true + :command command + :options (normalize-opts opts) + :args (vec args) + :summary summary}) + +(defn- missing-graph-result + [summary] + {:ok? false + :error {:code :missing-graph + :message "graph name is required"} + :summary summary}) + +(defn- missing-content-result + [summary] + {:ok? false + :error {:code :missing-content + :message "content is required"} + :summary summary}) + +(defn- missing-target-result + [summary] + {:ok? false + :error {:code :missing-target + :message "block or page is required"} + :summary summary}) + +(defn- missing-search-result + [summary] + {:ok? false + :error {:code :missing-search-text + :message "search text is required"} + :summary summary}) + +(defn- help-result + [summary] + {:ok? false + :help? true + :summary summary}) + +(defn- invalid-options-result + [summary message] + {:ok? false + :error {:code :invalid-options + :message message} + :summary summary}) + +(defn- unknown-command-result + [summary message] + {:ok? false + :error {:code :unknown-command + :message message} + :summary summary}) + +(defn- command-entry + [cmds command desc spec] + (let [spec* (merge-spec spec)] + {:cmds cmds + :desc desc + :spec spec* + :restrict true + :fn (fn [{:keys [opts args]}] + {:command command + :cmds cmds + :spec spec* + :opts opts + :args args})})) + +(def ^:private table + [(command-entry ["graph" "list"] :graph-list "List graphs" {}) + (command-entry ["graph" "create"] :graph-create "Create graph" graph-spec) + (command-entry ["graph" "switch"] :graph-switch "Switch current graph" graph-spec) + (command-entry ["graph" "remove"] :graph-remove "Remove graph" graph-spec) + (command-entry ["graph" "validate"] :graph-validate "Validate graph" graph-spec) + (command-entry ["graph" "info"] :graph-info "Graph metadata" graph-spec) + (command-entry ["block" "add"] :add "Add blocks" content-add-spec) + (command-entry ["block" "remove"] :remove "Remove block or page" content-remove-spec) + (command-entry ["block" "search"] :search "Search blocks" content-search-spec) + (command-entry ["block" "tree"] :tree "Show tree" content-tree-spec)]) + +(def ^:private global-aliases + (->> global-spec + (keep (fn [[k {:keys [alias]}]] + (when alias + [alias k]))) + (into {}))) + +(def ^:private global-flag-options + (->> global-spec + (keep (fn [[k {:keys [coerce]}]] + (when (= coerce :boolean) k))) + (set))) + +(defn- global-opt-key + [token] + (cond + (string/starts-with? token "--") + (keyword (subs token 2)) + + (and (string/starts-with? token "-") + (= 2 (count token))) + (get global-aliases (keyword (subs token 1))) + + :else nil)) + +(defn- parse-leading-global-opts [args] - (let [{:keys [options arguments errors summary]} (cli/parse-opts args cli-options) - command-str (first arguments) - command-args (vec (rest arguments)) - command (get command->keyword command-str)] + (loop [remaining args + opts {}] + (if (empty? remaining) + {:opts opts :args []} + (let [token (first remaining)] + (if-let [opt-key (global-opt-key token)] + (if (contains? global-flag-options opt-key) + (recur (rest remaining) (assoc opts opt-key true)) + (if-let [value (second remaining)] + (recur (drop 2 remaining) (assoc opts opt-key value)) + {:opts opts :args (rest remaining)})) + {:opts opts :args remaining}))))) + +(defn- unknown-command-message + [{:keys [dispatch wrong-input]}] + (string/join " " (cond-> (vec dispatch) + wrong-input (conj wrong-input)))) + +(defn- finalize-command + [summary {:keys [command opts args cmds spec]}] + (let [opts (normalize-opts opts) + args (vec args) + cmd-summary (command-summary {:cmds cmds :spec spec}) + graph (or (:graph opts) (:repo opts)) + has-args? (seq args) + has-content? (or (seq (:content opts)) + (seq (:blocks opts)) + (seq (:blocks-file opts)) + has-args?)] (cond - (seq errors) - {:ok? false - :error {:code :invalid-options - :message (string/join "\n" errors)} - :summary summary} + (:help opts) + (help-result cmd-summary) - (:help options) - {:ok? false - :help? true - :summary summary} + (and (#{:graph-create :graph-switch :graph-remove :graph-validate} command) + (not (seq graph))) + (missing-graph-result summary) - (nil? command-str) - {:ok? false - :error {:code :missing-command - :message "missing command"} - :summary summary} + (and (= command :add) (not has-content?)) + (missing-content-result summary) - (nil? command) - {:ok? false - :error {:code :unknown-command - :message (str "unknown command: " command-str)} - :summary summary} + (and (= command :remove) (not (or (seq (:block opts)) (seq (:page opts))))) + (missing-target-result summary) + + (and (= command :tree) (not (or (seq (:block opts)) (seq (:page opts))))) + (missing-target-result summary) + + (and (= command :search) (not (or (seq (:text opts)) has-args?))) + (missing-search-result summary) :else - {:ok? true - :command command - :options options - :args command-args - :summary summary}))) + (ok-result command opts args summary)))) + +(defn- cli-error->result + [summary {:keys [msg]}] + (invalid-options-result summary (or msg "invalid options"))) + +(defn parse-args + [raw-args] + (let [summary (top-level-summary table) + {:keys [opts args]} (parse-leading-global-opts raw-args)] + (if (empty? args) + (if (:help opts) + (help-result summary) + {:ok? false + :error {:code :missing-command + :message "missing command"} + :summary summary}) + (if (and (= 1 (count args)) (#{"graph" "block"} (first args))) + (help-result (group-summary (first args) table)) + (try + (let [result (cli/dispatch table args {:spec global-spec})] + (if (nil? result) + (unknown-command-result summary (str "unknown command: " (string/join " " args))) + (finalize-command summary (update result :opts #(merge opts (or % {})))))) + (catch :default e + (let [{:keys [cause] :as data} (ex-data e)] + (cond + (= cause :input-exhausted) + (if (:help opts) + (help-result summary) + {:ok? false + :error {:code :missing-command + :message "missing command"} + :summary summary}) + + (= cause :no-match) + (unknown-command-result summary (str "unknown command: " (unknown-command-message data))) + + (some? data) + (cli-error->result summary data) + + :else + (unknown-command-result summary (str "unknown command: " (string/join " " args))))))))))) (defn- graph->repo [graph] diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 7aaf2239fa..78ca0edef9 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -12,6 +12,18 @@ (when (and (some? value) (not (string/blank? value))) (js/parseInt value 10))) +(def ^:private output-formats + #{:human :json :edn}) + +(defn- parse-output-format + [value] + (let [kw (cond + (keyword? value) value + (string? value) (-> value string/trim string/lower-case keyword) + :else nil)] + (when (output-formats kw) + kw))) + (defn- default-config-path [] (node-path/join (.homedir os) ".logseq" "cli.edn")) @@ -57,6 +69,9 @@ (seq (gobj/get env "LOGSEQ_CLI_RETRIES")) (assoc :retries (parse-int (gobj/get env "LOGSEQ_CLI_RETRIES"))) + (seq (gobj/get env "LOGSEQ_CLI_OUTPUT")) + (assoc :output-format (parse-output-format (gobj/get env "LOGSEQ_CLI_OUTPUT"))) + (seq (gobj/get env "LOGSEQ_CLI_CONFIG")) (assoc :config-path (gobj/get env "LOGSEQ_CLI_CONFIG"))))) @@ -78,7 +93,14 @@ (:config-path env) (:config-path defaults)) file-config (or (read-config-file config-path) {}) + output-format (or (parse-output-format (:output-format opts)) + (parse-output-format (:output opts)) + (parse-output-format (:output-format env)) + (parse-output-format (:output env)) + (parse-output-format (:output-format file-config)) + (parse-output-format (:output file-config))) merged (merge defaults file-config env opts {:config-path config-path}) derived (build-base-url merged)] (cond-> merged + output-format (assoc :output-format output-format) (seq derived) (assoc :base-url derived)))) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index af5d4869fe..168debdafe 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq-cli [options]" "" - "Commands: graph-list, graph-create, graph-switch, graph-remove, graph-validate, graph-info, add, remove, search, tree" + "Commands: graph list, graph create, graph switch, graph remove, graph validate, graph info, block add, block remove, block search, block tree" "" "Options:" summary])) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index ae27b07842..c8276a8171 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1,14 +1,79 @@ (ns logseq.cli.commands-test - (:require [cljs.test :refer [deftest is testing]] + (:require [clojure.string :as string] + [cljs.test :refer [deftest is testing]] [logseq.cli.commands :as commands])) +(deftest test-help-output + (testing "top-level help lists subcommand groups" + (let [result (commands/parse-args ["--help"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "graph")) + (is (string/includes? summary "block"))))) + (deftest test-parse-args - (testing "rejects removed commands" - (doseq [command ["ping" "status" "query" "export"]] + (testing "graph group shows subcommands" + (let [result (commands/parse-args ["graph"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "graph list")) + (is (string/includes? summary "graph create")))) + + (testing "block group shows subcommands" + (let [result (commands/parse-args ["block"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "block add")) + (is (string/includes? summary "block search")))) + + (testing "graph group aligns subcommand columns" + (let [result (commands/parse-args ["graph"]) + summary (:summary result) + subcommand-lines (let [lines (string/split-lines summary) + start (inc (.indexOf lines "Subcommands:"))] + (->> lines + (drop start) + (take-while (complement string/blank?)))) + desc-starts (->> subcommand-lines + (keep (fn [line] + (when-let [[_ desc] (re-matches #"^\s+.*?\s{2,}(.+)$" line)] + (.indexOf line desc)))))] + (is (seq subcommand-lines)) + (is (apply = desc-starts)))) + + (testing "block group aligns subcommand columns" + (let [result (commands/parse-args ["block"]) + summary (:summary result) + subcommand-lines (let [lines (string/split-lines summary) + start (inc (.indexOf lines "Subcommands:"))] + (->> lines + (drop start) + (take-while (complement string/blank?)))) + desc-starts (->> subcommand-lines + (keep (fn [line] + (when-let [[_ desc] (re-matches #"^\s+.*?\s{2,}(.+)$" line)] + (.indexOf line desc)))))] + (is (seq subcommand-lines)) + (is (apply = desc-starts)))) + + (testing "rejects legacy commands" + (doseq [command ["graph-list" "graph-create" "graph-switch" "graph-remove" + "graph-validate" "graph-info" "add" "remove" "search" "tree" + "ping" "status" "query" "export"]] (let [result (commands/parse-args [command])] (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) + (testing "rejects removed commands" + (let [result (commands/parse-args ["graph" "wat"])] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code]))))) + + (testing "rejects removed group commands" + (let [result (commands/parse-args ["content" "add"])] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code]))))) + (testing "errors on missing command" (let [result (commands/parse-args [])] (is (false? (:ok? result))) @@ -17,9 +82,142 @@ (testing "errors on unknown command" (let [result (commands/parse-args ["wat"])] (is (false? (:ok? result))) - (is (= :unknown-command (get-in result [:error :code])))))) + (is (= :unknown-command (get-in result [:error :code]))))) -(deftest test-graph-commands + (testing "global output option is accepted" + (let [result (commands/parse-args ["--output" "json" "graph" "list"])] + (is (true? (:ok? result))) + (is (= "json" (get-in result [:options :output])))))) + +(deftest test-graph-subcommand-parse + (testing "graph list parses" + (let [result (commands/parse-args ["graph" "list"])] + (is (true? (:ok? result))) + (is (= :graph-list (:command result))))) + + (testing "graph create requires graph option" + (let [result (commands/parse-args ["graph" "create"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + + (testing "graph create parses with graph option" + (let [result (commands/parse-args ["graph" "create" "--graph" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-create (:command result))) + (is (= "demo" (get-in result [:options :graph]))))) + + (testing "graph switch requires graph option" + (let [result (commands/parse-args ["graph" "switch"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + + (testing "graph switch parses with graph option" + (let [result (commands/parse-args ["graph" "switch" "--graph" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-switch (:command result))) + (is (= "demo" (get-in result [:options :graph]))))) + + (testing "graph remove requires graph option" + (let [result (commands/parse-args ["graph" "remove"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + + (testing "graph remove parses with graph option" + (let [result (commands/parse-args ["graph" "remove" "--graph" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-remove (:command result))) + (is (= "demo" (get-in result [:options :graph]))))) + + (testing "graph validate requires graph option" + (let [result (commands/parse-args ["graph" "validate"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + + (testing "graph validate parses with graph option" + (let [result (commands/parse-args ["graph" "validate" "--graph" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-validate (:command result))) + (is (= "demo" (get-in result [:options :graph]))))) + + (testing "graph info parses without graph option" + (let [result (commands/parse-args ["graph" "info"])] + (is (true? (:ok? result))) + (is (= :graph-info (:command result))))) + + (testing "graph info parses with graph option" + (let [result (commands/parse-args ["graph" "info" "--graph" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-info (:command result))) + (is (= "demo" (get-in result [:options :graph]))))) + + (testing "graph subcommands reject unknown flags" + (doseq [subcommand ["list" "create" "switch" "remove" "validate" "info"]] + (let [result (commands/parse-args ["graph" subcommand "--wat"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))))) + + (testing "graph subcommands accept output option" + (let [result (commands/parse-args ["graph" "list" "--output" "edn"])] + (is (true? (:ok? result))) + (is (= "edn" (get-in result [:options :output]))))) + +(deftest test-block-subcommand-parse + (testing "block add requires content source" + (let [result (commands/parse-args ["block" "add"])] + (is (false? (:ok? result))) + (is (= :missing-content (get-in result [:error :code]))))) + + (testing "block add parses with content" + (let [result (commands/parse-args ["block" "add" "--content" "hello"])] + (is (true? (:ok? result))) + (is (= :add (:command result))) + (is (= "hello" (get-in result [:options :content]))))) + + (testing "block remove requires target" + (let [result (commands/parse-args ["block" "remove"])] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "block remove parses with block" + (let [result (commands/parse-args ["block" "remove" "--block" "demo"])] + (is (true? (:ok? result))) + (is (= :remove (:command result))) + (is (= "demo" (get-in result [:options :block]))))) + + (testing "block search requires text" + (let [result (commands/parse-args ["block" "search"])] + (is (false? (:ok? result))) + (is (= :missing-search-text (get-in result [:error :code]))))) + + (testing "block search parses with text" + (let [result (commands/parse-args ["block" "search" "--text" "hello"])] + (is (true? (:ok? result))) + (is (= :search (:command result))) + (is (= "hello" (get-in result [:options :text]))))) + + (testing "block tree requires target" + (let [result (commands/parse-args ["block" "tree"])] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "block tree parses with page" + (let [result (commands/parse-args ["block" "tree" "--page" "Home"])] + (is (true? (:ok? result))) + (is (= :tree (:command result))) + (is (= "Home" (get-in result [:options :page]))))) + + (testing "block subcommands reject unknown flags" + (doseq [subcommand ["add" "remove" "search" "tree"]] + (let [result (commands/parse-args ["block" subcommand "--wat"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) + + (testing "block subcommands accept output option" + (let [result (commands/parse-args ["block" "search" "--text" "hello" "--output" "json"])] + (is (true? (:ok? result))) + (is (= "json" (get-in result [:options :output])))))) + +(deftest test-build-action (testing "graph-list uses list-db" (let [parsed {:ok? true :command :graph-list :options {}} result (commands/build-action parsed {})] @@ -42,9 +240,8 @@ (let [parsed {:ok? true :command :graph-info :options {}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= :graph-info (get-in result [:action :type])))))) + (is (= :graph-info (get-in result [:action :type]))))) -(deftest test-content-commands (testing "add requires content" (let [parsed {:ok? true :command :add :options {}} result (commands/build-action parsed {:repo "demo"})] diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index ea974adf3e..6387e102c5 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -27,25 +27,29 @@ ":auth-token \"file-token\" " ":repo \"file-repo\" " ":timeout-ms 111 " - ":retries 1}")) + ":retries 1 " + ":output-format :edn}")) env {"LOGSEQ_DB_WORKER_URL" "http://env:9999" "LOGSEQ_DB_WORKER_AUTH_TOKEN" "env-token" "LOGSEQ_CLI_REPO" "env-repo" "LOGSEQ_CLI_TIMEOUT_MS" "222" - "LOGSEQ_CLI_RETRIES" "2"} + "LOGSEQ_CLI_RETRIES" "2" + "LOGSEQ_CLI_OUTPUT" "json"} opts {:config-path cfg-path :base-url "http://cli:1234" :auth-token "cli-token" :repo "cli-repo" :timeout-ms 333 - :retries 3} + :retries 3 + :output-format :human} result (with-env env #(config/resolve-config opts))] (is (= cfg-path (:config-path result))) (is (= "http://cli:1234" (:base-url result))) (is (= "cli-token" (:auth-token result))) (is (= "cli-repo" (:repo result))) (is (= 333 (:timeout-ms result))) - (is (= 3 (:retries result))))) + (is (= 3 (:retries result))) + (is (= :human (:output-format result))))) (deftest test-host-port-derived-base-url (let [result (config/resolve-config {:host "127.0.0.2" :port 9200})] @@ -61,6 +65,14 @@ (is (= "http://env:9999" (:base-url result))) (is (= "env-repo" (:repo result))))) +(deftest test-output-format-env-overrides-file + (let [dir (node-helper/create-tmp-dir) + cfg-path (node-path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :edn}") + env {"LOGSEQ_CLI_OUTPUT" "json"} + result (with-env env #(config/resolve-config {:config-path cfg-path}))] + (is (= :json (:output-format result))))) + (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index db06a6cf47..d28f058a63 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -3,23 +3,33 @@ [logseq.cli.format :as format])) (deftest test-format-success - (testing "json output" + (testing "json output via output-format" (let [result (format/format-result {:status :ok :data {:message "ok"}} - {:json? true})] + {:output-format :json})] (is (= "{\"status\":\"ok\",\"data\":{\"message\":\"ok\"}}" result)))) - (testing "human output" + (testing "edn output via output-format" (let [result (format/format-result {:status :ok :data {:message "ok"}} - {:json? false})] + {:output-format :edn})] + (is (= "{:status :ok, :data {:message \"ok\"}}" result)))) + + (testing "human output (default)" + (let [result (format/format-result {:status :ok :data {:message "ok"}} + {:output-format nil})] (is (= "ok" result))))) (deftest test-format-error - (testing "json error" + (testing "json error via output-format" (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} - {:json? true})] + {:output-format :json})] (is (= "{\"status\":\"error\",\"error\":{\"code\":\"boom\",\"message\":\"nope\"}}" result)))) - (testing "human error" + (testing "edn error via output-format" (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} - {:json? false})] + {:output-format :edn})] + (is (= "{:status :error, :error {:code :boom, :message \"nope\"}}" result)))) + + (testing "human error (default)" + (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} + {:output-format nil})] (is (= "error: nope" result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9a141b7030..9c861c5cae 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -9,7 +9,7 @@ (defn- run-cli [args url cfg-path] - (cli-main/run! (vec (concat args ["--base-url" url "--config" cfg-path "--json"])) + (cli-main/run! (vec (concat args ["--base-url" url "--config" cfg-path "--output" "json"])) {:exit? false})) (defn- parse-json-output @@ -24,7 +24,7 @@ :data-dir data-dir}) url (str "http://127.0.0.1:" (:port daemon)) cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - result (run-cli ["graph-list"] url cfg-path) + result (run-cli ["graph" "list"] url cfg-path) payload (parse-json-output result)] (is (= 0 (:exit-code result))) (is (= "ok" (:status payload))) @@ -44,9 +44,9 @@ url (str "http://127.0.0.1:" (:port daemon)) cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{}") - create-result (run-cli ["graph-create" "--graph" "demo-graph"] url cfg-path) + create-result (run-cli ["graph" "create" "--graph" "demo-graph"] url cfg-path) create-payload (parse-json-output create-result) - info-result (run-cli ["graph-info"] url cfg-path) + info-result (run-cli ["graph" "info"] url cfg-path) info-payload (parse-json-output info-result)] (is (= 0 (:exit-code create-result))) (is (= "ok" (:status create-payload))) @@ -68,15 +68,15 @@ url (str "http://127.0.0.1:" (:port daemon)) cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{}") - _ (run-cli ["graph-create" "--graph" "content-graph"] url cfg-path) - add-result (run-cli ["add" "--page" "TestPage" "--content" "hello world"] url cfg-path) + _ (run-cli ["graph" "create" "--graph" "content-graph"] url cfg-path) + add-result (run-cli ["block" "add" "--page" "TestPage" "--content" "hello world"] url cfg-path) _ (parse-json-output add-result) - search-result (run-cli ["search" "--text" "hello world"] url cfg-path) + search-result (run-cli ["block" "search" "--text" "hello world"] url cfg-path) search-payload (parse-json-output search-result) - tree-result (run-cli ["tree" "--page" "TestPage" "--format" "json"] url cfg-path) + tree-result (run-cli ["block" "tree" "--page" "TestPage" "--format" "json"] url cfg-path) tree-payload (parse-json-output tree-result) block-uuid (get-in tree-payload [:data :root :children 0 :uuid]) - remove-result (run-cli ["remove" "--block" (str block-uuid)] url cfg-path) + remove-result (run-cli ["block" "remove" "--block" (str block-uuid)] url cfg-path) remove-payload (parse-json-output remove-result)] (is (= 0 (:exit-code add-result))) (is (= "ok" (:status search-payload))) From 0bb650fad1fb5ead691642ce3b91a9ad96e576bb Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 15 Jan 2026 23:09:32 +0800 Subject: [PATCH 009/375] impl 002-logseq-cli-subcommands.md (2) --- .../agent-guide/002-logseq-cli-subcommands.md | 7 ++-- src/main/logseq/cli/commands.cljs | 2 +- src/main/logseq/cli/config.cljs | 1 - src/main/logseq/cli/format.cljs | 3 +- src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/config_test.cljs | 14 +++++++ src/test/logseq/cli/format_test.cljs | 7 ++++ src/test/logseq/cli/integration_test.cljs | 39 +++++++++++++++++-- 8 files changed, 63 insertions(+), 12 deletions(-) diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index ca58e2009f..d3af8abe6f 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -104,9 +104,8 @@ Each subcommand uses a nested path and its own options. ## Status -- Completed: Plan tasks 1-6. -- In progress: Plan task 7. -- Pending: Plan tasks 8-19. +- Completed: Plan tasks 1-10, 12-19. +- Skipped: Plan task 11 (red-phase confirmation no longer applicable after parser swap). ## Edge cases @@ -130,7 +129,7 @@ Expected output includes a failing assertion about subcommand help text and ends Run the full unit test suite in green phase. ```bash -bb dev:test -v logseq.cli.* +bb dev:test -r logseq.cli.* ``` Expected output includes 0 failures and 0 errors. diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 6c103a518c..3b4d89868d 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -710,7 +710,7 @@ :tree (-> (p/let [tree-data (fetch-tree config action) - format (or (:format action) (when (:json? config) "json"))] + format (:format action)] (case format "edn" {:status :ok diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 78ca0edef9..57cdc7eef7 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -85,7 +85,6 @@ (let [defaults {:base-url "http://127.0.0.1:9101" :timeout-ms 10000 :retries 0 - :json? false :output-format nil :config-path (default-config-path)} env (env-config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index e9631f734d..5cbb5fe625 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -42,11 +42,10 @@ (= status :error) (assoc :error error)))) (defn format-result - [result {:keys [json? output-format]}] + [result {:keys [output-format]}] (let [format (cond (= output-format :edn) :edn (= output-format :json) :json - json? :json :else :human)] (case format :json (->json result) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 168debdafe..187acaca4c 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -30,7 +30,7 @@ (p/resolved {:exit-code 1 :output (format/format-result {:status :error :error (:error parsed)} - {:json? false})}) + {})}) :else (let [cfg (config/resolve-config (:options parsed)) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 6387e102c5..c37e015683 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -73,6 +73,20 @@ result (with-env env #(config/resolve-config {:config-path cfg-path}))] (is (= :json (:output-format result))))) +(deftest test-output-format-precedence + (let [dir (node-helper/create-tmp-dir) + cfg-path (node-path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :edn}") + env {"LOGSEQ_CLI_OUTPUT" "json"} + result (with-env env #(config/resolve-config {:config-path cfg-path + :output "human"}))] + (is (= :human (:output-format result))))) + +(deftest test-output-format-overrides-output + (let [result (config/resolve-config {:output-format :edn + :output "json"})] + (is (= :edn (:output-format result))))) + (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index d28f058a63..32e099764d 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -18,6 +18,13 @@ {:output-format nil})] (is (= "ok" result))))) +(deftest test-format-ignores-legacy-json-flag + (testing "json? flag does not override output-format" + (let [result (format/format-result {:status :ok :data {:message "ok"}} + {:output-format nil + :json? true})] + (is (= "ok" result))))) + (deftest test-format-error (testing "json error via output-format" (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9c861c5cae..32f5edae7d 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1,5 +1,7 @@ (ns logseq.cli.integration-test - (:require [cljs.test :refer [deftest is async]] + (:require [cljs.reader :as reader] + [cljs.test :refer [deftest is async]] + [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker.db-worker-node :as db-worker-node] [logseq.cli.main :as cli-main] @@ -9,13 +11,20 @@ (defn- run-cli [args url cfg-path] - (cli-main/run! (vec (concat args ["--base-url" url "--config" cfg-path "--output" "json"])) - {:exit? false})) + (let [args-with-output (if (some #{"--output"} args) + args + (concat args ["--output" "json"]))] + (cli-main/run! (vec (concat args-with-output ["--base-url" url "--config" cfg-path])) + {:exit? false}))) (defn- parse-json-output [result] (js->clj (js/JSON.parse (:output result)) :keywordize-keys true)) +(defn- parse-edn-output + [result] + (reader/read-string (:output result))) + (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -88,3 +97,27 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest test-cli-output-formats-graph-list + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" + :port 0 + :data-dir data-dir}) + url (str "http://127.0.0.1:" (:port daemon)) + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + json-result (run-cli ["graph" "list" "--output" "json"] url cfg-path) + json-payload (parse-json-output json-result) + edn-result (run-cli ["graph" "list" "--output" "edn"] url cfg-path) + edn-payload (parse-edn-output edn-result) + human-result (run-cli ["graph" "list" "--output" "human"] url cfg-path)] + (is (= 0 (:exit-code json-result))) + (is (= "ok" (:status json-payload))) + (is (= 0 (:exit-code edn-result))) + (is (= :ok (:status edn-payload))) + (is (not (string/starts-with? (:output human-result) "{:status"))) + (p/let [_ ((:stop! daemon))] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) From 9fe28950d30a651b73cf9eec10f5a142349a9b12 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 15 Jan 2026 23:15:35 +0800 Subject: [PATCH 010/375] fix compile warnings --- src/main/frontend/worker/db_worker_node.cljs | 2 +- src/main/frontend/worker/platform/node.cljs | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 59b56ed7c8..a6271feb8c 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -88,7 +88,7 @@ (swap! *sse-clients disj res)))) (defn- buffer data))))) (defn- remove-vfs! - [pool] + [^js pool] (when pool (fs/rm (.-repoDir pool) #js {:recursive true :force true}))) From 76c60e5e39fa64607384c3348a3093a0c9a86fcd Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 16 Jan 2026 21:43:56 +0800 Subject: [PATCH 011/375] update package.json --- package.json | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/package.json b/package.json index b00c94c092..4186c8f5c2 100644 --- a/package.json +++ b/package.json @@ -3,6 +3,12 @@ "version": "0.0.1", "private": true, "main": "static/electron.js", + "bin": { + "logseq": "static/logseq-cli.js" + }, + "files": [ + "static/db-worker-node.js" + ], "engines": { "node": ">=22.20.0" }, From be40f5ade63e851e9789fa10af04369a204e6b37 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 16 Jan 2026 22:44:58 +0800 Subject: [PATCH 012/375] add 003-db-worker-node-cli-orchestration.md --- .gitignore | 1 + .../003-db-worker-node-cli-orchestration.md | 115 ++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 docs/agent-guide/003-db-worker-node-cli-orchestration.md diff --git a/.gitignore b/.gitignore index 4d22d6e9e8..44c09b6fab 100644 --- a/.gitignore +++ b/.gitignore @@ -82,3 +82,4 @@ clj-e2e/e2e-dump .dir-locals.el .projectile deps/db-sync/data +*.map diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md new file mode 100644 index 0000000000..27e650ff31 --- /dev/null +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -0,0 +1,115 @@ +# db-worker-node and logseq-cli Orchestration Plan + +Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, refactor db-worker-node to be repo-bound with locking, make logseq-cli fully manage db-worker-node lifecycle, and add server subcommands for management. + +## Background and Current State (from existing code) + +- `db-worker-node` currently accepts `--repo` but it is optional; it can open/switch graphs via `thread-api/create-or-open-db` at runtime. Entrypoint: `src/main/frontend/worker/db_worker_node.cljs`. +- `logseq-cli` connects to an existing db-worker-node via `base-url`; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`. +- Tests explicitly start db-worker-node (`src/test/logseq/cli/integration_test.cljs`, `src/test/frontend/worker/db_worker_node_test.cljs`). + +## Requirements + +1. Refactor `db-worker-node`: startup must require `--repo`; on startup it must open or create that graph; it must not switch graphs at runtime; it must create a lock file so a graph can be served by only one db-worker-node instance; it only needs to bind to localhost. +2. In `logseq-cli`, all commands requiring `--repo` or any graph operations must connect to or create the corresponding db-worker-node server. +3. db-worker-node server must not be started manually; logseq-cli is fully responsible. +4. Add `server` subcommand(s) to logseq-cli for managing db-worker-node servers. + +## Scope / Non-goals + +- In scope: db-worker-node startup and lifecycle, repo binding enforcement, lock files, CLI server orchestration, management commands, tests/docs. +- Out of scope: changing db-worker core query/write protocol; changing db-worker-node HTTP API semantics beyond repo binding constraints. + +## Proposed Design + +- **Repo-bound server**: db-worker-node opens a single repo at startup and refuses repo changes for the lifetime of the process. It only listens on localhost. +- **Lock file**: each repo directory has a lock file to ensure one server per graph. Lock contains metadata for status/cleanup; db-worker-node handles it by default, and logseq-cli handles cases db-worker-node cannot. +- **CLI orchestration**: logseq-cli discovers/starts/reuses db-worker-node servers per repo. It is the only entrypoint for starting servers. +- **Server subcommands**: add `server list|status|start|stop|restart` (or similar) to manage servers explicitly. + +## Detailed Design + +### 1) db-worker-node: required repo, repo binding, lock file + +Files: +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/platform/node.cljs` (for data-dir / repo dir resolution) +- Optional new helper: `src/main/frontend/worker/db_worker_node_lock.cljs` + +Key changes: +- **Argument parsing**: `--repo` becomes required. If missing, print help and exit non-zero. Host binding is restricted to localhost (e.g., `127.0.0.1`) regardless of input. +- **Startup flow**: replace `/db-worker.lock`). + - Content: JSON `{repo, pid, host, port, startedAt}`. + - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”. + - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM. + - Stale lock: if lock exists but pid is dead, allow replacement (db-worker-node first; CLI can repair when server cannot). + +### 2) logseq-cli: auto start/reuse db-worker-node per repo + +Files: +- `src/main/logseq/cli/commands.cljs` +- `src/main/logseq/cli/main.cljs` +- `src/main/logseq/cli/config.cljs` +- New: `src/main/logseq/cli/server.cljs` (process management + lock handling) + +Key changes: +- **Repo resolution**: for all graph/content commands, require `--repo` or resolved repo from config; otherwise error. +- **Ensure server** (new helper `ensure-server!`): + 1. Derive data-dir, repo dir, and lock file path from repo. + 2. If lock file exists, read host/port/pid; probe `/healthz` + `/readyz`. + 3. If healthy, reuse existing server; set `config :base-url` dynamically. + 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. + 5. Spawn via `child_process.spawn`: `node ./static/db-worker-node.js --repo --data-dir <...> --host 127.0.0.1 --port 0`. + 6. Resolve actual port from lock file or server output. +- **base-url usage**: dynamically set based on repo server; no longer required from user. If `--base-url` is provided, decide if it is ignored or overrides orchestration (see Compatibility section). + +### 3) CLI `server` subcommands + +Suggested command group: +- `server list`: list servers from lock files (repo, pid, port, status). +- `server start --repo `: start server for repo. +- `server stop --repo `: stop server (SIGTERM or `/v1/shutdown`). +- `server restart --repo `: stop + start. + +Implementation notes: +- `start|stop|restart` require `--repo`. +- `list` scans data-dir for repo directories, reads lock files, and verifies status. +- Consider adding `/v1/shutdown` in db-worker-node for graceful stop. + +## Compatibility / Migration + +- No need to preserve compatibility for existing env vars, config keys, or flags related to db-worker-node or CLI server connectivity; remove them if they are no longer needed. + +## Test Plan + +- **Unit tests**: + - CLI: repo resolution, server orchestration logic, lock parsing, error codes (`src/test/logseq/cli/*`). + - db-worker-node: repo required, repo mismatch rejection, lock file create/cleanup (`src/test/frontend/worker/db_worker_node_test.cljs`). +- **Integration tests**: + - CLI runs graph/content commands without manual server startup (`src/test/logseq/cli/integration_test.cljs`). + - Concurrent startup of same repo must fail due to lock. + +## Milestones + +1. **db-worker-node binding & lock file**: repo required + repo enforcement + lock creation/cleanup. +2. **CLI server module**: `ensure-server!` with lock/health checks and spawning. +3. **CLI command updates**: graph/content commands require repo and auto-start server; add `server` subcommands. +4. **Tests + docs**: update/extend tests and adjust CLI docs (`docs/cli/logseq-cli.md`). + +## Open Questions + +1. Should `graph list` require `--repo`? If not, define a “global” server or out-of-band access to data-dir. + - Answer: No --repo needed, using 'out-of-band access to data-dir' way +2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). + - lockfile name:`db-worker.lock`, + - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-/db-worker.lock`). + - only consider linux/macos for now +3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle. +4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI? +5. db-worker-node servers should keep running unless `logseq-cli server stop` is invoked or the process exits unexpectedly; in the latter case, CLI should handle lockfile cleanup on restart. From 44d2f3ebe2b5c34359464c0454eca72d52e49349 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 17 Jan 2026 21:59:01 +0800 Subject: [PATCH 013/375] 003-db-worker-node-cli-orchestration.md (2) --- deps/cli/src/logseq/cli.cljs | 4 +- deps/db/src/logseq/db/sqlite/gc.cljs | 16 +- docs/agent-guide/001-logseq-cli.md | 8 +- .../agent-guide/002-logseq-cli-subcommands.md | 19 +- .../003-db-worker-node-cli-orchestration.md | 12 +- .../task--db-worker-nodejs-compatible.md | 3 +- docs/cli/logseq-cli.md | 35 +- src/main/frontend/persist_db/node.cljs | 4 +- src/main/frontend/worker/db_worker_node.cljs | 184 ++++++--- .../frontend/worker/db_worker_node_lock.cljs | 97 +++++ .../frontend/worker/platform/browser.cljs | 2 +- src/main/frontend/worker/platform/node.cljs | 4 +- src/main/logseq/cli/commands.cljs | 348 ++++++++++++------ src/main/logseq/cli/config.cljs | 21 +- src/main/logseq/cli/main.cljs | 2 +- src/main/logseq/cli/server.cljs | 321 ++++++++++++++++ .../frontend/worker/db_worker_node_test.cljs | 95 ++++- src/test/logseq/cli/commands_test.cljs | 138 +++++-- src/test/logseq/cli/config_test.cljs | 26 +- src/test/logseq/cli/integration_test.cljs | 88 ++--- src/test/logseq/cli/server_test.cljs | 53 +++ src/test/logseq/cli/transport_test.cljs | 2 +- 22 files changed, 1157 insertions(+), 325 deletions(-) create mode 100644 src/main/frontend/worker/db_worker_node_lock.cljs create mode 100644 src/main/logseq/cli/server.cljs create mode 100644 src/test/logseq/cli/server_test.cljs diff --git a/deps/cli/src/logseq/cli.cljs b/deps/cli/src/logseq/cli.cljs index cfcd4557fc..e9dc1af7b0 100644 --- a/deps/cli/src/logseq/cli.cljs +++ b/deps/cli/src/logseq/cli.cljs @@ -109,7 +109,7 @@ :args->opts [:args] :require [:args] :coerce {:args []} :spec cli-spec/append} {:cmds ["mcp-server"] :desc "Run a MCP server" - :description "Run a MCP server against a local graph if --graph is given or against the current in-app graph. For the in-app graph, the API server must be on in the app. By default the MCP server runs as a HTTP Streamable server. Use --stdio to run it as a stdio server." + :description "Run a MCP server against a local graph if --repo is given or against the current in-app graph. For the in-app graph, the API server must be on in the app. By default the MCP server runs as a HTTP Streamable server. Use --stdio to run it as a stdio server." :fn (lazy-load-fn 'logseq.cli.commands.mcp-server/start) :spec cli-spec/mcp-server} {:cmds ["validate"] :desc "Validate DB graph" @@ -159,4 +159,4 @@ (nbb.error/print-error-report e) (js/process.exit 1)))) -#js {:main -main} \ No newline at end of file +#js {:main -main} diff --git a/deps/db/src/logseq/db/sqlite/gc.cljs b/deps/db/src/logseq/db/sqlite/gc.cljs index 08295ca919..9e6ace61ca 100644 --- a/deps/db/src/logseq/db/sqlite/gc.cljs +++ b/deps/db/src/logseq/db/sqlite/gc.cljs @@ -57,12 +57,12 @@ (println :debug :db-gc "There's no garbage data that's need to be collected."))))) (defn- get-unused-addresses-node-version - [db] - (let [schema (let [stmt (.prepare db "select content from kvs where addr = ?") + [^js db] + (let [schema (let [^js stmt (.prepare db "select content from kvs where addr = ?") content (.-content (.get stmt 0))] (sqlite-util/transit-read content)) internal-addrs (set [0 1 (:eavt schema) (:avet schema) (:aevt schema)]) - non-refed-addrs (let [stmt (.prepare db get-non-refed-addrs-sql)] + non-refed-addrs (let [^js stmt (.prepare db get-non-refed-addrs-sql)] (->> (.all ^object stmt) bean/->clj (map :addr) @@ -70,13 +70,13 @@ (set/difference non-refed-addrs internal-addrs))) (defn- get-unused-addresses-node-walk-version - [db] - (let [schema (let [stmt (.prepare db "select content from kvs where addr = ?") + [^js db] + (let [schema (let [^js stmt (.prepare db "select content from kvs where addr = ?") content (.-content (.get stmt 0))] (sqlite-util/transit-read content)) set-addresses #{(:eavt schema) (:avet schema) (:aevt schema)} internal-addresses (conj set-addresses 0 1) - parent->children (let [stmt (.prepare db "select addr, addresses from kvs")] + parent->children (let [^js stmt (.prepare db "select addr, addresses from kvs")] (->> (.all ^object stmt) bean/->clj (map (fn [{:keys [addr addresses]}] @@ -95,13 +95,13 @@ (let [unused-addresses (if walk? (get-unused-addresses-node-walk-version db) (get-unused-addresses-node-version db)) - addrs-count (let [stmt (.prepare db "select count(*) as c from kvs")] + addrs-count (let [^js stmt (.prepare db "select count(*) as c from kvs")] (.-c (.get stmt)))] (println :debug "addrs total count: " addrs-count) (if (seq unused-addresses) (do (println :debug :db-gc :unused-addresses-count (count unused-addresses)) - (let [stmt (.prepare db "Delete from kvs where addr = ?") + (let [^js stmt (.prepare db "Delete from kvs where addr = ?") delete (.transaction db (fn [addrs] diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md index bb4e10f165..aca73ea487 100644 --- a/docs/agent-guide/001-logseq-cli.md +++ b/docs/agent-guide/001-logseq-cli.md @@ -28,19 +28,19 @@ NOTE: I will write *all* tests before I add any implementation behavior. ## Architecture sketch The CLI is a Node program that parses flags, loads config, and sends requests to db-worker-node. -The db-worker-node server is already built from the :db-worker-node shadow-cljs target and listens on a TCP port. +The db-worker-node server is already built from the :db-worker-node shadow-cljs target and listens on a random localhost TCP port recorded in the lock file. ASCII diagram: +--------------+ HTTP or WS +---------------------+ | logseq-cli | -----------------------> | db-worker-node | -| node script | <----------------------- | server on port 9101 | +| node script | <----------------------- | server on random port | +--------------+ +---------------------+ ## Assumptions The db-worker-node server exposes a stable API for a small set of requests needed by the CLI. -The CLI will default to localhost:9101 unless configured otherwise. +The CLI always uses localhost and discovers the server port from the lock file. The CLI will use JSON for request and response bodies for ease of scripting. ## Implementation plan @@ -98,7 +98,7 @@ Open follow-ups (optional): ## Edge cases -The db-worker-node server is not running or is listening on a different port. +The db-worker-node server is not running or the lock file points to a stale server. The response payload is invalid JSON or missing fields. The request times out or the server closes the connection early. The user passes incompatible flags or unknown commands. diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index d3af8abe6f..5824f5c475 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -25,8 +25,8 @@ NOTE: I will write all tests before I add any implementation behavior. ## Architecture sketch +--------------+ HTTP +---------------------+ -| logseq-cli | -----------------> | db-worker-node | -| node script | <----------------- | server on port 9101 | +| logseq-cli | -----------------> | db-worker-node | +| node script | <----------------- | server on random port | +--------------+ +---------------------+ ## Command and output surface @@ -56,9 +56,6 @@ Global options apply to all subcommands and are parsed before subcommand options | --- | --- | --- | | --help | Show help | Available at top level and per subcommand. | | --config PATH | Config file path | Defaults to ~/.logseq/cli.edn. | -| --base-url URL | Server URL | Overrides host/port. | -| --host HOST | Server host | Combined with --port. | -| --port PORT | Server port | Combined with --host. | | --auth-token TOKEN | Auth token | Sent as header. | | --repo REPO | Graph name | Used as current repo. | | --timeout-ms MS | Request timeout | Integer milliseconds. | @@ -70,11 +67,11 @@ Each subcommand uses a nested path and its own options. | Subcommand path | Required args | Options | Notes | | --- | --- | --- | --- | | graph list | none | --output | Lists all graphs. | -| graph create | none | --graph GRAPH, --output | Creates and switches graph. | -| graph switch | none | --graph GRAPH, --output | Switches current graph. | -| graph remove | none | --graph GRAPH, --output | Removes graph. | -| graph validate | none | --graph GRAPH, --output | Validates graph. | -| graph info | none | --graph GRAPH, --output | Shows metadata, defaults to config repo if omitted. | +| graph create | none | --repo GRAPH, --output | Creates and switches graph. | +| graph switch | none | --repo GRAPH, --output | Switches current graph. | +| graph remove | none | --repo GRAPH, --output | Removes graph. | +| graph validate | none | --repo GRAPH, --output | Validates graph. | +| graph info | none | --repo GRAPH, --output | Shows metadata, defaults to config repo if omitted. | | block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. | | block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. | | block search | none | --text TEXT, --limit N, --output | Search text is required. | @@ -145,7 +142,7 @@ Expected output includes successful linting and tests with exit code 0. ## Testing Details The unit tests will exercise parsing and output formatting behavior without mocking internal parser details. -The integration tests will start db-worker-node on a test port and invoke the CLI entrypoint with subcommands to verify end-to-end behavior. +The integration tests will start db-worker-node on a random port and invoke the CLI entrypoint with subcommands to verify end-to-end behavior. ## Implementation Details diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index 27e650ff31..6303688cd9 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -5,7 +5,7 @@ Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, re ## Background and Current State (from existing code) - `db-worker-node` currently accepts `--repo` but it is optional; it can open/switch graphs via `thread-api/create-or-open-db` at runtime. Entrypoint: `src/main/frontend/worker/db_worker_node.cljs`. -- `logseq-cli` connects to an existing db-worker-node via `base-url`; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`. +- `logseq-cli` connects to an existing db-worker-node on localhost using the port recorded in the lock file; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`. - Tests explicitly start db-worker-node (`src/test/logseq/cli/integration_test.cljs`, `src/test/frontend/worker/db_worker_node_test.cljs`). ## Requirements @@ -62,12 +62,12 @@ Key changes: - **Repo resolution**: for all graph/content commands, require `--repo` or resolved repo from config; otherwise error. - **Ensure server** (new helper `ensure-server!`): 1. Derive data-dir, repo dir, and lock file path from repo. - 2. If lock file exists, read host/port/pid; probe `/healthz` + `/readyz`. - 3. If healthy, reuse existing server; set `config :base-url` dynamically. + 2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`. + 3. If healthy, reuse existing server; build the connection URL from localhost and the lock file port. 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. - 5. Spawn via `child_process.spawn`: `node ./static/db-worker-node.js --repo --data-dir <...> --host 127.0.0.1 --port 0`. - 6. Resolve actual port from lock file or server output. -- **base-url usage**: dynamically set based on repo server; no longer required from user. If `--base-url` is provided, decide if it is ignored or overrides orchestration (see Compatibility section). + 5. Spawn via `child_process.spawn`: `node ./static/db-worker-node.js --repo --data-dir <...>`. + 6. Resolve actual port from the lock file written by db-worker-node. +- **Connection URL**: derived from the repo lock file; host is always localhost and the port is always discovered from the lock file. ### 3) CLI `server` subcommands diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index 99ee3e28e8..fc80681804 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -128,8 +128,7 @@ The db-worker should be runnable as a standalone process for Node.js environment ### Entry Point - Provide a CLI entry (example: `bin/logseq-db-worker` or `node dist/db-worker-node.js`). - CLI flags (suggested): - - `--host` (default `127.0.0.1`) - - `--port` (default `9101`) + - Binds to localhost on a random port and records it in the repo lock file. - `--data-dir` (path for sqlite files, required or default to `~/.logseq/db-worker`) - `--repo` (optional: auto-open a repo on boot) - `--rtc-ws-url` (optional) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 6b1f1ee56b..21dcdc2ce3 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -1,6 +1,6 @@ # Logseq CLI (Node) -The Logseq CLI is a Node.js program compiled from ClojureScript that connects to the db-worker-node server. +The Logseq CLI is a Node.js program compiled from ClojureScript that connects to a db-worker-node server managed by the CLI. ## Build the CLI @@ -8,17 +8,14 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to clojure -M:cljs compile logseq-cli ``` -## Start db-worker-node (in another terminal) +## db-worker-node lifecycle -```bash -clojure -M:cljs compile db-worker-node -node ./static/db-worker-node.js -``` +`logseq-cli` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. ## Run the CLI ```bash -node ./static/logseq-cli.js graph list --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js graph list ``` ## Configuration @@ -26,9 +23,9 @@ node ./static/logseq-cli.js graph list --base-url http://127.0.0.1:9101 Optional configuration file: `~/.logseq/cli.edn` Supported keys include: -- `:base-url` - `:auth-token` - `:repo` +- `:data-dir` - `:timeout-ms` - `:retries` - `:output-format` (use `:json` or `:edn` for scripting) @@ -39,11 +36,20 @@ CLI flags take precedence over environment variables, which take precedence over Graph commands: - `graph list` - list all db graphs -- `graph create --graph ` - create a new db graph and switch to it -- `graph switch --graph ` - switch current graph -- `graph remove --graph ` - remove a graph -- `graph validate --graph ` - validate graph data -- `graph info [--graph ]` - show graph metadata (defaults to current graph) +- `graph create --repo ` - create a new db graph and switch to it +- `graph switch --repo ` - switch current graph +- `graph remove --repo ` - remove a graph +- `graph validate --repo ` - validate graph data +- `graph info [--repo ]` - show graph metadata (defaults to current graph) + +For any command that requires `--repo`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). + +Server commands: +- `server list` - list running db-worker-node servers +- `server status --repo ` - show server status for a graph +- `server start --repo ` - start db-worker-node for a graph +- `server stop --repo ` - stop db-worker-node for a graph +- `server restart --repo ` - restart db-worker-node for a graph Block commands: - `block add --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given @@ -71,8 +77,9 @@ Output formats: Examples: ```bash -node ./static/logseq-cli.js graph create --graph demo --base-url http://127.0.0.1:9101 +node ./static/logseq-cli.js graph create --repo demo node ./static/logseq-cli.js block add --page TestPage --content "hello world" node ./static/logseq-cli.js block search --text "hello" node ./static/logseq-cli.js block tree --page TestPage --format json --output json +node ./static/logseq-cli.js server list ``` diff --git a/src/main/frontend/persist_db/node.cljs b/src/main/frontend/persist_db/node.cljs index 19986a41b7..54f6fb166d 100644 --- a/src/main/frontend/persist_db/node.cljs +++ b/src/main/frontend/persist_db/node.cljs @@ -38,7 +38,7 @@ [method url headers body] (p/create (fn [resolve reject] - (let [req (.request (request-module url) + (let [^js req (.request (request-module url) #js {:method method :hostname (.-hostname url) :port (or (.-port url) (if (= "https:" (.-protocol url)) 443 80)) @@ -84,7 +84,7 @@ headers (base-headers auth-token) buffer (atom "") handler (or event-handler (fn [_type _payload _wrapped-worker] nil))] - (let [req (.request + (let [^js req (.request (request-module url) #js {:method "GET" :hostname (.-hostname url) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index a6271feb8c..afbbc792ea 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -3,6 +3,7 @@ (:require ["http" :as http] [clojure.string :as string] [frontend.worker.db-core :as db-core] + [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.platform.node :as platform-node] [frontend.worker.state :as worker-state] [goog.object :as gobj] @@ -13,6 +14,7 @@ (defonce ^:private *ready? (atom false)) (defonce ^:private *sse-clients (atom #{})) +(defonce ^:private *lock-info (atom nil)) (defn- send-json! [^js res status payload] @@ -50,8 +52,6 @@ opts (let [[flag value & remaining] args] (case flag - "--host" (recur remaining (assoc opts :host value)) - "--port" (recur remaining (assoc opts :port (js/parseInt value 10))) "--data-dir" (recur remaining (assoc opts :data-dir value)) "--repo" (recur remaining (assoc opts :repo value)) "--rtc-ws-url" (recur remaining (assoc opts :rtc-ws-url value)) @@ -68,8 +68,8 @@ (defn- handle-event! [type payload] - (let [event (js/JSON.stringify (clj->js {:type type - :payload (encode-event-payload payload)})) + (let [event (js/JSON.stringify (clj->js {:type type} + :payload (encode-event-payload payload))) message (str "data: " event "\n\n")] (doseq [^js res @*sse-clients] (try @@ -109,10 +109,38 @@ [proxy rtc-ws-url] ( (default 127.0.0.1)") - (println " --port (default 9101)") (println " --data-dir (default ~/.logseq/db-worker)") - (println " --repo (optional)") + (println " --repo (required)") (println " --rtc-ws-url (optional)") (println " --log-level (default info)") (println " --auth-token (optional)")) (defn start-daemon! - [{:keys [host port data-dir repo rtc-ws-url auth-token]}] - (let [host (or host "127.0.0.1") - port (or port 9101)] - (reset! *ready? false) - (set-main-thread-stub!) - (p/let [platform (platform-node/node-platform {:data-dir data-dir - :event-fn handle-event!}) - proxy (db-core/init-core! platform) - _ ( (p/let [platform (platform-node/node-platform {:data-dir data-dir + :event-fn handle-event!}) + proxy (db-core/init-core! platform) + _ (clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true))) + +(defn remove-lock! + [path] + (when (and (seq path) (fs/existsSync path)) + (fs/unlinkSync path))) + +(defn create-lock! + [{:keys [data-dir repo host port]}] + (p/create + (fn [resolve reject] + (try + (let [data-dir (resolve-data-dir data-dir) + path (lock-path data-dir repo) + existing (read-lock path)] + (when (and existing (pid-alive? (:pid existing))) + (throw (ex-info "graph already locked" {:code :repo-locked :lock existing}))) + (when existing + (remove-lock! path)) + (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) + (let [fd (fs/openSync path "wx") + lock {:repo repo + :pid (.-pid js/process) + :host host + :port port + :startedAt (.toISOString (js/Date.))}] + (try + (fs/writeFileSync fd (js/JSON.stringify (clj->js lock))) + (finally + (fs/closeSync fd))) + (resolve lock))) + (catch :default e + (log/error :db-worker-node-lock-create-failed e) + (reject e)))))) + +(defn update-lock! + [path lock] + (p/create + (fn [resolve reject] + (try + (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) + (resolve lock) + (catch :default e + (log/error :db-worker-node-lock-update-failed e) + (reject e)))))) + +(defn ensure-lock! + [{:keys [data-dir repo host port]}] + (let [data-dir (resolve-data-dir data-dir) + path (lock-path data-dir repo)] + (p/let [lock (create-lock! {:data-dir data-dir + :repo repo + :host host + :port port})] + {:path path + :lock lock}))) diff --git a/src/main/frontend/worker/platform/browser.cljs b/src/main/frontend/worker/platform/browser.cljs index ff914e29b8..32a4b2b572 100644 --- a/src/main/frontend/worker/platform/browser.cljs +++ b/src/main/frontend/worker/platform/browser.cljs @@ -98,7 +98,7 @@ (defn- open-sqlite-db [{:keys [sqlite pool path mode]}] (if pool - (new (.-OpfsSAHPoolDb pool) path) + (new (.-OpfsSAHPoolDb ^js pool) path) (let [^js DB (.-DB ^js (.-oo1 ^js sqlite))] (new DB path (or mode "c"))))) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index 3c59589de6..afa84644cf 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -77,7 +77,7 @@ (p/catch (fn [_] false))))) (defn- exec-sql - [db opts-or-sql] + [^js db opts-or-sql] (if (string? opts-or-sql) (.exec db opts-or-sql) (let [sql (gobj/get opts-or-sql "sql") @@ -94,7 +94,7 @@ (gobj/set out normalized value))) out) bind) - stmt (.prepare db sql)] + ^js stmt (.prepare db sql)] (if (= row-mode "array") (do (.raw stmt) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 3b4d89868d..360c0cad46 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -6,6 +6,7 @@ [cljs.reader :as reader] [clojure.string :as string] [logseq.cli.config :as cli-config] + [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [logseq.common.config :as common-config] [logseq.common.util :as common-util] @@ -17,20 +18,17 @@ :desc "Show help" :coerce :boolean} :config {:desc "Path to cli.edn"} - :base-url {:desc "Base URL for db-worker-node"} - :host {:desc "Host for db-worker-node"} - :port {:desc "Port for db-worker-node" - :coerce :long} :auth-token {:desc "Auth token for db-worker-node"} :repo {:desc "Graph name"} + :data-dir {:desc "Path to db-worker data dir"} :timeout-ms {:desc "Request timeout in ms" :coerce :long} :retries {:desc "Retry count for requests" :coerce :long} :output {:desc "Output format (human, json, edn)"}}) -(def ^:private graph-spec - {:graph {:desc "Graph name"}}) +(def ^:private server-spec + {:repo {:desc "Graph name"}}) (def ^:private content-add-spec {:content {:desc "Block content for add"} @@ -126,6 +124,13 @@ :message "graph name is required"} :summary summary}) +(defn- missing-repo-result + [summary] + {:ok? false + :error {:code :missing-repo + :message "repo is required"} + :summary summary}) + (defn- missing-content-result [summary] {:ok? false @@ -183,11 +188,16 @@ (def ^:private table [(command-entry ["graph" "list"] :graph-list "List graphs" {}) - (command-entry ["graph" "create"] :graph-create "Create graph" graph-spec) - (command-entry ["graph" "switch"] :graph-switch "Switch current graph" graph-spec) - (command-entry ["graph" "remove"] :graph-remove "Remove graph" graph-spec) - (command-entry ["graph" "validate"] :graph-validate "Validate graph" graph-spec) - (command-entry ["graph" "info"] :graph-info "Graph metadata" graph-spec) + (command-entry ["graph" "create"] :graph-create "Create graph" {}) + (command-entry ["graph" "switch"] :graph-switch "Switch current graph" {}) + (command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) + (command-entry ["graph" "validate"] :graph-validate "Validate graph" {}) + (command-entry ["graph" "info"] :graph-info "Graph metadata" {}) + (command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) + (command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec) + (command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) + (command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec) + (command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec) (command-entry ["block" "add"] :add "Add blocks" content-add-spec) (command-entry ["block" "remove"] :remove "Remove block or page" content-remove-spec) (command-entry ["block" "search"] :search "Search blocks" content-search-spec) @@ -243,7 +253,7 @@ (let [opts (normalize-opts opts) args (vec args) cmd-summary (command-summary {:cmds cmds :spec spec}) - graph (or (:graph opts) (:repo opts)) + graph (:repo opts) has-args? (seq args) has-content? (or (seq (:content opts)) (seq (:blocks opts)) @@ -269,6 +279,10 @@ (and (= command :search) (not (or (seq (:text opts)) has-args?))) (missing-search-result summary) + (and (#{:server-status :server-start :server-stop :server-restart} command) + (not (seq (:repo opts)))) + (missing-repo-result summary) + :else (ok-result command opts args summary)))) @@ -287,7 +301,7 @@ :error {:code :missing-command :message "missing command"} :summary summary}) - (if (and (= 1 (count args)) (#{"graph" "block"} (first args))) + (if (and (= 1 (count args)) (#{"graph" "block" "server"} (first args))) (help-result (group-summary (first args) table)) (try (let [result (cli/dispatch table args {:spec global-spec})] @@ -326,10 +340,21 @@ (when (seq repo) (string/replace-first repo common-config/db-version-prefix ""))) +(defn- ensure-existing-graph + [action config] + (if (and (:repo action) (not (:allow-missing-graph action))) + (p/let [graphs (cli-server/list-graphs config) + graph (repo->graph (:repo action))] + (if (some #(= graph %) graphs) + {:ok? true} + {:ok? false + :error {:code :graph-not-exists + :message "graph not exists"}})) + (p/resolved {:ok? true}))) + (defn- pick-graph [options command-args config] - (or (:graph options) - (:repo options) + (or (:repo options) (first command-args) (:repo config))) @@ -501,10 +526,7 @@ (case command :graph-list {:ok? true - :action {:type :invoke - :method "thread-api/list-db" - :direct-pass? false - :args []}} + :action {:type :graph-list}} :graph-create (if-not (seq graph) @@ -514,6 +536,8 @@ :method "thread-api/create-or-open-db" :direct-pass? false :args [repo {}] + :repo repo + :allow-missing-graph true :persist-repo (repo->graph repo)}}) :graph-switch @@ -531,7 +555,8 @@ :action {:type :invoke :method "thread-api/unsafe-unlink-db" :direct-pass? false - :args [repo]}}) + :args [repo] + :repo repo}}) :graph-validate (if-not (seq repo) @@ -540,7 +565,8 @@ :action {:type :invoke :method "thread-api/validate-db" :direct-pass? false - :args [repo]}}) + :args [repo] + :repo repo}}) :graph-info (if-not (seq repo) @@ -550,6 +576,45 @@ :repo repo :graph (repo->graph repo)}}))) +(defn- build-server-action + [command repo] + (case command + :server-list + {:ok? true + :action {:type :server-list}} + + :server-status + (if-not (seq repo) + (missing-repo-error "repo is required for server status") + {:ok? true + :action {:type :server-status + :repo repo}}) + + :server-start + (if-not (seq repo) + (missing-repo-error "repo is required for server start") + {:ok? true + :action {:type :server-start + :repo repo}}) + + :server-stop + (if-not (seq repo) + (missing-repo-error "repo is required for server stop") + {:ok? true + :action {:type :server-stop + :repo repo}}) + + :server-restart + (if-not (seq repo) + (missing-repo-error "repo is required for server restart") + {:ok? true + :action {:type :server-restart + :repo repo}}) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown server command: " command)}})) + (defn- build-add-action [options args repo] (if-not (seq repo) @@ -623,11 +688,15 @@ parsed (let [{:keys [command options args]} parsed graph (pick-graph options args config) - repo (resolve-repo graph)] + repo (resolve-repo graph) + server-repo (resolve-repo (:repo options))] (case command (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) (build-graph-action command graph repo) + (:server-list :server-status :server-start :server-stop :server-restart) + (build-server-action command server-repo) + :add (build-add-action options args repo) @@ -644,87 +713,160 @@ :error {:code :unknown-command :message (str "unknown command: " command)}})))) +(defn- execute-graph-list + [_action config] + (let [graphs (cli-server/list-graphs config)] + {:status :ok + :data {:graphs graphs}})) + +(defn- execute-invoke + [action config] + (-> (p/let [cfg (if-let [repo (:repo action)] + (cli-server/ensure-server! config repo) + (p/resolved config)) + result (transport/invoke cfg + (:method action) + (:direct-pass? action) + (:args action))] + (when-let [repo (:persist-repo action)] + (cli-config/update-config! config {:repo repo})) + (if-let [write (:write action)] + (let [{:keys [format path]} write] + (transport/write-output {:format format :path path :data result}) + {:status :ok + :data {:message (str "wrote " path)}}) + {:status :ok :data {:result result}})))) + +(defn- execute-graph-switch + [action config] + (-> (p/let [graphs (cli-server/list-graphs config) + graph (:graph action)] + (if-not (some #(= graph %) graphs) + {:status :error + :error {:code :graph-not-found + :message (str "graph not found: " graph)}} + (p/let [_ (cli-server/ensure-server! config (:repo action))] + (cli-config/update-config! config {:repo graph}) + {:status :ok + :data {:message (str "switched to " graph)}}))))) + +(defn- execute-graph-info + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + created (transport/invoke cfg "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) + schema (transport/invoke cfg "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/schema-version])] + {:status :ok + :data {:graph (:graph action) + :logseq.kv/graph-created-at (:kv/value created) + :logseq.kv/schema-version (:kv/value schema)}}))) + +(defn- execute-add + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + target-id (resolve-add-target cfg action) + ops [[:insert-blocks [(:blocks action) + target-id + {:sibling? false + :bottom? true + :outliner-op :insert-blocks}]]] + result (transport/invoke cfg "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] + {:status :ok + :data {:result result}}))) + +(defn- execute-remove + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + result (perform-remove cfg action)] + {:status :ok + :data {:result result}}))) + +(defn- execute-search + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + query '[:find ?e ?title + :in $ ?q + :where + [?e :block/title ?title] + [(clojure.string/includes? ?title ?q)]] + results (transport/invoke cfg "thread-api/q" false [(:repo action) [query (:text action)]]) + mapped (mapv (fn [[id title]] {:db/id id :block/title title}) results) + limited (if (some? (:limit action)) (vec (take (:limit action) mapped)) mapped)] + {:status :ok + :data {:results limited}}))) + +(defn- execute-tree + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + tree-data (fetch-tree cfg action) + format (:format action)] + (case format + "edn" + {:status :ok + :data tree-data + :output-format :edn} + + "json" + {:status :ok + :data tree-data + :output-format :json} + + {:status :ok + :data {:message (tree->text tree-data)}})))) + +(defn- server-result->response + [result] + (if (:ok? result) + {:status :ok + :data (:data result)} + {:status :error + :error (:error result)})) + +(defn- execute-server-list + [_action config] + (-> (p/let [servers (cli-server/list-servers config)] + {:status :ok + :data {:servers servers}}))) + +(defn- execute-server-status + [action config] + (-> (p/let [result (cli-server/server-status config (:repo action))] + (server-result->response result)))) + +(defn- execute-server-start + [action config] + (-> (p/let [result (cli-server/start-server! config (:repo action))] + (server-result->response result)))) + +(defn- execute-server-stop + [action config] + (-> (p/let [result (cli-server/stop-server! config (:repo action))] + (server-result->response result)))) + +(defn- execute-server-restart + [action config] + (-> (p/let [result (cli-server/restart-server! config (:repo action))] + (server-result->response result)))) + (defn execute [action config] - (case (:type action) - :invoke - (-> (p/let [result (transport/invoke config - (:method action) - (:direct-pass? action) - (:args action))] - (when-let [repo (:persist-repo action)] - (cli-config/update-config! config {:repo repo})) - (if-let [write (:write action)] - (let [{:keys [format path]} write] - (transport/write-output {:format format :path path :data result}) - {:status :ok - :data {:message (str "wrote " path)}}) - {:status :ok :data {:result result}}))) - - :graph-switch - (-> (p/let [exists? (transport/invoke config "thread-api/db-exists" false [(:repo action)])] - (if-not exists? + (-> (p/let [check (ensure-existing-graph action config)] + (if-not (:ok? check) + {:status :error + :error (:error check)} + (case (:type action) + :graph-list (execute-graph-list action config) + :invoke (execute-invoke action config) + :graph-switch (execute-graph-switch action config) + :graph-info (execute-graph-info action config) + :add (execute-add action config) + :remove (execute-remove action config) + :search (execute-search action config) + :tree (execute-tree action config) + :server-list (execute-server-list action config) + :server-status (execute-server-status action config) + :server-start (execute-server-start action config) + :server-stop (execute-server-stop action config) + :server-restart (execute-server-restart action config) {:status :error - :error {:code :graph-not-found - :message (str "graph not found: " (:graph action))}} - (p/let [_ (transport/invoke config "thread-api/create-or-open-db" false [(:repo action) {}])] - (cli-config/update-config! config {:repo (:graph action)}) - {:status :ok - :data {:message (str "switched to " (:graph action))}})))) - - :graph-info - (-> (p/let [created (transport/invoke config "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) - schema (transport/invoke config "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/schema-version])] - {:status :ok - :data {:graph (:graph action) - :logseq.kv/graph-created-at (:kv/value created) - :logseq.kv/schema-version (:kv/value schema)}})) - - :add - (-> (p/let [target-id (resolve-add-target config action) - ops [[:insert-blocks [(:blocks action) - target-id - {:sibling? false - :bottom? true - :outliner-op :insert-blocks}]]] - result (transport/invoke config "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] - {:status :ok - :data {:result result}})) - - :remove - (-> (p/let [result (perform-remove config action)] - {:status :ok - :data {:result result}})) - - :search - (-> (p/let [query '[:find ?e ?title - :in $ ?q - :where - [?e :block/title ?title] - [(clojure.string/includes? ?title ?q)]] - results (transport/invoke config "thread-api/q" false [(:repo action) [query (:text action)]]) - mapped (mapv (fn [[id title]] {:db/id id :block/title title}) results) - limited (if (some? (:limit action)) (vec (take (:limit action) mapped)) mapped)] - {:status :ok - :data {:results limited}})) - - :tree - (-> (p/let [tree-data (fetch-tree config action) - format (:format action)] - (case format - "edn" - {:status :ok - :data tree-data - :output-format :edn} - - "json" - {:status :ok - :data tree-data - :output-format :json} - - {:status :ok - :data {:message (tree->text tree-data)}}))) - - {:status :error - :error {:code :unknown-action - :message "unknown action"}})) + :error {:code :unknown-action + :message "unknown action"}}))))) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 57cdc7eef7..ae03dfc039 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -54,15 +54,15 @@ [] (let [env (.-env js/process)] (cond-> {} - (seq (gobj/get env "LOGSEQ_DB_WORKER_URL")) - (assoc :base-url (gobj/get env "LOGSEQ_DB_WORKER_URL")) - (seq (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) (assoc :auth-token (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) (seq (gobj/get env "LOGSEQ_CLI_REPO")) (assoc :repo (gobj/get env "LOGSEQ_CLI_REPO")) + (seq (gobj/get env "LOGSEQ_CLI_DATA_DIR")) + (assoc :data-dir (gobj/get env "LOGSEQ_CLI_DATA_DIR")) + (seq (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS")) (assoc :timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS"))) @@ -75,17 +75,12 @@ (seq (gobj/get env "LOGSEQ_CLI_CONFIG")) (assoc :config-path (gobj/get env "LOGSEQ_CLI_CONFIG"))))) -(defn- build-base-url - [{:keys [host port]}] - (when (or (seq host) (some? port)) - (str "http://" (or host "127.0.0.1") ":" (or port 9101)))) - (defn resolve-config [opts] - (let [defaults {:base-url "http://127.0.0.1:9101" - :timeout-ms 10000 + (let [defaults {:timeout-ms 10000 :retries 0 :output-format nil + :data-dir "~/.logseq/db-worker" :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) @@ -98,8 +93,6 @@ (parse-output-format (:output env)) (parse-output-format (:output-format file-config)) (parse-output-format (:output file-config))) - merged (merge defaults file-config env opts {:config-path config-path}) - derived (build-base-url merged)] + merged (merge defaults file-config env opts {:config-path config-path})] (cond-> merged - output-format (assoc :output-format output-format) - (seq derived) (assoc :base-url derived)))) + output-format (assoc :output-format output-format)))) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 187acaca4c..bd1659dc98 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq-cli [options]" "" - "Commands: graph list, graph create, graph switch, graph remove, graph validate, graph info, block add, block remove, block search, block tree" + "Commands: graph list, graph create, graph switch, graph remove, graph validate, graph info, server list, server status, server start, server stop, server restart, block add, block remove, block search, block tree" "" "Options:" summary])) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs new file mode 100644 index 0000000000..9063677371 --- /dev/null +++ b/src/main/logseq/cli/server.cljs @@ -0,0 +1,321 @@ +(ns logseq.cli.server + "db-worker-node lifecycle orchestration for logseq-cli." + (:require ["child_process" :as child-process] + ["fs" :as fs] + ["http" :as http] + ["os" :as os] + ["path" :as node-path] + [clojure.string :as string] + [frontend.worker.db-worker-node :as db-worker-node] + [frontend.worker-common.util :as worker-util] + [lambdaisland.glogi :as log] + [promesa.core :as p])) + +(defonce ^:private *inproc-servers (atom {})) + +(defn- inproc-enabled? + [] + (boolean (.-DEBUG js/goog))) + +(defn- expand-home + [path] + (if (string/starts-with? path "~") + (node-path/join (.homedir os) (subs path 1)) + path)) + +(defn resolve-data-dir + [config] + (expand-home (or (:data-dir config) "~/.logseq/db-worker"))) + +(defn- repo-dir + [data-dir repo] + (let [pool-name (worker-util/get-pool-name repo)] + (node-path/join data-dir (str "." pool-name)))) + +(defn lock-path + [data-dir repo] + (node-path/join (repo-dir data-dir repo) "db-worker.lock")) + +(defn- pid-alive? + [pid] + (when (number? pid) + (try + (.kill js/process pid 0) + true + (catch :default _ false)))) + +(defn- read-lock + [path] + (when (and (seq path) (fs/existsSync path)) + (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true))) + +(defn- remove-lock! + [path] + (when (and (seq path) (fs/existsSync path)) + (fs/unlinkSync path))) + +(defn- base-url + [{:keys [host port]}] + (str "http://" host ":" port)) + +(defn- http-request + [{:keys [method host port path headers body timeout-ms]}] + (p/create + (fn [resolve reject] + (let [timeout-ms (or timeout-ms 5000) + req (.request + http + #js {:method method + :hostname host + :port port + :path path + :headers (clj->js (or headers {}))} + (fn [^js res] + (let [chunks (array)] + (.on res "data" (fn [chunk] (.push chunks chunk))) + (.on res "end" (fn [] + (let [buf (js/Buffer.concat chunks)] + (resolve {:status (.-statusCode res) + :body (.toString buf "utf8")})))) + (.on res "error" reject)))) + timeout-id (js/setTimeout + (fn [] + (.destroy req) + (reject (ex-info "request timeout" {:code :timeout}))) + timeout-ms)] + (.on req "error" (fn [err] + (js/clearTimeout timeout-id) + (reject err))) + (when body + (.write req body)) + (.end req) + (.on req "response" (fn [_] + (js/clearTimeout timeout-id))))))) + +(defn- ready? + [{:keys [host port]}] + (-> (p/let [{:keys [status]} (http-request {:method "GET" + :host host + :port port + :path "/readyz" + :timeout-ms 1000})] + (= 200 status)) + (p/catch (fn [_] false)))) + +(defn- healthy? + [{:keys [host port]}] + (-> (p/let [{:keys [status]} (http-request {:method "GET" + :host host + :port port + :path "/healthz" + :timeout-ms 1000})] + (= 200 status)) + (p/catch (fn [_] false)))) + +(defn- valid-lock? + [lock] + (and (seq (:host lock)) + (pos-int? (:port lock)))) + +(defn- cleanup-stale-lock! + [path lock] + (cond + (nil? lock) + (p/resolved nil) + + (not (pid-alive? (:pid lock))) + (do + (remove-lock! path) + (p/resolved nil)) + + (not (valid-lock? lock)) + (do + (remove-lock! path) + (p/resolved nil)) + + :else + (p/let [healthy (healthy? lock)] + (when-not healthy + (remove-lock! path))))) + +(defn- wait-for + [pred-fn {:keys [timeout-ms interval-ms] + :or {timeout-ms 8000 + interval-ms 200}}] + (p/create + (fn [resolve reject] + (let [start (js/Date.now) + tick (fn tick [] + (p/let [ok? (pred-fn)] + (if ok? + (resolve true) + (if (> (- (js/Date.now) start) timeout-ms) + (reject (ex-info "timeout" {:code :timeout})) + (js/setTimeout tick interval-ms)))))] + (tick))))) + +(defn- wait-for-lock + [path] + (wait-for (fn [] + (p/resolved (and (fs/existsSync path) + (let [lock (read-lock path)] + (pos-int? (:port lock)))))) + {:timeout-ms 8000 + :interval-ms 200})) + +(defn- wait-for-ready + [lock] + (wait-for (fn [] (ready? lock)) + {:timeout-ms 8000 + :interval-ms 250})) + +(defn- spawn-server! + [{:keys [repo data-dir]}] + (let [script (node-path/join (js/process.cwd) "static" "db-worker-node.js") + args #js [script "--repo" repo "--data-dir" data-dir] + child (.spawn child-process "node" args #js {:detached true + :stdio "ignore"})] + (.unref child) + child)) + +(defn- start-inproc-server! + [{:keys [repo data-dir]}] + (p/let [daemon (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo})] + (swap! *inproc-servers assoc repo daemon) + daemon)) + +(defn- ensure-server-started! + [config repo] + (let [data-dir (resolve-data-dir config) + path (lock-path data-dir repo)] + (p/let [existing (read-lock path) + _ (cleanup-stale-lock! path existing) + _ (when (not (fs/existsSync path)) + (if (inproc-enabled?) + (start-inproc-server! {:repo repo :data-dir data-dir}) + (spawn-server! {:repo repo :data-dir data-dir})) + (wait-for-lock path)) + lock (read-lock path)] + (when-not lock + (throw (ex-info "db-worker-node failed to start" {:code :server-start-failed}))) + (p/let [_ (wait-for-ready lock)] + lock)))) + +(defn ensure-server! + [config repo] + (p/let [lock (ensure-server-started! config repo)] + (assoc config :base-url (base-url lock)))) + +(defn- shutdown! + [{:keys [host port]}] + (p/let [{:keys [status]} (http-request {:method "POST" + :host host + :port port + :path "/v1/shutdown" + :headers {"Content-Type" "application/json"} + :timeout-ms 1000})] + (= 200 status))) + +(defn stop-server! + [config repo] + (let [data-dir (resolve-data-dir config) + path (lock-path data-dir repo) + lock (read-lock path)] + (if-not lock + (p/resolved {:ok? false + :error {:code :server-not-found + :message "server is not running"}}) + (-> (p/let [_ (shutdown! lock)] + (wait-for (fn [] + (p/resolved (not (fs/existsSync path)))) + {:timeout-ms 5000 + :interval-ms 200}) + (swap! *inproc-servers dissoc repo) + {:ok? true + :data {:repo repo}}) + (p/catch (fn [_] + (when (and (pid-alive? (:pid lock)) + (not= (:pid lock) (.-pid js/process))) + (try + (.kill js/process (:pid lock) "SIGTERM") + (catch :default e + (log/warn :cli-server-stop-sigterm-failed e)))) + (when-not (pid-alive? (:pid lock)) + (remove-lock! path)) + (if (fs/existsSync path) + {:ok? false + :error {:code :server-stop-timeout + :message "timed out stopping server"}} + (do + (swap! *inproc-servers dissoc repo) + {:ok? true + :data {:repo repo}})))))))) + +(defn start-server! + [config repo] + (p/let [_ (ensure-server-started! config repo)] + {:ok? true + :data {:repo repo}})) + +(defn restart-server! + [config repo] + (-> (p/let [_ (stop-server! config repo)] + (start-server! config repo)) + (p/catch (fn [_] + (start-server! config repo))))) + +(defn server-status + [config repo] + (let [data-dir (resolve-data-dir config) + path (lock-path data-dir repo) + lock (read-lock path)] + (if-not lock + (p/resolved {:ok? true + :data {:repo repo + :status :stopped}}) + (p/let [ready (ready? lock)] + {:ok? true + :data {:repo repo + :status (if ready :ready :starting) + :host (:host lock) + :port (:port lock) + :pid (:pid lock) + :started-at (:startedAt lock)}})))) + +(defn list-servers + [config] + (let [data-dir (resolve-data-dir config) + entries (when (fs/existsSync data-dir) + (fs/readdirSync data-dir #js {:withFileTypes true}))] + (p/all + (for [^js entry entries + :when (.isDirectory entry) + :let [name (.-name entry) + lock (read-lock (node-path/join data-dir name "db-worker.lock"))] + :when lock] + (p/let [ready (ready? lock)] + {:repo (:repo lock) + :host (:host lock) + :port (:port lock) + :pid (:pid lock) + :status (if ready :ready :starting)}))))) + +(defn list-graphs + [config] + (let [data-dir (resolve-data-dir config) + db-dir-prefix ".logseq-pool-" + entries (when (fs/existsSync data-dir) + (fs/readdirSync data-dir #js {:withFileTypes true}))] + (->> entries + (filter #(.isDirectory ^js %)) + (map (fn [^js dirent] + (.-name dirent))) + (filter #(string/starts-with? % db-dir-prefix)) + (map (fn [dir-name] + (-> dir-name + (string/replace-first db-dir-prefix "") + (string/replace "+3A+" ":") + (string/replace "++" "/")))) + (vec)))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index e71cdda65b..0edbbbd77f 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -3,10 +3,14 @@ [cljs.test :refer [async deftest is]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] + [frontend.worker-common.util :as worker-util] [frontend.worker.db-worker-node :as db-worker-node] + [goog.object :as gobj] [logseq.db :as ldb] [logseq.db.sqlite.util :as sqlite-util] - [promesa.core :as p])) + [promesa.core :as p] + ["fs" :as fs] + ["path" :as node-path])) (defn- http-request [opts body] @@ -56,6 +60,37 @@ (is (:ok parsed)) (ldb/read-transit-str (:resultTransit parsed))))) +(defn- invoke-raw + [host port method args] + (let [payload (js/JSON.stringify + (clj->js {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)}))] + (http-request {:hostname host + :port port + :path "/v1/invoke" + :method "POST" + :headers {"Content-Type" "application/json"}} + payload))) + +(defn- lock-path + [data-dir repo] + (let [pool-name (worker-util/get-pool-name repo) + repo-dir (node-path/join data-dir (str "." pool-name))] + (node-path/join repo-dir "db-worker.lock"))) + +(deftest db-worker-node-parse-args-ignores-host-and-port + (let [parse-args #'db-worker-node/parse-args + result (parse-args #js ["node" "db-worker-node.js" + "--host" "0.0.0.0" + "--port" "1234" + "--repo" "logseq_db_parse_args" + "--data-dir" "/tmp/db-worker"])] + (is (nil? (:host result))) + (is (nil? (:port result))) + (is (= "logseq_db_parse_args" (:repo result))) + (is (= "/tmp/db-worker" (:data-dir result))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) @@ -66,9 +101,8 @@ block-uuid (random-uuid)] (-> (p/let [{:keys [host port stop!]} (db-worker-node/start-daemon! - {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) + {:data-dir data-dir + :repo repo}) health (http-get host port "/healthz") ready (http-get host port "/readyz") _ (do @@ -88,6 +122,11 @@ (subs repo (count prefix)) repo)] (is (some #(= expected-name (:name %)) dbs)))) + lock-file (lock-path data-dir repo) + _ (is (fs/existsSync lock-file)) + lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + _ (is (= repo (gobj/get lock-contents "repo"))) + _ (is (= host (gobj/get lock-contents "host"))) _ (invoke host port "thread-api/transact" [repo [{:block/uuid page-uuid @@ -119,5 +158,51 @@ (p/finally (fn [] (if-let [stop! (:stop! @daemon)] (-> (stop!) - (p/finally (fn [] (done)))) + (p/finally (fn [] + (is (not (fs/existsSync (lock-path data-dir repo)))) + (done)))) + (done)))))))) + +(deftest db-worker-node-repo-mismatch-test + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-repo-mismatch") + repo (str "logseq_db_mismatch_" (subs (str (random-uuid)) 0 8)) + other-repo (str repo "_other")] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + {:keys [status body]} (invoke-raw host port "thread-api/create-or-open-db" [other-repo {}]) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (is (= 409 status)) + (is (= false (:ok parsed))) + (is (= "repo-mismatch" (get-in parsed [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-lock-prevents-multiple-daemons + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-lock") + repo (str "logseq_db_lock_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!})] + (-> (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + (p/then (fn [_] + (is false "expected lock error"))) + (p/catch (fn [e] + (is (= :repo-locked (-> (ex-data e) :code))))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index c8276a8171..1ff60c10cd 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1,7 +1,9 @@ (ns logseq.cli.commands-test - (:require [clojure.string :as string] - [cljs.test :refer [deftest is testing]] - [logseq.cli.commands :as commands])) + (:require [cljs.test :refer [async deftest is testing]] + [clojure.string :as string] + [logseq.cli.commands :as commands] + [logseq.cli.server :as cli-server] + [promesa.core :as p])) (deftest test-help-output (testing "top-level help lists subcommand groups" @@ -9,7 +11,8 @@ summary (:summary result)] (is (true? (:help? result))) (is (string/includes? summary "graph")) - (is (string/includes? summary "block"))))) + (is (string/includes? summary "block")) + (is (string/includes? summary "server"))))) (deftest test-parse-args (testing "graph group shows subcommands" @@ -26,6 +29,13 @@ (is (string/includes? summary "block add")) (is (string/includes? summary "block search")))) + (testing "server group shows subcommands" + (let [result (commands/parse-args ["server"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "server list")) + (is (string/includes? summary "server start")))) + (testing "graph group aligns subcommand columns" (let [result (commands/parse-args ["graph"]) summary (:summary result) @@ -56,6 +66,21 @@ (is (seq subcommand-lines)) (is (apply = desc-starts)))) + (testing "server group aligns subcommand columns" + (let [result (commands/parse-args ["server"]) + summary (:summary result) + subcommand-lines (let [lines (string/split-lines summary) + start (inc (.indexOf lines "Subcommands:"))] + (->> lines + (drop start) + (take-while (complement string/blank?)))) + desc-starts (->> subcommand-lines + (keep (fn [line] + (when-let [[_ desc] (re-matches #"^\s+.*?\s{2,}(.+)$" line)] + (.indexOf line desc)))))] + (is (seq subcommand-lines)) + (is (apply = desc-starts)))) + (testing "rejects legacy commands" (doseq [command ["graph-list" "graph-create" "graph-switch" "graph-remove" "graph-validate" "graph-info" "add" "remove" "search" "tree" @@ -95,72 +120,94 @@ (is (true? (:ok? result))) (is (= :graph-list (:command result))))) - (testing "graph create requires graph option" + (testing "graph create requires repo option" (let [result (commands/parse-args ["graph" "create"])] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) - (testing "graph create parses with graph option" - (let [result (commands/parse-args ["graph" "create" "--graph" "demo"])] + (testing "graph create parses with repo option" + (let [result (commands/parse-args ["graph" "create" "--repo" "demo"])] (is (true? (:ok? result))) (is (= :graph-create (:command result))) - (is (= "demo" (get-in result [:options :graph]))))) + (is (= "demo" (get-in result [:options :repo]))))) - (testing "graph switch requires graph option" + (testing "graph switch requires repo option" (let [result (commands/parse-args ["graph" "switch"])] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) - (testing "graph switch parses with graph option" - (let [result (commands/parse-args ["graph" "switch" "--graph" "demo"])] + (testing "graph switch parses with repo option" + (let [result (commands/parse-args ["graph" "switch" "--repo" "demo"])] (is (true? (:ok? result))) (is (= :graph-switch (:command result))) - (is (= "demo" (get-in result [:options :graph]))))) + (is (= "demo" (get-in result [:options :repo]))))) - (testing "graph remove requires graph option" + (testing "graph remove requires repo option" (let [result (commands/parse-args ["graph" "remove"])] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) - (testing "graph remove parses with graph option" - (let [result (commands/parse-args ["graph" "remove" "--graph" "demo"])] + (testing "graph remove parses with repo option" + (let [result (commands/parse-args ["graph" "remove" "--repo" "demo"])] (is (true? (:ok? result))) (is (= :graph-remove (:command result))) - (is (= "demo" (get-in result [:options :graph]))))) + (is (= "demo" (get-in result [:options :repo]))))) - (testing "graph validate requires graph option" + (testing "graph validate requires repo option" (let [result (commands/parse-args ["graph" "validate"])] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) - (testing "graph validate parses with graph option" - (let [result (commands/parse-args ["graph" "validate" "--graph" "demo"])] + (testing "graph validate parses with repo option" + (let [result (commands/parse-args ["graph" "validate" "--repo" "demo"])] (is (true? (:ok? result))) (is (= :graph-validate (:command result))) - (is (= "demo" (get-in result [:options :graph]))))) + (is (= "demo" (get-in result [:options :repo]))))) - (testing "graph info parses without graph option" + (testing "graph info parses without repo option" (let [result (commands/parse-args ["graph" "info"])] (is (true? (:ok? result))) (is (= :graph-info (:command result))))) - (testing "graph info parses with graph option" - (let [result (commands/parse-args ["graph" "info" "--graph" "demo"])] + (testing "graph info parses with repo option" + (let [result (commands/parse-args ["graph" "info" "--repo" "demo"])] (is (true? (:ok? result))) (is (= :graph-info (:command result))) - (is (= "demo" (get-in result [:options :graph]))))) + (is (= "demo" (get-in result [:options :repo]))))) (testing "graph subcommands reject unknown flags" (doseq [subcommand ["list" "create" "switch" "remove" "validate" "info"]] (let [result (commands/parse-args ["graph" subcommand "--wat"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))))) + (is (= :invalid-options (get-in result [:error :code])))))) + (testing "graph subcommands accept output option" (let [result (commands/parse-args ["graph" "list" "--output" "edn"])] (is (true? (:ok? result))) (is (= "edn" (get-in result [:options :output]))))) + (testing "server list parses" + (let [result (commands/parse-args ["server" "list"])] + (is (true? (:ok? result))) + (is (= :server-list (:command result))))) + + (testing "server start requires repo" + (let [result (commands/parse-args ["server" "start"])] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + + (testing "server start parses with repo" + (let [result (commands/parse-args ["server" "start" "--repo" "demo"])] + (is (true? (:ok? result))) + (is (= :server-start (:command result))) + (is (= "demo" (get-in result [:options :repo]))))) + + (testing "server stop parses with repo" + (let [result (commands/parse-args ["server" "stop" "--repo" "demo"])] + (is (true? (:ok? result))) + (is (= :server-stop (:command result)))))) + (deftest test-block-subcommand-parse (testing "block add requires content source" (let [result (commands/parse-args ["block" "add"])] @@ -222,16 +269,34 @@ (let [parsed {:ok? true :command :graph-list :options {}} result (commands/build-action parsed {})] (is (true? (:ok? result))) - (is (= "thread-api/list-db" (get-in result [:action :method]))))) + (is (= :graph-list (get-in result [:action :type]))))) - (testing "graph-create requires graph name" + (testing "server list builds action" + (let [parsed {:ok? true :command :server-list :options {}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :server-list (get-in result [:action :type]))))) + + (testing "server start requires repo" + (let [parsed {:ok? true :command :server-start :options {}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + + (testing "server stop builds action" + (let [parsed {:ok? true :command :server-stop :options {:repo "demo"}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :server-stop (get-in result [:action :type]))))) + + (testing "graph-create requires repo name" (let [parsed {:ok? true :command :graph-create :options {}} result (commands/build-action parsed {})] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) (testing "graph-switch uses graph name" - (let [parsed {:ok? true :command :graph-switch :options {:graph "demo"}} + (let [parsed {:ok? true :command :graph-switch :options {:repo "demo"}} result (commands/build-action parsed {})] (is (true? (:ok? result))) (is (= :graph-switch (get-in result [:action :type]))))) @@ -271,3 +336,20 @@ result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code])))))) + +(deftest test-execute-requires-existing-graph + (async done + (with-redefs [cli-server/list-graphs (fn [_] []) + cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))] + (-> (p/let [result (commands/execute {:type :search + :repo "logseq_db_missing" + :text "hello"} + {})] + (is (= :error (:status result))) + (is (= :graph-not-exists (get-in result [:error :code]))) + (is (= "graph not exists" (get-in result [:error :message]))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index c37e015683..c408e4bd8f 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -23,47 +23,43 @@ (let [dir (node-helper/create-tmp-dir) cfg-path (node-path/join dir "cli.edn") _ (fs/writeFileSync cfg-path - (str "{:base-url \"http://file:7777\" " - ":auth-token \"file-token\" " + (str "{:auth-token \"file-token\" " ":repo \"file-repo\" " + ":data-dir \"file-data\" " ":timeout-ms 111 " ":retries 1 " ":output-format :edn}")) - env {"LOGSEQ_DB_WORKER_URL" "http://env:9999" - "LOGSEQ_DB_WORKER_AUTH_TOKEN" "env-token" + env {"LOGSEQ_DB_WORKER_AUTH_TOKEN" "env-token" "LOGSEQ_CLI_REPO" "env-repo" + "LOGSEQ_CLI_DATA_DIR" "env-data" "LOGSEQ_CLI_TIMEOUT_MS" "222" "LOGSEQ_CLI_RETRIES" "2" "LOGSEQ_CLI_OUTPUT" "json"} opts {:config-path cfg-path - :base-url "http://cli:1234" :auth-token "cli-token" :repo "cli-repo" + :data-dir "cli-data" :timeout-ms 333 :retries 3 :output-format :human} result (with-env env #(config/resolve-config opts))] (is (= cfg-path (:config-path result))) - (is (= "http://cli:1234" (:base-url result))) (is (= "cli-token" (:auth-token result))) (is (= "cli-repo" (:repo result))) + (is (= "cli-data" (:data-dir result))) (is (= 333 (:timeout-ms result))) (is (= 3 (:retries result))) (is (= :human (:output-format result))))) -(deftest test-host-port-derived-base-url - (let [result (config/resolve-config {:host "127.0.0.2" :port 9200})] - (is (= "http://127.0.0.2:9200" (:base-url result))))) - (deftest test-env-overrides-file (let [dir (node-helper/create-tmp-dir) cfg-path (node-path/join dir "cli.edn") - _ (fs/writeFileSync cfg-path "{:base-url \"http://file:7777\" :repo \"file-repo\"}") - env {"LOGSEQ_DB_WORKER_URL" "http://env:9999" - "LOGSEQ_CLI_REPO" "env-repo"} + _ (fs/writeFileSync cfg-path "{:repo \"file-repo\" :data-dir \"file-data\"}") + env {"LOGSEQ_CLI_REPO" "env-repo" + "LOGSEQ_CLI_DATA_DIR" "env-data"} result (with-env env #(config/resolve-config {:config-path cfg-path}))] - (is (= "http://env:9999" (:base-url result))) - (is (= "env-repo" (:repo result))))) + (is (= "env-repo" (:repo result))) + (is (= "env-data" (:data-dir result))))) (deftest test-output-format-env-overrides-file (let [dir (node-helper/create-tmp-dir) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 32f5edae7d..0ef7058f6a 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -3,19 +3,24 @@ [cljs.test :refer [deftest is async]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] - [frontend.worker.db-worker-node :as db-worker-node] [logseq.cli.main :as cli-main] [promesa.core :as p] ["fs" :as fs] ["path" :as node-path])) (defn- run-cli - [args url cfg-path] + [args data-dir cfg-path] (let [args-with-output (if (some #{"--output"} args) args - (concat args ["--output" "json"]))] - (cli-main/run! (vec (concat args-with-output ["--base-url" url "--config" cfg-path])) - {:exit? false}))) + (concat args ["--output" "json"])) + global-opts ["--data-dir" data-dir "--config" cfg-path] + final-args (vec (concat global-opts args-with-output))] + (-> (cli-main/run! final-args {:exit? false}) + (p/then (fn [result] + (let [res (if (map? result) + result + (js->clj result :keywordize-keys true))] + res)))))) (defn- parse-json-output [result] @@ -28,18 +33,13 @@ (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) - url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - result (run-cli ["graph" "list"] url cfg-path) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + result (run-cli ["graph" "list"] data-dir cfg-path) payload (parse-json-output result)] (is (= 0 (:exit-code result))) (is (= "ok" (:status payload))) (is (contains? payload :data)) - (p/let [_ ((:stop! daemon))] - (done))) + (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -47,23 +47,22 @@ (deftest test-cli-graph-create-and-info (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) - url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{}") - create-result (run-cli ["graph" "create" "--graph" "demo-graph"] url cfg-path) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--repo" "demo-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) - info-result (run-cli ["graph" "info"] url cfg-path) - info-payload (parse-json-output info-result)] + info-result (run-cli ["graph" "info"] data-dir cfg-path) + info-payload (parse-json-output info-result) + stop-result (run-cli ["server" "stop" "--repo" "demo-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-result))) (is (= "ok" (:status create-payload))) (is (= 0 (:exit-code info-result))) (is (= "ok" (:status info-payload))) (is (= "demo-graph" (get-in info-payload [:data :graph]))) - (p/let [_ ((:stop! daemon))] - (done))) + (is (= 0 (:exit-code stop-result))) + (is (= "ok" (:status stop-payload))) + (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -71,29 +70,27 @@ (deftest test-cli-add-search-tree-remove (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) - url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{}") - _ (run-cli ["graph" "create" "--graph" "content-graph"] url cfg-path) - add-result (run-cli ["block" "add" "--page" "TestPage" "--content" "hello world"] url cfg-path) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) + add-result (run-cli ["block" "add" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) _ (parse-json-output add-result) - search-result (run-cli ["block" "search" "--text" "hello world"] url cfg-path) + search-result (run-cli ["block" "search" "--text" "hello world"] data-dir cfg-path) search-payload (parse-json-output search-result) - tree-result (run-cli ["block" "tree" "--page" "TestPage" "--format" "json"] url cfg-path) + tree-result (run-cli ["block" "tree" "--page" "TestPage" "--format" "json"] data-dir cfg-path) tree-payload (parse-json-output tree-result) block-uuid (get-in tree-payload [:data :root :children 0 :uuid]) - remove-result (run-cli ["block" "remove" "--block" (str block-uuid)] url cfg-path) - remove-payload (parse-json-output remove-result)] + remove-result (run-cli ["block" "remove" "--block" (str block-uuid)] data-dir cfg-path) + remove-payload (parse-json-output remove-result) + stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-result))) (is (= "ok" (:status search-payload))) (is (seq (get-in search-payload [:data :results]))) (is (= "ok" (:status tree-payload))) (is (= "ok" (:status remove-payload))) - (p/let [_ ((:stop! daemon))] - (done))) + (is (= "ok" (:status stop-payload))) + (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -101,23 +98,18 @@ (deftest test-cli-output-formats-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [daemon (db-worker-node/start-daemon! {:host "127.0.0.1" - :port 0 - :data-dir data-dir}) - url (str "http://127.0.0.1:" (:port daemon)) - cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - json-result (run-cli ["graph" "list" "--output" "json"] url cfg-path) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + json-result (run-cli ["graph" "list" "--output" "json"] data-dir cfg-path) json-payload (parse-json-output json-result) - edn-result (run-cli ["graph" "list" "--output" "edn"] url cfg-path) + edn-result (run-cli ["graph" "list" "--output" "edn"] data-dir cfg-path) edn-payload (parse-edn-output edn-result) - human-result (run-cli ["graph" "list" "--output" "human"] url cfg-path)] + human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path)] (is (= 0 (:exit-code json-result))) (is (= "ok" (:status json-payload))) (is (= 0 (:exit-code edn-result))) (is (= :ok (:status edn-payload))) (is (not (string/starts-with? (:output human-result) "{:status"))) - (p/let [_ ((:stop! daemon))] - (done))) + (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs new file mode 100644 index 0000000000..2d91b2ae89 --- /dev/null +++ b/src/test/logseq/cli/server_test.cljs @@ -0,0 +1,53 @@ +(ns logseq.cli.server-test + (:require [cljs.test :refer [async deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.server :as cli-server] + [promesa.core :as p] + [clojure.string :as string] + ["fs" :as fs] + ["path" :as node-path] + ["child_process" :as child-process])) + +(deftest spawn-server-omits-host-and-port-flags + (let [spawn-server! #'cli-server/spawn-server! + captured (atom nil) + original-spawn (.-spawn child-process)] + (set! (.-spawn child-process) + (fn [cmd args opts] + (reset! captured {:cmd cmd + :args (vec (js->clj args)) + :opts (js->clj opts :keywordize-keys true)}) + (js-obj "unref" (fn [] nil)))) + (try + (spawn-server! {:repo "logseq_db_spawn_test" + :data-dir "/tmp/logseq-db-worker"}) + (is (= "node" (:cmd @captured))) + (is (some #{"--repo"} (:args @captured))) + (is (some #{"--data-dir"} (:args @captured))) + (is (not-any? #{"--host" "--port"} (:args @captured))) + (finally + (set! (.-spawn child-process) original-spawn))))) + +(deftest ensure-server-repairs-stale-lock + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server") + repo (str "logseq_db_stale_" (subs (str (random-uuid)) 0 8)) + path (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :startedAt (.toISOString (js/Date.))}] + (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) + (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) + (-> (p/let [cfg (cli-server/ensure-server! {:data-dir data-dir} repo) + _ (is (string/starts-with? (:base-url cfg) "http://127.0.0.1:")) + lock-data (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true) + _ (is (pos-int? (:port lock-data))) + stop-result (cli-server/stop-server! {:data-dir data-dir} repo)] + (is (:ok? stop-result)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index f429287c08..cc2207dda5 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -24,7 +24,7 @@ (async done (let [calls (atom 0)] (-> (p/let [{:keys [url stop!]} (start-server - (fn [_req res] + (fn [_req ^js res] (let [attempt (swap! calls inc)] (if (= attempt 1) (do From 66278e768782f1b61a2de3e8003e9181359d95d7 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 17 Jan 2026 23:25:11 +0800 Subject: [PATCH 014/375] fix: bb process/shell throws 'Operations not permitted(sysctl fail)' in agent sandbox --- scripts/src/logseq/tasks/common_errors.clj | 4 ++-- scripts/src/logseq/tasks/dev.clj | 17 +++++++++-------- scripts/src/logseq/tasks/dev/desktop.clj | 12 ++++++------ scripts/src/logseq/tasks/dev/lint.clj | 10 +++++----- scripts/src/logseq/tasks/dev/mobile.clj | 22 +++++++++++----------- scripts/src/logseq/tasks/lang.clj | 2 +- 6 files changed, 34 insertions(+), 33 deletions(-) diff --git a/scripts/src/logseq/tasks/common_errors.clj b/scripts/src/logseq/tasks/common_errors.clj index 862a78006c..c459d3fa2d 100644 --- a/scripts/src/logseq/tasks/common_errors.clj +++ b/scripts/src/logseq/tasks/common_errors.clj @@ -7,9 +7,9 @@ (defn check-common-errors [] (let [prompt (String. (fs/read-all-bytes "prompts/review.md")) - diff (:out (shell {:out :string} "git diff --no-prefix -U100 -- '*.cljs'"))] + diff (:out (shell {:out :string :shutdown nil} "git diff --no-prefix -U100 -- '*.cljs'"))] (when-not (string/blank? diff) (let [command (format "gh models run openai/gpt-5 \"%s\"" (str prompt (format "\n\n %s" diff)))] - (shell command))))) + (shell {:shutdown nil} command))))) diff --git a/scripts/src/logseq/tasks/dev.clj b/scripts/src/logseq/tasks/dev.clj index 331dae4f34..e151331afb 100644 --- a/scripts/src/logseq/tasks/dev.clj +++ b/scripts/src/logseq/tasks/dev.clj @@ -17,9 +17,9 @@ (defn test "Run tests. Pass args through to cmd 'yarn cljs:run-test'" [& args] - (shell "yarn cljs:test") + (shell {:shutdown nil} "yarn cljs:test") (let [args* (or (seq args) ["-e" "long" "-e" "fix-me"])] - (apply shell "yarn cljs:run-test" args*))) + (apply shell {:shutdown nil} "yarn cljs:run-test" args*))) (defn test-no-worker "Run tests without compiling worker namespaces. Pass args through to cmd 'yarn cljs:run-test-no-worker'" @@ -51,11 +51,12 @@ (let [config-edn ".clj-kondo/metosin/malli-types/config.edn" compile-cmd "clojure -M:cljs compile gen-malli-kondo-config"] (println compile-cmd) - (shell compile-cmd) + (shell {:shutdown nil} compile-cmd) (println "generate kondo config: " config-edn) (io/make-parents config-edn) (let [config (with-out-str - (pp/pprint (edn/read-string (:out (shell {:out :string} "node ./static/gen-malli-kondo-config.js")))))] + (pp/pprint (edn/read-string (:out (shell {:out :string :shutdown nil} + "node ./static/gen-malli-kondo-config.js")))))] (spit config-edn config)))) (defn diff-datoms @@ -82,19 +83,19 @@ (fs/glob "." "{src/main,deps/graph-parser/src}/**")))))] (do (println "Building publishing js asset...") - (shell "clojure -M:cljs release publishing db-worker inference-worker")) + (shell {:shutdown nil} "clojure -M:cljs release publishing db-worker inference-worker")) (println "Publishing js asset is up to date"))) (defn publishing-backend "Builds publishing backend and copies over supporting frontend assets" [& args] - (apply shell {:dir "deps/publishing" :extra-env {"ORIGINAL_PWD" (fs/cwd)}} + (apply shell {:dir "deps/publishing" :extra-env {"ORIGINAL_PWD" (fs/cwd)} :shutdown nil} "yarn -s nbb-logseq -cp src:../graph-parser/src script/publishing.cljs" (into ["static"] args))) (defn watch-publishing-frontend [& _args] - (shell "npx shadow-cljs watch publishing")) + (shell {:shutdown nil} "npx shadow-cljs watch publishing")) (defn watch-publishing-backend "Builds publishing backend once watch-publishing-frontend has built initial frontend" @@ -116,4 +117,4 @@ (doseq [file-graph file-graphs] (let [db-graph (fs/path parent-graph-dir (fs/file-name file-graph))] (println "Importing" (str db-graph) "...") - (apply shell "bb" "dev:import" file-graph db-graph (concat import-options ["--validate"])))))) + (apply shell {:shutdown nil} "bb" "dev:db-import" file-graph db-graph (concat import-options ["--validate"])))))) diff --git a/scripts/src/logseq/tasks/dev/desktop.clj b/scripts/src/logseq/tasks/dev/desktop.clj index 6ac70a12ca..2682d95793 100644 --- a/scripts/src/logseq/tasks/dev/desktop.clj +++ b/scripts/src/logseq/tasks/dev/desktop.clj @@ -7,15 +7,15 @@ (defn watch "Watches environment to reload cljs, css and other assets" [] - (shell "yarn electron-watch")) + (shell {:shutdown nil} "yarn electron-watch")) (defn open-dev-electron-app "Opens dev-electron-app when watch process has built main.js" [] (let [start-time (java.time.Instant/now)] (dotimes [_n 1000] - (if (and (fs/exists? "static/js/main.js") - (task-util/file-modified-later-than? "static/js/main.js" start-time)) - (shell "yarn dev-electron-app") - (println "Waiting for app to build...")) - (Thread/sleep 1000)))) + (if (and (fs/exists? "static/js/main.js") + (task-util/file-modified-later-than? "static/js/main.js" start-time)) + (shell {:shutdown nil} "yarn dev-electron-app") + (println "Waiting for app to build...")) + (Thread/sleep 1000)))) diff --git a/scripts/src/logseq/tasks/dev/lint.clj b/scripts/src/logseq/tasks/dev/lint.clj index 8dff77c741..645f065f9b 100644 --- a/scripts/src/logseq/tasks/dev/lint.clj +++ b/scripts/src/logseq/tasks/dev/lint.clj @@ -18,14 +18,14 @@ "bb lang:validate-translations" "bb lint:ns-docstrings"]] (println cmd) - (shell cmd))) + (shell {:shutdown nil} cmd))) (defn kondo-git-changes "Run clj-kondo across dirs and only for files that git diff detects as unstaged changes" [] (let [kondo-dirs ["src" "deps/common" "deps/db" "deps/graph-parser" "deps/outliner" "deps/publishing" "deps/publish" "deps/cli"] dir-regex (re-pattern (str "^(" (string/join "|" kondo-dirs) ")")) - dir-to-files (->> (shell {:out :string} "git diff --name-only") + dir-to-files (->> (shell {:out :string :shutdown nil} "git diff --name-only") :out string/split-lines (filter #(re-find #"\.(cljs|clj|cljc)$" %)) @@ -38,13 +38,13 @@ files (mapv #(string/replace-first % (str dir "/") "") files*) cmd (str "cd " dir " && clj-kondo --lint " (string/join " " files)) _ (println cmd) - res (apply shell {:dir dir :continue :true} "clj-kondo --lint" files)] + res (apply shell {:dir dir :continue :true :shutdown nil} "clj-kondo --lint" files)] (when (pos? (:exit res)) (System/exit (:exit res))))) (println "No clj* files have changed to lint.")))) (defn- validate-frontend-not-in-workers [] - (let [res (shell {:out :string} + (let [res (shell {:out :string :shutdown nil} "git grep -h" "\\[frontend.*:as" "src/main/frontend/worker" "src/main/frontend/worker_common" "src/main/frontend/inference_worker") req-lines (->> (:out res) @@ -60,7 +60,7 @@ (defn- validate-workers-not-in-frontend [] - (let [res (shell {:out :string :continue true} + (let [res (shell {:out :string :continue true :shutdown nil} "grep -r --exclude-dir=worker --exclude-dir=inference_worker" "\\[frontend.worker.*:" "src/main/frontend") ;; allow reset-file b/c it's only affects tests allowed-exceptions #{"src/main/frontend/handler/file_based/file.cljs: [frontend.worker.file.reset :as file-reset]"} diff --git a/scripts/src/logseq/tasks/dev/mobile.clj b/scripts/src/logseq/tasks/dev/mobile.clj index 315062dc9c..c900acebf8 100644 --- a/scripts/src/logseq/tasks/dev/mobile.clj +++ b/scripts/src/logseq/tasks/dev/mobile.clj @@ -12,7 +12,7 @@ (loop [n 1000] (if (and (fs/exists? "static/js/main.js") (task-util/file-modified-later-than? "static/js/main.js" start-time)) - (shell cmd) + (shell {:shutdown nil} cmd) (println "Waiting for app to build...")) (Thread/sleep 1000) (when-not (or (and (fs/exists? "ios/App/App/public/js/main.js") @@ -24,11 +24,11 @@ (defn- set-system-env "Updates capacitor.config.ts serve url with IP from ifconfig" [] - (let [ip (string/trim (:out (or (shell {:out :string :continue true} "ipconfig getifaddr en0") - (shell {:out :string} "ipconfig getifaddr en1")))) + (let [ip (string/trim (:out (or (shell {:out :string :continue true :shutdown nil} "ipconfig getifaddr en0") + (shell {:out :string :shutdown nil} "ipconfig getifaddr en1")))) logseq-app-server-url (format "%s://%s:%s" "http" ip "3001")] (println "Server URL:" logseq-app-server-url) - (shell "git checkout capacitor.config.ts") + (shell {:shutdown nil} "git checkout capacitor.config.ts") (let [new-body (-> (slurp "capacitor.config.ts") (string/replace "// , server:" " , server:") (string/replace "// url:" " url:") @@ -46,28 +46,28 @@ (doseq [cmd ["yarn clean" "yarn app-watch"]] (println cmd) - (shell cmd))) + (shell {:shutdown nil} cmd))) (defn npx-cap-run-ios "Copy assets files to iOS build directory, and run app in Xcode" [] (open-dev-app "npx cap sync ios") - (shell "npx cap open ios")) + (shell {:shutdown nil} "npx cap open ios")) (defn npx-cap-run-android "Copy assets files to Android build directory, and run app in Android Studio" [] (open-dev-app "npx cap sync android") - (shell "npx cap open android")) + (shell {:shutdown nil} "npx cap open android")) (defn run-ios-release "Build iOS app release" [] - (shell "git checkout capacitor.config.ts") - (shell "yarn run-ios-release")) + (shell {:shutdown nil} "git checkout capacitor.config.ts") + (shell {:shutdown nil} "yarn run-ios-release")) (defn run-android-release "Build Android app release" [] - (shell "git checkout capacitor.config.ts") - (shell "yarn run-android-release")) + (shell {:shutdown nil} "git checkout capacitor.config.ts") + (shell {:shutdown nil} "yarn run-android-release")) diff --git a/scripts/src/logseq/tasks/lang.clj b/scripts/src/logseq/tasks/lang.clj index c534875d94..39907563f9 100644 --- a/scripts/src/logseq/tasks/lang.clj +++ b/scripts/src/logseq/tasks/lang.clj @@ -150,7 +150,7 @@ the ones defined for the default :en lang. This catches translations that have been added in UI but don't have an entry or translations no longer used in the UI" [{:keys [fix?]}] - (let [actual-dicts (->> (shell {:out :string} + (let [actual-dicts (->> (shell {:out :string :shutdown nil} ;; This currently assumes all ui translations ;; use (t and src/main. This can easily be ;; tweaked as needed From ff630c1503ff9ba694d99289ba50908c11911635 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 14:51:25 +0800 Subject: [PATCH 015/375] add 004-logseq-cli-verb-subcommands.md --- .../004-logseq-cli-verb-subcommands.md | 223 ++++++++++++++++++ 1 file changed, 223 insertions(+) create mode 100644 docs/agent-guide/004-logseq-cli-verb-subcommands.md diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md new file mode 100644 index 0000000000..37d3efc1a0 --- /dev/null +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -0,0 +1,223 @@ +# Logseq CLI Verb-First Subcommands Implementation Plan + +Goal: Refactor logseq-cli to remove the block subcommand group and replace it with verb-first subcommands for list, add, remove, search, and show. + +Architecture: Keep the babashka/cli dispatch table but reorganize it into verb-first subcommands, and route each verb to typed actions for pages, blocks, tags, and properties through existing db-worker-node thread-apis. + +Tech Stack: ClojureScript, babashka/cli, db-worker-node thread-apis, Datascript queries. + +Related: Builds on docs/agent-guide/003-db-worker-node-cli-orchestration.md and docs/agent-guide/002-logseq-cli-subcommands.md. + +## Problem statement + +The current CLI uses a block subcommand group, which makes the interface noun-first and inconsistent with graph and server commands. + +We need to make the CLI verb-first, so that content operations are consistent with other tooling and easier to extend for new resource types. + +The refactor must preserve existing behaviors for add, remove, search, and tree while adding list commands for pages, tags, and properties, and renaming tree to show. + +## Testing Plan + +I will add unit tests that cover verb-first parsing, group help output, and option validation for list, add, remove, search, and tree. + +I will add unit tests that assert list option parsing for each list subtype and that invalid flag combinations are rejected. + +I will add integration tests that run list, add, remove, search, and tree against a real db-worker-node with a test graph and assert output shapes. + +I will follow @test-driven-development for all behavior changes. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Command surface + +The block subcommand group is removed and replaced with verb-first subcommands. + +Help output groups commands into two sections, with Graph Inspect and Edit first, and Graph Management last. + +Group names and order: + +| Group | Commands | Order | +| --- | --- | --- | +| Graph Inspect and Edit | list, add, remove, search, show | First | +| Graph Management | graph, server | Last | + +| Command | Subcommand | Purpose | +| --- | --- | --- | +| list | page | List pages | +| list | tag | List tags | +| list | property | List properties | +| list | block | List blocks | +| add | block | Add blocks | +| add | page | Create page | +| remove | block | Remove block | +| remove | page | Remove page | +| search | none | Search across resources | +| show | none | Show block tree | + +Global options remain unchanged and are shared across all commands. + +## List options detail + +The list command should expose a consistent shape across resource types. + +Common list options: + +| Option | Applies to | Purpose | Notes | +| --- | --- | --- | --- | +| --expand | page, tag, property | Include expanded metadata | Maps to existing api-list-* expand behavior. | +| --limit N | page, tag, property, block | Limit results | Implemented in CLI after fetch unless server supports it. | +| --offset N | page, tag, property, block | Offset results | Implemented in CLI after fetch unless server supports it. | +| --sort FIELD | page, tag, property, block | Sort results | Field whitelist per type. | +| --order asc|desc | page, tag, property, block | Sort direction | Defaults to asc. | +| --output FORMAT | all | Output format | Existing output handling. | + +List page options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-journal | Include journal pages | Default is include all. | +| --journal-only | Only journal pages | Requires journal detection in api-list-pages. | +| --include-hidden | Include hidden pages | Requires a flag to bypass entity-util/hidden? filtering. | +| --updated-after ISO8601 | Filter by updated-at | Compare to :block/updated-at. | +| --created-after ISO8601 | Filter by created-at | Compare to :block/created-at. | +| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | + +List tag options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-built-in | Include built-in classes | Built-in tags are currently included by default, clarify behavior. | +| --with-properties | Include class properties | Uses :logseq.property.class/properties when expanded. | +| --with-extends | Include class extends | Uses :logseq.property.class/extends when expanded. | +| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | + +List property options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-built-in | Include built-in properties | Built-in properties are currently included by default, clarify behavior. | +| --with-classes | Include property classes | Uses :logseq.property/classes when expanded. | +| --with-type | Include property type | Uses :logseq.property/type when expanded. | +| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | + +List block is removed to avoid overlap with search. + +## Search options detail + +Search has no subcommands and searches across pages, blocks, tags, and properties by default. + +| Option | Purpose | Notes | +| --- | --- | --- | +| --text QUERY | Search text | Required unless positional args are used. | +| --type page|block|tag|property|all | Restrict types | Default is all. | +| --tag NAME | Restrict to a specific tag | Tag is a class page, e.g. Page, Asset, Task. | +| --limit N | Limit results | Apply after merging type results. | +| --case-sensitive | Case sensitive search | Default is case-insensitive. | +| --include-content | Search block content, not just title | Requires query expansion. | +| --sort updated-at|created-at | Sort results | Default is relevance or stable order. | +| --order asc|desc | Sort direction | Defaults to desc for time sorts. | + +## Tree options detail + +Show has no subcommands and returns the block tree for a page or block. + +| Option | Purpose | Notes | +| --- | --- | --- | +| --id ID | Tree root by :db/id | Mutually exclusive with other identifiers. | +| --uuid UUID | Tree root by :block/uuid | Mutually exclusive with other identifiers. | +| --page-name NAME | Tree root by :block/title for a #Page block | Must be a page. | +| --level N | Limit tree depth | N >= 1, default 10. | +| --format text|json|edn | Output format | Existing behavior. | + +## Plan + +1. Review current CLI command parsing and action routing in src/main/logseq/cli/commands.cljs to map block group behavior to verb-first commands. +2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and tree. +3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, list property, and list block. +4. Add failing unit tests for add page, add tag, add property, remove tag, and remove property parse and validation behavior. +5. Add failing unit tests that assert search defaults to all types and respects --type and --include-content options. +6. Add failing unit tests that assert tree accepts --page or --block and rejects missing targets. +7. Run bb dev:test -v logseq.cli.commands-test/test-parse-args and confirm failures are about the new verbs and options. +8. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. +9. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last. +10. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface. +11. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags. +12. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties. +13. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs. +14. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --include-content. +15. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. +16. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. +21. Run bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search and confirm failures before implementation. +22. Implement behavior for list, add, remove, search, and tree until all tests pass. +23. Update docs/cli/logseq-cli.md with new verb-first commands and examples. +24. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors. +25. Run bb dev:lint-and-test and confirm a zero exit code. + +## Edge cases + +Missing subcommand should return group help for list, add, or remove, and still exit with a non-zero status. + +Unknown subcommands should return a helpful error message that lists the valid subcommands. + +Add block should still default to today’s journal page when no page is provided and no parent is provided. + +Search across all types should avoid duplicate hits when a tag or property is also a page with the same title. + +Show should return a deterministic order based on :block/order. + +## Testing commands and expected output + +Run a single unit test in red phase. + +```bash +bb dev:test -v logseq.cli.commands-test/test-parse-args +``` + +Expected output includes failing assertions about the new verb-first commands and ends with a non-zero exit code. + +Run the integration tests in red phase. + +```bash +bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search +``` + +Expected output includes failing assertions about list and search output and ends with a non-zero exit code. + +Run the full suite in green phase. + +```bash +bb dev:test -r logseq.cli.* +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and tests after all changes. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +The unit tests will validate parsing, help output, and option validation for each new verb-first command. + +The integration tests will create a temporary graph, add pages, tags, and properties, and verify list, search, and tree output against db-worker-node behavior. + +## Implementation Details + +- Replace block group entries with list, add, remove, search, and show in src/main/logseq/cli/commands.cljs. +- Add list subtype specs and validation, including common list options and per-type field filtering in src/main/logseq/cli/commands.cljs. +- Extend search to combine page, block, tag, and property queries and to enforce --type and --tag behavior in src/main/logseq/cli/commands.cljs. +- Preserve existing add block and remove block behavior while changing only the command paths and option names. +- Rename tree to show and add id, uuid, page-name, and level parsing in src/main/logseq/cli/commands.cljs. +- Update docs/cli/logseq-cli.md to show new usage and examples. + +## Question + +Add tag, remove tag, add property, remove property are Implement Later. + +Rename for page, tag, and property is Implement Later. + +--- From 07b70240a65ad5de06d564a9696d5b09b9742059 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 15:46:10 +0800 Subject: [PATCH 016/375] impl 004-logseq-cli-verb-subcommands.md --- deps/cli/src/logseq/cli/common/mcp/tools.cljs | 54 +- .../004-logseq-cli-verb-subcommands.md | 54 +- docs/cli/logseq-cli.md | 42 +- src/main/logseq/cli/commands.cljs | 703 +++++++++++++++--- src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 320 ++++---- src/test/logseq/cli/integration_test.cljs | 41 +- 7 files changed, 909 insertions(+), 307 deletions(-) diff --git a/deps/cli/src/logseq/cli/common/mcp/tools.cljs b/deps/cli/src/logseq/cli/common/mcp/tools.cljs index f9f8ef838e..f5bd6949cc 100644 --- a/deps/cli/src/logseq/cli/common/mcp/tools.cljs +++ b/deps/cli/src/logseq/cli/common/mcp/tools.cljs @@ -18,9 +18,14 @@ (defn list-properties "Main fn for ListProperties tool" - [db {:keys [expand]}] + [db {:keys [expand include-built-in] :as options}] + (ensure-db-graph db) + (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] (->> (d/datoms db :avet :block/tags :logseq.class/Property) (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) (map (fn [e] (if expand @@ -35,13 +40,18 @@ (:logseq.property/description e) (update :logseq.property/description db-property/property-value-content)) {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))}))))) + :block/uuid (str (:block/uuid e))})))))) (defn list-tags "Main fn for ListTags tool" - [db {:keys [expand]}] + [db {:keys [expand include-built-in] :as options}] + (ensure-db-graph db) + (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] (->> (d/datoms db :avet :block/tags :logseq.class/Tag) (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) (map (fn [e] (if expand (cond-> (into {} e) @@ -59,7 +69,7 @@ (:logseq.property/description e) (update :logseq.property/description db-property/property-value-content)) {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))}))))) + :block/uuid (str (:block/uuid e))})))))) (defn- get-page-blocks [db page-id] @@ -91,12 +101,40 @@ (dissoc :block/children :block/page)) (get-page-blocks db (:db/id page)))})) +(defn- parse-time + [value] + (cond + (number? value) value + (string? value) (let [ms (js/Date.parse value)] + (when-not (js/isNaN ms) ms)) + :else nil)) + (defn list-pages "Main fn for ListPages tool" - [db {:keys [expand]}] + [db {:keys [expand include-hidden include-journal journal-only created-after updated-after] :as options}] + (ensure-db-graph db) + (let [include-hidden? (boolean include-hidden) + include-journal? (if (contains? options :include-journal) include-journal true) + journal-only? (boolean journal-only) + created-after-ms (parse-time created-after) + updated-after-ms (parse-time updated-after)] (->> (d/datoms db :avet :block/name) (map #(d/entity db (:e %))) - (remove entity-util/hidden?) + (remove (fn [e] + (and (not include-hidden?) + (entity-util/hidden? e)))) + (remove (fn [e] + (let [is-journal? (ldb/journal? e)] + (cond + journal-only? (not is-journal?) + (false? include-journal?) is-journal? + :else false)))) + (remove (fn [e] + (and created-after-ms + (<= (:block/created-at e 0) created-after-ms)))) + (remove (fn [e] + (and updated-after-ms + (<= (:block/updated-at e 0) updated-after-ms)))) (map (fn [e] (if expand (-> e @@ -105,7 +143,7 @@ (select-keys [:block/uuid :block/title :block/created-at :block/updated-at]) (update :block/uuid str)) {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))}))))) + :block/uuid (str (:block/uuid e))})))))) ;; upsert-nodes tool ;; ================= @@ -395,4 +433,4 @@ [conn operations* {:keys [dry-run] :as opts}] (let [import-edn (build-upsert-nodes-edn @conn operations*)] (when-not dry-run (import-edn-data conn import-edn)) - (summarize-upsert-operations operations* opts))) \ No newline at end of file + (summarize-upsert-operations operations* opts))) diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md index 37d3efc1a0..dd21e48368 100644 --- a/docs/agent-guide/004-logseq-cli-verb-subcommands.md +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -46,7 +46,6 @@ Group names and order: | list | page | List pages | | list | tag | List tags | | list | property | List properties | -| list | block | List blocks | | add | block | Add blocks | | add | page | Create page | | remove | block | Remove block | @@ -65,10 +64,10 @@ Common list options: | Option | Applies to | Purpose | Notes | | --- | --- | --- | --- | | --expand | page, tag, property | Include expanded metadata | Maps to existing api-list-* expand behavior. | -| --limit N | page, tag, property, block | Limit results | Implemented in CLI after fetch unless server supports it. | -| --offset N | page, tag, property, block | Offset results | Implemented in CLI after fetch unless server supports it. | -| --sort FIELD | page, tag, property, block | Sort results | Field whitelist per type. | -| --order asc|desc | page, tag, property, block | Sort direction | Defaults to asc. | +| --limit N | page, tag, property | Limit results | Implemented in CLI after fetch unless server supports it. | +| --offset N | page, tag, property | Offset results | Implemented in CLI after fetch unless server supports it. | +| --sort FIELD | page, tag, property | Sort results | Field whitelist per type. | +| --order asc|desc | page, tag, property | Sort direction | Defaults to asc. | | --output FORMAT | all | Output format | Existing output handling. | List page options: @@ -132,26 +131,25 @@ Show has no subcommands and returns the block tree for a page or block. ## Plan 1. Review current CLI command parsing and action routing in src/main/logseq/cli/commands.cljs to map block group behavior to verb-first commands. -2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and tree. -3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, list property, and list block. -4. Add failing unit tests for add page, add tag, add property, remove tag, and remove property parse and validation behavior. -5. Add failing unit tests that assert search defaults to all types and respects --type and --include-content options. -6. Add failing unit tests that assert tree accepts --page or --block and rejects missing targets. -7. Run bb dev:test -v logseq.cli.commands-test/test-parse-args and confirm failures are about the new verbs and options. -8. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. -9. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last. -10. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface. -11. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags. -12. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties. -13. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs. -14. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --include-content. -15. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. -16. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. -21. Run bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search and confirm failures before implementation. -22. Implement behavior for list, add, remove, search, and tree until all tests pass. -23. Update docs/cli/logseq-cli.md with new verb-first commands and examples. -24. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors. -25. Run bb dev:lint-and-test and confirm a zero exit code. +2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and show. +3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, and list property. +4. Add failing unit tests that assert search defaults to all types and respects --type and --include-content options. +5. Add failing unit tests that assert show accepts --page-name, --uuid, or --id and rejects missing targets. +6. Run bb dev:test -v logseq.cli.commands-test/test-parse-args and confirm failures are about the new verbs and options. +7. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. +8. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last. +9. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface. +10. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags. +11. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties. +12. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs. +13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --include-content. +14. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. +15. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. +16. Run bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search and confirm failures before implementation. +17. Implement behavior for list, add, remove, search, and show until all tests pass. +18. Update docs/cli/logseq-cli.md with new verb-first commands and examples. +19. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors. +20. Run bb dev:lint-and-test and confirm a zero exit code. ## Edge cases @@ -178,7 +176,7 @@ Expected output includes failing assertions about the new verb-first commands an Run the integration tests in red phase. ```bash -bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search +bb dev:test -v logseq.cli.integration-test/test-cli-list-add-search-show-remove ``` Expected output includes failing assertions about list and search output and ends with a non-zero exit code. @@ -216,8 +214,6 @@ The integration tests will create a temporary graph, add pages, tags, and proper ## Question -Add tag, remove tag, add property, remove property are Implement Later. - -Rename for page, tag, and property is Implement Later. +None. --- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 21dcdc2ce3..2d09e810f9 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -51,24 +51,34 @@ Server commands: - `server stop --repo ` - stop db-worker-node for a graph - `server restart --repo ` - restart db-worker-node for a graph -Block commands: -- `block add --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given -- `block add --blocks [--page ] [--parent ]` - insert blocks via EDN vector -- `block add --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file -- `block remove --block ` - remove a block and its children -- `block remove --page ` - remove a page and its children -- `block search --text [--limit ]` - search block titles (Datalog includes?) -- `block tree --page [--format text|json|edn]` - show page tree -- `block tree --block [--format text|json|edn]` - show block tree +Inspect and edit commands: +- `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages +- `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags +- `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties +- `add block --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given +- `add block --blocks [--page ] [--parent ]` - insert blocks via EDN vector +- `add block --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file +- `add page --page ` - create a page +- `remove block --block ` - remove a block and its children +- `remove page --page ` - remove a page and its children +- `search --text [--type page|block|tag|property|all] [--include-content] [--limit ]` - search across pages, blocks, tags, and properties +- `show --page-name [--format text|json|edn] [--level ]` - show page tree +- `show --uuid [--format text|json|edn] [--level ]` - show block tree +- `show --id [--format text|json|edn] [--level ]` - show block tree by db/id Help output: ``` Subcommands: - block add [options] Add blocks - block remove [options] Remove block or page - block search [options] Search blocks - block tree [options] Show tree + list page [options] List pages + list tag [options] List tags + list property [options] List properties + add block [options] Add blocks + add page [options] Create page + remove block [options] Remove block + remove page [options] Remove page + search [options] Search graph + show [options] Show tree ``` Output formats: @@ -78,8 +88,8 @@ Examples: ```bash node ./static/logseq-cli.js graph create --repo demo -node ./static/logseq-cli.js block add --page TestPage --content "hello world" -node ./static/logseq-cli.js block search --text "hello" -node ./static/logseq-cli.js block tree --page TestPage --format json --output json +node ./static/logseq-cli.js add block --page TestPage --content "hello world" +node ./static/logseq-cli.js search --text "hello" +node ./static/logseq-cli.js show --page-name TestPage --format json --output json node ./static/logseq-cli.js server list ``` diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 360c0cad46..4026b2c088 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -37,19 +37,78 @@ :page {:desc "Page name"} :parent {:desc "Parent block UUID for add"}}) -(def ^:private content-remove-spec - {:block {:desc "Block UUID"} - :page {:desc "Page name"}}) +(def ^:private add-page-spec + {:page {:desc "Page name"}}) -(def ^:private content-search-spec - {:text {:desc "Search text"} +(def ^:private remove-block-spec + {:block {:desc "Block UUID"}}) + +(def ^:private remove-page-spec + {:page {:desc "Page name"}}) + +(def ^:private list-common-spec + {:expand {:desc "Include expanded metadata" + :coerce :boolean} :limit {:desc "Limit results" - :coerce :long}}) + :coerce :long} + :offset {:desc "Offset results" + :coerce :long} + :sort {:desc "Sort field"} + :order {:desc "Sort order (asc, desc)"}}) -(def ^:private content-tree-spec - {:block {:desc "Block UUID"} - :page {:desc "Page name"} - :format {:desc "Output format (tree)"}}) +(def ^:private list-page-spec + (merge list-common-spec + {:include-journal {:desc "Include journal pages" + :coerce :boolean} + :journal-only {:desc "Only journal pages" + :coerce :boolean} + :include-hidden {:desc "Include hidden pages" + :coerce :boolean} + :updated-after {:desc "Filter by updated-at (ISO8601)"} + :created-after {:desc "Filter by created-at (ISO8601)"} + :fields {:desc "Select output fields (comma separated)"}})) + +(def ^:private list-tag-spec + (merge list-common-spec + {:include-built-in {:desc "Include built-in tags" + :coerce :boolean} + :with-properties {:desc "Include tag properties" + :coerce :boolean} + :with-extends {:desc "Include tag extends" + :coerce :boolean} + :fields {:desc "Select output fields (comma separated)"}})) + +(def ^:private list-property-spec + (merge list-common-spec + {:include-built-in {:desc "Include built-in properties" + :coerce :boolean} + :with-classes {:desc "Include property classes" + :coerce :boolean} + :with-type {:desc "Include property type" + :coerce :boolean} + :fields {:desc "Select output fields (comma separated)"}})) + +(def ^:private search-spec + {:text {:desc "Search text"} + :type {:desc "Search types (page, block, tag, property, all)"} + :tag {:desc "Restrict to a specific tag"} + :limit {:desc "Limit results" + :coerce :long} + :case-sensitive {:desc "Case sensitive search" + :coerce :boolean} + :include-content {:desc "Search block content" + :coerce :boolean} + :sort {:desc "Sort field (updated-at, created-at)"} + :order {:desc "Sort order (asc, desc)"}}) + +(def ^:private show-spec + {:id {:desc "Block db/id" + :coerce :long} + :uuid {:desc "Block UUID"} + :page-name {:desc "Page name"} + :level {:desc "Limit tree depth" + :coerce :long} + :format {:desc "Output format (text, json, edn)"}}) (defn- format-commands [table] @@ -82,14 +141,21 @@ (defn- top-level-summary [table] - (string/join "\n" - ["Usage: logseq-cli [options]" - "" - "Commands:" - (format-commands table) - "" - "Options:" - (cli/format-opts {:spec global-spec})])) + (let [groups [{:title "Graph Inspect and Edit" + :commands #{"list" "add" "remove" "search" "show"}} + {:title "Graph Management" + :commands #{"graph" "server"}}] + render-group (fn [{:keys [title commands]}] + (let [entries (filter #(contains? commands (first (:cmds %))) table)] + (string/join "\n" [title (format-commands entries)])))] + (string/join "\n" + ["Usage: logseq-cli [options]" + "" + "Commands:" + (string/join "\n\n" (map render-group groups)) + "" + "Options:" + (cli/format-opts {:spec global-spec})]))) (defn- command-summary [{:keys [cmds spec]}] @@ -145,6 +211,13 @@ :message "block or page is required"} :summary summary}) +(defn- missing-page-name-result + [summary] + {:ok? false + :error {:code :missing-page-name + :message "page name is required"} + :summary summary}) + (defn- missing-search-result [summary] {:ok? false @@ -172,6 +245,67 @@ :message message} :summary summary}) +(def ^:private list-sort-fields + {:list-page #{"title" "created-at" "updated-at"} + :list-tag #{"name" "title"} + :list-property #{"name" "title"}}) + +(def ^:private show-formats + #{"text" "json" "edn"}) + +(def ^:private search-types + #{"page" "block" "tag" "property" "all"}) + +(defn- invalid-list-options? + [command opts] + (let [{:keys [order include-journal journal-only]} opts + sort-field (:sort opts) + allowed (get list-sort-fields command)] + (cond + (and include-journal journal-only) + "include-journal and journal-only are mutually exclusive" + + (and (seq sort-field) (not (contains? allowed sort-field))) + (str "invalid sort field: " sort-field) + + (and (seq order) (not (#{"asc" "desc"} order))) + (str "invalid order: " order) + + :else + nil))) + +(defn- invalid-show-options? + [opts] + (let [format (:format opts) + level (:level opts)] + (cond + (and (seq format) (not (contains? show-formats (string/lower-case format)))) + (str "invalid format: " format) + + (and (some? level) (< level 1)) + "level must be >= 1" + + :else + nil))) + +(defn- invalid-search-options? + [opts] + (let [type (:type opts) + order (:order opts) + sort-field (:sort opts)] + (cond + (and (seq type) (not (contains? search-types type))) + (str "invalid type: " type) + + (and (seq sort-field) (not (#{"updated-at" "created-at"} sort-field))) + (str "invalid sort field: " sort-field) + + (and (seq order) (not (#{"asc" "desc"} order))) + (str "invalid order: " order) + + :else + nil))) + (defn- command-entry [cmds command desc spec] (let [spec* (merge-spec spec)] @@ -198,10 +332,15 @@ (command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) (command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec) (command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec) - (command-entry ["block" "add"] :add "Add blocks" content-add-spec) - (command-entry ["block" "remove"] :remove "Remove block or page" content-remove-spec) - (command-entry ["block" "search"] :search "Search blocks" content-search-spec) - (command-entry ["block" "tree"] :tree "Show tree" content-tree-spec)]) + (command-entry ["list" "page"] :list-page "List pages" list-page-spec) + (command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec) + (command-entry ["list" "property"] :list-property "List properties" list-property-spec) + (command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) + (command-entry ["add" "page"] :add-page "Create page" add-page-spec) + (command-entry ["remove" "block"] :remove-block "Remove block" remove-block-spec) + (command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec) + (command-entry ["search"] :search "Search graph" search-spec) + (command-entry ["show"] :show "Show tree" show-spec)]) (def ^:private global-aliases (->> global-spec @@ -258,7 +397,8 @@ has-content? (or (seq (:content opts)) (seq (:blocks opts)) (seq (:blocks-file opts)) - has-args?)] + has-args?) + show-targets (filter some? [(:id opts) (:uuid opts) (:page-name opts)])] (cond (:help opts) (help-result cmd-summary) @@ -267,18 +407,37 @@ (not (seq graph))) (missing-graph-result summary) - (and (= command :add) (not has-content?)) + (and (= command :add-block) (not has-content?)) (missing-content-result summary) - (and (= command :remove) (not (or (seq (:block opts)) (seq (:page opts))))) + (and (= command :add-page) (not (seq (:page opts)))) + (missing-page-name-result summary) + + (and (= command :remove-block) (not (seq (:block opts)))) (missing-target-result summary) - (and (= command :tree) (not (or (seq (:block opts)) (seq (:page opts))))) + (and (= command :remove-page) (not (seq (:page opts)))) (missing-target-result summary) + (and (= command :show) (empty? show-targets)) + (missing-target-result summary) + + (and (= command :show) (> (count show-targets) 1)) + (invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") + (and (= command :search) (not (or (seq (:text opts)) has-args?))) (missing-search-result summary) + (and (#{:list-page :list-tag :list-property} command) + (invalid-list-options? command opts)) + (invalid-options-result summary (invalid-list-options? command opts)) + + (and (= command :show) (invalid-show-options? opts)) + (invalid-options-result summary (invalid-show-options? opts)) + + (and (= command :search) (invalid-search-options? opts)) + (invalid-options-result summary (invalid-search-options? opts)) + (and (#{:server-status :server-start :server-stop :server-restart} command) (not (seq (:repo opts)))) (missing-repo-result summary) @@ -301,7 +460,7 @@ :error {:code :missing-command :message "missing command"} :summary summary}) - (if (and (= 1 (count args)) (#{"graph" "block" "server"} (first args))) + (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "remove"} (first args))) (help-result (group-summary (first args) table)) (try (let [result (cli/dispatch table args {:spec global-spec})] @@ -457,38 +616,56 @@ (mapv first rows)))) (defn- build-tree - [blocks root-id] + [blocks root-id max-depth] (let [parent->children (group-by #(get-in % [:block/parent :db/id]) blocks) sort-children (fn [children] (vec (sort-by :block/order children))) - build (fn build [parent-id] + build (fn build [parent-id depth] (mapv (fn [b] - (let [children (build (:db/id b))] + (let [children (build (:db/id b) (inc depth))] (cond-> b (seq children) (assoc :block/children children)))) - (sort-children (get parent->children parent-id))))] - (build root-id))) + (if (and max-depth (>= depth max-depth)) + [] + (sort-children (get parent->children parent-id)))))] + (build root-id 1))) (defn- fetch-tree - [config {:keys [repo block page]}] - (if (seq block) - (if-not (common-util/uuid-string? block) - (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + [config {:keys [repo id uuid page-name level]}] + (let [max-depth (or level 10)] + (cond + (some? id) (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] - [:block/uuid (uuid block)]])] + [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks (:db/id entity))] + children (build-tree blocks (:db/id entity) max-depth)] {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found}))))) - (p/let [page-entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title] [:block/name page]])] - (if-let [page-id (:db/id page-entity)] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks page-id)] - {:root (assoc page-entity :block/children children)}) - (throw (ex-info "page not found" {:code :page-not-found})))))) + (throw (ex-info "block not found" {:code :block-not-found})))) + + (seq uuid) + (if-not (common-util/uuid-string? uuid) + (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + (p/let [entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid (uuid uuid)]])] + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found}))))) + + (seq page-name) + (p/let [page-entity (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/uuid :block/title] [:block/name page-name]])] + (if-let [page-id (:db/id page-entity)] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks page-id max-depth)] + {:root (assoc page-entity :block/children children)}) + (throw (ex-info "page not found" {:code :page-not-found})))) + + :else + (p/rejected (ex-info "block or page required" {:code :missing-target}))))) (defn- tree->text [{:keys [root]}] @@ -521,6 +698,81 @@ :error {:code :missing-repo :message message}}) +(def ^:private list-page-field-map + {"title" :block/title + "uuid" :block/uuid + "created-at" :block/created-at + "updated-at" :block/updated-at}) + +(def ^:private list-tag-field-map + {"name" :block/title + "title" :block/title + "uuid" :block/uuid + "properties" :logseq.property.class/properties + "extends" :logseq.property.class/extends + "description" :logseq.property/description}) + +(def ^:private list-property-field-map + {"name" :block/title + "title" :block/title + "uuid" :block/uuid + "classes" :logseq.property/classes + "type" :logseq.property/type + "description" :logseq.property/description}) + +(defn- parse-field-list + [fields] + (when (seq fields) + (->> (string/split fields #",") + (map string/trim) + (remove string/blank?) + vec))) + +(defn- apply-fields + [items fields field-map] + (if (seq fields) + (let [keys (->> fields + (map #(get field-map %)) + (remove nil?) + vec)] + (if (seq keys) + (mapv #(select-keys % keys) items) + items)) + items)) + +(defn- apply-sort + [items sort-field order field-map] + (if (seq sort-field) + (let [sort-key (get field-map sort-field) + sorted (if sort-key + (sort-by #(get % sort-key) items) + items) + sorted (if (= "desc" order) (reverse sorted) sorted)] + (vec sorted)) + (vec items))) + +(defn- apply-offset-limit + [items offset limit] + (cond-> items + (some? offset) (->> (drop offset) vec) + (some? limit) (->> (take limit) vec))) + +(defn- prepare-tag-item + [item {:keys [expand with-properties with-extends]}] + (if expand + (cond-> item + (not with-properties) (dissoc :logseq.property.class/properties) + (not with-extends) (dissoc :logseq.property.class/extends)) + item)) + +(defn- prepare-property-item + [item {:keys [expand with-classes with-type]}] + (if expand + (cond-> item + (not with-classes) (dissoc :logseq.property/classes) + (not with-type) (dissoc :logseq.property/type)) + item)) + (defn- build-graph-action [command graph repo] (case command @@ -615,7 +867,7 @@ :error {:code :unknown-command :message (str "unknown server command: " command)}})) -(defn- build-add-action +(defn- build-add-block-action [options args repo] (if-not (seq repo) (missing-repo-error "repo is required for add") @@ -626,28 +878,64 @@ (if-not (:ok? vector-result) vector-result {:ok? true - :action {:type :add + :action {:type :add-block :repo repo :graph (repo->graph repo) :page (:page options) :parent (:parent options) :blocks (:value vector-result)}})))))) -(defn- build-remove-action +(defn- build-add-page-action + [options repo] + (if-not (seq repo) + (missing-repo-error "repo is required for add") + (let [page (some-> (:page options) string/trim)] + (if (seq page) + {:ok? true + :action {:type :add-page + :repo repo + :graph (repo->graph repo) + :page page}} + {:ok? false + :error {:code :missing-page-name + :message "page name is required"}})))) + +(defn- build-remove-block-action [options repo] (if-not (seq repo) (missing-repo-error "repo is required for remove") - (let [block (:block options) - page (:page options)] - (if (or (seq block) (seq page)) + (let [block (some-> (:block options) string/trim)] + (if (seq block) {:ok? true - :action {:type :remove + :action {:type :remove-block + :repo repo + :block block}} + {:ok? false + :error {:code :missing-target + :message "block is required"}})))) + +(defn- build-remove-page-action + [options repo] + (if-not (seq repo) + (missing-repo-error "repo is required for remove") + (let [page (some-> (:page options) string/trim)] + (if (seq page) + {:ok? true + :action {:type :remove-page :repo repo - :block block :page page}} {:ok? false :error {:code :missing-target - :message "block or page is required"}})))) + :message "page is required"}})))) + +(defn- build-list-action + [command options repo] + (if-not (seq repo) + (missing-repo-error "repo is required for list") + {:ok? true + :action {:type command + :repo repo + :options options}})) (defn- build-search-action [options args repo] @@ -659,28 +947,35 @@ :action {:type :search :repo repo :text text - :limit (:limit options)}} + :search-type (:type options) + :tag (:tag options) + :limit (:limit options) + :case-sensitive (:case-sensitive options) + :include-content (:include-content options) + :sort (:sort options) + :order (:order options)}} {:ok? false :error {:code :missing-search-text :message "search text is required"}})))) -(defn- build-tree-action +(defn- build-show-action [options repo] (if-not (seq repo) - (missing-repo-error "repo is required for tree") - (let [block (:block options) - page (:page options) - target (or block page)] - (if (seq target) - {:ok? true - :action {:type :tree - :repo repo - :block block - :page page - :format (some-> (:format options) string/lower-case)}} + (missing-repo-error "repo is required for show") + (let [format (some-> (:format options) string/lower-case) + targets (filter some? [(:id options) (:uuid options) (:page-name options)])] + (if (empty? targets) {:ok? false :error {:code :missing-target - :message "block or page is required"}})))) + :message "block or page is required"}} + {:ok? true + :action {:type :show + :repo repo + :id (:id options) + :uuid (:uuid options) + :page-name (:page-name options) + :level (:level options) + :format format}})))) (defn build-action [parsed config] @@ -697,17 +992,26 @@ (:server-list :server-status :server-start :server-stop :server-restart) (build-server-action command server-repo) - :add - (build-add-action options args repo) + (:list-page :list-tag :list-property) + (build-list-action command options repo) - :remove - (build-remove-action options repo) + :add-block + (build-add-block-action options args repo) + + :add-page + (build-add-page-action options repo) + + :remove-block + (build-remove-block-action options repo) + + :remove-page + (build-remove-page-action options repo) :search (build-search-action options args repo) - :tree - (build-tree-action options repo) + :show + (build-show-action options repo) {:ok? false :error {:code :unknown-command @@ -760,7 +1064,57 @@ :logseq.kv/graph-created-at (:kv/value created) :logseq.kv/schema-version (:kv/value schema)}}))) -(defn- execute-add +(defn- execute-list-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg "thread-api/api-list-pages" false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + sorted (apply-sort items (:sort options) order list-page-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-page-field-map) + limited)] + {:status :ok + :data {:items final}}))) + +(defn- execute-list-tag + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg "thread-api/api-list-tags" false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + prepared (mapv #(prepare-tag-item % options) items) + sorted (apply-sort prepared (:sort options) order list-tag-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-tag-field-map) + limited)] + {:status :ok + :data {:items final}}))) + +(defn- execute-list-property + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg "thread-api/api-list-properties" false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + prepared (mapv #(prepare-property-item % options) items) + sorted (apply-sort prepared (:sort options) order list-property-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-property-field-map) + limited)] + {:status :ok + :data {:items final}}))) + +(defn- execute-add-block [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) target-id (resolve-add-target cfg action) @@ -773,6 +1127,14 @@ {:status :ok :data {:result result}}))) +(defn- execute-add-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + ops [[:create-page [(:page action) {}]]] + result (transport/invoke cfg "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] + {:status :ok + :data {:result result}}))) + (defn- execute-remove [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -780,21 +1142,179 @@ {:status :ok :data {:result result}}))) +(defn- query-pages + [cfg repo text case-sensitive?] + (let [query (if case-sensitive? + '[:find ?e ?title ?uuid ?updated ?created + :in $ ?q + :where + [?e :block/name ?name] + [?e :block/title ?title] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? ?title ?q)]] + '[:find ?e ?title ?uuid ?updated ?created + :in $ ?q + :where + [?e :block/name ?name] + [?e :block/title ?title] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? (string/lower-case ?title) ?q)]]) + q* (if case-sensitive? text (string/lower-case text))] + (transport/invoke cfg "thread-api/q" false [repo [query q*]]))) + +(defn- query-blocks + [cfg repo text case-sensitive? tag include-content?] + (let [has-tag? (seq tag) + content-attr (if include-content? :block/content :block/title) + query (cond + (and case-sensitive? has-tag?) + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q ?tag-name + :where + [?tag :block/name ?tag-name] + [?e :block/tags ?tag] + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? ?value ?q)]] + + case-sensitive? + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q + :where + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? ?value ?q)]] + + has-tag? + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q ?tag-name + :where + [?tag :block/name ?tag-name] + [?e :block/tags ?tag] + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? (string/lower-case ?value) ?q)]] + + :else + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q + :where + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(string/includes? (string/lower-case ?value) ?q)]]) + q* (if case-sensitive? text (string/lower-case text)) + tag-name (some-> tag string/lower-case)] + (if has-tag? + (transport/invoke cfg "thread-api/q" false [repo [query q* tag-name]]) + (transport/invoke cfg "thread-api/q" false [repo [query q*]])))) + +(defn- normalize-search-types + [type] + (let [type (or type "all")] + (case type + "page" [:page] + "block" [:block] + "tag" [:tag] + "property" [:property] + [:page :block :tag :property]))) + +(defn- search-sort-key + [item sort-field] + (case sort-field + "updated-at" (:updated-at item) + "created-at" (:created-at item) + nil)) + (defn- execute-search [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - query '[:find ?e ?title - :in $ ?q - :where - [?e :block/title ?title] - [(clojure.string/includes? ?title ?q)]] - results (transport/invoke cfg "thread-api/q" false [(:repo action) [query (:text action)]]) - mapped (mapv (fn [[id title]] {:db/id id :block/title title}) results) - limited (if (some? (:limit action)) (vec (take (:limit action) mapped)) mapped)] + types (normalize-search-types (:search-type action)) + case-sensitive? (boolean (:case-sensitive action)) + text (:text action) + tag (:tag action) + page-results (when (some #{:page} types) + (p/let [rows (query-pages cfg (:repo action) text case-sensitive?)] + (mapv (fn [[id title uuid updated created]] + {:type "page" + :db/id id + :title title + :uuid (str uuid) + :updated-at updated + :created-at created}) + rows))) + include-content? (boolean (:include-content action)) + block-results (when (some #{:block} types) + (p/let [rows (query-blocks cfg (:repo action) text case-sensitive? tag include-content?)] + (mapv (fn [[id content uuid updated created]] + {:type "block" + :db/id id + :content content + :uuid (str uuid) + :updated-at updated + :created-at created}) + rows))) + tag-results (when (some #{:tag} types) + (p/let [items (transport/invoke cfg "thread-api/api-list-tags" false + [(:repo action) {:expand true :include-built-in true}]) + q* (if case-sensitive? text (string/lower-case text))] + (->> items + (filter (fn [item] + (let [title (:block/title item)] + (if case-sensitive? + (string/includes? title q*) + (string/includes? (string/lower-case title) q*))))) + (mapv (fn [item] + {:type "tag" + :title (:block/title item) + :uuid (:block/uuid item)}))))) + property-results (when (some #{:property} types) + (p/let [items (transport/invoke cfg "thread-api/api-list-properties" false + [(:repo action) {:expand true :include-built-in true}]) + q* (if case-sensitive? text (string/lower-case text))] + (->> items + (filter (fn [item] + (let [title (:block/title item)] + (if case-sensitive? + (string/includes? title q*) + (string/includes? (string/lower-case title) q*))))) + (mapv (fn [item] + {:type "property" + :title (:block/title item) + :uuid (:block/uuid item)}))))) + results (->> (concat (or page-results []) + (or block-results []) + (or tag-results []) + (or property-results [])) + (distinct) + vec) + sorted (if-let [sort-field (:sort action)] + (let [order (or (:order action) "desc")] + (->> results + (sort-by #(search-sort-key % sort-field)) + (cond-> (= order "desc") reverse) + vec)) + results) + limited (if (some? (:limit action)) (vec (take (:limit action) sorted)) sorted)] {:status :ok :data {:results limited}}))) -(defn- execute-tree +(defn- execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) tree-data (fetch-tree cfg action) @@ -858,10 +1378,15 @@ :invoke (execute-invoke action config) :graph-switch (execute-graph-switch action config) :graph-info (execute-graph-info action config) - :add (execute-add action config) - :remove (execute-remove action config) + :list-page (execute-list-page action config) + :list-tag (execute-list-tag action config) + :list-property (execute-list-property action config) + :add-block (execute-add-block action config) + :add-page (execute-add-page action config) + :remove-block (execute-remove action config) + :remove-page (execute-remove action config) :search (execute-search action config) - :tree (execute-tree action config) + :show (execute-show action config) :server-list (execute-server-list action config) :server-status (execute-server-status action config) :server-start (execute-server-start action config) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index bd1659dc98..ff5ef19464 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq-cli [options]" "" - "Commands: graph list, graph create, graph switch, graph remove, graph validate, graph info, server list, server status, server start, server stop, server restart, block add, block remove, block search, block tree" + "Commands: list page, list tag, list property, add block, add page, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 1ff60c10cd..225939a66f 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -6,12 +6,18 @@ [promesa.core :as p])) (deftest test-help-output - (testing "top-level help lists subcommand groups" + (testing "top-level help lists command groups" (let [result (commands/parse-args ["--help"]) summary (:summary result)] (is (true? (:help? result))) + (is (string/includes? summary "Graph Inspect and Edit")) + (is (string/includes? summary "Graph Management")) + (is (string/includes? summary "list")) + (is (string/includes? summary "add")) + (is (string/includes? summary "remove")) + (is (string/includes? summary "search")) + (is (string/includes? summary "show")) (is (string/includes? summary "graph")) - (is (string/includes? summary "block")) (is (string/includes? summary "server"))))) (deftest test-parse-args @@ -22,12 +28,27 @@ (is (string/includes? summary "graph list")) (is (string/includes? summary "graph create")))) - (testing "block group shows subcommands" - (let [result (commands/parse-args ["block"]) + (testing "list group shows subcommands" + (let [result (commands/parse-args ["list"]) summary (:summary result)] (is (true? (:help? result))) - (is (string/includes? summary "block add")) - (is (string/includes? summary "block search")))) + (is (string/includes? summary "list page")) + (is (string/includes? summary "list tag")) + (is (string/includes? summary "list property")))) + + (testing "add group shows subcommands" + (let [result (commands/parse-args ["add"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "add block")) + (is (string/includes? summary "add page")))) + + (testing "remove group shows subcommands" + (let [result (commands/parse-args ["remove"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "remove block")) + (is (string/includes? summary "remove page")))) (testing "server group shows subcommands" (let [result (commands/parse-args ["server"]) @@ -51,23 +72,8 @@ (is (seq subcommand-lines)) (is (apply = desc-starts)))) - (testing "block group aligns subcommand columns" - (let [result (commands/parse-args ["block"]) - summary (:summary result) - subcommand-lines (let [lines (string/split-lines summary) - start (inc (.indexOf lines "Subcommands:"))] - (->> lines - (drop start) - (take-while (complement string/blank?)))) - desc-starts (->> subcommand-lines - (keep (fn [line] - (when-let [[_ desc] (re-matches #"^\s+.*?\s{2,}(.+)$" line)] - (.indexOf line desc)))))] - (is (seq subcommand-lines)) - (is (apply = desc-starts)))) - - (testing "server group aligns subcommand columns" - (let [result (commands/parse-args ["server"]) + (testing "list group aligns subcommand columns" + (let [result (commands/parse-args ["list"]) summary (:summary result) subcommand-lines (let [lines (string/split-lines summary) start (inc (.indexOf lines "Subcommands:"))] @@ -83,21 +89,21 @@ (testing "rejects legacy commands" (doseq [command ["graph-list" "graph-create" "graph-switch" "graph-remove" - "graph-validate" "graph-info" "add" "remove" "search" "tree" + "graph-validate" "graph-info" "block" "tree" "ping" "status" "query" "export"]] (let [result (commands/parse-args [command])] (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) - (testing "rejects removed commands" - (let [result (commands/parse-args ["graph" "wat"])] - (is (false? (:ok? result))) - (is (= :unknown-command (get-in result [:error :code]))))) - (testing "rejects removed group commands" - (let [result (commands/parse-args ["content" "add"])] - (is (false? (:ok? result))) - (is (= :unknown-command (get-in result [:error :code]))))) + (doseq [args [["block" "add"] + ["block" "remove"] + ["block" "search"] + ["block" "tree"] + ["content" "add"]]] + (let [result (commands/parse-args args)] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code])))))) (testing "errors on missing command" (let [result (commands/parse-args [])] @@ -114,153 +120,153 @@ (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) -(deftest test-graph-subcommand-parse - (testing "graph list parses" - (let [result (commands/parse-args ["graph" "list"])] +(deftest test-list-subcommand-parse + (testing "list page parses" + (let [result (commands/parse-args ["list" "page" + "--expand" + "--include-journal" + "--limit" "10" + "--offset" "5" + "--sort" "updated-at" + "--order" "desc" + "--fields" "title,updated-at"])] (is (true? (:ok? result))) - (is (= :graph-list (:command result))))) + (is (= :list-page (:command result))) + (is (true? (get-in result [:options :expand]))) + (is (true? (get-in result [:options :include-journal]))) + (is (= 10 (get-in result [:options :limit]))) + (is (= 5 (get-in result [:options :offset]))) + (is (= "updated-at" (get-in result [:options :sort]))) + (is (= "desc" (get-in result [:options :order]))) + (is (= "title,updated-at" (get-in result [:options :fields]))))) - (testing "graph create requires repo option" - (let [result (commands/parse-args ["graph" "create"])] + (testing "list tag parses" + (let [result (commands/parse-args ["list" "tag" + "--expand" + "--include-built-in" + "--with-properties" + "--with-extends" + "--fields" "name,properties"])] + (is (true? (:ok? result))) + (is (= :list-tag (:command result))) + (is (true? (get-in result [:options :expand]))) + (is (true? (get-in result [:options :include-built-in]))) + (is (true? (get-in result [:options :with-properties]))) + (is (true? (get-in result [:options :with-extends]))) + (is (= "name,properties" (get-in result [:options :fields]))))) + + (testing "list property parses" + (let [result (commands/parse-args ["list" "property" + "--expand" + "--include-built-in" + "--with-classes" + "--with-type" + "--fields" "name,type"])] + (is (true? (:ok? result))) + (is (= :list-property (:command result))) + (is (true? (get-in result [:options :expand]))) + (is (true? (get-in result [:options :include-built-in]))) + (is (true? (get-in result [:options :with-classes]))) + (is (true? (get-in result [:options :with-type]))) + (is (= "name,type" (get-in result [:options :fields])))))) + +(deftest test-list-subcommand-validation + (testing "list page rejects mutually exclusive journal flags" + (let [result (commands/parse-args ["list" "page" + "--include-journal" + "--journal-only"])] (is (false? (:ok? result))) - (is (= :missing-graph (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code]))))) - (testing "graph create parses with repo option" - (let [result (commands/parse-args ["graph" "create" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :graph-create (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "graph switch requires repo option" - (let [result (commands/parse-args ["graph" "switch"])] + (testing "list page rejects invalid sort field" + (let [result (commands/parse-args ["list" "page" "--sort" "wat"])] (is (false? (:ok? result))) - (is (= :missing-graph (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code]))))) - (testing "graph switch parses with repo option" - (let [result (commands/parse-args ["graph" "switch" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :graph-switch (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "graph remove requires repo option" - (let [result (commands/parse-args ["graph" "remove"])] + (testing "list tag rejects invalid sort field" + (let [result (commands/parse-args ["list" "tag" "--sort" "wat"])] (is (false? (:ok? result))) - (is (= :missing-graph (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code]))))) - (testing "graph remove parses with repo option" - (let [result (commands/parse-args ["graph" "remove" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :graph-remove (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "graph validate requires repo option" - (let [result (commands/parse-args ["graph" "validate"])] + (testing "list property rejects invalid sort field" + (let [result (commands/parse-args ["list" "property" "--sort" "wat"])] (is (false? (:ok? result))) - (is (= :missing-graph (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code])))))) - (testing "graph validate parses with repo option" - (let [result (commands/parse-args ["graph" "validate" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :graph-validate (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "graph info parses without repo option" - (let [result (commands/parse-args ["graph" "info"])] - (is (true? (:ok? result))) - (is (= :graph-info (:command result))))) - - (testing "graph info parses with repo option" - (let [result (commands/parse-args ["graph" "info" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :graph-info (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "graph subcommands reject unknown flags" - (doseq [subcommand ["list" "create" "switch" "remove" "validate" "info"]] - (let [result (commands/parse-args ["graph" subcommand "--wat"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code])))))) - - - (testing "graph subcommands accept output option" - (let [result (commands/parse-args ["graph" "list" "--output" "edn"])] - (is (true? (:ok? result))) - (is (= "edn" (get-in result [:options :output]))))) - - (testing "server list parses" - (let [result (commands/parse-args ["server" "list"])] - (is (true? (:ok? result))) - (is (= :server-list (:command result))))) - - (testing "server start requires repo" - (let [result (commands/parse-args ["server" "start"])] - (is (false? (:ok? result))) - (is (= :missing-repo (get-in result [:error :code]))))) - - (testing "server start parses with repo" - (let [result (commands/parse-args ["server" "start" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :server-start (:command result))) - (is (= "demo" (get-in result [:options :repo]))))) - - (testing "server stop parses with repo" - (let [result (commands/parse-args ["server" "stop" "--repo" "demo"])] - (is (true? (:ok? result))) - (is (= :server-stop (:command result)))))) - -(deftest test-block-subcommand-parse - (testing "block add requires content source" - (let [result (commands/parse-args ["block" "add"])] +(deftest test-verb-subcommand-parse + (testing "add block requires content source" + (let [result (commands/parse-args ["add" "block"])] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) - (testing "block add parses with content" - (let [result (commands/parse-args ["block" "add" "--content" "hello"])] + (testing "add block parses with content" + (let [result (commands/parse-args ["add" "block" "--content" "hello"])] (is (true? (:ok? result))) - (is (= :add (:command result))) + (is (= :add-block (:command result))) (is (= "hello" (get-in result [:options :content]))))) - (testing "block remove requires target" - (let [result (commands/parse-args ["block" "remove"])] + (testing "add page requires page name" + (let [result (commands/parse-args ["add" "page"])] + (is (false? (:ok? result))) + (is (= :missing-page-name (get-in result [:error :code]))))) + + (testing "add page parses with name" + (let [result (commands/parse-args ["add" "page" "--page" "Home"])] + (is (true? (:ok? result))) + (is (= :add-page (:command result))) + (is (= "Home" (get-in result [:options :page]))))) + + (testing "remove block requires target" + (let [result (commands/parse-args ["remove" "block"])] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "block remove parses with block" - (let [result (commands/parse-args ["block" "remove" "--block" "demo"])] + (testing "remove block parses with block" + (let [result (commands/parse-args ["remove" "block" "--block" "demo"])] (is (true? (:ok? result))) - (is (= :remove (:command result))) + (is (= :remove-block (:command result))) (is (= "demo" (get-in result [:options :block]))))) - (testing "block search requires text" - (let [result (commands/parse-args ["block" "search"])] + (testing "remove page parses with page" + (let [result (commands/parse-args ["remove" "page" "--page" "Home"])] + (is (true? (:ok? result))) + (is (= :remove-page (:command result))) + (is (= "Home" (get-in result [:options :page]))))) + + (testing "search requires text" + (let [result (commands/parse-args ["search"])] (is (false? (:ok? result))) (is (= :missing-search-text (get-in result [:error :code]))))) - (testing "block search parses with text" - (let [result (commands/parse-args ["block" "search" "--text" "hello"])] + (testing "search parses with text" + (let [result (commands/parse-args ["search" "--text" "hello"])] (is (true? (:ok? result))) (is (= :search (:command result))) (is (= "hello" (get-in result [:options :text]))))) - (testing "block tree requires target" - (let [result (commands/parse-args ["block" "tree"])] + (testing "show requires target" + (let [result (commands/parse-args ["show"])] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "block tree parses with page" - (let [result (commands/parse-args ["block" "tree" "--page" "Home"])] + (testing "show parses with page name" + (let [result (commands/parse-args ["show" "--page-name" "Home"])] (is (true? (:ok? result))) - (is (= :tree (:command result))) - (is (= "Home" (get-in result [:options :page]))))) + (is (= :show (:command result))) + (is (= "Home" (get-in result [:options :page-name]))))) - (testing "block subcommands reject unknown flags" - (doseq [subcommand ["add" "remove" "search" "tree"]] - (let [result (commands/parse-args ["block" subcommand "--wat"])] + (testing "verb subcommands reject unknown flags" + (doseq [args [["list" "page" "--wat"] + ["add" "block" "--wat"] + ["remove" "block" "--wat"] + ["search" "--wat"] + ["show" "--wat"]]] + (let [result (commands/parse-args args)] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) - (testing "block subcommands accept output option" - (let [result (commands/parse-args ["block" "search" "--text" "hello" "--output" "json"])] + (testing "verb subcommands accept output option" + (let [result (commands/parse-args ["search" "--text" "hello" "--output" "json"])] (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) @@ -307,20 +313,32 @@ (is (true? (:ok? result))) (is (= :graph-info (get-in result [:action :type]))))) - (testing "add requires content" - (let [parsed {:ok? true :command :add :options {}} + (testing "list page requires repo" + (let [parsed {:ok? true :command :list-page :options {}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + + (testing "add block requires content" + (let [parsed {:ok? true :command :add-block :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) - (testing "add builds insert-blocks op" - (let [parsed {:ok? true :command :add :options {:content "hello"}} + (testing "add block builds insert-blocks op" + (let [parsed {:ok? true :command :add-block :options {:content "hello"}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= :add (get-in result [:action :type]))))) + (is (= :add-block (get-in result [:action :type]))))) - (testing "remove requires target" - (let [parsed {:ok? true :command :remove :options {}} + (testing "add page requires name" + (let [parsed {:ok? true :command :add-page :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-page-name (get-in result [:error :code]))))) + + (testing "remove block requires target" + (let [parsed {:ok? true :command :remove-block :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) @@ -331,8 +349,8 @@ (is (false? (:ok? result))) (is (= :missing-search-text (get-in result [:error :code]))))) - (testing "tree requires target" - (let [parsed {:ok? true :command :tree :options {}} + (testing "show requires target" + (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code])))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 0ef7058f6a..683c667e3b 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -67,28 +67,43 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-search-tree-remove +(deftest test-cli-list-add-search-show-remove (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) - add-result (run-cli ["block" "add" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) - _ (parse-json-output add-result) - search-result (run-cli ["block" "search" "--text" "hello world"] data-dir cfg-path) + add-page-result (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + add-page-payload (parse-json-output add-page-result) + list-page-result (run-cli ["list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + list-property-result (run-cli ["list" "property"] data-dir cfg-path) + list-property-payload (parse-json-output list-property-result) + add-block-result (run-cli ["add" "block" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) + _ (parse-json-output add-block-result) + search-result (run-cli ["search" "--text" "hello world" "--include-content"] data-dir cfg-path) search-payload (parse-json-output search-result) - tree-result (run-cli ["block" "tree" "--page" "TestPage" "--format" "json"] data-dir cfg-path) - tree-payload (parse-json-output tree-result) - block-uuid (get-in tree-payload [:data :root :children 0 :uuid]) - remove-result (run-cli ["block" "remove" "--block" (str block-uuid)] data-dir cfg-path) - remove-payload (parse-json-output remove-result) + show-result (run-cli ["show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) + show-payload (parse-json-output show-result) + remove-page-result (run-cli ["remove" "page" "--page" "TestPage"] data-dir cfg-path) + remove-page-payload (parse-json-output remove-page-result) stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code add-result))) + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (= "ok" (:status list-page-payload))) + (is (vector? (get-in list-page-payload [:data :items]))) + (is (= "ok" (:status list-tag-payload))) + (is (vector? (get-in list-tag-payload [:data :items]))) + (is (= "ok" (:status list-property-payload))) + (is (vector? (get-in list-property-payload [:data :items]))) (is (= "ok" (:status search-payload))) - (is (seq (get-in search-payload [:data :results]))) - (is (= "ok" (:status tree-payload))) - (is (= "ok" (:status remove-payload))) + (is (vector? (get-in search-payload [:data :results]))) + (is (= "ok" (:status show-payload))) + (is (contains? (get-in show-payload [:data :root]) :uuid)) + (is (= "ok" (:status remove-page-payload))) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From d09f97446ed19a24cb4e7c35550106440d497cdc Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 16:03:52 +0800 Subject: [PATCH 017/375] add 005-logseq-cli-output-and-db-worker-node-log.md --- ...ogseq-cli-output-and-db-worker-node-log.md | 137 ++++++++++++++++++ 1 file changed, 137 insertions(+) create mode 100644 docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md new file mode 100644 index 0000000000..06ad319da9 --- /dev/null +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -0,0 +1,137 @@ +# Logseq CLI Output and db-worker-node Log Implementation Plan + +Goal: Improve all logseq-cli human output for clarity and usability, and write db-worker-node logs to //db-worker-node-YYYYMMDD.log with retention of the most recent 7 logs. + +Architecture: Add a dedicated human output formatter with per-command renderers and consistent error messaging in the CLI formatting layer. +Add a file-based glogi appender for db-worker-node that writes logs into the graph-specific data directory using dated filenames and retention while keeping console logging behavior explicit and configurable. + +Tech Stack: ClojureScript, babashka/cli, lambdaisland.glogi, Node.js fs/path. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Problem statement + +The current logseq-cli human output is mostly raw pr-str output, which is hard to read and inconsistent across commands. +Users need clearer, command-specific summaries, stable table formats, and more helpful error messages for CLI usage. +The db-worker-node process currently logs only to console, but operational debugging requires a per-graph log file stored under the data directory with simple retention. + +## Testing Plan + +I will add unit tests for human output formatting functions to ensure stable, readable rendering for each command result shape. +I will add unit tests for error formatting to ensure consistent human output for common failure cases. +I will add an integration test that starts db-worker-node and verifies that a log file is created at //db-worker-node-YYYYMMDD.log. +I will add an integration test that exercises a log-producing db-worker-node action and asserts the log file contains the expected log entries. +I will follow @test-driven-development for all behavior changes. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +The CLI outputs structured results and the formatter converts them into human-friendly text based on the command and payload. +The db-worker-node daemon configures glogi to append to a per-graph log file under the repo-specific data directory. + +ASCII diagram: + ++-----------------+ format-result +------------------------+ +| logseq-cli | --------------------> | human output formatter | +| result payloads | <-------------------- | command renderers | ++-----------------+ +------------------------+ + ++------------------+ glogi appender +-------------------------------------+ +| db-worker-node | -----------------> | //db-worker-node-YYYYMMDD.log | ++------------------+ +-------------------------------------+ + +## Implementation plan + +1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps. +2. Read @test-driven-development guidelines and confirm the red phase will include all CLI output and log file tests first. +3. Review existing CLI output shapes in src/main/logseq/cli/commands.cljs to catalog the current :data payloads by command. +4. Review current formatting in src/main/logseq/cli/format.cljs and identify all human output paths that need command-specific rendering. +5. Define a human output specification table in docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md that maps each command to its target human output layout. +6. Add unit test scaffolding for CLI formatting in src/test/logseq/cli/format_test.cljs (or a new namespace) using representative :status/:data payloads. +7. Write a failing unit test for list commands to ensure human output renders a table with a header and row count. +8. Write a failing unit test for add/remove commands to ensure human output renders a succinct success line with key identifiers. +9. Write a failing unit test for graph management commands to ensure human output includes graph name and status text. +10. Write a failing unit test for server commands to ensure human output includes repo, status, host, and port when available. +11. Write a failing unit test for search and show commands to ensure human output includes result counts and stable ordering. +12. Write a failing unit test for error formatting to ensure error codes and helpful hints are included in human output. +13. Add a failing integration test in src/test/frontend/worker/db_worker_node_test.cljs (or a new namespace) that starts db-worker-node and asserts the log file exists at //db-worker-node-YYYYMMDD.log. +14. Add a failing integration test that performs a db-worker-node action and asserts at least one log line is appended to the log file. +15. Implement a command-aware human formatter in src/main/logseq/cli/format.cljs, using a dispatch on command or data shape. +16. Update src/main/logseq/cli/main.cljs to pass command context into the formatter so it can choose the correct renderer. +17. Normalize CLI result payloads in src/main/logseq/cli/commands.cljs to include explicit command identifiers where needed for formatting. +18. Ensure human output uses consistent spacing, headers, and ordering for list output, and avoids raw EDN dumps in normal cases. +19. Add a utility for table rendering with fixed column widths and truncation behavior in src/main/logseq/cli/format.cljs or a new helper namespace. +20. Implement db-worker-node log file setup in src/main/frontend/worker/db_worker_node.cljs using lambdaisland.glogi appenders. +21. Compute the log path using frontend.worker.db-worker-node-lock/repo-dir and ensure the directory exists before writing. +22. Configure glogi to append to //db-worker-node-YYYYMMDD.log and define whether console logging remains enabled. +23. Update help text in src/main/frontend/worker/db_worker_node.cljs to document the log file location and log-level flag behavior. +24. Update docs/cli/logseq-cli.md with the new human output expectations and any new formatting options. +25. Run unit tests in the red phase to confirm failures, then implement minimal changes to make them pass. +26. Run bb dev:test -v logseq.cli.* and bb dev:test -v frontend.worker.db-worker-node-test in the green phase. +27. Run bb dev:lint-and-test after all changes to validate lint and unit tests. + +## Edge cases + +The repo name contains characters that change the pool directory name, so the log file path must use worker-util/get-pool-name consistently. +The data directory is on a filesystem without write permissions, which should surface a clear error message and non-zero exit code. +Multiple db-worker-node instances for different repos should not overwrite each other’s log files. +The log file should be created even if no requests are served yet and only startup logs are emitted. +Human output should remain stable when list results are empty or fields are missing. +The human formatter should avoid printing large nested maps by default for search or show results. + +## Testing commands and expected output + +Run a focused unit test during the red phase. + +```bash +bb dev:test -v logseq.cli.format-test/test-human-output-list +``` + +Expected output includes a failing assertion and exits with a non-zero status code. + +Run the db-worker-node log integration test in the green phase. + +```bash +bb dev:test -v frontend.worker.db-worker-node-test/test-log-file-created +``` + +Expected output includes 0 failures and 0 errors. + +Run the full lint and unit test suite when all changes are complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +I will validate human output formatting by asserting on complete rendered strings for representative payloads instead of inspecting internal formatting helpers. +I will validate db-worker-node logging by checking file existence, dated filename format, and that only the most recent 7 log files remain after multiple startups. +I will assert that a known log event is present after a startup or invoke action. + +## Implementation Details + +- Add a command-aware human output renderer that produces tables, summaries, and success lines based on command and result payloads. +- Standardize human error output to include error codes, messages, and actionable hints when possible. +- Ensure human output defaults to stable ordering and includes a count line for list and search commands. +- Add a table rendering helper with column width limits and truncation rules. +- Pass command context through CLI result objects so the formatter can select the correct renderer. +- Configure db-worker-node glogi to append logs to //db-worker-node-YYYYMMDD.log. +- Enforce log retention by keeping only the most recent 7 dated log files per graph directory. +- Ensure the log directory exists before log initialization and keep the log file path deterministic. +- Document log file location and new human output behavior in CLI documentation. +- Keep JSON and EDN outputs unchanged for scripting compatibility. +- Preserve existing exit codes and error handling semantics in the CLI. + +## Question + +Should the human output include color or ANSI styling, or should it remain plain text for maximal portability. +Answer: Remain plain text for maximal portability. +Should db-worker-node log to both console and file, or file-only to avoid duplicate logs in CLI output. +Answer: File-only to avoid duplicate logs in CLI output. +Is log rotation or size management required for db-worker-node.log, or is simple append-only acceptable. +Answer: Use dated log filenames and keep only the most recent 7 log files. + +--- From e7dd64348cef7a0ec22693fd2854d0afe542d516 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 18:09:23 +0800 Subject: [PATCH 018/375] impl 005-logseq-cli-output-and-db-worker-node-log.md (1) --- deps/cli/src/logseq/cli/common/mcp/tools.cljs | 148 ++++++----- ...ogseq-cli-output-and-db-worker-node-log.md | 23 ++ docs/cli/logseq-cli.md | 1 + src/main/frontend/worker/db_worker_node.cljs | 85 +++++- src/main/logseq/cli/commands.cljs | 70 ++--- src/main/logseq/cli/format.cljs | 246 +++++++++++++++++- src/main/logseq/cli/main.cljs | 8 +- src/main/logseq/cli/server.cljs | 27 +- .../frontend/worker/db_worker_node_test.cljs | 61 +++++ src/test/logseq/cli/format_test.cljs | 131 +++++++++- src/test/logseq/cli/integration_test.cljs | 45 ++++ src/test/logseq/cli/server_test.cljs | 8 +- 12 files changed, 707 insertions(+), 146 deletions(-) diff --git a/deps/cli/src/logseq/cli/common/mcp/tools.cljs b/deps/cli/src/logseq/cli/common/mcp/tools.cljs index f5bd6949cc..c9c84b9200 100644 --- a/deps/cli/src/logseq/cli/common/mcp/tools.cljs +++ b/deps/cli/src/logseq/cli/common/mcp/tools.cljs @@ -16,60 +16,71 @@ [malli.core :as m] [malli.error :as me])) +(defn- ensure-db-graph + [db] + (when-not (ldb/db-based-graph? db) + (throw (ex-info "This tool must be called on a DB graph" {})))) + +(defn- minimal-list-item + [e] + (cond-> {:db/id (:db/id e) + :block/title (:block/title e) + :block/created-at (:block/created-at e) + :block/updated-at (:block/updated-at e)} + (:db/ident e) (assoc :db/ident (:db/ident e)))) + (defn list-properties "Main fn for ListProperties tool" [db {:keys [expand include-built-in] :as options}] (ensure-db-graph db) (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] - (->> (d/datoms db :avet :block/tags :logseq.class/Property) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-built-in?) - (ldb/built-in? e)))) - #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) - (map (fn [e] - (if expand - (cond-> (into {} e) - true - (dissoc e :block/tags :block/order :block/refs :block/name :db/index - :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) - true - (update :block/uuid str) - (:logseq.property/classes e) - (update :logseq.property/classes #(mapv :db/ident %)) - (:logseq.property/description e) - (update :logseq.property/description db-property/property-value-content)) - {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))})))))) + (->> (d/datoms db :avet :block/tags :logseq.class/Property) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) + #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name :db/index + :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) + true + (update :block/uuid str) + (:logseq.property/classes e) + (update :logseq.property/classes #(mapv :db/ident %)) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + (minimal-list-item e))))))) (defn list-tags "Main fn for ListTags tool" [db {:keys [expand include-built-in] :as options}] (ensure-db-graph db) (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] - (->> (d/datoms db :avet :block/tags :logseq.class/Tag) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-built-in?) - (ldb/built-in? e)))) - (map (fn [e] - (if expand - (cond-> (into {} e) - true - (dissoc e :block/tags :block/order :block/refs :block/name - :logseq.property.embedding/hnsw-label-updated-at) - true - (update :block/uuid str) - (:logseq.property.class/extends e) - (update :logseq.property.class/extends #(mapv :db/ident %)) - (:logseq.property.class/properties e) - (update :logseq.property.class/properties #(mapv :db/ident %)) - (:logseq.property.view/type e) - (assoc :logseq.property.view/type (:db/ident (:logseq.property.view/type e))) - (:logseq.property/description e) - (update :logseq.property/description db-property/property-value-content)) - {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))})))))) + (->> (d/datoms db :avet :block/tags :logseq.class/Tag) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name + :logseq.property.embedding/hnsw-label-updated-at) + true + (update :block/uuid str) + (:logseq.property.class/extends e) + (update :logseq.property.class/extends #(mapv :db/ident %)) + (:logseq.property.class/properties e) + (update :logseq.property.class/properties #(mapv :db/ident %)) + (:logseq.property.view/type e) + (assoc :logseq.property.view/type (:db/ident (:logseq.property.view/type e))) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + (minimal-list-item e))))))) (defn- get-page-blocks [db page-id] @@ -118,32 +129,31 @@ journal-only? (boolean journal-only) created-after-ms (parse-time created-after) updated-after-ms (parse-time updated-after)] - (->> (d/datoms db :avet :block/name) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-hidden?) - (entity-util/hidden? e)))) - (remove (fn [e] - (let [is-journal? (ldb/journal? e)] - (cond - journal-only? (not is-journal?) - (false? include-journal?) is-journal? - :else false)))) - (remove (fn [e] - (and created-after-ms - (<= (:block/created-at e 0) created-after-ms)))) - (remove (fn [e] - (and updated-after-ms - (<= (:block/updated-at e 0) updated-after-ms)))) - (map (fn [e] - (if expand - (-> e - ;; Until there are options to limit pages, return minimal info to avoid - ;; exceeding max payload size - (select-keys [:block/uuid :block/title :block/created-at :block/updated-at]) - (update :block/uuid str)) - {:block/title (:block/title e) - :block/uuid (str (:block/uuid e))})))))) + (->> (d/datoms db :avet :block/name) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-hidden?) + (entity-util/hidden? e)))) + (remove (fn [e] + (let [is-journal? (ldb/journal? e)] + (cond + journal-only? (not is-journal?) + (false? include-journal?) is-journal? + :else false)))) + (remove (fn [e] + (and created-after-ms + (<= (:block/created-at e 0) created-after-ms)))) + (remove (fn [e] + (and updated-after-ms + (<= (:block/updated-at e 0) updated-after-ms)))) + (map (fn [e] + (if expand + (-> e + ;; Until there are options to limit pages, return minimal info to avoid + ;; exceeding max payload size + (select-keys [:db/id :db/ident :block/uuid :block/title :block/created-at :block/updated-at]) + (update :block/uuid str)) + (minimal-list-item e))))))) ;; upsert-nodes tool ;; ================= diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md index 06ad319da9..f2527db35e 100644 --- a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -9,6 +9,29 @@ Tech Stack: ClojureScript, babashka/cli, lambdaisland.glogi, Node.js fs/path. Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md. +## Human Output Specification + +Target: plain text, no ANSI colors. Each command has a stable layout and ordering. + +| Command | OK output (human) | Empty output | Notes | +| --- | --- | --- | --- | +| graph list | Table with header `GRAPH` and rows of graph names, followed by `Count: N` | Header + `Count: 0` | Data from `{:graphs [...]}` | +| graph create | `Graph created: ` | n/a | Use graph name from action/options | +| graph switch | `Graph switched: ` | n/a | Use graph name from action/options | +| graph remove | `Graph removed: ` | n/a | Use graph name from action/options | +| graph validate | `Graph validated: ` | n/a | Use graph name from action/options | +| graph info | Lines: `Graph: `, `Created at: `, `Schema version: ` | n/a | Use `:logseq.kv/*` data; show `-` if missing | +| server list | Table with header `REPO STATUS HOST PORT PID`, rows for servers, followed by `Count: N` | Header + `Count: 0` | Data from `{:servers [...]}` | +| server status/start/stop/restart | `Server : ` + details line `Host: Port: ` when available | n/a | Use `:status` keyword where present | +| list page/tag/property | Table with header (fields vary by command) and rows, followed by `Count: N` | Header + `Count: 0` | Defaults: page/tag/property `ID TITLE UPDATED-AT CREATED-AT` (ID uses `:db/id`); if `:db/ident` present, include `IDENT` column | +| add block | `Added blocks: (repo: )` | n/a | Count = number of blocks submitted | +| add page | `Added page: (repo: )` | n/a | | +| remove block | `Removed block: (repo: )` | n/a | Prefer UUID if available | +| remove page | `Removed page: (repo: )` | n/a | | +| search | Table with header `TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT`, rows in stable order, followed by `Count: N` | Header + `Count: 0` | For block rows use content snippet; for tag/property rows omit timestamps | +| show (text) | Raw tree text (no table), trimmed | n/a | For `--format json|edn`, keep existing structured output | +| errors | `Error (): ` + optional `Hint: ` line | n/a | Ensure error codes are stable and consistent | + ## Problem statement The current logseq-cli human output is mostly raw pr-str output, which is hard to read and inconsistent across commands. diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 2d09e810f9..4f09c628b4 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -83,6 +83,7 @@ Subcommands: Output formats: - Global `--output ` (also accepted per subcommand) +- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. Examples: diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index afbbc792ea..7ef7b14b10 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -1,6 +1,8 @@ (ns frontend.worker.db-worker-node "Node.js daemon entrypoint for db-worker." - (:require ["http" :as http] + (:require ["fs" :as fs] + ["http" :as http] + ["path" :as node-path] [clojure.string :as string] [frontend.worker.db-core :as db-core] [frontend.worker.db-worker-node-lock :as db-lock] @@ -8,13 +10,13 @@ [frontend.worker.state :as worker-state] [goog.object :as gobj] [lambdaisland.glogi :as log] - [lambdaisland.glogi.console :as glogi-console] [logseq.db :as ldb] [promesa.core :as p])) (defonce ^:private *ready? (atom false)) (defonce ^:private *sse-clients (atom #{})) (defonce ^:private *lock-info (atom nil)) +(defonce ^:private *file-handler (atom nil)) (defn- send-json! [^js res status payload] @@ -222,15 +224,82 @@ (println " --repo (required)") (println " --rtc-ws-url (optional)") (println " --log-level (default info)") + (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)") (println " --auth-token (optional)")) +(defn- pad2 + [value] + (if (< value 10) + (str "0" value) + (str value))) + +(defn- yyyymmdd + [^js date] + (str (.getFullYear date) + (pad2 (inc (.getMonth date))) + (pad2 (.getDate date)))) + +(defn- log-path + [data-dir repo] + (let [data-dir (db-lock/resolve-data-dir data-dir) + repo-dir (db-lock/repo-dir data-dir repo) + date-str (yyyymmdd (js/Date.))] + (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) + +(defn- log-files + [repo-dir] + (->> (when (fs/existsSync repo-dir) + (fs/readdirSync repo-dir)) + (filter (fn [^js name] + (re-matches #"db-worker-node-\d{8}\.log" name))) + (sort))) + +(defn- enforce-log-retention! + [repo-dir] + (let [files (log-files repo-dir) + excess (max 0 (- (count files) 7))] + (doseq [name (take excess files)] + (fs/unlinkSync (node-path/join repo-dir name))))) + +(defn- format-log-line + [{:keys [time level message logger-name exception]}] + (let [ts (.toISOString (js/Date. time)) + base (str ts + " [" + (name level) + "] [" + logger-name + "] " + (pr-str message))] + (str base (when exception (str " " (pr-str exception))) "\n"))) + +(defn- install-file-logger! + [{:keys [data-dir repo log-level]}] + (let [data-dir (db-lock/resolve-data-dir data-dir) + repo-dir (db-lock/repo-dir data-dir repo) + file-path (log-path data-dir repo)] + (fs/mkdirSync repo-dir #js {:recursive true}) + (fs/writeFileSync file-path "" #js {:flag "a"}) + (enforce-log-retention! repo-dir) + (when-let [handler @*file-handler] + (log/remove-handler handler)) + (let [handler (fn [record] + (fs/appendFileSync file-path (format-log-line record)))] + (reset! *file-handler handler) + (log/add-handler handler)) + (log/set-levels {:glogi/root log-level}) + file-path)) + (defn start-daemon! - [{:keys [data-dir repo rtc-ws-url auth-token]}] + [{:keys [data-dir repo rtc-ws-url auth-token log-level]}] (let [host "127.0.0.1" port 0] (if-not (seq repo) (p/rejected (ex-info "repo is required" {:code :missing-repo})) (do + (install-file-logger! {:data-dir data-dir + :repo repo + :log-level (keyword (or log-level "info"))}) (reset! *ready? false) (set-main-thread-stub!) (-> (p/let [platform (platform-node/node-platform {:data-dir data-dir @@ -288,22 +357,20 @@ (defn main [] - (let [{:keys [data-dir repo rtc-ws-url log-level auth-token help?]} - (parse-args (.-argv js/process)) - log-level (keyword (or log-level "info"))] + (let [{:keys [data-dir repo rtc-ws-url auth-token help?] :as opts} + (parse-args (.-argv js/process))] (when help? (show-help!) (.exit js/process 0)) (when-not (seq repo) (show-help!) (.exit js/process 1)) - (glogi-console/install!) - (log/set-levels {:glogi/root log-level}) (p/let [{:keys [stop!] :as daemon} (start-daemon! {:data-dir data-dir :repo repo :rtc-ws-url rtc-ws-url - :auth-token auth-token})] + :auth-token auth-token + :log-level (:log-level opts)})] (log/info :db-worker-node-ready {:host (:host daemon) :port (:port daemon)}) (let [shutdown (fn [] (-> (stop!) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 4026b2c088..9232c87b1c 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -778,17 +778,20 @@ (case command :graph-list {:ok? true - :action {:type :graph-list}} + :action {:type :graph-list + :command :graph-list}} :graph-create (if-not (seq graph) (missing-graph-error) {:ok? true :action {:type :invoke + :command :graph-create :method "thread-api/create-or-open-db" :direct-pass? false :args [repo {}] :repo repo + :graph (repo->graph repo) :allow-missing-graph true :persist-repo (repo->graph repo)}}) @@ -797,6 +800,7 @@ (missing-graph-error) {:ok? true :action {:type :graph-switch + :command :graph-switch :repo repo :graph (repo->graph repo)}}) @@ -805,26 +809,31 @@ (missing-graph-error) {:ok? true :action {:type :invoke + :command :graph-remove :method "thread-api/unsafe-unlink-db" :direct-pass? false :args [repo] - :repo repo}}) + :repo repo + :graph (repo->graph repo)}}) :graph-validate (if-not (seq repo) (missing-graph-error) {:ok? true :action {:type :invoke + :command :graph-validate :method "thread-api/validate-db" :direct-pass? false :args [repo] - :repo repo}}) + :repo repo + :graph (repo->graph repo)}}) :graph-info (if-not (seq repo) (missing-graph-error) {:ok? true :action {:type :graph-info + :command :graph-info :repo repo :graph (repo->graph repo)}}))) @@ -1369,29 +1378,32 @@ (defn execute [action config] - (-> (p/let [check (ensure-existing-graph action config)] - (if-not (:ok? check) - {:status :error - :error (:error check)} - (case (:type action) - :graph-list (execute-graph-list action config) - :invoke (execute-invoke action config) - :graph-switch (execute-graph-switch action config) - :graph-info (execute-graph-info action config) - :list-page (execute-list-page action config) - :list-tag (execute-list-tag action config) - :list-property (execute-list-property action config) - :add-block (execute-add-block action config) - :add-page (execute-add-page action config) - :remove-block (execute-remove action config) - :remove-page (execute-remove action config) - :search (execute-search action config) - :show (execute-show action config) - :server-list (execute-server-list action config) - :server-status (execute-server-status action config) - :server-start (execute-server-start action config) - :server-stop (execute-server-stop action config) - :server-restart (execute-server-restart action config) - {:status :error - :error {:code :unknown-action - :message "unknown action"}}))))) + (-> (p/let [check (ensure-existing-graph action config) + result (if-not (:ok? check) + {:status :error + :error (:error check)} + (case (:type action) + :graph-list (execute-graph-list action config) + :invoke (execute-invoke action config) + :graph-switch (execute-graph-switch action config) + :graph-info (execute-graph-info action config) + :list-page (execute-list-page action config) + :list-tag (execute-list-tag action config) + :list-property (execute-list-property action config) + :add-block (execute-add-block action config) + :add-page (execute-add-page action config) + :remove-block (execute-remove action config) + :remove-page (execute-remove action config) + :search (execute-search action config) + :show (execute-show action config) + :server-list (execute-server-list action config) + :server-status (execute-server-status action config) + :server-start (execute-server-start action config) + :server-stop (execute-server-stop action config) + :server-restart (execute-server-restart action config) + {:status :error + :error {:code :unknown-action + :message "unknown action"}}))] + (assoc result + :command (or (:command action) (:type action)) + :context (select-keys action [:repo :graph :page :block :blocks]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 5cbb5fe625..d6fac1bc2a 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.format "Formatting helpers for CLI output." - (:require [clojure.walk :as walk])) + (:require [clojure.string :as string] + [clojure.walk :as walk])) (defn- normalize-json [value] @@ -22,18 +23,239 @@ (set! (.-error obj) (clj->js (normalize-json (update error :code name))))) (js/JSON.stringify obj))) +(defn- pad-right + [value width] + (let [text (str value) + missing (- width (count text))] + (if (pos? missing) + (str text (apply str (repeat missing " "))) + text))) + +(defn- normalize-cell + [value] + (cond + (nil? value) "-" + (keyword? value) (str value) + :else (str value))) + +(defn- render-table + [headers rows] + (let [normalized-rows (mapv (fn [row] + (mapv normalize-cell row)) + rows) + trim-right (fn [value] + (string/replace value #"\s+$" "")) + widths (mapv (fn [idx header] + (apply max (count header) + (map #(count (nth % idx)) normalized-rows))) + (range (count headers)) + headers) + render-row (fn [row] + (->> (map pad-right row widths) + (string/join " ") + (trim-right))) + lines (cons (render-row headers) + (map render-row normalized-rows))] + (string/join "\n" lines))) + +(defn- format-counted-table + [headers rows] + (str (render-table headers rows) + "\n" + "Count: " + (count rows))) + +(defn- error-hint + [{:keys [code]}] + (case code + :missing-graph "Use --graph " + :missing-repo "Use --repo " + :missing-content "Use --content or pass content as args" + :missing-search-text "Provide search text or --text" + nil)) + +(defn- format-error + [error] + (let [{:keys [code message]} error + hint (error-hint error)] + (cond-> (str "Error (" (name (or code :error)) "): " message) + hint (str "\nHint: " hint)))) + +(defn- maybe-ident-header + [items] + (when (some :db/ident items) + ["IDENT"])) + +(defn- parse-ts + [value] + (cond + (number? value) value + (string? value) (let [ms (js/Date.parse value)] + (when-not (js/isNaN ms) ms)) + :else nil)) + +(defn- human-ago + [value now-ms] + (if-let [ts (parse-ts value)] + (let [diff-ms (max 0 (- now-ms ts)) + secs (js/Math.floor (/ diff-ms 1000)) + mins (js/Math.floor (/ secs 60)) + hours (js/Math.floor (/ mins 60)) + days (js/Math.floor (/ hours 24)) + months (js/Math.floor (/ days 30)) + years (js/Math.floor (/ days 365))] + (cond + (< secs 60) (str secs "s ago") + (< mins 60) (str mins "m ago") + (< hours 24) (str hours "h ago") + (< days 30) (str days "d ago") + (< months 12) (str months "mo ago") + :else (str years "y ago"))) + "-")) + +(defn- format-list-row + [item include-ident? now-ms] + (let [base [(or (:db/id item) (:id item)) + (or (:title item) (:block/title item) (:name item))] + with-ident (cond-> base + include-ident? (conj (:db/ident item))) + updated (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms) + created (human-ago (or (:created-at item) (:block/created-at item)) now-ms)] + (conj with-ident updated created))) + +(defn- format-list-page + [items now-ms] + (let [items (or items []) + include-ident? (boolean (some :db/ident items)) + headers (into ["ID" "TITLE"] + (concat (or (maybe-ident-header items) []) + ["UPDATED-AT" "CREATED-AT"]))] + (format-counted-table + headers + (mapv #(format-list-row % include-ident? now-ms) items)))) + +(defn- format-list-tag-or-property + [items now-ms] + (let [items (or items []) + include-ident? (boolean (some :db/ident items)) + headers (into ["ID" "TITLE"] + (concat (or (maybe-ident-header items) []) + ["UPDATED-AT" "CREATED-AT"]))] + (format-counted-table + headers + (mapv #(format-list-row % include-ident? now-ms) items)))) + +(defn- format-graph-list + [graphs] + (format-counted-table + ["GRAPH"] + (mapv (fn [graph] [graph]) (or graphs [])))) + +(defn- format-server-list + [servers] + (format-counted-table + ["REPO" "STATUS" "HOST" "PORT" "PID"] + (mapv (fn [server] + [(:repo server) + (:status server) + (:host server) + (:port server) + (:pid server)]) + (or servers [])))) + +(defn- format-search-results + [results] + (format-counted-table + ["TYPE" "TITLE/CONTENT" "UUID" "UPDATED-AT" "CREATED-AT"] + (mapv (fn [item] + [(:type item) + (or (:title item) (:content item)) + (:uuid item) + (:updated-at item) + (:created-at item)]) + (or results [])))) + +(defn- format-graph-info + [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]}] + (string/join "\n" + [(str "Graph: " (or graph "-")) + (str "Created at: " (or graph-created-at "-")) + (str "Schema version: " (or schema-version "-"))])) + +(defn- format-server-status + [{:keys [repo status host port]}] + (string/join "\n" + (cond-> [(str "Server " (name (or status :unknown)) ": " repo)] + (and host port) (conj (str "Host: " host " Port: " port))))) + +(defn- format-server-action + [command {:keys [repo status host port]}] + (let [status (or status + (case command + :server-start :started + :server-stop :stopped + :server-restart :restarted + :unknown))] + (string/join "\n" + (cond-> [(str "Server " (name status) ": " repo)] + (and host port) (conj (str "Host: " host " Port: " port)))))) + +(defn- format-add-block + [{:keys [repo blocks]}] + (str "Added blocks: " (count blocks) " (repo: " repo ")")) + +(defn- format-add-page + [{:keys [repo page]}] + (str "Added page: " page " (repo: " repo ")")) + +(defn- format-remove-page + [{:keys [repo page]}] + (str "Removed page: " page " (repo: " repo ")")) + +(defn- format-remove-block + [{:keys [repo block]}] + (str "Removed block: " block " (repo: " repo ")")) + +(defn- format-graph-action + [command {:keys [graph]}] + (let [verb (case command + :graph-create "created" + :graph-switch "switched" + :graph-remove "removed" + :graph-validate "validated" + "updated")] + (str "Graph " verb ": " graph))) + (defn- ->human - [{:keys [status data error]}] - (case status - :ok - (if (and (map? data) (contains? data :message)) - (:message data) - (pr-str data)) + [{:keys [status data error command context]} {:keys [now-ms]}] + (let [now-ms (or now-ms (js/Date.now))] + (case status + :ok + (case command + :graph-list (format-graph-list (:graphs data)) + :graph-info (format-graph-info data) + (:graph-create :graph-switch :graph-remove :graph-validate) + (format-graph-action command context) + :server-list (format-server-list (:servers data)) + :server-status (format-server-status data) + (:server-start :server-stop :server-restart) + (format-server-action command data) + :list-page (format-list-page (:items data) now-ms) + (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) + :add-block (format-add-block context) + :add-page (format-add-page context) + :remove-page (format-remove-page context) + :remove-block (format-remove-block context) + :search (format-search-results (:results data)) + :show (or (:message data) (pr-str data)) + (if (and (map? data) (contains? data :message)) + (:message data) + (pr-str data))) - :error - (str "error: " (:message error)) + :error + (format-error error) - (pr-str {:status status :data data :error error}))) + (pr-str {:status status :data data :error error})))) (defn- ->edn [{:keys [status data error]}] @@ -42,7 +264,7 @@ (= status :error) (assoc :error error)))) (defn format-result - [result {:keys [output-format]}] + [result {:keys [output-format now-ms] :as opts}] (let [format (cond (= output-format :edn) :edn (= output-format :json) :json @@ -50,4 +272,4 @@ (case format :json (->json result) :edn (->edn result) - (->human result)))) + (->human result opts)))) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index ff5ef19464..8d72d1bca5 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -29,7 +29,8 @@ (not (:ok? parsed)) (p/resolved {:exit-code 1 :output (format/format-result {:status :error - :error (:error parsed)} + :error (:error parsed) + :command (:command parsed)} {})}) :else @@ -38,7 +39,10 @@ (if-not (:ok? action-result) (p/resolved {:exit-code 1 :output (format/format-result {:status :error - :error (:error action-result)} + :error (:error action-result) + :command (:command parsed) + :context (select-keys (:options parsed) + [:repo :graph :page :block])} cfg)}) (-> (commands/execute (:action action-result) cfg) (p/then (fn [result] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 9063677371..b28a3123f4 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -6,17 +6,10 @@ ["os" :as os] ["path" :as node-path] [clojure.string :as string] - [frontend.worker.db-worker-node :as db-worker-node] [frontend.worker-common.util :as worker-util] [lambdaisland.glogi :as log] [promesa.core :as p])) -(defonce ^:private *inproc-servers (atom {})) - -(defn- inproc-enabled? - [] - (boolean (.-DEBUG js/goog))) - (defn- expand-home [path] (if (string/starts-with? path "~") @@ -172,20 +165,13 @@ (defn- spawn-server! [{:keys [repo data-dir]}] - (let [script (node-path/join (js/process.cwd) "static" "db-worker-node.js") + (let [script (node-path/join js/__dirname "db-worker-node.js") args #js [script "--repo" repo "--data-dir" data-dir] child (.spawn child-process "node" args #js {:detached true :stdio "ignore"})] (.unref child) child)) -(defn- start-inproc-server! - [{:keys [repo data-dir]}] - (p/let [daemon (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo})] - (swap! *inproc-servers assoc repo daemon) - daemon)) - (defn- ensure-server-started! [config repo] (let [data-dir (resolve-data-dir config) @@ -193,9 +179,7 @@ (p/let [existing (read-lock path) _ (cleanup-stale-lock! path existing) _ (when (not (fs/existsSync path)) - (if (inproc-enabled?) - (start-inproc-server! {:repo repo :data-dir data-dir}) - (spawn-server! {:repo repo :data-dir data-dir})) + (spawn-server! {:repo repo :data-dir data-dir}) (wait-for-lock path)) lock (read-lock path)] (when-not lock @@ -232,7 +216,6 @@ (p/resolved (not (fs/existsSync path)))) {:timeout-ms 5000 :interval-ms 200}) - (swap! *inproc-servers dissoc repo) {:ok? true :data {:repo repo}}) (p/catch (fn [_] @@ -248,10 +231,8 @@ {:ok? false :error {:code :server-stop-timeout :message "timed out stopping server"}} - (do - (swap! *inproc-servers dissoc repo) - {:ok? true - :data {:repo repo}})))))))) + {:ok? true + :data {:repo repo}}))))))) (defn start-server! [config repo] diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 0edbbbd77f..9ebc3870fc 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -79,6 +79,67 @@ repo-dir (node-path/join data-dir (str "." pool-name))] (node-path/join repo-dir "db-worker.lock"))) +(defn- pad2 + [value] + (if (< value 10) + (str "0" value) + (str value))) + +(defn- yyyymmdd + [^js date] + (str (.getFullYear date) + (pad2 (inc (.getMonth date))) + (pad2 (.getDate date)))) + +(defn- log-path + [data-dir repo] + (let [pool-name (worker-util/get-pool-name repo) + repo-dir (node-path/join data-dir (str "." pool-name)) + date-str (yyyymmdd (js/Date.))] + (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) + +(deftest db-worker-node-creates-log-file + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-log") + repo (str "logseq_db_log_" (subs (str (random-uuid)) 0 8)) + log-file (log-path data-dir repo)] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (p/delay 50)] + (is (fs/existsSync log-file))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-log-file-has-entries + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-log-entries") + repo (str "logseq_db_log_entries_" (subs (str (random-uuid)) 0 8)) + log-file (log-path data-dir repo)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + _ (p/delay 50) + contents (when (fs/existsSync log-file) + (.toString (fs/readFileSync log-file) "utf8"))] + (is (fs/existsSync log-file)) + (is (pos? (count contents)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-parse-args-ignores-host-and-port (let [parse-args #'db-worker-node/parse-args result (parse-args #js ["node" "db-worker-node.js" diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 32e099764d..069d84f5bb 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -39,4 +39,133 @@ (testing "human error (default)" (let [result (format/format-result {:status :error :error {:code :boom :message "nope"}} {:output-format nil})] - (is (= "error: nope" result))))) + (is (= "Error (boom): nope" result))))) + +(deftest test-human-output-list-page + (testing "list page renders a table with count" + (let [result (format/format-result {:status :ok + :command :list-page + :data {:items [{:db/id 1 + :title "Alpha" + :updated-at 90000 + :created-at 40000}]}} + {:output-format nil + :now-ms 100000})] + (is (= (str "ID TITLE UPDATED-AT CREATED-AT\n" + "1 Alpha 10s ago 1m ago\n" + "Count: 1") + result))))) + +(deftest test-human-output-list-tag-property + (testing "list tag uses ID column from :db/id" + (let [result (format/format-result {:status :ok + :command :list-tag + :data {:items [{:block/title "Tag" + :db/id 42 + :block/created-at 40000 + :block/updated-at 90000 + :db/ident :logseq.class/Tag}]}} + {:output-format nil + :now-ms 100000})] + (is (= (str "ID TITLE IDENT UPDATED-AT CREATED-AT\n" + "42 Tag :logseq.class/Tag 10s ago 1m ago\n" + "Count: 1") + result)))) + + (testing "list property uses ID column from :db/id" + (let [result (format/format-result {:status :ok + :command :list-property + :data {:items [{:block/title "Prop" + :db/id 99 + :block/created-at 40000 + :block/updated-at 90000}]}} + {:output-format nil + :now-ms 100000})] + (is (= (str "ID TITLE UPDATED-AT CREATED-AT\n" + "99 Prop 10s ago 1m ago\n" + "Count: 1") + result))))) + +(deftest test-human-output-add-remove + (testing "add block renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :add-block + :context {:repo "demo-repo" + :blocks ["a" "b"]} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Added blocks: 2 (repo: demo-repo)" result)))) + + (testing "remove page renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :remove-page + :context {:repo "demo-repo" + :page "Home"} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Removed page: Home (repo: demo-repo)" result))))) + +(deftest test-human-output-graph-info + (testing "graph info includes key metadata lines" + (let [result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :logseq.kv/graph-created-at 123 + :logseq.kv/schema-version 2}} + {:output-format nil})] + (is (= (str "Graph: demo-graph\n" + "Created at: 123\n" + "Schema version: 2") + result))))) + +(deftest test-human-output-server-status + (testing "server status includes repo, status, host, port" + (let [result (format/format-result {:status :ok + :command :server-status + :data {:repo "demo-repo" + :status :ready + :host "127.0.0.1" + :port 1234}} + {:output-format nil})] + (is (= (str "Server ready: demo-repo\n" + "Host: 127.0.0.1 Port: 1234") + result))))) + +(deftest test-human-output-search-and-show + (testing "search renders a table with count" + (let [result (format/format-result {:status :ok + :command :search + :data {:results [{:type "page" + :title "Alpha" + :uuid "u1" + :updated-at 3 + :created-at 1} + {:type "block" + :content "Note" + :uuid "u2" + :updated-at 4 + :created-at 2}]}} + {:output-format nil})] + (is (= (str "TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT\n" + "page Alpha u1 3 1\n" + "block Note u2 4 2\n" + "Count: 2") + result)))) + + (testing "show renders text payloads directly" + (let [result (format/format-result {:status :ok + :command :show + :data {:message "Line 1\nLine 2"}} + {:output-format nil})] + (is (= "Line 1\nLine 2" result))))) + +(deftest test-human-output-error-formatting + (testing "errors include code and hint when available" + (let [result (format/format-result {:status :error + :command :graph-create + :error {:code :missing-graph + :message "graph name is required"}} + {:output-format nil})] + (is (= (str "Error (missing-graph): graph name is required\n" + "Hint: Use --graph ") + result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 683c667e3b..4f51bf2ae2 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -128,3 +128,48 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest test-cli-list-outputs-include-id + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "list-id-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + list-property-result (run-cli ["list" "property"] data-dir cfg-path) + list-property-payload (parse-json-output list-property-result) + stop-result (run-cli ["server" "stop" "--repo" "list-id-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status list-page-payload))) + (is (every? #(contains? % :id) (get-in list-page-payload [:data :items]))) + (is (= "ok" (:status list-tag-payload))) + (is (every? #(contains? % :id) (get-in list-tag-payload [:data :items]))) + (is (= "ok" (:status list-property-payload))) + (is (every? #(contains? % :id) (get-in list-property-payload [:data :items]))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-list-page-human-output + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "human-list-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page" "--output" "human"] data-dir cfg-path) + output (:output list-page-result)] + (is (= 0 (:exit-code list-page-result))) + (is (string/includes? output "TITLE")) + (is (string/includes? output "TestPage")) + (is (string/includes? output "Count:")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index 2d91b2ae89..a995cfea6c 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -11,7 +11,8 @@ (deftest spawn-server-omits-host-and-port-flags (let [spawn-server! #'cli-server/spawn-server! captured (atom nil) - original-spawn (.-spawn child-process)] + original-spawn (.-spawn child-process) + original-cwd (.cwd js/process)] (set! (.-spawn child-process) (fn [cmd args opts] (reset! captured {:cmd cmd @@ -19,15 +20,20 @@ :opts (js->clj opts :keywordize-keys true)}) (js-obj "unref" (fn [] nil)))) (try + (.chdir js/process "/") (spawn-server! {:repo "logseq_db_spawn_test" :data-dir "/tmp/logseq-db-worker"}) (is (= "node" (:cmd @captured))) + (is (= (node-path/join js/__dirname "db-worker-node.js") + (first (:args @captured)))) (is (some #{"--repo"} (:args @captured))) (is (some #{"--data-dir"} (:args @captured))) (is (not-any? #{"--host" "--port"} (:args @captured))) (finally + (.chdir js/process original-cwd) (set! (.-spawn child-process) original-spawn))))) + (deftest ensure-server-repairs-stale-lock (async done (let [data-dir (node-helper/create-tmp-dir "cli-server") From ab89d5e1d657ae32bacc8ae9299f37d383985dc3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 20:35:13 +0800 Subject: [PATCH 019/375] impl 005-logseq-cli-output-and-db-worker-node-log.md (2) --- ...ogseq-cli-output-and-db-worker-node-log.md | 2 +- docs/cli/logseq-cli.md | 15 ++++- src/main/logseq/cli/commands.cljs | 64 +++++++++++++------ src/main/logseq/cli/format.cljs | 10 +-- src/main/logseq/cli/main.cljs | 2 +- src/main/logseq/cli/server.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 22 +++++-- src/test/logseq/cli/format_test.cljs | 7 +- src/test/logseq/cli/integration_test.cljs | 37 +++++++++++ 9 files changed, 122 insertions(+), 39 deletions(-) diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md index f2527db35e..b05ebf25b9 100644 --- a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -20,7 +20,7 @@ Target: plain text, no ANSI colors. Each command has a stable layout and orderin | graph switch | `Graph switched: ` | n/a | Use graph name from action/options | | graph remove | `Graph removed: ` | n/a | Use graph name from action/options | | graph validate | `Graph validated: ` | n/a | Use graph name from action/options | -| graph info | Lines: `Graph: `, `Created at: `, `Schema version: ` | n/a | Use `:logseq.kv/*` data; show `-` if missing | +| graph info | Lines: `Graph: `, `Created at: `, `Schema version: ` | n/a | Use `:logseq.kv/*` data; show `-` if missing; `Created at` should use the same human-friendly relative format as list outputs | | server list | Table with header `REPO STATUS HOST PORT PID`, rows for servers, followed by `Count: N` | Header + `Count: 0` | Data from `{:servers [...]}` | | server status/start/stop/restart | `Server : ` + details line `Host: Port: ` when available | n/a | Use `:status` keyword where present | | list page/tag/property | Table with header (fields vary by command) and rows, followed by `Count: N` | Header + `Count: 0` | Defaults: page/tag/property `ID TITLE UPDATED-AT CREATED-AT` (ID uses `:db/id`); if `:db/ident` present, include `IDENT` column | diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 4f09c628b4..0e17b7bd82 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -1,6 +1,6 @@ # Logseq CLI (Node) -The Logseq CLI is a Node.js program compiled from ClojureScript that connects to a db-worker-node server managed by the CLI. +The Logseq CLI is a Node.js program compiled from ClojureScript that connects to a db-worker-node server managed by the CLI. When installed, the CLI binary name is `logseq`. ## Build the CLI @@ -10,12 +10,18 @@ clojure -M:cljs compile logseq-cli ## db-worker-node lifecycle -`logseq-cli` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. +`logseq` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. ## Run the CLI ```bash node ./static/logseq-cli.js graph list + +If installed globally, run: + +```bash +logseq graph list +``` ``` ## Configuration @@ -81,9 +87,12 @@ Subcommands: show [options] Show tree ``` +Options grouping: +- Help output separates **Global options** (apply to all commands) and **Command options** (command-specific flags). + Output formats: - Global `--output ` (also accepted per subcommand) -- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. Examples: diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 9232c87b1c..3bebb76c2e 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -131,13 +131,16 @@ [group table] (let [group-table (filter #(= group (first (:cmds %))) table)] (string/join "\n" - [(str "Usage: logseq-cli " group " [options]") + [(str "Usage: logseq " group " [options]") "" "Subcommands:" (format-commands group-table) "" - "Options:" - (cli/format-opts {:spec global-spec})]))) + "Global options:" + (cli/format-opts {:spec global-spec}) + "" + "Command options:" + (str " See `logseq " group " --help`")]))) (defn- top-level-summary [table] @@ -149,21 +152,28 @@ (let [entries (filter #(contains? commands (first (:cmds %))) table)] (string/join "\n" [title (format-commands entries)])))] (string/join "\n" - ["Usage: logseq-cli [options]" + ["Usage: logseq [options]" "" "Commands:" (string/join "\n\n" (map render-group groups)) "" - "Options:" - (cli/format-opts {:spec global-spec})]))) + "Global options:" + (cli/format-opts {:spec global-spec}) + "" + "Command options:" + " See `logseq --help`"]))) (defn- command-summary [{:keys [cmds spec]}] - (string/join "\n" - [(str "Usage: logseq-cli " (string/join " " cmds) " [options]") - "" - "Options:" - (cli/format-opts {:spec spec})])) + (let [command-spec (apply dissoc spec (keys global-spec))] + (string/join "\n" + [(str "Usage: logseq " (string/join " " cmds) " [options]") + "" + "Global options:" + (cli/format-opts {:spec global-spec}) + "" + "Command options:" + (cli/format-opts {:spec command-spec})]))) (defn- merge-spec [spec] @@ -631,29 +641,43 @@ (build root-id 1))) (defn- fetch-tree - [config {:keys [repo id uuid page-name level]}] - (let [max-depth (or level 10)] + [config {:keys [repo id page-name level] :as opts}] + (let [max-depth (or level 10) + uuid-str (:uuid opts)] (cond (some? id) (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks (:db/id entity) max-depth)] {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found})))) + (if (:db/id entity) + (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found}))))) - (seq uuid) - (if-not (common-util/uuid-string? uuid) + (seq uuid-str) + (if-not (common-util/uuid-string? uuid-str) (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title {:block/page [:db/id :block/title]}] - [:block/uuid (uuid uuid)]])] + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid (uuid uuid-str)]]) + entity (if (:db/id entity) + entity + (transport/invoke config "thread-api/pull" false + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid uuid-str]]))] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks (:db/id entity) max-depth)] {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found}))))) + (if (:db/id entity) + (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found})))))) (seq page-name) (p/let [page-entity (transport/invoke config "thread-api/pull" false diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index d6fac1bc2a..3a620f7704 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -176,10 +176,12 @@ (or results [])))) (defn- format-graph-info - [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]}] + [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]} now-ms] (string/join "\n" [(str "Graph: " (or graph "-")) - (str "Created at: " (or graph-created-at "-")) + (str "Created at: " (if (some? graph-created-at) + (human-ago graph-created-at now-ms) + "-")) (str "Schema version: " (or schema-version "-"))])) (defn- format-server-status @@ -233,7 +235,7 @@ :ok (case command :graph-list (format-graph-list (:graphs data)) - :graph-info (format-graph-info data) + :graph-info (format-graph-info data now-ms) (:graph-create :graph-switch :graph-remove :graph-validate) (format-graph-action command context) :server-list (format-server-list (:servers data)) @@ -264,7 +266,7 @@ (= status :error) (assoc :error error)))) (defn format-result - [result {:keys [output-format now-ms] :as opts}] + [result {:keys [output-format] :as opts}] (let [format (cond (= output-format :edn) :edn (= output-format :json) :json diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 8d72d1bca5..59ff135756 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -10,7 +10,7 @@ (defn- usage [summary] (string/join "\n" - ["logseq-cli [options]" + ["logseq [options]" "" "Commands: list page, list tag, list property, add block, add page, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, server list, server status, server start, server stop, server restart" "" diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index b28a3123f4..f835fcbc60 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -1,5 +1,5 @@ (ns logseq.cli.server - "db-worker-node lifecycle orchestration for logseq-cli." + "db-worker-node lifecycle orchestration for logseq." (:require ["child_process" :as child-process] ["fs" :as fs] ["http" :as http] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 225939a66f..599c8c51a7 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -18,9 +18,14 @@ (is (string/includes? summary "search")) (is (string/includes? summary "show")) (is (string/includes? summary "graph")) - (is (string/includes? summary "server"))))) + (is (string/includes? summary "server")))) -(deftest test-parse-args + (testing "top-level help separates global and command options" + (let [summary (:summary (commands/parse-args ["--help"]))] + (is (string/includes? summary "Global options:")) + (is (string/includes? summary "Command options:"))))) + +(deftest test-parse-args-help (testing "graph group shows subcommands" (let [result (commands/parse-args ["graph"]) summary (:summary result)] @@ -34,7 +39,9 @@ (is (true? (:help? result))) (is (string/includes? summary "list page")) (is (string/includes? summary "list tag")) - (is (string/includes? summary "list property")))) + (is (string/includes? summary "list property")) + (is (string/includes? summary "Global options:")) + (is (string/includes? summary "Command options:")))) (testing "add group shows subcommands" (let [result (commands/parse-args ["add"]) @@ -55,8 +62,9 @@ summary (:summary result)] (is (true? (:help? result))) (is (string/includes? summary "server list")) - (is (string/includes? summary "server start")))) + (is (string/includes? summary "server start"))))) +(deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" (let [result (commands/parse-args ["graph"]) summary (:summary result) @@ -85,8 +93,9 @@ (when-let [[_ desc] (re-matches #"^\s+.*?\s{2,}(.+)$" line)] (.indexOf line desc)))))] (is (seq subcommand-lines)) - (is (apply = desc-starts)))) + (is (apply = desc-starts))))) +(deftest test-parse-args-errors (testing "rejects legacy commands" (doseq [command ["graph-list" "graph-create" "graph-switch" "graph-remove" "graph-validate" "graph-info" "block" "tree" @@ -113,8 +122,9 @@ (testing "errors on unknown command" (let [result (commands/parse-args ["wat"])] (is (false? (:ok? result))) - (is (= :unknown-command (get-in result [:error :code]))))) + (is (= :unknown-command (get-in result [:error :code])))))) +(deftest test-parse-args-global-options (testing "global output option is accepted" (let [result (commands/parse-args ["--output" "json" "graph" "list"])] (is (true? (:ok? result))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 069d84f5bb..8b79d2e628 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -110,11 +110,12 @@ (let [result (format/format-result {:status :ok :command :graph-info :data {:graph "demo-graph" - :logseq.kv/graph-created-at 123 + :logseq.kv/graph-created-at 40000 :logseq.kv/schema-version 2}} - {:output-format nil})] + {:output-format nil + :now-ms 100000})] (is (= (str "Graph: demo-graph\n" - "Created at: 123\n" + "Created at: 1m ago\n" "Schema version: 2") result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 4f51bf2ae2..0df87c97dd 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -173,3 +173,40 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest test-cli-show-page-block-by-id-and-uuid + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "show-page-block-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page" "--expand"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-item (some (fn [item] + (when (= "TestPage" (or (:block/title item) (:title item))) + item)) + (get-in list-page-payload [:data :items])) + page-id (or (:db/id page-item) (:id page-item)) + page-uuid (or (:block/uuid page-item) (:uuid page-item)) + show-by-id-result (run-cli ["show" "--id" (str page-id) "--format" "json"] data-dir cfg-path) + show-by-id-payload (parse-json-output show-by-id-result) + show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) "--format" "json"] data-dir cfg-path) + show-by-uuid-payload (parse-json-output show-by-uuid-result) + stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status list-page-payload))) + (is (some? page-item)) + (is (some? page-id)) + (is (some? page-uuid)) + (is (= "ok" (:status show-by-id-payload))) + (is (= (str page-uuid) (str (or (get-in show-by-id-payload [:data :root :uuid]) + (get-in show-by-id-payload [:data :root :block/uuid]))))) + (is (= "ok" (:status show-by-uuid-payload))) + (is (= (str page-uuid) (str (or (get-in show-by-uuid-payload [:data :root :uuid]) + (get-in show-by-uuid-payload [:data :root :block/uuid]))))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) From 0e1a2f8ca88385fe2c52dd6fce520735d6c9573a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 18 Jan 2026 21:18:03 +0800 Subject: [PATCH 020/375] impl 005-logseq-cli-output-and-db-worker-node-log.md (3) --- ...ogseq-cli-output-and-db-worker-node-log.md | 2 +- docs/cli/logseq-cli.md | 12 ++++++ src/main/logseq/cli/commands.cljs | 40 ++++++++++++++----- .../frontend/worker/db_worker_node_test.cljs | 28 +++++++++++++ src/test/logseq/cli/commands_test.cljs | 38 ++++++++++++++++++ src/test/logseq/cli/format_test.cljs | 18 ++++++--- 6 files changed, 123 insertions(+), 15 deletions(-) diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md index b05ebf25b9..3d4774ac6b 100644 --- a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -29,7 +29,7 @@ Target: plain text, no ANSI colors. Each command has a stable layout and orderin | remove block | `Removed block: (repo: )` | n/a | Prefer UUID if available | | remove page | `Removed page: (repo: )` | n/a | | | search | Table with header `TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT`, rows in stable order, followed by `Count: N` | Header + `Count: 0` | For block rows use content snippet; for tag/property rows omit timestamps | -| show (text) | Raw tree text (no table), trimmed | n/a | For `--format json|edn`, keep existing structured output | +| show (text) | Raw tree text with `:db/id` as first column, trimmed | n/a | Tree lines should be prefixed by `:db/id` followed by the tree glyphs. Example:

`id1 block1`
`id2 ├── b2`
`id3 │ └── b3`
`id4 ├── b4`
`id5 │ ├── b5`
`id6 │ │ └── b6`
`id7 │ └── b7`
`id8 └── b8`

If a block title spans multiple lines, show the first line on the tree line and indent the remaining lines under the glyph column. Example:

`168 Jan 18th, 2026`
`169 ├── b1`
`173 ├── aaaxx`
`174 ├── block-line1`
` │ block-line2`
`175 └── cccc`

For `--format json|edn`, keep existing structured output | | errors | `Error (): ` + optional `Hint: ` line | n/a | Ensure error codes are stable and consistent | ## Problem statement diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 0e17b7bd82..9f65165fe1 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -93,6 +93,18 @@ Options grouping: Output formats: - Global `--output ` (also accepted per subcommand) - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- `show` human output prints the `:db/id` as the first column followed by a tree: + +``` +id1 block1 +id2 ├── b2 +id3 │ └── b3 +id4 ├── b4 +id5 │ ├── b5 +id6 │ │ └── b6 +id7 │ └── b7 +id8 └── b8 +``` Examples: diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 3bebb76c2e..dcbcd584f6 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -693,15 +693,37 @@ (defn- tree->text [{:keys [root]}] - (let [title (or (:block/title root) (:block/name root) (str (:block/uuid root))) - lines (atom [title]) - walk (fn walk [node depth] - (doseq [child (:block/children node)] - (let [prefix (apply str (repeat depth " ")) - label (or (:block/title child) (:block/name child) (str (:block/uuid child)))] - (swap! lines conj (str prefix "- " label))) - (walk child (inc depth))))] - (walk root 1) + (let [label (fn [node] + (or (:block/title node) (:block/name node) (str (:block/uuid node)))) + node-id (fn [node] + (or (:db/id node) "-")) + id-padding (fn [node] + (apply str (repeat (inc (count (str (node-id node)))) " "))) + split-lines (fn [value] + (string/split (or value "") #"\n")) + lines (atom []) + walk (fn walk [node prefix] + (let [children (:block/children node) + total (count children)] + (doseq [[idx child] (map-indexed vector children)] + (let [last-child? (= idx (dec total)) + branch (if last-child? "└── " "├── ") + next-prefix (str prefix (if last-child? " " "│ ")) + rows (split-lines (label child)) + first-row (first rows) + rest-rows (rest rows) + line (str (node-id child) " " prefix branch first-row)] + (swap! lines conj line) + (doseq [row rest-rows] + (swap! lines conj (str (id-padding child) next-prefix row))) + (walk child next-prefix)))))] + (let [rows (split-lines (label root)) + first-row (first rows) + rest-rows (rest rows)] + (swap! lines conj (str (node-id root) " " first-row)) + (doseq [row rest-rows] + (swap! lines conj (str (id-padding root) row)))) + (walk root "") (string/join "\n" @lines))) (defn- resolve-repo diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 9ebc3870fc..84be0e282b 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -140,6 +140,34 @@ (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) +(deftest db-worker-node-log-retention + (let [enforce-log-retention! #'db-worker-node/enforce-log-retention! + data-dir (node-helper/create-tmp-dir "db-worker-log-retention") + repo (str "logseq_db_log_retention_" (subs (str (random-uuid)) 0 8)) + pool-name (worker-util/get-pool-name repo) + repo-dir (node-path/join data-dir (str "." pool-name)) + days ["20240101" "20240102" "20240103" "20240104" "20240105" + "20240106" "20240107" "20240108" "20240109"] + make-log (fn [day] + (node-path/join repo-dir (str "db-worker-node-" day ".log")))] + (fs/mkdirSync repo-dir #js {:recursive true}) + (doseq [day days] + (fs/writeFileSync (make-log day) "log\n")) + (enforce-log-retention! repo-dir) + (let [remaining (->> (fs/readdirSync repo-dir) + (filter (fn [^js name] + (re-matches #"db-worker-node-\d{8}\.log" name))) + (sort))] + (is (= 7 (count remaining))) + (is (= ["db-worker-node-20240103.log" + "db-worker-node-20240104.log" + "db-worker-node-20240105.log" + "db-worker-node-20240106.log" + "db-worker-node-20240107.log" + "db-worker-node-20240108.log" + "db-worker-node-20240109.log"] + remaining))))) + (deftest db-worker-node-parse-args-ignores-host-and-port (let [parse-args #'db-worker-node/parse-args result (parse-args #js ["node" "db-worker-node.js" diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 599c8c51a7..95ff0d4227 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -130,6 +130,44 @@ (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) +(deftest test-tree->text-format + (testing "show tree text uses db/id with tree glyphs" + (let [tree->text #'commands/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child A" + :block/children [{:db/id 3 + :block/title "Grandchild A1"}]} + {:db/id 4 + :block/title "Child B"}]}}] + (is (= (str "1 Root\n" + "2 ├── Child A\n" + "3 │ └── Grandchild A1\n" + "4 └── Child B") + (tree->text tree-data)))))) + +(deftest test-tree->text-multiline + (testing "show tree text renders multiline blocks under glyph column" + (let [tree->text #'commands/tree->text + tree-data {:root {:db/id 168 + :block/title "Jan 18th, 2026" + :block/children [{:db/id 169 + :block/title "b1"} + {:db/id 173 + :block/title "aaaxx"} + {:db/id 174 + :block/title "block-line1\nblock-line2"} + {:db/id 175 + :block/title "cccc"}]}}] + (is (= (str "168 Jan 18th, 2026\n" + "169 ├── b1\n" + "173 ├── aaaxx\n" + "174 ├── block-line1\n" + " │ block-line2\n" + "175 └── cccc") + (tree->text tree-data)))))) + (deftest test-list-subcommand-parse (testing "list page parses" (let [result (commands/parse-args ["list" "page" diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 8b79d2e628..e3b8dbacab 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -145,12 +145,20 @@ :content "Note" :uuid "u2" :updated-at 4 - :created-at 2}]}} + :created-at 2} + {:type "tag" + :title "Taggy" + :uuid "u3"} + {:type "property" + :title "Prop" + :uuid "u4"}]}} {:output-format nil})] - (is (= (str "TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT\n" - "page Alpha u1 3 1\n" - "block Note u2 4 2\n" - "Count: 2") + (is (= (str "TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT\n" + "page Alpha u1 3 1\n" + "block Note u2 4 2\n" + "tag Taggy u3 - -\n" + "property Prop u4 - -\n" + "Count: 4") result)))) (testing "show renders text payloads directly" From add9319b244fcf8a139cae3ae1934f675bcd8e1a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 19 Jan 2026 15:39:42 +0800 Subject: [PATCH 021/375] add&impl 006-logseq-cli-import-export.md --- .../006-logseq-cli-import-export.md | 83 +++++++ docs/cli/logseq-cli.md | 7 +- src/main/frontend/worker/db_core.cljs | 32 +++ src/main/logseq/cli/commands.cljs | 170 ++++++++++++++- src/main/logseq/cli/format.cljs | 10 + src/main/logseq/cli/main.cljs | 15 +- src/main/logseq/cli/transport.cljs | 23 +- .../frontend/worker/db_worker_node_test.cljs | 121 +++++++++++ src/test/logseq/cli/commands_test.cljs | 203 +++++++++++++++++- src/test/logseq/cli/format_test.cljs | 17 ++ src/test/logseq/cli/integration_test.cljs | 94 +++++++- src/test/logseq/cli/transport_test.cljs | 32 ++- 12 files changed, 783 insertions(+), 24 deletions(-) create mode 100644 docs/agent-guide/006-logseq-cli-import-export.md diff --git a/docs/agent-guide/006-logseq-cli-import-export.md b/docs/agent-guide/006-logseq-cli-import-export.md new file mode 100644 index 0000000000..f84ab5b0d7 --- /dev/null +++ b/docs/agent-guide/006-logseq-cli-import-export.md @@ -0,0 +1,83 @@ +# Logseq CLI Import/Export Plan + +Goal: Add logseq-cli support for import/export with EDN and SQLite formats using the existing db-worker-node server. + +Architecture: Extend logseq-cli command parsing and execution to invoke db-worker-node thread APIs for export and import, with minimal new APIs to handle EDN import and SQLite binary payloads over HTTP. + +Tech Stack: ClojureScript, babashka/cli, db-worker-node HTTP /v1/invoke, datascript, sqlite-export helpers, Node fs/path. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Requirements + +- Import types: edn, sqlite. +- Export types: edn, sqlite. +- CLI must work against db-worker-node with repo binding and lock file behavior. +- Output files must be written to user-specified paths with clear success/error messages. + +## Proposed CLI UX + +Prefer graph-scoped subcommands to keep import/export with graph management: + +- `logseq graph export --type edn --output [--repo ]` +- `logseq graph export --type sqlite --output [--repo ]` +- `logseq graph import --type edn --input --repo ` +- `logseq graph import --type sqlite --input --repo ` + +Notes: +- `graph import` only supports importing into a new graph name; it must not overwrite an existing graph. +- `--repo` is required for import, and required unless the current graph is set in config for export. + +## Current Capabilities (Baseline) + +- db-worker-node supports `:thread-api/export-edn`, `:thread-api/export-db`, and `:thread-api/import-db`. +- logseq-cli can invoke db-worker-node via `src/main/logseq/cli/transport.cljs` and write files with `transport/write-output`. +- EDN import exists in the app layer (`frontend.handler.db-based.import`) but not in db-worker-node. +- SQLite import in db-worker-node writes the sqlite file, but CLI needs a full reload step to reflect new data. + +## Implementation Plan + +1. Review existing CLI command table and action pipeline in `src/main/logseq/cli/commands.cljs` and `src/main/logseq/cli/main.cljs` to locate insertion points for new graph import/export actions. +2. Add new CLI specs for import/export flags (type, input, output, export-type, mode) in `src/main/logseq/cli/commands.cljs`. +3. Extend the command table with `graph import` and `graph export` entries and ensure help output includes them. +4. Add action builders for import/export that validate repo presence, file paths, and allowed types (edn, sqlite). +5. Add CLI helpers for reading input files and writing output files in `src/main/logseq/cli/transport.cljs` (or a new helper namespace), keeping the existing `write-output` behavior for EDN and DB files. +6. Implement export execution: + - EDN: call `thread-api/export-edn` (graph-only) and write EDN file. + - SQLite: call a new db-worker-node export API that returns a base64 string or transit-safe binary; decode to Buffer and write `.sqlite` file. +7. Implement import execution: + - EDN: read EDN file, pass data to a new db-worker-node `thread-api/import-edn` (see below), and return a summary message. + - SQLite: read file as Buffer, pass to a new db-worker-node `thread-api/import-sqlite` (or reuse `import-db` with a wrapper that closes/reopens the repo). + - Always stop and restart the db-worker-node server from the CLI around import to ensure a clean reload. +8. Add db-worker-node thread APIs in `src/main/frontend/worker/db_core.cljs`: + - `:thread-api/import-edn` to convert export EDN into tx data via `logseq.db.sqlite.export/build-import` and transact with `:tx-meta` including `::sqlite-export/imported-data? true` so the pipeline rebuilds refs. + - `:thread-api/export-db-base64` (or similar) to return a base64 string for SQLite export over HTTP. + - `:thread-api/import-db-base64` (or similar) to accept base64 input, close existing sqlite connections, import db data, and reopen the repo (or invoke `:thread-api/create-or-open-db` with `:import-type :sqlite-db`). +9. Update db-worker-node server validation (repo binding) if the new thread APIs need special argument shapes. +10. Update CLI output formatting in `src/main/logseq/cli/format.cljs` to print concise success lines like `Exported to ` and `Imported from `. +11. Update documentation in `docs/cli/logseq-cli.md` with new commands, examples, and file format notes. + +## Testing Plan + +- Add CLI parsing tests for `graph import` and `graph export` options in `src/test/logseq/cli/commands_test.cljs` (or a new namespace). +- Add integration tests in `src/test/logseq/cli/integration_test.cljs` to: + - export EDN and SQLite from a test graph and assert output files exist and are non-empty. + - import EDN into a new graph and verify a known page/block exists via CLI `show` or `list`. + - import SQLite into a new graph and verify graph metadata or page count. +- Add db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` for the new import/export thread APIs (EDN build-import path and base64 DB export/import). +- Follow @test-driven-development: write failing tests before implementation. + +## Edge Cases + +- Large SQLite exports may exceed JSON limits if not base64/transit encoded; ensure streaming-safe or chunked base64 handling. +- Import should fail fast if the repo is missing and `--repo` is not provided, or if input file does not exist. +- SQLite import while the repo is open must close/reopen connections to avoid stale datascript state. +- EDN import should validate the export shape and surface readable errors when EDN is invalid or incompatible. +- Overwrite behavior should be explicit for SQLite imports to prevent accidental data loss. + +## Decisions + +1. `graph import` only imports into a new graph; it must not overwrite an existing graph. +2. No `--mode` flag; both EDN and SQLite imports are replace-style imports. +3. CLI always stops and restarts db-worker-node around imports. +4. `graph export --type edn` is graph-only for now (no page/view/blocks). diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 9f65165fe1..4022aa95f2 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -47,8 +47,10 @@ Graph commands: - `graph remove --repo ` - remove a graph - `graph validate --repo ` - validate graph data - `graph info [--repo ]` - show graph metadata (defaults to current graph) +- `graph export --type edn|sqlite --output [--repo ]` - export a graph to EDN or SQLite +- `graph import --type edn|sqlite --input --repo ` - import a graph from EDN or SQLite (new graph only) -For any command that requires `--repo`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). +For any command that requires `--repo`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). `graph import` fails if the target graph already exists. Server commands: - `server list` - list running db-worker-node servers @@ -92,6 +94,7 @@ Options grouping: Output formats: - Global `--output ` (also accepted per subcommand) +- For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - `show` human output prints the `:db/id` as the first column followed by a tree: @@ -110,6 +113,8 @@ Examples: ```bash node ./static/logseq-cli.js graph create --repo demo +node ./static/logseq-cli.js graph export --type edn --output /tmp/demo.edn --repo demo +node ./static/logseq-cli.js graph import --type edn --input /tmp/demo.edn --repo demo-import node ./static/logseq-cli.js add block --page TestPage --content "hello world" node ./static/logseq-cli.js search --text "hello" node ./static/logseq-cli.js show --page-name TestPage --format json --output json diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 6b6a50f609..d6ed898222 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -579,6 +579,17 @@ (p/let [data (> table @@ -228,6 +236,27 @@ :message "page name is required"} :summary summary}) +(defn- missing-type-result + [summary] + {:ok? false + :error {:code :missing-type + :message "type is required"} + :summary summary}) + +(defn- missing-input-result + [summary] + {:ok? false + :error {:code :missing-input + :message "input is required"} + :summary summary}) + +(defn- missing-output-result + [summary] + {:ok? false + :error {:code :missing-output + :message "output is required"} + :summary summary}) + (defn- missing-search-result [summary] {:ok? false @@ -266,6 +295,13 @@ (def ^:private search-types #{"page" "block" "tag" "property" "all"}) +(def ^:private import-export-types + #{"edn" "sqlite"}) + +(defn- normalize-import-export-type + [value] + (some-> value string/lower-case string/trim)) + (defn- invalid-list-options? [command opts] (let [{:keys [order include-journal journal-only]} opts @@ -337,6 +373,8 @@ (command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) (command-entry ["graph" "validate"] :graph-validate "Validate graph" {}) (command-entry ["graph" "info"] :graph-info "Graph metadata" {}) + (command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec) + (command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec) (command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) (command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec) (command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) @@ -448,6 +486,29 @@ (and (= command :search) (invalid-search-options? opts)) (invalid-options-result summary (invalid-search-options? opts)) + (and (= command :graph-export) (not (seq (normalize-import-export-type (:type opts))))) + (missing-type-result summary) + + (and (= command :graph-export) (not (seq (:output opts)))) + (missing-output-result summary) + + (and (= command :graph-export) + (not (contains? import-export-types (normalize-import-export-type (:type opts))))) + (invalid-options-result summary (str "invalid type: " (:type opts))) + + (and (= command :graph-import) (not (seq (normalize-import-export-type (:type opts))))) + (missing-type-result summary) + + (and (= command :graph-import) (not (seq (:input opts)))) + (missing-input-result summary) + + (and (= command :graph-import) (not (seq (:repo opts)))) + (missing-repo-result summary) + + (and (= command :graph-import) + (not (contains? import-export-types (normalize-import-export-type (:type opts))))) + (invalid-options-result summary (str "invalid type: " (:type opts))) + (and (#{:server-status :server-start :server-stop :server-restart} command) (not (seq (:repo opts)))) (missing-repo-result summary) @@ -521,6 +582,18 @@ :message "graph not exists"}})) (p/resolved {:ok? true}))) +(defn- ensure-missing-graph + [action config] + (if (and (= :graph-import (:type action)) (:repo action)) + (p/let [graphs (cli-server/list-graphs config) + graph (repo->graph (:repo action))] + (if (some #(= graph %) graphs) + {:ok? false + :error {:code :graph-exists + :message "graph already exists"}} + {:ok? true})) + (p/resolved {:ok? true}))) + (defn- pick-graph [options command-args config] (or (:repo options) @@ -1044,6 +1117,30 @@ (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) (build-graph-action command graph repo) + :graph-export + (let [export-type (normalize-import-export-type (:type options))] + (if-not (seq repo) + (missing-repo-error "repo is required for export") + {:ok? true + :action {:type :graph-export + :repo repo + :graph (repo->graph repo) + :export-type export-type + :output (:output options)}})) + + :graph-import + (let [import-repo (resolve-repo (:repo options)) + import-type (normalize-import-export-type (:type options))] + (if-not (seq import-repo) + (missing-repo-error "repo is required for import") + {:ok? true + :action {:type :graph-import + :repo import-repo + :graph (repo->graph import-repo) + :import-type import-type + :input (:input options) + :allow-missing-graph true}})) + (:server-list :server-status :server-start :server-stop :server-restart) (build-server-action command server-repo) @@ -1119,6 +1216,51 @@ :logseq.kv/graph-created-at (:kv/value created) :logseq.kv/schema-version (:kv/value schema)}}))) +(defn- execute-graph-export + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + export-type (:export-type action) + export-result (case export-type + "edn" + (transport/invoke cfg + "thread-api/export-edn" + false + [(:repo action) {:export-type :graph}]) + "sqlite" + (transport/invoke cfg + "thread-api/export-db-base64" + true + [(:repo action)]) + (throw (ex-info "unsupported export type" {:export-type export-type}))) + data (if (= export-type "sqlite") + (js/Buffer.from export-result "base64") + export-result) + format (if (= export-type "sqlite") :sqlite :edn)] + (transport/write-output {:format format :path (:output action) :data data}) + {:status :ok + :data {:message (str "wrote " (:output action))}}))) + +(defn- execute-graph-import + [action config] + (-> (p/let [_ (cli-server/stop-server! config (:repo action)) + cfg (cli-server/ensure-server! config (:repo action)) + import-type (:import-type action) + input-data (case import-type + "edn" (transport/read-input {:format :edn :path (:input action)}) + "sqlite" (transport/read-input {:format :sqlite :path (:input action)}) + (throw (ex-info "unsupported import type" {:import-type import-type}))) + payload (if (= import-type "sqlite") + (.toString (js/Buffer.from input-data) "base64") + input-data) + method (if (= import-type "sqlite") + "thread-api/import-db-base64" + "thread-api/import-edn") + direct-pass? (= import-type "sqlite") + _ (transport/invoke cfg method direct-pass? [(:repo action) payload]) + _ (cli-server/restart-server! config (:repo action))] + {:status :ok + :data {:message (str "imported " import-type " from " (:input action))}}))) + (defn- execute-list-page [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -1208,7 +1350,7 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? ?title ?q)]] + [(clojure.string/includes? ?title ?q)]] '[:find ?e ?title ?uuid ?updated ?created :in $ ?q :where @@ -1217,10 +1359,12 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? (string/lower-case ?title) ?q)]]) + [(clojure.string/includes? (clojure.string/lower-case ?title) ?q)]]) q* (if case-sensitive? text (string/lower-case text))] (transport/invoke cfg "thread-api/q" false [repo [query q*]]))) + +#_{:clj-kondo/ignore [:aliased-namespace-symbol]} (defn- query-blocks [cfg repo text case-sensitive? tag include-content?] (let [has-tag? (seq tag) @@ -1237,7 +1381,7 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? ?value ?q)]] + [(clojure.string/includes? ?value ?q)]] case-sensitive? `[:find ?e ?value ?uuid ?updated ?created @@ -1248,7 +1392,7 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? ?value ?q)]] + [(clojure.string/includes? ?value ?q)]] has-tag? `[:find ?e ?value ?uuid ?updated ?created @@ -1261,7 +1405,7 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? (string/lower-case ?value) ?q)]] + [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]] :else `[:find ?e ?value ?uuid ?updated ?created @@ -1272,7 +1416,7 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(string/includes? (string/lower-case ?value) ?q)]]) + [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]]) q* (if case-sensitive? text (string/lower-case text)) tag-name (some-> tag string/lower-case)] (if has-tag? @@ -1424,15 +1568,25 @@ (defn execute [action config] - (-> (p/let [check (ensure-existing-graph action config) - result (if-not (:ok? check) + (-> (p/let [missing-check (ensure-missing-graph action config) + check (ensure-existing-graph action config) + result (cond + (not (:ok? missing-check)) + {:status :error + :error (:error missing-check)} + + (not (:ok? check)) {:status :error :error (:error check)} + + :else (case (:type action) :graph-list (execute-graph-list action config) :invoke (execute-invoke action config) :graph-switch (execute-graph-switch action config) :graph-info (execute-graph-info action config) + :graph-export (execute-graph-export action config) + :graph-import (execute-graph-import action config) :list-page (execute-list-page action config) :list-tag (execute-list-tag action config) :list-property (execute-list-property action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 3a620f7704..e70cc0bfa9 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -218,6 +218,14 @@ [{:keys [repo block]}] (str "Removed block: " block " (repo: " repo ")")) +(defn- format-graph-export + [{:keys [export-type output]}] + (str "Exported " export-type " to " output)) + +(defn- format-graph-import + [{:keys [import-type input]}] + (str "Imported " import-type " from " input)) + (defn- format-graph-action [command {:keys [graph]}] (let [verb (case command @@ -248,6 +256,8 @@ :add-page (format-add-page context) :remove-page (format-remove-page context) :remove-block (format-remove-block context) + :graph-export (format-graph-export context) + :graph-import (format-graph-import context) :search (format-search-results (:results data)) :show (or (:message data) (pr-str data)) (if (and (map? data) (contains? data :message)) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 59ff135756..eb5f1ab1cb 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) @@ -52,9 +52,16 @@ {:exit-code 0 :output (format/format-result result opts)}))) (p/catch (fn [error] - (let [message (or (some-> (ex-data error) :message) - (.-message error) - (str error))] + (let [data (ex-data error) + message (cond + (and (= :http-error (:code data)) (seq (:body data))) + (str "http request failed (" (:status data) "): " (:body data)) + + (some? (:message data)) + (:message data) + + :else + (or (.-message error) (str error)))] {:exit-code 1 :output (format/format-result {:status :error :error {:code :exception diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index eb342bac65..b24ad93645 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.transport "HTTP transport for communicating with db-worker-node." - (:require [clojure.string :as string] + (:require [cljs.reader :as reader] + [clojure.string :as string] [logseq.db :as ldb] [promesa.core :as p] ["fs" :as fs] @@ -120,4 +121,24 @@ (js/Buffer.from data))] (fs/writeFileSync path buffer)) + :sqlite + (let [buffer (if (instance? js/Buffer data) + data + (js/Buffer.from data))] + (fs/writeFileSync path buffer)) + (throw (ex-info "unsupported output format" {:format format})))) + +(defn read-input + [{:keys [format path]}] + (case format + :edn + (reader/read-string (.toString (fs/readFileSync path) "utf8")) + + :db + (fs/readFileSync path) + + :sqlite + (fs/readFileSync path) + + (throw (ex-info "unsupported input format" {:format format})))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 84be0e282b..3e889483dd 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -252,6 +252,127 @@ (done)))) (done)))))))) +(deftest db-worker-node-import-edn + (async done + (let [daemon-a (atom nil) + daemon-b (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-import-edn") + repo-a (str "logseq_db_import_edn_a_" (subs (str (random-uuid)) 0 8)) + repo-b (str "logseq_db_import_edn_b_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-a}) + _ (reset! daemon-a {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) + _ (invoke host port "thread-api/transact" + [repo-a + [{:block/uuid page-uuid + :block/title "Import Page" + :block/name "import-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now}] + {} + nil]) + export-edn (invoke host port "thread-api/export-edn" [repo-a {:export-type :graph}])] + (is (map? export-edn)) + (p/let [_ ((:stop! @daemon-a)) + {:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-b}) + _ (reset! daemon-b {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) + _ (invoke host port "thread-api/import-edn" [repo-b export-edn]) + result (invoke host port "thread-api/q" + [repo-b + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + "Import Page"]])] + (is (seq result)))) + (p/catch (fn [e] + (println "[db-worker-node-test] import-edn error:" e) + (is false (str e)))) + (p/finally (fn [] + (let [stop-a (:stop! @daemon-a) + stop-b (:stop! @daemon-b)] + (cond + (and stop-a stop-b) + (-> (stop-a) + (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) + + stop-a + (-> (stop-a) (p/finally (fn [] (done)))) + + stop-b + (-> (stop-b) (p/finally (fn [] (done)))) + + :else + (done))))))))) + +(deftest db-worker-node-import-db-base64 + (async done + (let [daemon-a (atom nil) + daemon-b (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-import-sqlite") + repo-a (str "logseq_db_import_sqlite_a_" (subs (str (random-uuid)) 0 8)) + repo-b (str "logseq_db_import_sqlite_b_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-a}) + _ (reset! daemon-a {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) + _ (invoke host port "thread-api/transact" + [repo-a + [{:block/uuid page-uuid + :block/title "SQLite Import Page" + :block/name "sqlite-import-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now}] + {} + nil]) + export-base64 (invoke host port "thread-api/export-db-base64" [repo-a])] + (is (string? export-base64)) + (is (pos? (count export-base64))) + (p/let [_ ((:stop! @daemon-a)) + {:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-b}) + _ (reset! daemon-b {:stop! stop!}) + _ (invoke host port "thread-api/import-db-base64" [repo-b export-base64]) + _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) + result (invoke host port "thread-api/q" + [repo-b + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + "SQLite Import Page"]])] + (is (seq result)))) + (p/catch (fn [e] + (println "[db-worker-node-test] import-sqlite error:" e) + (is false (str e)))) + (p/finally (fn [] + (let [stop-a (:stop! @daemon-a) + stop-b (:stop! @daemon-b)] + (cond + (and stop-a stop-b) + (-> (stop-a) + (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) + + stop-a + (-> (stop-a) (p/finally (fn [] (done)))) + + stop-b + (-> (stop-b) (p/finally (fn [] (done)))) + + :else + (done))))))))) + (deftest db-worker-node-repo-mismatch-test (async done (let [daemon (atom nil) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 95ff0d4227..1d6bbe4657 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -3,6 +3,7 @@ [clojure.string :as string] [logseq.cli.commands :as commands] [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] [promesa.core :as p])) (deftest test-help-output @@ -31,7 +32,9 @@ summary (:summary result)] (is (true? (:help? result))) (is (string/includes? summary "graph list")) - (is (string/includes? summary "graph create")))) + (is (string/includes? summary "graph create")) + (is (string/includes? summary "graph export")) + (is (string/includes? summary "graph import")))) (testing "list group shows subcommands" (let [result (commands/parse-args ["list"]) @@ -303,6 +306,51 @@ (is (= :show (:command result))) (is (= "Home" (get-in result [:options :page-name]))))) + (testing "graph export parses with type and output" + (let [result (commands/parse-args ["graph" "export" + "--type" "edn" + "--output" "export.edn"])] + (is (true? (:ok? result))) + (is (= :graph-export (:command result))) + (is (= "edn" (get-in result [:options :type]))) + (is (= "export.edn" (get-in result [:options :output]))))) + + (testing "graph import parses with type, input, and repo" + (let [result (commands/parse-args ["graph" "import" + "--type" "sqlite" + "--input" "import.sqlite" + "--repo" "demo"])] + (is (true? (:ok? result))) + (is (= :graph-import (:command result))) + (is (= "sqlite" (get-in result [:options :type]))) + (is (= "import.sqlite" (get-in result [:options :input]))) + (is (= "demo" (get-in result [:options :repo]))))) + + (testing "graph export requires type" + (let [result (commands/parse-args ["graph" "export" "--output" "export.edn"])] + (is (false? (:ok? result))) + (is (= :missing-type (get-in result [:error :code]))))) + + (testing "graph export requires output" + (let [result (commands/parse-args ["graph" "export" "--type" "edn"])] + (is (false? (:ok? result))) + (is (= :missing-output (get-in result [:error :code]))))) + + (testing "graph import requires repo" + (let [result (commands/parse-args ["graph" "import" + "--type" "edn" + "--input" "import.edn"])] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + + (testing "graph import rejects unknown type" + (let [result (commands/parse-args ["graph" "import" + "--type" "zip" + "--input" "import.zip" + "--repo" "demo"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "verb subcommands reject unknown flags" (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] @@ -361,6 +409,22 @@ (is (true? (:ok? result))) (is (= :graph-info (get-in result [:action :type]))))) + (testing "graph export uses config repo" + (let [parsed {:ok? true + :command :graph-export + :options {:type "edn" :output "export.edn"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :graph-export (get-in result [:action :type]))))) + + (testing "graph import requires repo" + (let [parsed {:ok? true + :command :graph-import + :options {:type "edn" :input "import.edn"}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code]))))) + (testing "list page requires repo" (let [parsed {:ok? true :command :list-page :options {}} result (commands/build-action parsed {})] @@ -419,3 +483,140 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest test-execute-graph-import-rejects-existing-graph + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server!] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))) + (-> (p/let [result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :allow-missing-graph true} + {})] + (is (= :error (:status result))) + (is (= :graph-exists (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (done))))))) + +(deftest test-execute-graph-export + (async done + (let [invoke-calls (atom []) + write-calls (atom []) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + orig-write-output transport/write-output] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [config _] + (assoc config :base-url "http://127.0.0.1:9999"))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (if (= method "thread-api/export-db-base64") + "c3FsaXRl" + {:exported true}))) + (set! transport/write-output (fn [opts] + (swap! write-calls conj opts))) + (-> (p/let [edn-result (commands/execute {:type :graph-export + :repo "logseq_db_demo" + :graph "demo" + :export-type "edn" + :output "/tmp/export.edn" + :allow-missing-graph true} + {}) + sqlite-result (commands/execute {:type :graph-export + :repo "logseq_db_demo" + :graph "demo" + :export-type "sqlite" + :output "/tmp/export.sqlite" + :allow-missing-graph true} + {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= [["thread-api/export-edn" false ["logseq_db_demo" {:export-type :graph}]] + ["thread-api/export-db-base64" true ["logseq_db_demo"]]] + @invoke-calls)) + (is (= 2 (count @write-calls))) + (let [[edn-write sqlite-write] @write-calls] + (is (= {:format :edn :path "/tmp/export.edn" :data {:exported true}} + edn-write)) + (is (= :sqlite (:format sqlite-write))) + (is (= "/tmp/export.sqlite" (:path sqlite-write))) + (is (= "sqlite" (.toString (:data sqlite-write) "utf8"))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (set! transport/write-output orig-write-output) + (done))))))) + +(deftest test-execute-graph-import + (async done + (let [invoke-calls (atom []) + read-calls (atom []) + stop-calls (atom []) + restart-calls (atom []) + orig-list-graphs cli-server/list-graphs + orig-stop-server! cli-server/stop-server! + orig-restart-server! cli-server/restart-server! + orig-ensure-server! cli-server/ensure-server! + orig-read-input transport/read-input + orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] [])) + (set! cli-server/stop-server! (fn [_ repo] + (swap! stop-calls conj repo) + (p/resolved {:ok? true}))) + (set! cli-server/restart-server! (fn [_ repo] + (swap! restart-calls conj repo) + (p/resolved {:ok? true}))) + (set! cli-server/ensure-server! (fn [config _] + (assoc config :base-url "http://127.0.0.1:9999"))) + (set! transport/read-input (fn [{:keys [format path]}] + (swap! read-calls conj [format path]) + (if (= format :edn) + {:page "Import Page"} + (js/Buffer.from "sqlite" "utf8")))) + (set! transport/invoke (fn [_ method _ args] + (swap! invoke-calls conj [method args]) + {:ok true})) + (-> (p/let [edn-result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :graph "demo" + :import-type "edn" + :input "/tmp/import.edn" + :allow-missing-graph true} + {}) + sqlite-result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :graph "demo" + :import-type "sqlite" + :input "/tmp/import.sqlite" + :allow-missing-graph true} + {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= [[:edn "/tmp/import.edn"] + [:sqlite "/tmp/import.sqlite"]] + @read-calls)) + (is (= [["thread-api/import-edn" ["logseq_db_demo" {:page "Import Page"}]] + ["thread-api/import-db-base64" ["logseq_db_demo" "c3FsaXRl"]]] + @invoke-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/stop-server! orig-stop-server!) + (set! cli-server/restart-server! orig-restart-server!) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/read-input orig-read-input) + (set! transport/invoke orig-invoke) + (done))))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index e3b8dbacab..60bc40559b 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -105,6 +105,23 @@ {:output-format nil})] (is (= "Removed page: Home (repo: demo-repo)" result))))) +(deftest test-human-output-graph-import-export + (testing "graph export renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :graph-export + :context {:export-type "edn" + :output "/tmp/export.edn"}} + {:output-format nil})] + (is (= "Exported edn to /tmp/export.edn" result)))) + + (testing "graph import renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :graph-import + :context {:import-type "sqlite" + :input "/tmp/import.sqlite"}} + {:output-format nil})] + (is (= "Imported sqlite from /tmp/import.sqlite" result))))) + (deftest test-human-output-graph-info (testing "graph info includes key metadata lines" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 0df87c97dd..fde2079428 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -73,21 +73,21 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) - add-page-result (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + add-page-result (run-cli ["--repo" "content-graph" "add" "page" "--page" "TestPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) - list-page-result (run-cli ["list" "page"] data-dir cfg-path) + list-page-result (run-cli ["--repo" "content-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) - list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--repo" "content-graph" "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) - list-property-result (run-cli ["list" "property"] data-dir cfg-path) + list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["add" "block" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) _ (parse-json-output add-block-result) - search-result (run-cli ["search" "--text" "hello world" "--include-content"] data-dir cfg-path) + search-result (run-cli ["--repo" "content-graph" "search" "--text" "hello world"] data-dir cfg-path) search-payload (parse-json-output search-result) - show-result (run-cli ["show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) + show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) show-payload (parse-json-output show-result) - remove-page-result (run-cli ["remove" "page" "--page" "TestPage"] data-dir cfg-path) + remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--page" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -210,3 +210,81 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest test-cli-graph-export-import-edn + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-export-edn")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + export-graph "export-edn-graph" + import-graph "import-edn-graph" + export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") + _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "page" "--page" "ExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--page" "ExportPage" "--content" "Export content"] data-dir cfg-path) + export-result (run-cli ["--repo" export-graph + "graph" "export" + "--type" "edn" + "--output" export-path] data-dir cfg-path) + export-payload (parse-json-output export-result) + _ (run-cli ["--repo" import-graph + "graph" "import" + "--type" "edn" + "--input" export-path] data-dir cfg-path) + list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-payload (parse-json-output list-result) + stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + (is (= 0 (:exit-code export-result))) + (is (= "ok" (:status export-payload))) + (is (fs/existsSync export-path)) + (is (pos? (.-size (fs/statSync export-path)))) + (is (= "ok" (:status list-payload))) + (is (some (fn [item] + (= "ExportPage" (or (:title item) (:block/title item)))) + (get-in list-payload [:data :items]))) + (is (= 0 (:exit-code stop-export))) + (is (= 0 (:exit-code stop-import))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-graph-export-import-sqlite + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-export-sqlite")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + export-graph "export-sqlite-graph" + import-graph "import-sqlite-graph" + export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") + _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--page" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) + export-result (run-cli ["--repo" export-graph + "graph" "export" + "--type" "sqlite" + "--output" export-path] data-dir cfg-path) + export-payload (parse-json-output export-result) + _ (run-cli ["--repo" import-graph + "graph" "import" + "--type" "sqlite" + "--input" export-path] data-dir cfg-path) + list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-payload (parse-json-output list-result) + stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + (is (= 0 (:exit-code export-result))) + (is (= "ok" (:status export-payload))) + (is (fs/existsSync export-path)) + (is (pos? (.-size (fs/statSync export-path)))) + (is (= "ok" (:status list-payload))) + (is (some (fn [item] + (= "SQLiteExportPage" (or (:title item) (:block/title item)))) + (get-in list-payload [:data :items]))) + (is (= 0 (:exit-code stop-export))) + (is (= 0 (:exit-code stop-import))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index cc2207dda5..03ea866e42 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -1,8 +1,17 @@ (ns logseq.cli.transport-test - (:require [cljs.test :refer [deftest is async]] + (:require [cljs.test :refer [deftest is async testing]] [promesa.core :as p] [logseq.cli.transport :as transport])) +(def ^:private fs (js/require "fs")) +(def ^:private os (js/require "os")) +(def ^:private path (js/require "path")) + +(defn- temp-path + [filename] + (let [dir (.mkdtempSync fs (.join path (.tmpdir os) "logseq-cli-"))] + (.join path dir filename))) + (defn- start-server [handler] (p/create @@ -62,3 +71,24 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done)))))) + +(deftest test-read-input + (testing "reads edn input" + (let [path (temp-path "input.edn")] + (.writeFileSync fs path "{:a 1}") + (is (= {:a 1} (transport/read-input {:format :edn :path path}))))) + + (testing "reads sqlite input as buffer" + (let [path (temp-path "input.sqlite") + buffer (js/Buffer.from "sqlite-data")] + (.writeFileSync fs path buffer) + (let [result (transport/read-input {:format :sqlite :path path})] + (is (instance? js/Buffer result)) + (is (= "sqlite-data" (.toString result "utf8"))))))) + +(deftest test-write-output + (testing "writes sqlite output as buffer" + (let [path (temp-path "output.sqlite") + buffer (js/Buffer.from "sqlite-export")] + (transport/write-output {:format :sqlite :path path :data buffer}) + (is (= "sqlite-export" (.toString (.readFileSync fs path) "utf8")))))) From 1abe01a942014b73e22757c36ee8652f451f574c Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 19 Jan 2026 15:42:47 +0800 Subject: [PATCH 022/375] add 007-logseq-cli-thread-api-and-command-split.md --- ...logseq-cli-thread-api-and-command-split.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md diff --git a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md new file mode 100644 index 0000000000..978b62ae0b --- /dev/null +++ b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md @@ -0,0 +1,76 @@ +# Logseq CLI Thread API Keywords And Command Split Implementation Plan + +Goal: Replace thread-api string usage with keywords, standardize CLI repo option to --repo, and split logseq.cli.commands into per-subcommand namespaces. + +Architecture: Update transport and db-worker-node boundaries to accept keyword methods while still serializing over HTTP. Refactor CLI command parsing into a shared dispatcher plus per-subcommand namespaces under a new command directory. Keep existing CLI behavior and output stable while updating option naming and error hints. + +Tech Stack: ClojureScript, babashka.cli, promesa, Logseq db-worker-node. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md, docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md, docs/agent-guide/006-logseq-cli-import-export.md. + +## Problem statement + +The current CLI and db-worker-node codebase mixes thread-api method strings with keyword-based APIs, which makes it easy to introduce mismatches and reduces consistency with the thread-api macro design. The CLI option naming also has legacy --graph hints and expectations that conflict with the newer --repo naming, which creates confusion for users and tests. The logseq.cli.commands namespace has grown large and mixes parsing, validation, and execution for multiple command groups, which makes maintenance and ownership difficult. + +## Testing Plan + +I will add unit tests to ensure all CLI thread-api method invocations use keywords and still serialize correctly through transport when invoking db-worker-node. I will add unit tests to ensure any error hint or help text for missing graph/repo uses --repo and no longer mentions --graph. I will add unit tests for command parsing and action building to cover the new per-subcommand namespaces and ensure summaries and help still match expected output. I will update db-worker-node tests to assert keyword method handling for repo validation and allowed non-repo methods. NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +1. Review @prompts/review.md to align plan with review expectations. + +2. Enumerate all thread-api string usages in CLI, db-worker-node, and tests using `rg -n -- "\"thread-api/" src/main src/test`. + +3. Enumerate all --graph references in CLI code and tests using `rg -n -- "--graph" src/main src/test` and confirm there are no docs references outside CLI. + +4. Define a small utility in `src/main/logseq/cli/transport.cljs` to normalize thread-api method arguments to keywords for callers while serializing over HTTP as string names. + +5. Add tests in `src/test/logseq/cli/transport_test.cljs` that pass a keyword method to transport invoke and assert the outgoing payload uses the string name and the response handling is unchanged. + +6. Update all CLI calls in `src/main/logseq/cli/commands.cljs` to pass keyword thread-api names to transport invoke and to any action maps that store method identifiers. + +7. Update CLI tests in `src/test/logseq/cli/commands_test.cljs` and `src/test/logseq/cli/integration_test.cljs` to expect keyword thread-api methods where they assert method calls. + +8. Update db-worker-node internal invocation and repo validation in `src/main/frontend/worker/db_worker_node.cljs` to coerce incoming method values to keywords for comparisons and logging, while still accepting string input from JSON payloads. + +9. Update any db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` that assert method strings to use keyword expectations and verify non-repo method handling. + +10. Replace any --graph option references in CLI formatting and tests by updating `src/main/logseq/cli/format.cljs` and `src/test/logseq/cli/format_test.cljs` to use --repo. + +11. Search for any `:graph` option wiring in `src/main/logseq/cli/commands.cljs` and remove CLI option parsing for --graph, including any help or usage text, while preserving graph-specific subcommands like `graph create`. + +12. Introduce a new directory `src/main/logseq/cli/command/` and split `logseq.cli.commands` into per-subcommand namespaces such as `logseq.cli.command.graph`, `logseq.cli.command.server`, `logseq.cli.command.list`, `logseq.cli.command.add`, `logseq.cli.command.remove`, `logseq.cli.command.search`, and `logseq.cli.command.show`. + +13. Create a small shared namespace like `src/main/logseq/cli/command/core.cljs` for global option spec, parsing helpers, summary formatting, and shared validation utilities, ensuring no behavior changes in parsing or summaries. + +14. Update `src/main/logseq/cli/commands.cljs` to become a thin facade that assembles tables from per-subcommand modules, delegates parse-args and build-action to those modules, and exposes execute dispatching without changing public API. + +15. Update `src/main/logseq/cli/main.cljs` requires to match any namespace changes and confirm the CLI usage summary still renders the same command list. + +16. Update tests in `src/test/logseq/cli/commands_test.cljs` to import any moved namespaces or use the facade namespace, and ensure all help summary snapshots still pass. + +17. Run unit tests for CLI and db-worker-node with `bb dev:test -v logseq.cli.commands-test`, `bb dev:test -v logseq.cli.transport-test`, and `bb dev:test -v frontend.worker.db-worker-node-test` and fix failures. + +18. Run the full lint and unit test suite with `bb dev:lint-and-test` after all changes are complete. + +## Testing Details + +The tests will focus on behavior by asserting that CLI invocations still produce the same actions and outputs while enforcing keyword-based thread-api methods and updated --repo hints. The db-worker-node tests will assert that repo validation and non-repo method bypass logic behave correctly when methods are provided as keywords or strings. The command split will be validated by reusing existing parse-args and summary tests to ensure no behavioral regression in user-facing help or dispatch. + +## Implementation Details + +- Normalize thread-api method values at transport and db-worker-node boundaries to accept keywords and serialize as strings over HTTP. +- Replace all explicit "thread-api/..." literals in CLI and db-worker-node call sites with :thread-api/... keywords. +- Remove --graph option handling and update error hints to use --repo. +- Preserve graph subcommands and graph naming semantics while standardizing on :repo options. +- Move per-subcommand parsing and execution helpers into `src/main/logseq/cli/command/` namespaces and keep `logseq.cli.commands` as a facade. +- Keep action map shapes stable to avoid downstream changes in format or execution. +- Update tests to match keyword method expectations and new module layout. +- Ensure public CLI output and behavior remain unchanged aside from --repo messaging. + +## Question + +Resolved: Remove --graph entirely and fail fast on any --graph usage. + +--- From 4b7946c3b8a2e9decd292024c1cebababfc5a4b2 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 19 Jan 2026 23:43:20 +0800 Subject: [PATCH 023/375] impl 007-logseq-cli-thread-api-and-command-split.md --- src/main/frontend/worker/db_worker_node.cljs | 75 +- src/main/logseq/cli/command/add.cljs | 147 ++ src/main/logseq/cli/command/core.cljs | 216 +++ src/main/logseq/cli/command/graph.cljs | 226 +++ src/main/logseq/cli/command/list.cljs | 212 +++ src/main/logseq/cli/command/remove.cljs | 81 + src/main/logseq/cli/command/search.cljs | 240 +++ src/main/logseq/cli/command/server.cljs | 96 ++ src/main/logseq/cli/command/show.cljs | 189 +++ src/main/logseq/cli/commands.cljs | 1450 ++--------------- src/main/logseq/cli/format.cljs | 2 +- src/main/logseq/cli/transport.cljs | 9 +- .../frontend/worker/db_worker_node_test.cljs | 16 + src/test/logseq/cli/commands_test.cljs | 33 +- src/test/logseq/cli/format_test.cljs | 2 +- src/test/logseq/cli/transport_test.cljs | 41 +- tmp_scripts/db-worker-smoke-test.clj | 93 -- tmp_scripts/db-worker-sse-smoke-test.clj | 53 - 18 files changed, 1636 insertions(+), 1545 deletions(-) create mode 100644 src/main/logseq/cli/command/add.cljs create mode 100644 src/main/logseq/cli/command/core.cljs create mode 100644 src/main/logseq/cli/command/graph.cljs create mode 100644 src/main/logseq/cli/command/list.cljs create mode 100644 src/main/logseq/cli/command/remove.cljs create mode 100644 src/main/logseq/cli/command/search.cljs create mode 100644 src/main/logseq/cli/command/server.cljs create mode 100644 src/main/logseq/cli/command/show.cljs delete mode 100644 tmp_scripts/db-worker-smoke-test.clj delete mode 100644 tmp_scripts/db-worker-sse-smoke-test.clj diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 7ef7b14b10..368b5fe188 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -68,6 +68,22 @@ payload (ldb/write-transit-str payload))) +(defn- normalize-method-kw + [method] + (cond + (keyword? method) method + (string? method) (keyword method) + (nil? method) nil + :else (keyword (str method)))) + +(defn- normalize-method-str + [method] + (cond + (keyword? method) (subs (str method) 1) + (string? method) method + (nil? method) nil + :else (str method))) + (defn- handle-event! [type payload] (let [event (js/JSON.stringify (clj->js {:type type} @@ -90,7 +106,7 @@ (swap! *sse-clients disj res)))) (defn- (.remoteInvoke proxy method (boolean direct-pass?) args') + (-> (.remoteInvoke proxy method-str (boolean direct-pass?) args') (p/finally (fn [] (js/clearTimeout timeout-id)))))) (defn- (p/let [body (clj payload :keywordize-keys true) + method-kw (normalize-method-kw method) + method-str (normalize-method-str method) direct-pass? (boolean directPass) args' (if direct-pass? args @@ -186,9 +207,9 @@ (if (string? args') (ldb/read-transit-str args') args'))] - (if-let [{:keys [status error]} (repo-error method args-for-validation bound-repo)] + (if-let [{:keys [status error]} (repo-error method-kw args-for-validation bound-repo)] (send-json! res status {:ok false :error error}) - (p/let [result (graph repo) + :page (:page options) + :parent (:parent options) + :blocks (:value vector-result)}})))))) + +(defn build-add-page-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for add"}} + (let [page (some-> (:page options) string/trim)] + (if (seq page) + {:ok? true + :action {:type :add-page + :repo repo + :graph (core/repo->graph repo) + :page page}} + {:ok? false + :error {:code :missing-page-name + :message "page name is required"}})))) + +(defn execute-add-block + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + target-id (resolve-add-target cfg action) + ops [[:insert-blocks [(:blocks action) + target-id + {:sibling? false + :bottom? true + :outliner-op :insert-blocks}]]] + result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])] + {:status :ok + :data {:result result}}))) + +(defn execute-add-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + ops [[:create-page [(:page action) {}]]] + result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])] + {:status :ok + :data {:result result}}))) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs new file mode 100644 index 0000000000..7fa0d7f1c6 --- /dev/null +++ b/src/main/logseq/cli/command/core.cljs @@ -0,0 +1,216 @@ +(ns logseq.cli.command.core + "Shared CLI parsing utilities." + (:require [babashka.cli :as cli] + [clojure.string :as string] + [logseq.common.config :as common-config])) + +(def ^:private global-spec* + {:help {:alias :h + :desc "Show help" + :coerce :boolean} + :config {:desc "Path to cli.edn"} + :auth-token {:desc "Auth token for db-worker-node"} + :repo {:desc "Graph name"} + :data-dir {:desc "Path to db-worker data dir"} + :timeout-ms {:desc "Request timeout in ms" + :coerce :long} + :retries {:desc "Retry count for requests" + :coerce :long} + :output {:desc "Output format (human, json, edn)"}}) + +(defn global-spec + [] + global-spec*) + +(defn- merge-spec + [spec] + (merge global-spec* (or spec {}))) + +(defn command-entry + [cmds command desc spec] + (let [spec* (merge-spec spec)] + {:cmds cmds + :command command + :desc desc + :spec spec* + :restrict true + :fn (fn [{:keys [opts args]}] + {:command command + :cmds cmds + :spec spec* + :opts opts + :args args})})) + +(defn- format-commands + [table] + (let [rows (->> table + (filter (comp seq :cmds)) + (map (fn [{:keys [cmds desc spec]}] + (let [command (str (string/join " " cmds) + (when (seq spec) " [options]"))] + {:command command + :desc desc})))) + width (apply max 0 (map (comp count :command) rows))] + (->> rows + (map (fn [{:keys [command desc]}] + (let [padding (apply str (repeat (- width (count command)) " "))] + (cond-> (str " " command padding) + (seq desc) (str " " desc))))) + (string/join "\n")))) + +(defn group-summary + [group table] + (let [group-table (filter #(= group (first (:cmds %))) table)] + (string/join "\n" + [(str "Usage: logseq " group " [options]") + "" + "Subcommands:" + (format-commands group-table) + "" + "Global options:" + (cli/format-opts {:spec global-spec*}) + "" + "Command options:" + (str " See `logseq " group " --help`")]))) + +(defn top-level-summary + [table] + (let [groups [{:title "Graph Inspect and Edit" + :commands #{"list" "add" "remove" "search" "show"}} + {:title "Graph Management" + :commands #{"graph" "server"}}] + render-group (fn [{:keys [title commands]}] + (let [entries (filter #(contains? commands (first (:cmds %))) table)] + (string/join "\n" [title (format-commands entries)])))] + (string/join "\n" + ["Usage: logseq [options]" + "" + "Commands:" + (string/join "\n\n" (map render-group groups)) + "" + "Global options:" + (cli/format-opts {:spec global-spec*}) + "" + "Command options:" + " See `logseq --help`"]))) + +(defn command-summary + [{:keys [cmds spec]}] + (let [command-spec (apply dissoc spec (keys global-spec*))] + (string/join "\n" + [(str "Usage: logseq " (string/join " " cmds) " [options]") + "" + "Global options:" + (cli/format-opts {:spec global-spec*}) + "" + "Command options:" + (cli/format-opts {:spec command-spec})]))) + +(defn normalize-opts + [opts] + (cond-> opts + (:config opts) (-> (assoc :config-path (:config opts)) + (dissoc :config)))) + +(defn ok-result + [command opts args summary] + {:ok? true + :command command + :options (normalize-opts opts) + :args (vec args) + :summary summary}) + +(defn help-result + [summary] + {:ok? false + :help? true + :summary summary}) + +(defn invalid-options-result + [summary message] + {:ok? false + :error {:code :invalid-options + :message message} + :summary summary}) + +(defn unknown-command-result + [summary message] + {:ok? false + :error {:code :unknown-command + :message message} + :summary summary}) + +(def ^:private global-aliases + (->> global-spec* + (keep (fn [[k {:keys [alias]}]] + (when alias + [alias k]))) + (into {}))) + +(def ^:private global-flag-options + (->> global-spec* + (keep (fn [[k {:keys [coerce]}]] + (when (= coerce :boolean) k))) + (set))) + +(defn- global-opt-key + [token] + (cond + (string/starts-with? token "--") + (keyword (subs token 2)) + + (and (string/starts-with? token "-") + (= 2 (count token))) + (get global-aliases (keyword (subs token 1))) + + :else nil)) + +(defn parse-leading-global-opts + [args] + (loop [remaining args + opts {}] + (if (empty? remaining) + {:opts opts :args []} + (let [token (first remaining)] + (if-let [opt-key (global-opt-key token)] + (if (contains? global-flag-options opt-key) + (recur (rest remaining) (assoc opts opt-key true)) + (if-let [value (second remaining)] + (recur (drop 2 remaining) (assoc opts opt-key value)) + {:opts opts :args (rest remaining)})) + {:opts opts :args remaining}))))) + +(defn legacy-graph-opt? + [raw-args] + (some (fn [token] + (or (= token "--graph") + (string/starts-with? token "--graph="))) + raw-args)) + +(defn cli-error->result + [summary {:keys [msg]}] + (invalid-options-result summary (or msg "invalid options"))) + +(defn graph->repo + [graph] + (when (seq graph) + (if (string/starts-with? graph common-config/db-version-prefix) + graph + (str common-config/db-version-prefix graph)))) + +(defn repo->graph + [repo] + (when (seq repo) + (string/replace-first repo common-config/db-version-prefix ""))) + +(defn resolve-repo + [graph] + (let [graph (some-> graph string/trim)] + (when (seq graph) + (graph->repo graph)))) + +(defn pick-graph + [options command-args config] + (or (:repo options) + (first command-args) + (:repo config))) diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs new file mode 100644 index 0000000000..8a23a08848 --- /dev/null +++ b/src/main/logseq/cli/command/graph.cljs @@ -0,0 +1,226 @@ +(ns logseq.cli.command.graph + "Graph-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.config :as cli-config] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) + +(def ^:private graph-export-spec + {:type {:desc "Export type (edn, sqlite)"} + :output {:desc "Output path"}}) + +(def ^:private graph-import-spec + {:type {:desc "Import type (edn, sqlite)"} + :input {:desc "Input path"}}) + +(def entries + [(core/command-entry ["graph" "list"] :graph-list "List graphs" {}) + (core/command-entry ["graph" "create"] :graph-create "Create graph" {}) + (core/command-entry ["graph" "switch"] :graph-switch "Switch current graph" {}) + (core/command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) + (core/command-entry ["graph" "validate"] :graph-validate "Validate graph" {}) + (core/command-entry ["graph" "info"] :graph-info "Graph metadata" {}) + (core/command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec) + (core/command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec)]) + +(def ^:private import-export-types* + #{"edn" "sqlite"}) + +(defn import-export-types + [] + import-export-types*) + +(defn normalize-import-export-type + [value] + (some-> value string/lower-case string/trim)) + +(defn- missing-graph-error + [] + {:ok? false + :error {:code :missing-graph + :message "graph name is required"}}) + +(defn build-graph-action + [command graph repo] + (case command + :graph-list + {:ok? true + :action {:type :graph-list + :command :graph-list}} + + :graph-create + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :command :graph-create + :method :thread-api/create-or-open-db + :direct-pass? false + :args [repo {}] + :repo repo + :graph (core/repo->graph repo) + :allow-missing-graph true + :persist-repo (core/repo->graph repo)}}) + + :graph-switch + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :graph-switch + :command :graph-switch + :repo repo + :graph (core/repo->graph repo)}}) + + :graph-remove + (if-not (seq graph) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :command :graph-remove + :method :thread-api/unsafe-unlink-db + :direct-pass? false + :args [repo] + :repo repo + :graph (core/repo->graph repo)}}) + + :graph-validate + (if-not (seq repo) + (missing-graph-error) + {:ok? true + :action {:type :invoke + :command :graph-validate + :method :thread-api/validate-db + :direct-pass? false + :args [repo] + :repo repo + :graph (core/repo->graph repo)}}) + + :graph-info + (if-not (seq repo) + (missing-graph-error) + {:ok? true + :action {:type :graph-info + :command :graph-info + :repo repo + :graph (core/repo->graph repo)}}))) + +(defn build-export-action + [repo export-type output] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for export"}} + {:ok? true + :action {:type :graph-export + :repo repo + :graph (core/repo->graph repo) + :export-type export-type + :output output}})) + +(defn build-import-action + [repo import-type input] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for import"}} + {:ok? true + :action {:type :graph-import + :repo repo + :graph (core/repo->graph repo) + :import-type import-type + :input input + :allow-missing-graph true}})) + +(defn execute-graph-list + [_action config] + (let [graphs (cli-server/list-graphs config)] + {:status :ok + :data {:graphs graphs}})) + +(defn execute-invoke + [action config] + (-> (p/let [cfg (if-let [repo (:repo action)] + (cli-server/ensure-server! config repo) + (p/resolved config)) + result (transport/invoke cfg + (:method action) + (:direct-pass? action) + (:args action))] + (when-let [repo (:persist-repo action)] + (cli-config/update-config! config {:repo repo})) + (if-let [write (:write action)] + (let [{:keys [format path]} write] + (transport/write-output {:format format :path path :data result}) + {:status :ok + :data {:message (str "wrote " path)}}) + {:status :ok :data {:result result}})))) + +(defn execute-graph-switch + [action config] + (-> (p/let [graphs (cli-server/list-graphs config) + graph (:graph action)] + (if-not (some #(= graph %) graphs) + {:status :error + :error {:code :graph-not-found + :message (str "graph not found: " graph)}} + (p/let [_ (cli-server/ensure-server! config (:repo action))] + (cli-config/update-config! config {:repo graph}) + {:status :ok + :data {:message (str "switched to " graph)}}))))) + +(defn execute-graph-info + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + created (transport/invoke cfg :thread-api/pull false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) + schema (transport/invoke cfg :thread-api/pull false [(:repo action) [:kv/value] :logseq.kv/schema-version])] + {:status :ok + :data {:graph (:graph action) + :logseq.kv/graph-created-at (:kv/value created) + :logseq.kv/schema-version (:kv/value schema)}}))) + +(defn execute-graph-export + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + export-type (:export-type action) + export-result (case export-type + "edn" + (transport/invoke cfg + :thread-api/export-edn + false + [(:repo action) {:export-type :graph}]) + "sqlite" + (transport/invoke cfg + :thread-api/export-db-base64 + true + [(:repo action)]) + (throw (ex-info "unsupported export type" {:export-type export-type}))) + data (if (= export-type "sqlite") + (js/Buffer.from export-result "base64") + export-result) + format (if (= export-type "sqlite") :sqlite :edn)] + (transport/write-output {:format format :path (:output action) :data data}) + {:status :ok + :data {:message (str "wrote " (:output action))}}))) + +(defn execute-graph-import + [action config] + (-> (p/let [_ (cli-server/stop-server! config (:repo action)) + cfg (cli-server/ensure-server! config (:repo action)) + import-type (:import-type action) + input-data (case import-type + "edn" (transport/read-input {:format :edn :path (:input action)}) + "sqlite" (transport/read-input {:format :sqlite :path (:input action)}) + (throw (ex-info "unsupported import type" {:import-type import-type}))) + payload (if (= import-type "sqlite") + (.toString (js/Buffer.from input-data) "base64") + input-data) + method (if (= import-type "sqlite") + :thread-api/import-db-base64 + :thread-api/import-edn) + direct-pass? (= import-type "sqlite") + _ (transport/invoke cfg method direct-pass? [(:repo action) payload]) + _ (cli-server/restart-server! config (:repo action))] + {:status :ok + :data {:message (str "imported " import-type " from " (:input action))}}))) diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs new file mode 100644 index 0000000000..6053e7e167 --- /dev/null +++ b/src/main/logseq/cli/command/list.cljs @@ -0,0 +1,212 @@ +(ns logseq.cli.command.list + "List-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) + +(def ^:private list-common-spec + {:expand {:desc "Include expanded metadata" + :coerce :boolean} + :limit {:desc "Limit results" + :coerce :long} + :offset {:desc "Offset results" + :coerce :long} + :sort {:desc "Sort field"} + :order {:desc "Sort order (asc, desc)"}}) + +(def ^:private list-page-spec + (merge list-common-spec + {:include-journal {:desc "Include journal pages" + :coerce :boolean} + :journal-only {:desc "Only journal pages" + :coerce :boolean} + :include-hidden {:desc "Include hidden pages" + :coerce :boolean} + :updated-after {:desc "Filter by updated-at (ISO8601)"} + :created-after {:desc "Filter by created-at (ISO8601)"} + :fields {:desc "Select output fields (comma separated)"}})) + +(def ^:private list-tag-spec + (merge list-common-spec + {:include-built-in {:desc "Include built-in tags" + :coerce :boolean} + :with-properties {:desc "Include tag properties" + :coerce :boolean} + :with-extends {:desc "Include tag extends" + :coerce :boolean} + :fields {:desc "Select output fields (comma separated)"}})) + +(def ^:private list-property-spec + (merge list-common-spec + {:include-built-in {:desc "Include built-in properties" + :coerce :boolean} + :with-classes {:desc "Include property classes" + :coerce :boolean} + :with-type {:desc "Include property type" + :coerce :boolean} + :fields {:desc "Select output fields (comma separated)"}})) + +(def entries + [(core/command-entry ["list" "page"] :list-page "List pages" list-page-spec) + (core/command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec) + (core/command-entry ["list" "property"] :list-property "List properties" list-property-spec)]) + +(def ^:private list-sort-fields + {:list-page #{"title" "created-at" "updated-at"} + :list-tag #{"name" "title"} + :list-property #{"name" "title"}}) + +(defn invalid-options? + [command opts] + (let [{:keys [order include-journal journal-only]} opts + sort-field (:sort opts) + allowed (get list-sort-fields command)] + (cond + (and include-journal journal-only) + "include-journal and journal-only are mutually exclusive" + + (and (seq sort-field) (not (contains? allowed sort-field))) + (str "invalid sort field: " sort-field) + + (and (seq order) (not (#{"asc" "desc"} order))) + (str "invalid order: " order) + + :else nil))) + +(defn build-action + [command options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for list"}} + {:ok? true + :action {:type command + :repo repo + :options options}})) + +(def ^:private list-page-field-map + {"title" :block/title + "uuid" :block/uuid + "created-at" :block/created-at + "updated-at" :block/updated-at}) + +(def ^:private list-tag-field-map + {"name" :block/title + "title" :block/title + "uuid" :block/uuid + "properties" :logseq.property.class/properties + "extends" :logseq.property.class/extends + "description" :logseq.property/description}) + +(def ^:private list-property-field-map + {"name" :block/title + "title" :block/title + "uuid" :block/uuid + "classes" :logseq.property/classes + "type" :logseq.property/type + "description" :logseq.property/description}) + +(defn- parse-field-list + [fields] + (when (seq fields) + (->> (string/split fields #",") + (map string/trim) + (remove string/blank?) + vec))) + +(defn- apply-fields + [items fields field-map] + (if (seq fields) + (let [keys (->> fields + (map #(get field-map %)) + (remove nil?) + vec)] + (if (seq keys) + (mapv #(select-keys % keys) items) + items)) + items)) + +(defn- apply-sort + [items sort-field order field-map] + (if (seq sort-field) + (let [sort-key (get field-map sort-field) + sorted (if sort-key + (sort-by #(get % sort-key) items) + items) + sorted (if (= "desc" order) (reverse sorted) sorted)] + (vec sorted)) + (vec items))) + +(defn- apply-offset-limit + [items offset limit] + (cond-> items + (some? offset) (->> (drop offset) vec) + (some? limit) (->> (take limit) vec))) + +(defn- prepare-tag-item + [item {:keys [expand with-properties with-extends]}] + (if expand + (cond-> item + (not with-properties) (dissoc :logseq.property.class/properties) + (not with-extends) (dissoc :logseq.property.class/extends)) + item)) + +(defn- prepare-property-item + [item {:keys [expand with-classes with-type]}] + (if expand + (cond-> item + (not with-classes) (dissoc :logseq.property/classes) + (not with-type) (dissoc :logseq.property/type)) + item)) + +(defn execute-list-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg :thread-api/api-list-pages false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + sorted (apply-sort items (:sort options) order list-page-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-page-field-map) + limited)] + {:status :ok + :data {:items final}}))) + +(defn execute-list-tag + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg :thread-api/api-list-tags false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + prepared (mapv #(prepare-tag-item % options) items) + sorted (apply-sort prepared (:sort options) order list-tag-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-tag-field-map) + limited)] + {:status :ok + :data {:items final}}))) + +(defn execute-list-property + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + options (:options action) + items (transport/invoke cfg :thread-api/api-list-properties false + [(:repo action) options]) + order (or (:order options) "asc") + fields (parse-field-list (:fields options)) + prepared (mapv #(prepare-property-item % options) items) + sorted (apply-sort prepared (:sort options) order list-property-field-map) + limited (apply-offset-limit sorted (:offset options) (:limit options)) + final (if (:expand options) + (apply-fields limited fields list-property-field-map) + limited)] + {:status :ok + :data {:items final}}))) diff --git a/src/main/logseq/cli/command/remove.cljs b/src/main/logseq/cli/command/remove.cljs new file mode 100644 index 0000000000..c2ec7df163 --- /dev/null +++ b/src/main/logseq/cli/command/remove.cljs @@ -0,0 +1,81 @@ +(ns logseq.cli.command.remove + "Remove-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] + [promesa.core :as p])) + +(def ^:private remove-block-spec + {:block {:desc "Block UUID"}}) + +(def ^:private remove-page-spec + {:page {:desc "Page name"}}) + +(def entries + [(core/command-entry ["remove" "block"] :remove-block "Remove block" remove-block-spec) + (core/command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec)]) + +(defn- perform-remove + [config {:keys [repo block page]}] + (cond + (seq block) + (if-not (common-util/uuid-string? block) + (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] [:block/uuid (uuid block)]])] + (if-let [id (:db/id entity)] + (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:delete-blocks [[id] {}]]] {}]) + (throw (ex-info "block not found" {:code :block-not-found}))))) + + (seq page) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] [:block/name page]])] + (if-let [page-uuid (:block/uuid entity)] + (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:delete-page [page-uuid]]] {}]) + (throw (ex-info "page not found" {:code :page-not-found})))) + + :else + (p/rejected (ex-info "block or page required" {:code :missing-target})))) + +(defn build-remove-block-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for remove"}} + (let [block (some-> (:block options) string/trim)] + (if (seq block) + {:ok? true + :action {:type :remove-block + :repo repo + :block block}} + {:ok? false + :error {:code :missing-target + :message "block is required"}})))) + +(defn build-remove-page-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for remove"}} + (let [page (some-> (:page options) string/trim)] + (if (seq page) + {:ok? true + :action {:type :remove-page + :repo repo + :page page}} + {:ok? false + :error {:code :missing-target + :message "page is required"}})))) + +(defn execute-remove + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + result (perform-remove cfg action)] + {:status :ok + :data {:result result}}))) diff --git a/src/main/logseq/cli/command/search.cljs b/src/main/logseq/cli/command/search.cljs new file mode 100644 index 0000000000..074422324a --- /dev/null +++ b/src/main/logseq/cli/command/search.cljs @@ -0,0 +1,240 @@ +(ns logseq.cli.command.search + "Search-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) + +(def ^:private search-spec + {:text {:desc "Search text"} + :type {:desc "Search types (page, block, tag, property, all)"} + :tag {:desc "Restrict to a specific tag"} + :limit {:desc "Limit results" + :coerce :long} + :case-sensitive {:desc "Case sensitive search" + :coerce :boolean} + :include-content {:desc "Search block content" + :coerce :boolean} + :sort {:desc "Sort field (updated-at, created-at)"} + :order {:desc "Sort order (asc, desc)"}}) + +(def entries + [(core/command-entry ["search"] :search "Search graph" search-spec)]) + +(def ^:private search-types + #{"page" "block" "tag" "property" "all"}) + +(defn invalid-options? + [opts] + (let [type (:type opts) + order (:order opts) + sort-field (:sort opts)] + (cond + (and (seq type) (not (contains? search-types type))) + (str "invalid type: " type) + + (and (seq sort-field) (not (#{"updated-at" "created-at"} sort-field))) + (str "invalid sort field: " sort-field) + + (and (seq order) (not (#{"asc" "desc"} order))) + (str "invalid order: " order) + + :else + nil))) + +(defn build-action + [options args repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for search"}} + (let [text (or (:text options) (string/join " " args))] + (if (seq text) + {:ok? true + :action {:type :search + :repo repo + :text text + :search-type (:type options) + :tag (:tag options) + :limit (:limit options) + :case-sensitive (:case-sensitive options) + :include-content (:include-content options) + :sort (:sort options) + :order (:order options)}} + {:ok? false + :error {:code :missing-search-text + :message "search text is required"}})))) + +(defn- query-pages + [cfg repo text case-sensitive?] + (let [query (if case-sensitive? + '[:find ?e ?title ?uuid ?updated ?created + :in $ ?q + :where + [?e :block/name ?name] + [?e :block/title ?title] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? ?title ?q)]] + '[:find ?e ?title ?uuid ?updated ?created + :in $ ?q + :where + [?e :block/name ?name] + [?e :block/title ?title] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? (clojure.string/lower-case ?title) ?q)]]) + q* (if case-sensitive? text (string/lower-case text))] + (transport/invoke cfg :thread-api/q false [repo [query q*]]))) + +#_{:clj-kondo/ignore [:aliased-namespace-symbol]} +(defn- query-blocks + [cfg repo text case-sensitive? tag include-content?] + (let [has-tag? (seq tag) + content-attr (if include-content? :block/content :block/title) + query (cond + (and case-sensitive? has-tag?) + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q ?tag-name + :where + [?tag :block/name ?tag-name] + [?e :block/tags ?tag] + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? ?value ?q)]] + + case-sensitive? + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q + :where + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? ?value ?q)]] + + has-tag? + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q ?tag-name + :where + [?tag :block/name ?tag-name] + [?e :block/tags ?tag] + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]] + + :else + `[:find ?e ?value ?uuid ?updated ?created + :in $ ?q + :where + [?e ~content-attr ?value] + [(missing? $ ?e :block/name)] + [?e :block/uuid ?uuid] + [(get-else $ ?e :block/updated-at 0) ?updated] + [(get-else $ ?e :block/created-at 0) ?created] + [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]]) + q* (if case-sensitive? text (string/lower-case text)) + tag-name (some-> tag string/lower-case)] + (if has-tag? + (transport/invoke cfg :thread-api/q false [repo [query q* tag-name]]) + (transport/invoke cfg :thread-api/q false [repo [query q*]])))) + +(defn- normalize-search-types + [type] + (let [type (or type "all")] + (case type + "page" [:page] + "block" [:block] + "tag" [:tag] + "property" [:property] + [:page :block :tag :property]))) + +(defn- search-sort-key + [item sort-field] + (case sort-field + "updated-at" (:updated-at item) + "created-at" (:created-at item) + nil)) + +(defn execute-search + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + types (normalize-search-types (:search-type action)) + case-sensitive? (boolean (:case-sensitive action)) + text (:text action) + tag (:tag action) + page-results (when (some #{:page} types) + (p/let [rows (query-pages cfg (:repo action) text case-sensitive?)] + (mapv (fn [[id title uuid updated created]] + {:type "page" + :db/id id + :title title + :uuid (str uuid) + :updated-at updated + :created-at created}) + rows))) + include-content? (boolean (:include-content action)) + block-results (when (some #{:block} types) + (p/let [rows (query-blocks cfg (:repo action) text case-sensitive? tag include-content?)] + (mapv (fn [[id content uuid updated created]] + {:type "block" + :db/id id + :content content + :uuid (str uuid) + :updated-at updated + :created-at created}) + rows))) + tag-results (when (some #{:tag} types) + (p/let [items (transport/invoke cfg :thread-api/api-list-tags false + [(:repo action) {:expand true :include-built-in true}]) + q* (if case-sensitive? text (string/lower-case text))] + (->> items + (filter (fn [item] + (let [title (:block/title item)] + (if case-sensitive? + (string/includes? title q*) + (string/includes? (string/lower-case title) q*))))) + (mapv (fn [item] + {:type "tag" + :title (:block/title item) + :uuid (:block/uuid item)}))))) + property-results (when (some #{:property} types) + (p/let [items (transport/invoke cfg :thread-api/api-list-properties false + [(:repo action) {:expand true :include-built-in true}]) + q* (if case-sensitive? text (string/lower-case text))] + (->> items + (filter (fn [item] + (let [title (:block/title item)] + (if case-sensitive? + (string/includes? title q*) + (string/includes? (string/lower-case title) q*))))) + (mapv (fn [item] + {:type "property" + :title (:block/title item) + :uuid (:block/uuid item)}))))) + results (->> (concat (or page-results []) + (or block-results []) + (or tag-results []) + (or property-results [])) + (distinct) + vec) + sorted (if-let [sort-field (:sort action)] + (let [order (or (:order action) "desc")] + (->> results + (sort-by #(search-sort-key % sort-field)) + (cond-> (= order "desc") reverse) + vec)) + results) + limited (if (some? (:limit action)) (vec (take (:limit action) sorted)) sorted)] + {:status :ok + :data {:results limited}}))) diff --git a/src/main/logseq/cli/command/server.cljs b/src/main/logseq/cli/command/server.cljs new file mode 100644 index 0000000000..2d37477988 --- /dev/null +++ b/src/main/logseq/cli/command/server.cljs @@ -0,0 +1,96 @@ +(ns logseq.cli.command.server + "Server-related CLI commands." + (:require [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [promesa.core :as p])) + +(def ^:private server-spec + {:repo {:desc "Graph name"}}) + +(def entries + [(core/command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) + (core/command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec) + (core/command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) + (core/command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec) + (core/command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec)]) + +(defn build-action + [command repo] + (case command + :server-list + {:ok? true + :action {:type :server-list}} + + :server-status + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for server status"}} + {:ok? true + :action {:type :server-status + :repo repo}}) + + :server-start + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for server start"}} + {:ok? true + :action {:type :server-start + :repo repo}}) + + :server-stop + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for server stop"}} + {:ok? true + :action {:type :server-stop + :repo repo}}) + + :server-restart + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for server restart"}} + {:ok? true + :action {:type :server-restart + :repo repo}}) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown server command: " command)}})) + +(defn- server-result->response + [result] + (if (:ok? result) + {:status :ok + :data (:data result)} + {:status :error + :error (:error result)})) + +(defn execute-list + [_action config] + (-> (p/let [servers (cli-server/list-servers config)] + {:status :ok + :data {:servers servers}}))) + +(defn execute-status + [action config] + (-> (p/let [result (cli-server/server-status config (:repo action))] + (server-result->response result)))) + +(defn execute-start + [action config] + (-> (p/let [result (cli-server/start-server! config (:repo action))] + (server-result->response result)))) + +(defn execute-stop + [action config] + (-> (p/let [result (cli-server/stop-server! config (:repo action))] + (server-result->response result)))) + +(defn execute-restart + [action config] + (-> (p/let [result (cli-server/restart-server! config (:repo action))] + (server-result->response result)))) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs new file mode 100644 index 0000000000..85ed029851 --- /dev/null +++ b/src/main/logseq/cli/command/show.cljs @@ -0,0 +1,189 @@ +(ns logseq.cli.command.show + "Show-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] + [promesa.core :as p])) + +(def ^:private show-spec + {:id {:desc "Block db/id" + :coerce :long} + :uuid {:desc "Block UUID"} + :page-name {:desc "Page name"} + :level {:desc "Limit tree depth" + :coerce :long} + :format {:desc "Output format (text, json, edn)"}}) + +(def entries + [(core/command-entry ["show"] :show "Show tree" show-spec)]) + +(def ^:private show-formats + #{"text" "json" "edn"}) + +(defn invalid-options? + [opts] + (let [format (:format opts) + level (:level opts)] + (cond + (and (seq format) (not (contains? show-formats (string/lower-case format)))) + (str "invalid format: " format) + + (and (some? level) (< level 1)) + "level must be >= 1" + + :else + nil))) + +(def ^:private tree-block-selector + [:db/id :block/uuid :block/title :block/order {:block/parent [:db/id]}]) + +(defn- fetch-blocks-for-page + [config repo page-id] + (let [query [:find (list 'pull '?b tree-block-selector) + :in '$ '?page-id + :where ['?b :block/page '?page-id]]] + (p/let [rows (transport/invoke config :thread-api/q false [repo [query page-id]])] + (mapv first rows)))) + +(defn- build-tree + [blocks root-id max-depth] + (let [parent->children (group-by #(get-in % [:block/parent :db/id]) blocks) + sort-children (fn [children] + (vec (sort-by :block/order children))) + build (fn build [parent-id depth] + (mapv (fn [b] + (let [children (build (:db/id b) (inc depth))] + (cond-> b + (seq children) (assoc :block/children children)))) + (if (and max-depth (>= depth max-depth)) + [] + (sort-children (get parent->children parent-id)))))] + (build root-id 1))) + +(defn- fetch-tree + [config {:keys [repo id page-name level] :as opts}] + (let [max-depth (or level 10) + uuid-str (:uuid opts)] + (cond + (some? id) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (if (:db/id entity) + (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found}))))) + + (seq uuid-str) + (if-not (common-util/uuid-string? uuid-str) + (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid (uuid uuid-str)]]) + entity (if (:db/id entity) + entity + (transport/invoke config :thread-api/pull false + [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [:block/uuid uuid-str]]))] + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (if (:db/id entity) + (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "block not found" {:code :block-not-found})))))) + + (seq page-name) + (p/let [page-entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/title] [:block/name page-name]])] + (if-let [page-id (:db/id page-entity)] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks page-id max-depth)] + {:root (assoc page-entity :block/children children)}) + (throw (ex-info "page not found" {:code :page-not-found})))) + + :else + (p/rejected (ex-info "block or page required" {:code :missing-target}))))) + +(defn tree->text + [{:keys [root]}] + (let [label (fn [node] + (or (:block/title node) (:block/name node) (str (:block/uuid node)))) + node-id (fn [node] + (or (:db/id node) "-")) + id-padding (fn [node] + (apply str (repeat (inc (count (str (node-id node)))) " "))) + split-lines (fn [value] + (string/split (or value "") #"\n")) + lines (atom []) + walk (fn walk [node prefix] + (let [children (:block/children node) + total (count children)] + (doseq [[idx child] (map-indexed vector children)] + (let [last-child? (= idx (dec total)) + branch (if last-child? "└── " "├── ") + next-prefix (str prefix (if last-child? " " "│ ")) + rows (split-lines (label child)) + first-row (first rows) + rest-rows (rest rows) + line (str (node-id child) " " prefix branch first-row)] + (swap! lines conj line) + (doseq [row rest-rows] + (swap! lines conj (str (id-padding child) next-prefix row))) + (walk child next-prefix)))))] + (let [rows (split-lines (label root)) + first-row (first rows) + rest-rows (rest rows)] + (swap! lines conj (str (node-id root) " " first-row)) + (doseq [row rest-rows] + (swap! lines conj (str (id-padding root) row)))) + (walk root "") + (string/join "\n" @lines))) + +(defn build-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for show"}} + (let [format (some-> (:format options) string/lower-case) + targets (filter some? [(:id options) (:uuid options) (:page-name options)])] + (if (empty? targets) + {:ok? false + :error {:code :missing-target + :message "block or page is required"}} + {:ok? true + :action {:type :show + :repo repo + :id (:id options) + :uuid (:uuid options) + :page-name (:page-name options) + :level (:level options) + :format format}})))) + +(defn execute-show + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + tree-data (fetch-tree cfg action) + format (:format action)] + (case format + "edn" + {:status :ok + :data tree-data + :output-format :edn} + + "json" + {:status :ok + :data tree-data + :output-format :json} + + {:status :ok + :data {:message (tree->text tree-data)}})))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 2f61fae6ff..bff5d14b9a 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -1,205 +1,22 @@ (ns logseq.cli.commands "Command parsing and action building for the Logseq CLI." - (:require ["fs" :as fs] - [babashka.cli :as cli] - [cljs-time.coerce :as tc] - [cljs.reader :as reader] + (:require [babashka.cli :as cli] [clojure.string :as string] - [logseq.cli.config :as cli-config] + [logseq.cli.command.add :as add-command] + [logseq.cli.command.core :as command-core] + [logseq.cli.command.graph :as graph-command] + [logseq.cli.command.list :as list-command] + [logseq.cli.command.remove :as remove-command] + [logseq.cli.command.search :as search-command] + [logseq.cli.command.server :as server-command] + [logseq.cli.command.show :as show-command] [logseq.cli.server :as cli-server] - [logseq.cli.transport :as transport] - [logseq.common.config :as common-config] - [logseq.common.util :as common-util] - [logseq.common.util.date-time :as date-time-util] [promesa.core :as p])) (def ^:private global-spec - {:help {:alias :h - :desc "Show help" - :coerce :boolean} - :config {:desc "Path to cli.edn"} - :auth-token {:desc "Auth token for db-worker-node"} - :repo {:desc "Graph name"} - :data-dir {:desc "Path to db-worker data dir"} - :timeout-ms {:desc "Request timeout in ms" - :coerce :long} - :retries {:desc "Retry count for requests" - :coerce :long} - :output {:desc "Output format (human, json, edn)"}}) + (command-core/global-spec)) -(def ^:private server-spec - {:repo {:desc "Graph name"}}) - -(def ^:private content-add-spec - {:content {:desc "Block content for add"} - :blocks {:desc "EDN vector of blocks for add"} - :blocks-file {:desc "EDN file of blocks for add"} - :page {:desc "Page name"} - :parent {:desc "Parent block UUID for add"}}) - -(def ^:private add-page-spec - {:page {:desc "Page name"}}) - -(def ^:private remove-block-spec - {:block {:desc "Block UUID"}}) - -(def ^:private remove-page-spec - {:page {:desc "Page name"}}) - -(def ^:private list-common-spec - {:expand {:desc "Include expanded metadata" - :coerce :boolean} - :limit {:desc "Limit results" - :coerce :long} - :offset {:desc "Offset results" - :coerce :long} - :sort {:desc "Sort field"} - :order {:desc "Sort order (asc, desc)"}}) - -(def ^:private list-page-spec - (merge list-common-spec - {:include-journal {:desc "Include journal pages" - :coerce :boolean} - :journal-only {:desc "Only journal pages" - :coerce :boolean} - :include-hidden {:desc "Include hidden pages" - :coerce :boolean} - :updated-after {:desc "Filter by updated-at (ISO8601)"} - :created-after {:desc "Filter by created-at (ISO8601)"} - :fields {:desc "Select output fields (comma separated)"}})) - -(def ^:private list-tag-spec - (merge list-common-spec - {:include-built-in {:desc "Include built-in tags" - :coerce :boolean} - :with-properties {:desc "Include tag properties" - :coerce :boolean} - :with-extends {:desc "Include tag extends" - :coerce :boolean} - :fields {:desc "Select output fields (comma separated)"}})) - -(def ^:private list-property-spec - (merge list-common-spec - {:include-built-in {:desc "Include built-in properties" - :coerce :boolean} - :with-classes {:desc "Include property classes" - :coerce :boolean} - :with-type {:desc "Include property type" - :coerce :boolean} - :fields {:desc "Select output fields (comma separated)"}})) - -(def ^:private search-spec - {:text {:desc "Search text"} - :type {:desc "Search types (page, block, tag, property, all)"} - :tag {:desc "Restrict to a specific tag"} - :limit {:desc "Limit results" - :coerce :long} - :case-sensitive {:desc "Case sensitive search" - :coerce :boolean} - :include-content {:desc "Search block content" - :coerce :boolean} - :sort {:desc "Sort field (updated-at, created-at)"} - :order {:desc "Sort order (asc, desc)"}}) - -(def ^:private show-spec - {:id {:desc "Block db/id" - :coerce :long} - :uuid {:desc "Block UUID"} - :page-name {:desc "Page name"} - :level {:desc "Limit tree depth" - :coerce :long} - :format {:desc "Output format (text, json, edn)"}}) - -(def ^:private graph-export-spec - {:type {:desc "Export type (edn, sqlite)"} - :output {:desc "Output path"}}) - -(def ^:private graph-import-spec - {:type {:desc "Import type (edn, sqlite)"} - :input {:desc "Input path"}}) - -(defn- format-commands - [table] - (let [rows (->> table - (filter (comp seq :cmds)) - (map (fn [{:keys [cmds desc spec]}] - (let [command (str (string/join " " cmds) - (when (seq spec) " [options]"))] - {:command command - :desc desc})))) - width (apply max 0 (map (comp count :command) rows))] - (->> rows - (map (fn [{:keys [command desc]}] - (let [padding (apply str (repeat (- width (count command)) " "))] - (cond-> (str " " command padding) - (seq desc) (str " " desc))))) - (string/join "\n")))) - -(defn- group-summary - [group table] - (let [group-table (filter #(= group (first (:cmds %))) table)] - (string/join "\n" - [(str "Usage: logseq " group " [options]") - "" - "Subcommands:" - (format-commands group-table) - "" - "Global options:" - (cli/format-opts {:spec global-spec}) - "" - "Command options:" - (str " See `logseq " group " --help`")]))) - -(defn- top-level-summary - [table] - (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "search" "show"}} - {:title "Graph Management" - :commands #{"graph" "server"}}] - render-group (fn [{:keys [title commands]}] - (let [entries (filter #(contains? commands (first (:cmds %))) table)] - (string/join "\n" [title (format-commands entries)])))] - (string/join "\n" - ["Usage: logseq [options]" - "" - "Commands:" - (string/join "\n\n" (map render-group groups)) - "" - "Global options:" - (cli/format-opts {:spec global-spec}) - "" - "Command options:" - " See `logseq --help`"]))) - -(defn- command-summary - [{:keys [cmds spec]}] - (let [command-spec (apply dissoc spec (keys global-spec))] - (string/join "\n" - [(str "Usage: logseq " (string/join " " cmds) " [options]") - "" - "Global options:" - (cli/format-opts {:spec global-spec}) - "" - "Command options:" - (cli/format-opts {:spec command-spec})]))) - -(defn- merge-spec - [spec] - (merge global-spec (or spec {}))) - -(defn- normalize-opts - [opts] - (cond-> opts - (:config opts) (-> (assoc :config-path (:config opts)) - (dissoc :config)))) - -(defn- ok-result - [command opts args summary] - {:ok? true - :command command - :options (normalize-opts opts) - :args (vec args) - :summary summary}) +;; Parsing helpers and summaries are in logseq.cli.command.core. (defn- missing-graph-result [summary] @@ -264,171 +81,20 @@ :message "search text is required"} :summary summary}) -(defn- help-result - [summary] - {:ok? false - :help? true - :summary summary}) +;; Error helpers are in logseq.cli.command.core. -(defn- invalid-options-result - [summary message] - {:ok? false - :error {:code :invalid-options - :message message} - :summary summary}) - -(defn- unknown-command-result - [summary message] - {:ok? false - :error {:code :unknown-command - :message message} - :summary summary}) - -(def ^:private list-sort-fields - {:list-page #{"title" "created-at" "updated-at"} - :list-tag #{"name" "title"} - :list-property #{"name" "title"}}) - -(def ^:private show-formats - #{"text" "json" "edn"}) - -(def ^:private search-types - #{"page" "block" "tag" "property" "all"}) - -(def ^:private import-export-types - #{"edn" "sqlite"}) - -(defn- normalize-import-export-type - [value] - (some-> value string/lower-case string/trim)) - -(defn- invalid-list-options? - [command opts] - (let [{:keys [order include-journal journal-only]} opts - sort-field (:sort opts) - allowed (get list-sort-fields command)] - (cond - (and include-journal journal-only) - "include-journal and journal-only are mutually exclusive" - - (and (seq sort-field) (not (contains? allowed sort-field))) - (str "invalid sort field: " sort-field) - - (and (seq order) (not (#{"asc" "desc"} order))) - (str "invalid order: " order) - - :else - nil))) - -(defn- invalid-show-options? - [opts] - (let [format (:format opts) - level (:level opts)] - (cond - (and (seq format) (not (contains? show-formats (string/lower-case format)))) - (str "invalid format: " format) - - (and (some? level) (< level 1)) - "level must be >= 1" - - :else - nil))) - -(defn- invalid-search-options? - [opts] - (let [type (:type opts) - order (:order opts) - sort-field (:sort opts)] - (cond - (and (seq type) (not (contains? search-types type))) - (str "invalid type: " type) - - (and (seq sort-field) (not (#{"updated-at" "created-at"} sort-field))) - (str "invalid sort field: " sort-field) - - (and (seq order) (not (#{"asc" "desc"} order))) - (str "invalid order: " order) - - :else - nil))) - -(defn- command-entry - [cmds command desc spec] - (let [spec* (merge-spec spec)] - {:cmds cmds - :desc desc - :spec spec* - :restrict true - :fn (fn [{:keys [opts args]}] - {:command command - :cmds cmds - :spec spec* - :opts opts - :args args})})) +;; Command-specific validation and entries are in subcommand namespaces. (def ^:private table - [(command-entry ["graph" "list"] :graph-list "List graphs" {}) - (command-entry ["graph" "create"] :graph-create "Create graph" {}) - (command-entry ["graph" "switch"] :graph-switch "Switch current graph" {}) - (command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) - (command-entry ["graph" "validate"] :graph-validate "Validate graph" {}) - (command-entry ["graph" "info"] :graph-info "Graph metadata" {}) - (command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec) - (command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec) - (command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) - (command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec) - (command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) - (command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec) - (command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec) - (command-entry ["list" "page"] :list-page "List pages" list-page-spec) - (command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec) - (command-entry ["list" "property"] :list-property "List properties" list-property-spec) - (command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) - (command-entry ["add" "page"] :add-page "Create page" add-page-spec) - (command-entry ["remove" "block"] :remove-block "Remove block" remove-block-spec) - (command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec) - (command-entry ["search"] :search "Search graph" search-spec) - (command-entry ["show"] :show "Show tree" show-spec)]) + (vec (concat graph-command/entries + server-command/entries + list-command/entries + add-command/entries + remove-command/entries + search-command/entries + show-command/entries))) -(def ^:private global-aliases - (->> global-spec - (keep (fn [[k {:keys [alias]}]] - (when alias - [alias k]))) - (into {}))) - -(def ^:private global-flag-options - (->> global-spec - (keep (fn [[k {:keys [coerce]}]] - (when (= coerce :boolean) k))) - (set))) - -(defn- global-opt-key - [token] - (cond - (string/starts-with? token "--") - (keyword (subs token 2)) - - (and (string/starts-with? token "-") - (= 2 (count token))) - (get global-aliases (keyword (subs token 1))) - - :else nil)) - -(defn- parse-leading-global-opts - [args] - (loop [remaining args - opts {}] - (if (empty? remaining) - {:opts opts :args []} - (let [token (first remaining)] - (if-let [opt-key (global-opt-key token)] - (if (contains? global-flag-options opt-key) - (recur (rest remaining) (assoc opts opt-key true)) - (if-let [value (second remaining)] - (recur (drop 2 remaining) (assoc opts opt-key value)) - {:opts opts :args (rest remaining)})) - {:opts opts :args remaining}))))) +;; Global option parsing lives in logseq.cli.command.core. (defn- unknown-command-message [{:keys [dispatch wrong-input]}] @@ -437,9 +103,9 @@ (defn- finalize-command [summary {:keys [command opts args cmds spec]}] - (let [opts (normalize-opts opts) + (let [opts (command-core/normalize-opts opts) args (vec args) - cmd-summary (command-summary {:cmds cmds :spec spec}) + cmd-summary (command-core/command-summary {:cmds cmds :spec spec}) graph (:repo opts) has-args? (seq args) has-content? (or (seq (:content opts)) @@ -449,7 +115,7 @@ show-targets (filter some? [(:id opts) (:uuid opts) (:page-name opts)])] (cond (:help opts) - (help-result cmd-summary) + (command-core/help-result cmd-summary) (and (#{:graph-create :graph-switch :graph-remove :graph-validate} command) (not (seq graph))) @@ -471,32 +137,33 @@ (missing-target-result summary) (and (= command :show) (> (count show-targets) 1)) - (invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") + (command-core/invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") (and (= command :search) (not (or (seq (:text opts)) has-args?))) (missing-search-result summary) (and (#{:list-page :list-tag :list-property} command) - (invalid-list-options? command opts)) - (invalid-options-result summary (invalid-list-options? command opts)) + (list-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (list-command/invalid-options? command opts)) - (and (= command :show) (invalid-show-options? opts)) - (invalid-options-result summary (invalid-show-options? opts)) + (and (= command :show) (show-command/invalid-options? opts)) + (command-core/invalid-options-result summary (show-command/invalid-options? opts)) - (and (= command :search) (invalid-search-options? opts)) - (invalid-options-result summary (invalid-search-options? opts)) + (and (= command :search) (search-command/invalid-options? opts)) + (command-core/invalid-options-result summary (search-command/invalid-options? opts)) - (and (= command :graph-export) (not (seq (normalize-import-export-type (:type opts))))) + (and (= command :graph-export) (not (seq (graph-command/normalize-import-export-type (:type opts))))) (missing-type-result summary) (and (= command :graph-export) (not (seq (:output opts)))) (missing-output-result summary) (and (= command :graph-export) - (not (contains? import-export-types (normalize-import-export-type (:type opts))))) - (invalid-options-result summary (str "invalid type: " (:type opts))) + (not (contains? (graph-command/import-export-types) + (graph-command/normalize-import-export-type (:type opts))))) + (command-core/invalid-options-result summary (str "invalid type: " (:type opts))) - (and (= command :graph-import) (not (seq (normalize-import-export-type (:type opts))))) + (and (= command :graph-import) (not (seq (graph-command/normalize-import-export-type (:type opts))))) (missing-type-result summary) (and (= command :graph-import) (not (seq (:input opts)))) @@ -506,75 +173,67 @@ (missing-repo-result summary) (and (= command :graph-import) - (not (contains? import-export-types (normalize-import-export-type (:type opts))))) - (invalid-options-result summary (str "invalid type: " (:type opts))) + (not (contains? (graph-command/import-export-types) + (graph-command/normalize-import-export-type (:type opts))))) + (command-core/invalid-options-result summary (str "invalid type: " (:type opts))) (and (#{:server-status :server-start :server-stop :server-restart} command) (not (seq (:repo opts)))) (missing-repo-result summary) :else - (ok-result command opts args summary)))) + (command-core/ok-result command opts args summary)))) -(defn- cli-error->result - [summary {:keys [msg]}] - (invalid-options-result summary (or msg "invalid options"))) +;; CLI error handling is in logseq.cli.command.core. (defn parse-args [raw-args] - (let [summary (top-level-summary table) - {:keys [opts args]} (parse-leading-global-opts raw-args)] + (let [summary (command-core/top-level-summary table) + legacy-graph-opt? (command-core/legacy-graph-opt? raw-args) + {:keys [opts args]} (command-core/parse-leading-global-opts raw-args)] + (if legacy-graph-opt? + (command-core/invalid-options-result summary "unknown option: --graph") (if (empty? args) (if (:help opts) - (help-result summary) + (command-core/help-result summary) {:ok? false :error {:code :missing-command :message "missing command"} - :summary summary}) + :summary summary}) (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "remove"} (first args))) - (help-result (group-summary (first args) table)) + (command-core/help-result (command-core/group-summary (first args) table)) (try (let [result (cli/dispatch table args {:spec global-spec})] (if (nil? result) - (unknown-command-result summary (str "unknown command: " (string/join " " args))) + (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args))) (finalize-command summary (update result :opts #(merge opts (or % {})))))) (catch :default e (let [{:keys [cause] :as data} (ex-data e)] (cond (= cause :input-exhausted) (if (:help opts) - (help-result summary) + (command-core/help-result summary) {:ok? false :error {:code :missing-command :message "missing command"} :summary summary}) (= cause :no-match) - (unknown-command-result summary (str "unknown command: " (unknown-command-message data))) + (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) (some? data) - (cli-error->result summary data) + (command-core/cli-error->result summary data) :else - (unknown-command-result summary (str "unknown command: " (string/join " " args))))))))))) + (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args)))))))))))) -(defn- graph->repo - [graph] - (when (seq graph) - (if (string/starts-with? graph common-config/db-version-prefix) - graph - (str common-config/db-version-prefix graph)))) - -(defn- repo->graph - [repo] - (when (seq repo) - (string/replace-first repo common-config/db-version-prefix ""))) +;; Repo/graph helpers live in logseq.cli.command.core. (defn- ensure-existing-graph [action config] (if (and (:repo action) (not (:allow-missing-graph action))) (p/let [graphs (cli-server/list-graphs config) - graph (repo->graph (:repo action))] + graph (command-core/repo->graph (:repo action))] (if (some #(= graph %) graphs) {:ok? true} {:ok? false @@ -586,7 +245,7 @@ [action config] (if (and (= :graph-import (:type action)) (:repo action)) (p/let [graphs (cli-server/list-graphs config) - graph (repo->graph (:repo action))] + graph (command-core/repo->graph (:repo action))] (if (some #(= graph %) graphs) {:ok? false :error {:code :graph-exists @@ -594,978 +253,71 @@ {:ok? true})) (p/resolved {:ok? true}))) -(defn- pick-graph - [options command-args config] - (or (:repo options) - (first command-args) - (:repo config))) +;; Repo selection lives in logseq.cli.command.core. -(defn- read-blocks - [options command-args] - (cond - (seq (:blocks options)) - {:ok? true :value (reader/read-string (:blocks options))} +;; Block parsing lives in logseq.cli.command.add. - (seq (:blocks-file options)) - (let [contents (.toString (fs/readFileSync (:blocks-file options)) "utf8")] - {:ok? true :value (reader/read-string contents)}) +;; Add/remove helpers live in logseq.cli.command.add/remove. - (seq (:content options)) - {:ok? true :value [{:block/title (:content options)}]} +;; Show helpers live in logseq.cli.command.show. - (seq command-args) - {:ok? true :value [{:block/title (string/join " " command-args)}]} - :else - {:ok? false - :error {:code :missing-content - :message "content is required"}})) -(defn- ensure-blocks - [value] - (if (vector? value) - {:ok? true :value value} - {:ok? false - :error {:code :invalid-blocks - :message "blocks must be a vector"}})) +;; Show helpers live in logseq.cli.command.show. -(defn- today-page-title - [config repo] - (p/let [journal (transport/invoke config "thread-api/pull" false - [repo [:logseq.property.journal/title-format] :logseq.class/Journal]) - formatter (or (:logseq.property.journal/title-format journal) "MMM do, yyyy") - now (tc/from-date (js/Date.))] - (date-time-util/format now formatter))) +;; Repo normalization lives in logseq.cli.command.core. -(defn- ensure-page! - [config repo page-name] - (p/let [page (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]])] - (if (:db/id page) - page - (p/let [_ (transport/invoke config "thread-api/apply-outliner-ops" false - [repo [[:create-page [page-name {}]]] {}])] - (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]]))))) - -(defn- resolve-add-target - [config {:keys [repo page parent]}] - (if (seq parent) - (if-not (common-util/uuid-string? parent) - (p/rejected (ex-info "parent must be a uuid" {:code :invalid-parent})) - (p/let [block (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title] [:block/uuid (uuid parent)]])] - (if-let [id (:db/id block)] - id - (throw (ex-info "parent block not found" {:code :parent-not-found}))))) - (p/let [page-name (if (seq page) page (today-page-title config repo)) - page-entity (ensure-page! config repo page-name)] - (or (:db/id page-entity) - (throw (ex-info "page not found" {:code :page-not-found})))))) - -(defn- perform-remove - [config {:keys [repo block page]}] - (cond - (seq block) - (if-not (common-util/uuid-string? block) - (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) - (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid] [:block/uuid (uuid block)]])] - (if-let [id (:db/id entity)] - (transport/invoke config "thread-api/apply-outliner-ops" false - [repo [[:delete-blocks [[id] {}]]] {}]) - (throw (ex-info "block not found" {:code :block-not-found}))))) - - (seq page) - (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid] [:block/name page]])] - (if-let [page-uuid (:block/uuid entity)] - (transport/invoke config "thread-api/apply-outliner-ops" false - [repo [[:delete-page [page-uuid]]] {}]) - (throw (ex-info "page not found" {:code :page-not-found})))) - - :else - (p/rejected (ex-info "block or page required" {:code :missing-target})))) - -(def ^:private tree-block-selector - [:db/id :block/uuid :block/title :block/order {:block/parent [:db/id]}]) - -(defn- fetch-blocks-for-page - [config repo page-id] - (let [query [:find (list 'pull '?b tree-block-selector) - :in '$ '?page-id - :where ['?b :block/page '?page-id]]] - (p/let [rows (transport/invoke config "thread-api/q" false [repo [query page-id]])] - (mapv first rows)))) - -(defn- build-tree - [blocks root-id max-depth] - (let [parent->children (group-by #(get-in % [:block/parent :db/id]) blocks) - sort-children (fn [children] - (vec (sort-by :block/order children))) - build (fn build [parent-id depth] - (mapv (fn [b] - (let [children (build (:db/id b) (inc depth))] - (cond-> b - (seq children) (assoc :block/children children)))) - (if (and max-depth (>= depth max-depth)) - [] - (sort-children (get parent->children parent-id)))))] - (build root-id 1))) - -(defn- fetch-tree - [config {:keys [repo id page-name level] :as opts}] - (let [max-depth (or level 10) - uuid-str (:uuid opts)] - (cond - (some? id) - (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] - (if-let [page-id (get-in entity [:block/page :db/id])] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (if (:db/id entity) - (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found}))))) - - (seq uuid-str) - (if-not (common-util/uuid-string? uuid-str) - (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) - (p/let [entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] - [:block/uuid (uuid uuid-str)]]) - entity (if (:db/id entity) - entity - (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] - [:block/uuid uuid-str]]))] - (if-let [page-id (get-in entity [:block/page :db/id])] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (if (:db/id entity) - (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found})))))) - - (seq page-name) - (p/let [page-entity (transport/invoke config "thread-api/pull" false - [repo [:db/id :block/uuid :block/title] [:block/name page-name]])] - (if-let [page-id (:db/id page-entity)] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks page-id max-depth)] - {:root (assoc page-entity :block/children children)}) - (throw (ex-info "page not found" {:code :page-not-found})))) - - :else - (p/rejected (ex-info "block or page required" {:code :missing-target}))))) - -(defn- tree->text - [{:keys [root]}] - (let [label (fn [node] - (or (:block/title node) (:block/name node) (str (:block/uuid node)))) - node-id (fn [node] - (or (:db/id node) "-")) - id-padding (fn [node] - (apply str (repeat (inc (count (str (node-id node)))) " "))) - split-lines (fn [value] - (string/split (or value "") #"\n")) - lines (atom []) - walk (fn walk [node prefix] - (let [children (:block/children node) - total (count children)] - (doseq [[idx child] (map-indexed vector children)] - (let [last-child? (= idx (dec total)) - branch (if last-child? "└── " "├── ") - next-prefix (str prefix (if last-child? " " "│ ")) - rows (split-lines (label child)) - first-row (first rows) - rest-rows (rest rows) - line (str (node-id child) " " prefix branch first-row)] - (swap! lines conj line) - (doseq [row rest-rows] - (swap! lines conj (str (id-padding child) next-prefix row))) - (walk child next-prefix)))))] - (let [rows (split-lines (label root)) - first-row (first rows) - rest-rows (rest rows)] - (swap! lines conj (str (node-id root) " " first-row)) - (doseq [row rest-rows] - (swap! lines conj (str (id-padding root) row)))) - (walk root "") - (string/join "\n" @lines))) - -(defn- resolve-repo - [graph] - (let [graph (some-> graph string/trim)] - (when (seq graph) - (graph->repo graph)))) - -(defn- missing-graph-error - [] - {:ok? false - :error {:code :missing-graph - :message "graph name is required"}}) - -(defn- missing-repo-error - [message] - {:ok? false - :error {:code :missing-repo - :message message}}) - -(def ^:private list-page-field-map - {"title" :block/title - "uuid" :block/uuid - "created-at" :block/created-at - "updated-at" :block/updated-at}) - -(def ^:private list-tag-field-map - {"name" :block/title - "title" :block/title - "uuid" :block/uuid - "properties" :logseq.property.class/properties - "extends" :logseq.property.class/extends - "description" :logseq.property/description}) - -(def ^:private list-property-field-map - {"name" :block/title - "title" :block/title - "uuid" :block/uuid - "classes" :logseq.property/classes - "type" :logseq.property/type - "description" :logseq.property/description}) - -(defn- parse-field-list - [fields] - (when (seq fields) - (->> (string/split fields #",") - (map string/trim) - (remove string/blank?) - vec))) - -(defn- apply-fields - [items fields field-map] - (if (seq fields) - (let [keys (->> fields - (map #(get field-map %)) - (remove nil?) - vec)] - (if (seq keys) - (mapv #(select-keys % keys) items) - items)) - items)) - -(defn- apply-sort - [items sort-field order field-map] - (if (seq sort-field) - (let [sort-key (get field-map sort-field) - sorted (if sort-key - (sort-by #(get % sort-key) items) - items) - sorted (if (= "desc" order) (reverse sorted) sorted)] - (vec sorted)) - (vec items))) - -(defn- apply-offset-limit - [items offset limit] - (cond-> items - (some? offset) (->> (drop offset) vec) - (some? limit) (->> (take limit) vec))) - -(defn- prepare-tag-item - [item {:keys [expand with-properties with-extends]}] - (if expand - (cond-> item - (not with-properties) (dissoc :logseq.property.class/properties) - (not with-extends) (dissoc :logseq.property.class/extends)) - item)) - -(defn- prepare-property-item - [item {:keys [expand with-classes with-type]}] - (if expand - (cond-> item - (not with-classes) (dissoc :logseq.property/classes) - (not with-type) (dissoc :logseq.property/type)) - item)) - -(defn- build-graph-action - [command graph repo] - (case command - :graph-list - {:ok? true - :action {:type :graph-list - :command :graph-list}} - - :graph-create - (if-not (seq graph) - (missing-graph-error) - {:ok? true - :action {:type :invoke - :command :graph-create - :method "thread-api/create-or-open-db" - :direct-pass? false - :args [repo {}] - :repo repo - :graph (repo->graph repo) - :allow-missing-graph true - :persist-repo (repo->graph repo)}}) - - :graph-switch - (if-not (seq graph) - (missing-graph-error) - {:ok? true - :action {:type :graph-switch - :command :graph-switch - :repo repo - :graph (repo->graph repo)}}) - - :graph-remove - (if-not (seq graph) - (missing-graph-error) - {:ok? true - :action {:type :invoke - :command :graph-remove - :method "thread-api/unsafe-unlink-db" - :direct-pass? false - :args [repo] - :repo repo - :graph (repo->graph repo)}}) - - :graph-validate - (if-not (seq repo) - (missing-graph-error) - {:ok? true - :action {:type :invoke - :command :graph-validate - :method "thread-api/validate-db" - :direct-pass? false - :args [repo] - :repo repo - :graph (repo->graph repo)}}) - - :graph-info - (if-not (seq repo) - (missing-graph-error) - {:ok? true - :action {:type :graph-info - :command :graph-info - :repo repo - :graph (repo->graph repo)}}))) - -(defn- build-server-action - [command repo] - (case command - :server-list - {:ok? true - :action {:type :server-list}} - - :server-status - (if-not (seq repo) - (missing-repo-error "repo is required for server status") - {:ok? true - :action {:type :server-status - :repo repo}}) - - :server-start - (if-not (seq repo) - (missing-repo-error "repo is required for server start") - {:ok? true - :action {:type :server-start - :repo repo}}) - - :server-stop - (if-not (seq repo) - (missing-repo-error "repo is required for server stop") - {:ok? true - :action {:type :server-stop - :repo repo}}) - - :server-restart - (if-not (seq repo) - (missing-repo-error "repo is required for server restart") - {:ok? true - :action {:type :server-restart - :repo repo}}) - - {:ok? false - :error {:code :unknown-command - :message (str "unknown server command: " command)}})) - -(defn- build-add-block-action - [options args repo] - (if-not (seq repo) - (missing-repo-error "repo is required for add") - (let [blocks-result (read-blocks options args)] - (if-not (:ok? blocks-result) - blocks-result - (let [vector-result (ensure-blocks (:value blocks-result))] - (if-not (:ok? vector-result) - vector-result - {:ok? true - :action {:type :add-block - :repo repo - :graph (repo->graph repo) - :page (:page options) - :parent (:parent options) - :blocks (:value vector-result)}})))))) - -(defn- build-add-page-action - [options repo] - (if-not (seq repo) - (missing-repo-error "repo is required for add") - (let [page (some-> (:page options) string/trim)] - (if (seq page) - {:ok? true - :action {:type :add-page - :repo repo - :graph (repo->graph repo) - :page page}} - {:ok? false - :error {:code :missing-page-name - :message "page name is required"}})))) - -(defn- build-remove-block-action - [options repo] - (if-not (seq repo) - (missing-repo-error "repo is required for remove") - (let [block (some-> (:block options) string/trim)] - (if (seq block) - {:ok? true - :action {:type :remove-block - :repo repo - :block block}} - {:ok? false - :error {:code :missing-target - :message "block is required"}})))) - -(defn- build-remove-page-action - [options repo] - (if-not (seq repo) - (missing-repo-error "repo is required for remove") - (let [page (some-> (:page options) string/trim)] - (if (seq page) - {:ok? true - :action {:type :remove-page - :repo repo - :page page}} - {:ok? false - :error {:code :missing-target - :message "page is required"}})))) - -(defn- build-list-action - [command options repo] - (if-not (seq repo) - (missing-repo-error "repo is required for list") - {:ok? true - :action {:type command - :repo repo - :options options}})) - -(defn- build-search-action - [options args repo] - (if-not (seq repo) - (missing-repo-error "repo is required for search") - (let [text (or (:text options) (string/join " " args))] - (if (seq text) - {:ok? true - :action {:type :search - :repo repo - :text text - :search-type (:type options) - :tag (:tag options) - :limit (:limit options) - :case-sensitive (:case-sensitive options) - :include-content (:include-content options) - :sort (:sort options) - :order (:order options)}} - {:ok? false - :error {:code :missing-search-text - :message "search text is required"}})))) - -(defn- build-show-action - [options repo] - (if-not (seq repo) - (missing-repo-error "repo is required for show") - (let [format (some-> (:format options) string/lower-case) - targets (filter some? [(:id options) (:uuid options) (:page-name options)])] - (if (empty? targets) - {:ok? false - :error {:code :missing-target - :message "block or page is required"}} - {:ok? true - :action {:type :show - :repo repo - :id (:id options) - :uuid (:uuid options) - :page-name (:page-name options) - :level (:level options) - :format format}})))) +;; Command-specific errors live in subcommand namespaces. (defn build-action [parsed config] (if-not (:ok? parsed) parsed (let [{:keys [command options args]} parsed - graph (pick-graph options args config) - repo (resolve-repo graph) - server-repo (resolve-repo (:repo options))] + graph (command-core/pick-graph options args config) + repo (command-core/resolve-repo graph) + server-repo (command-core/resolve-repo (:repo options))] (case command (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) - (build-graph-action command graph repo) + (graph-command/build-graph-action command graph repo) :graph-export - (let [export-type (normalize-import-export-type (:type options))] - (if-not (seq repo) - (missing-repo-error "repo is required for export") - {:ok? true - :action {:type :graph-export - :repo repo - :graph (repo->graph repo) - :export-type export-type - :output (:output options)}})) + (let [export-type (graph-command/normalize-import-export-type (:type options))] + (graph-command/build-export-action repo export-type (:output options))) :graph-import - (let [import-repo (resolve-repo (:repo options)) - import-type (normalize-import-export-type (:type options))] - (if-not (seq import-repo) - (missing-repo-error "repo is required for import") - {:ok? true - :action {:type :graph-import - :repo import-repo - :graph (repo->graph import-repo) - :import-type import-type - :input (:input options) - :allow-missing-graph true}})) + (let [import-repo (command-core/resolve-repo (:repo options)) + import-type (graph-command/normalize-import-export-type (:type options))] + (graph-command/build-import-action import-repo import-type (:input options))) (:server-list :server-status :server-start :server-stop :server-restart) - (build-server-action command server-repo) + (server-command/build-action command server-repo) (:list-page :list-tag :list-property) - (build-list-action command options repo) + (list-command/build-action command options repo) :add-block - (build-add-block-action options args repo) + (add-command/build-add-block-action options args repo) :add-page - (build-add-page-action options repo) + (add-command/build-add-page-action options repo) :remove-block - (build-remove-block-action options repo) + (remove-command/build-remove-block-action options repo) :remove-page - (build-remove-page-action options repo) + (remove-command/build-remove-page-action options repo) :search - (build-search-action options args repo) + (search-command/build-action options args repo) :show - (build-show-action options repo) + (show-command/build-action options repo) {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) -(defn- execute-graph-list - [_action config] - (let [graphs (cli-server/list-graphs config)] - {:status :ok - :data {:graphs graphs}})) - -(defn- execute-invoke - [action config] - (-> (p/let [cfg (if-let [repo (:repo action)] - (cli-server/ensure-server! config repo) - (p/resolved config)) - result (transport/invoke cfg - (:method action) - (:direct-pass? action) - (:args action))] - (when-let [repo (:persist-repo action)] - (cli-config/update-config! config {:repo repo})) - (if-let [write (:write action)] - (let [{:keys [format path]} write] - (transport/write-output {:format format :path path :data result}) - {:status :ok - :data {:message (str "wrote " path)}}) - {:status :ok :data {:result result}})))) - -(defn- execute-graph-switch - [action config] - (-> (p/let [graphs (cli-server/list-graphs config) - graph (:graph action)] - (if-not (some #(= graph %) graphs) - {:status :error - :error {:code :graph-not-found - :message (str "graph not found: " graph)}} - (p/let [_ (cli-server/ensure-server! config (:repo action))] - (cli-config/update-config! config {:repo graph}) - {:status :ok - :data {:message (str "switched to " graph)}}))))) - -(defn- execute-graph-info - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - created (transport/invoke cfg "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) - schema (transport/invoke cfg "thread-api/pull" false [(:repo action) [:kv/value] :logseq.kv/schema-version])] - {:status :ok - :data {:graph (:graph action) - :logseq.kv/graph-created-at (:kv/value created) - :logseq.kv/schema-version (:kv/value schema)}}))) - -(defn- execute-graph-export - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - export-type (:export-type action) - export-result (case export-type - "edn" - (transport/invoke cfg - "thread-api/export-edn" - false - [(:repo action) {:export-type :graph}]) - "sqlite" - (transport/invoke cfg - "thread-api/export-db-base64" - true - [(:repo action)]) - (throw (ex-info "unsupported export type" {:export-type export-type}))) - data (if (= export-type "sqlite") - (js/Buffer.from export-result "base64") - export-result) - format (if (= export-type "sqlite") :sqlite :edn)] - (transport/write-output {:format format :path (:output action) :data data}) - {:status :ok - :data {:message (str "wrote " (:output action))}}))) - -(defn- execute-graph-import - [action config] - (-> (p/let [_ (cli-server/stop-server! config (:repo action)) - cfg (cli-server/ensure-server! config (:repo action)) - import-type (:import-type action) - input-data (case import-type - "edn" (transport/read-input {:format :edn :path (:input action)}) - "sqlite" (transport/read-input {:format :sqlite :path (:input action)}) - (throw (ex-info "unsupported import type" {:import-type import-type}))) - payload (if (= import-type "sqlite") - (.toString (js/Buffer.from input-data) "base64") - input-data) - method (if (= import-type "sqlite") - "thread-api/import-db-base64" - "thread-api/import-edn") - direct-pass? (= import-type "sqlite") - _ (transport/invoke cfg method direct-pass? [(:repo action) payload]) - _ (cli-server/restart-server! config (:repo action))] - {:status :ok - :data {:message (str "imported " import-type " from " (:input action))}}))) - -(defn- execute-list-page - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) - items (transport/invoke cfg "thread-api/api-list-pages" false - [(:repo action) options]) - order (or (:order options) "asc") - fields (parse-field-list (:fields options)) - sorted (apply-sort items (:sort options) order list-page-field-map) - limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-page-field-map) - limited)] - {:status :ok - :data {:items final}}))) - -(defn- execute-list-tag - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) - items (transport/invoke cfg "thread-api/api-list-tags" false - [(:repo action) options]) - order (or (:order options) "asc") - fields (parse-field-list (:fields options)) - prepared (mapv #(prepare-tag-item % options) items) - sorted (apply-sort prepared (:sort options) order list-tag-field-map) - limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-tag-field-map) - limited)] - {:status :ok - :data {:items final}}))) - -(defn- execute-list-property - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) - items (transport/invoke cfg "thread-api/api-list-properties" false - [(:repo action) options]) - order (or (:order options) "asc") - fields (parse-field-list (:fields options)) - prepared (mapv #(prepare-property-item % options) items) - sorted (apply-sort prepared (:sort options) order list-property-field-map) - limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-property-field-map) - limited)] - {:status :ok - :data {:items final}}))) - -(defn- execute-add-block - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - target-id (resolve-add-target cfg action) - ops [[:insert-blocks [(:blocks action) - target-id - {:sibling? false - :bottom? true - :outliner-op :insert-blocks}]]] - result (transport/invoke cfg "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] - {:status :ok - :data {:result result}}))) - -(defn- execute-add-page - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - ops [[:create-page [(:page action) {}]]] - result (transport/invoke cfg "thread-api/apply-outliner-ops" false [(:repo action) ops {}])] - {:status :ok - :data {:result result}}))) - -(defn- execute-remove - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - result (perform-remove cfg action)] - {:status :ok - :data {:result result}}))) - -(defn- query-pages - [cfg repo text case-sensitive?] - (let [query (if case-sensitive? - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q - :where - [?e :block/name ?name] - [?e :block/title ?title] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?title ?q)]] - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q - :where - [?e :block/name ?name] - [?e :block/title ?title] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?title) ?q)]]) - q* (if case-sensitive? text (string/lower-case text))] - (transport/invoke cfg "thread-api/q" false [repo [query q*]]))) - - -#_{:clj-kondo/ignore [:aliased-namespace-symbol]} -(defn- query-blocks - [cfg repo text case-sensitive? tag include-content?] - (let [has-tag? (seq tag) - content-attr (if include-content? :block/content :block/title) - query (cond - (and case-sensitive? has-tag?) - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q ?tag-name - :where - [?tag :block/name ?tag-name] - [?e :block/tags ?tag] - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?value ?q)]] - - case-sensitive? - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q - :where - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?value ?q)]] - - has-tag? - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q ?tag-name - :where - [?tag :block/name ?tag-name] - [?e :block/tags ?tag] - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]] - - :else - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q - :where - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]]) - q* (if case-sensitive? text (string/lower-case text)) - tag-name (some-> tag string/lower-case)] - (if has-tag? - (transport/invoke cfg "thread-api/q" false [repo [query q* tag-name]]) - (transport/invoke cfg "thread-api/q" false [repo [query q*]])))) - -(defn- normalize-search-types - [type] - (let [type (or type "all")] - (case type - "page" [:page] - "block" [:block] - "tag" [:tag] - "property" [:property] - [:page :block :tag :property]))) - -(defn- search-sort-key - [item sort-field] - (case sort-field - "updated-at" (:updated-at item) - "created-at" (:created-at item) - nil)) - -(defn- execute-search - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - types (normalize-search-types (:search-type action)) - case-sensitive? (boolean (:case-sensitive action)) - text (:text action) - tag (:tag action) - page-results (when (some #{:page} types) - (p/let [rows (query-pages cfg (:repo action) text case-sensitive?)] - (mapv (fn [[id title uuid updated created]] - {:type "page" - :db/id id - :title title - :uuid (str uuid) - :updated-at updated - :created-at created}) - rows))) - include-content? (boolean (:include-content action)) - block-results (when (some #{:block} types) - (p/let [rows (query-blocks cfg (:repo action) text case-sensitive? tag include-content?)] - (mapv (fn [[id content uuid updated created]] - {:type "block" - :db/id id - :content content - :uuid (str uuid) - :updated-at updated - :created-at created}) - rows))) - tag-results (when (some #{:tag} types) - (p/let [items (transport/invoke cfg "thread-api/api-list-tags" false - [(:repo action) {:expand true :include-built-in true}]) - q* (if case-sensitive? text (string/lower-case text))] - (->> items - (filter (fn [item] - (let [title (:block/title item)] - (if case-sensitive? - (string/includes? title q*) - (string/includes? (string/lower-case title) q*))))) - (mapv (fn [item] - {:type "tag" - :title (:block/title item) - :uuid (:block/uuid item)}))))) - property-results (when (some #{:property} types) - (p/let [items (transport/invoke cfg "thread-api/api-list-properties" false - [(:repo action) {:expand true :include-built-in true}]) - q* (if case-sensitive? text (string/lower-case text))] - (->> items - (filter (fn [item] - (let [title (:block/title item)] - (if case-sensitive? - (string/includes? title q*) - (string/includes? (string/lower-case title) q*))))) - (mapv (fn [item] - {:type "property" - :title (:block/title item) - :uuid (:block/uuid item)}))))) - results (->> (concat (or page-results []) - (or block-results []) - (or tag-results []) - (or property-results [])) - (distinct) - vec) - sorted (if-let [sort-field (:sort action)] - (let [order (or (:order action) "desc")] - (->> results - (sort-by #(search-sort-key % sort-field)) - (cond-> (= order "desc") reverse) - vec)) - results) - limited (if (some? (:limit action)) (vec (take (:limit action) sorted)) sorted)] - {:status :ok - :data {:results limited}}))) - -(defn- execute-show - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - tree-data (fetch-tree cfg action) - format (:format action)] - (case format - "edn" - {:status :ok - :data tree-data - :output-format :edn} - - "json" - {:status :ok - :data tree-data - :output-format :json} - - {:status :ok - :data {:message (tree->text tree-data)}})))) - -(defn- server-result->response - [result] - (if (:ok? result) - {:status :ok - :data (:data result)} - {:status :error - :error (:error result)})) - -(defn- execute-server-list - [_action config] - (-> (p/let [servers (cli-server/list-servers config)] - {:status :ok - :data {:servers servers}}))) - -(defn- execute-server-status - [action config] - (-> (p/let [result (cli-server/server-status config (:repo action))] - (server-result->response result)))) - -(defn- execute-server-start - [action config] - (-> (p/let [result (cli-server/start-server! config (:repo action))] - (server-result->response result)))) - -(defn- execute-server-stop - [action config] - (-> (p/let [result (cli-server/stop-server! config (:repo action))] - (server-result->response result)))) - -(defn- execute-server-restart - [action config] - (-> (p/let [result (cli-server/restart-server! config (:repo action))] - (server-result->response result)))) - (defn execute [action config] (-> (p/let [missing-check (ensure-missing-graph action config) @@ -1581,26 +333,26 @@ :else (case (:type action) - :graph-list (execute-graph-list action config) - :invoke (execute-invoke action config) - :graph-switch (execute-graph-switch action config) - :graph-info (execute-graph-info action config) - :graph-export (execute-graph-export action config) - :graph-import (execute-graph-import action config) - :list-page (execute-list-page action config) - :list-tag (execute-list-tag action config) - :list-property (execute-list-property action config) - :add-block (execute-add-block action config) - :add-page (execute-add-page action config) - :remove-block (execute-remove action config) - :remove-page (execute-remove action config) - :search (execute-search action config) - :show (execute-show action config) - :server-list (execute-server-list action config) - :server-status (execute-server-status action config) - :server-start (execute-server-start action config) - :server-stop (execute-server-stop action config) - :server-restart (execute-server-restart action config) + :graph-list (graph-command/execute-graph-list action config) + :invoke (graph-command/execute-invoke action config) + :graph-switch (graph-command/execute-graph-switch action config) + :graph-info (graph-command/execute-graph-info action config) + :graph-export (graph-command/execute-graph-export action config) + :graph-import (graph-command/execute-graph-import action config) + :list-page (list-command/execute-list-page action config) + :list-tag (list-command/execute-list-tag action config) + :list-property (list-command/execute-list-property action config) + :add-block (add-command/execute-add-block action config) + :add-page (add-command/execute-add-page action config) + :remove-block (remove-command/execute-remove action config) + :remove-page (remove-command/execute-remove action config) + :search (search-command/execute-search action config) + :show (show-command/execute-show action config) + :server-list (server-command/execute-list action config) + :server-status (server-command/execute-status action config) + :server-start (server-command/execute-start action config) + :server-stop (server-command/execute-stop action config) + :server-restart (server-command/execute-restart action config) {:status :error :error {:code :unknown-action :message "unknown action"}}))] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index e70cc0bfa9..4b872ea8d6 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -68,7 +68,7 @@ (defn- error-hint [{:keys [code]}] (case code - :missing-graph "Use --graph " + :missing-graph "Use --repo " :missing-repo "Use --repo " :missing-content "Use --content or pass content as args" :missing-search-text "Provide search text or --text" diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index b24ad93645..fc39591fa5 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -90,11 +90,16 @@ [{:keys [base-url auth-token timeout-ms retries]} method direct-pass? args] (let [url (str (string/replace base-url #"/$" "") "/v1/invoke") + method* (cond + (keyword? method) (subs (str method) 1) + (string? method) method + (nil? method) nil + :else (str method)) payload (if direct-pass? - {:method method + {:method method* :directPass true :args args} - {:method method + {:method method* :directPass false :argsTransit (ldb/write-transit-str args)}) body (js/JSON.stringify (clj->js payload))] diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 3e889483dd..677d60dc03 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -180,6 +180,22 @@ (is (= "logseq_db_parse_args" (:repo result))) (is (= "/tmp/db-worker" (:data-dir result))))) +(deftest db-worker-node-repo-error-handles-keyword-methods + (let [repo-error #'db-worker-node/repo-error + bound-repo "logseq_db_bound"] + (is (nil? (repo-error :thread-api/list-db [] bound-repo))) + (is (nil? (repo-error "thread-api/list-db" [] bound-repo))) + (is (= {:status 400 + :error {:code :missing-repo + :message "repo is required"}} + (repo-error :thread-api/create-or-open-db [] bound-repo))) + (is (= {:status 409 + :error {:code :repo-mismatch + :message "repo does not match bound repo" + :repo "other" + :bound-repo bound-repo}} + (repo-error :thread-api/create-or-open-db ["other"] bound-repo))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 1d6bbe4657..7d99d921ab 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.commands-test (:require [cljs.test :refer [async deftest is testing]] [clojure.string :as string] + [logseq.cli.command.show :as show-command] [logseq.cli.commands :as commands] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] @@ -127,6 +128,13 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) +(deftest test-parse-args-rejects-graph-option + (testing "rejects legacy --graph option" + (let [result (commands/parse-args ["--graph" "demo" "graph" "list"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))) + (is (= "unknown option: --graph" (get-in result [:error :message])))))) + (deftest test-parse-args-global-options (testing "global output option is accepted" (let [result (commands/parse-args ["--output" "json" "graph" "list"])] @@ -135,7 +143,7 @@ (deftest test-tree->text-format (testing "show tree text uses db/id with tree glyphs" - (let [tree->text #'commands/tree->text + (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 1 :block/title "Root" :block/children [{:db/id 2 @@ -152,7 +160,7 @@ (deftest test-tree->text-multiline (testing "show tree text renders multiline blocks under glyph column" - (let [tree->text #'commands/tree->text + (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 168 :block/title "Jan 18th, 2026" :block/children [{:db/id 169 @@ -244,7 +252,7 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) -(deftest test-verb-subcommand-parse +(deftest test-verb-subcommand-parse-add-remove (testing "add block requires content source" (let [result (commands/parse-args ["add" "block"])] (is (false? (:ok? result))) @@ -282,8 +290,9 @@ (let [result (commands/parse-args ["remove" "page" "--page" "Home"])] (is (true? (:ok? result))) (is (= :remove-page (:command result))) - (is (= "Home" (get-in result [:options :page]))))) + (is (= "Home" (get-in result [:options :page])))))) +(deftest test-verb-subcommand-parse-search-show (testing "search requires text" (let [result (commands/parse-args ["search"])] (is (false? (:ok? result))) @@ -304,8 +313,9 @@ (let [result (commands/parse-args ["show" "--page-name" "Home"])] (is (true? (:ok? result))) (is (= :show (:command result))) - (is (= "Home" (get-in result [:options :page-name]))))) + (is (= "Home" (get-in result [:options :page-name])))))) +(deftest test-verb-subcommand-parse-graph-import-export (testing "graph export parses with type and output" (let [result (commands/parse-args ["graph" "export" "--type" "edn" @@ -349,8 +359,9 @@ "--input" "import.zip" "--repo" "demo"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-verb-subcommand-parse-flags (testing "verb subcommands reject unknown flags" (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] @@ -517,7 +528,7 @@ (assoc config :base-url "http://127.0.0.1:9999"))) (set! transport/invoke (fn [_ method direct-pass? args] (swap! invoke-calls conj [method direct-pass? args]) - (if (= method "thread-api/export-db-base64") + (if (= method :thread-api/export-db-base64) "c3FsaXRl" {:exported true}))) (set! transport/write-output (fn [opts] @@ -538,8 +549,8 @@ {})] (is (= :ok (:status edn-result))) (is (= :ok (:status sqlite-result))) - (is (= [["thread-api/export-edn" false ["logseq_db_demo" {:export-type :graph}]] - ["thread-api/export-db-base64" true ["logseq_db_demo"]]] + (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] + [:thread-api/export-db-base64 true ["logseq_db_demo"]]] @invoke-calls)) (is (= 2 (count @write-calls))) (let [[edn-write sqlite-write] @write-calls] @@ -605,8 +616,8 @@ (is (= [[:edn "/tmp/import.edn"] [:sqlite "/tmp/import.sqlite"]] @read-calls)) - (is (= [["thread-api/import-edn" ["logseq_db_demo" {:page "Import Page"}]] - ["thread-api/import-db-base64" ["logseq_db_demo" "c3FsaXRl"]]] + (is (= [[:thread-api/import-edn ["logseq_db_demo" {:page "Import Page"}]] + [:thread-api/import-db-base64 ["logseq_db_demo" "c3FsaXRl"]]] @invoke-calls)) (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 60bc40559b..7ab050f1d7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -193,5 +193,5 @@ :message "graph name is required"}} {:output-format nil})] (is (= (str "Error (missing-graph): graph name is required\n" - "Hint: Use --graph ") + "Hint: Use --repo ") result))))) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index 03ea866e42..5cb0f553a0 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -72,23 +72,46 @@ (is false (str "unexpected error: " e)) (done)))))) +(deftest test-invoke-accepts-keyword-method + (async done + (let [received (atom nil)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [^js req ^js res] + (let [chunks (array)] + (.on req "data" (fn [chunk] (.push chunks chunk))) + (.on req "end" (fn [] + (let [buf (js/Buffer.concat chunks) + payload (js/JSON.parse (.toString buf "utf8"))] + (reset! received (js->clj payload :keywordize-keys true)) + (.writeHead res 200 #js {"Content-Type" "application/json"}) + (.end res (js/JSON.stringify #js {:result "ok"})))))))) + result (transport/invoke {:base-url url} :thread-api/pull true ["repo" [:block/title]])] + (is (= "ok" result)) + (is (= "thread-api/pull" (:method @received))) + (is (= true (:directPass @received))) + (p/let [_ (stop!)] + (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-read-input (testing "reads edn input" - (let [path (temp-path "input.edn")] - (.writeFileSync fs path "{:a 1}") - (is (= {:a 1} (transport/read-input {:format :edn :path path}))))) + (let [file-path (temp-path "input.edn")] + (.writeFileSync fs file-path "{:a 1}") + (is (= {:a 1} (transport/read-input {:format :edn :path file-path}))))) (testing "reads sqlite input as buffer" - (let [path (temp-path "input.sqlite") + (let [file-path (temp-path "input.sqlite") buffer (js/Buffer.from "sqlite-data")] - (.writeFileSync fs path buffer) - (let [result (transport/read-input {:format :sqlite :path path})] + (.writeFileSync fs file-path buffer) + (let [result (transport/read-input {:format :sqlite :path file-path})] (is (instance? js/Buffer result)) (is (= "sqlite-data" (.toString result "utf8"))))))) (deftest test-write-output (testing "writes sqlite output as buffer" - (let [path (temp-path "output.sqlite") + (let [file-path (temp-path "output.sqlite") buffer (js/Buffer.from "sqlite-export")] - (transport/write-output {:format :sqlite :path path :data buffer}) - (is (= "sqlite-export" (.toString (.readFileSync fs path) "utf8")))))) + (transport/write-output {:format :sqlite :path file-path :data buffer}) + (is (= "sqlite-export" (.toString (.readFileSync fs file-path) "utf8")))))) diff --git a/tmp_scripts/db-worker-smoke-test.clj b/tmp_scripts/db-worker-smoke-test.clj deleted file mode 100644 index 0debde9741..0000000000 --- a/tmp_scripts/db-worker-smoke-test.clj +++ /dev/null @@ -1,93 +0,0 @@ -(require '[babashka.curl :as curl] - '[cheshire.core :as json] - '[cognitect.transit :as transit] - '[clojure.pprint :as pprint] - '[clojure.string :as string]) - -(def base-url (or (System/getenv "DB_WORKER_URL") "http://127.0.0.1:9101")) - -(defn write-transit [v] - (let [out (java.io.ByteArrayOutputStream.) - w (transit/writer out :json)] - (transit/write w v) - (.toString out "UTF-8"))) - -(defn read-transit [s] - (let [in (java.io.ByteArrayInputStream. (.getBytes s "UTF-8")) - r (transit/reader in :json)] - (transit/read r))) - -(defn invoke [method direct-pass? args] - (let [payload (if direct-pass? - {:method method :directPass true :args args} - {:method method :directPass false :argsTransit (write-transit args)}) - resp (curl/post (str base-url "/v1/invoke") - {:headers {"Content-Type" "application/json"} - :body (json/generate-string payload)}) - body (json/parse-string (:body resp) true)] - (if (<= 200 (:status resp) 299) - (if direct-pass? - (:result body) - (read-transit (:resultTransit body))) - (throw (ex-info "db-worker invoke failed" {:status (:status resp) :body (:body resp)}))))) - -(def suffix (subs (str (random-uuid)) 0 8)) -(def repo (str "logseq_db_smoke_" suffix)) -(def page-uuid (random-uuid)) -(def block-uuid (random-uuid)) -(def now (long (System/currentTimeMillis))) - -(println "== db-worker-node smoke test ==") -(println "Base URL:" base-url) -(println "Repo:" repo) -(println "Step 1/4: list-db (before)") -(println "Result:" (json/generate-string (invoke "thread-api/list-db" false []) - {:pretty true})) - -(println "Step 2/4: create-or-open-db") -(invoke "thread-api/create-or-open-db" false [repo {}]) -(println "Step 3/4: list-db (after)") -(println "Result:" (json/generate-string (invoke "thread-api/list-db" false []) - {:pretty true})) - -(println "Step 4/4: transact + q") -(invoke "thread-api/transact" false - [repo - [{:block/uuid page-uuid - :block/title "Smoke Page" - :block/name "smoke-page" - :block/tags #{:logseq.class/Page} - :block/created-at now - :block/updated-at now} - {:block/uuid block-uuid - :block/title "Smoke Test" - :block/page [:block/uuid page-uuid] - :block/parent [:block/uuid page-uuid] - :block/order "a0" - :block/created-at now - :block/updated-at now}] - {} - nil]) - -(let [query '[:find ?e - :in $ ?uuid - :where [?e :block/uuid ?uuid]] - result (invoke "thread-api/q" false [repo [query block-uuid]])] - (println "Query result:" result) - (when (empty? result) - (throw (ex-info "Query returned no results" {:uuid block-uuid})))) - -(let [page-query '[:find (pull ?e [:db/id :block/uuid :block/title :block/name :block/tags]) - :in $ ?uuid - :where [?e :block/uuid ?uuid]] - blocks-query '[:find (pull ?e [:db/id :block/uuid :block/title :block/order :block/parent]) - :in $ ?page-uuid - :where [?page :block/uuid ?page-uuid] - [?e :block/page ?page]] - page-result (invoke "thread-api/q" false [repo [page-query page-uuid]]) - blocks-result (invoke "thread-api/q" false [repo [blocks-query page-uuid]])] - (println "Page + blocks (pretty):") - (pprint/pprint {:page page-result - :blocks blocks-result})) - -(println "Smoke test OK") diff --git a/tmp_scripts/db-worker-sse-smoke-test.clj b/tmp_scripts/db-worker-sse-smoke-test.clj deleted file mode 100644 index c343acc996..0000000000 --- a/tmp_scripts/db-worker-sse-smoke-test.clj +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env bb -(require '[babashka.process :as process] - '[clojure.java.io :as io] - '[clojure.string :as string]) - -(def base-url (or (System/getenv "DB_WORKER_URL") - "http://127.0.0.1:9101")) -(def auth-token (System/getenv "DB_WORKER_AUTH_TOKEN")) -(def events-url (str (string/replace base-url #"/$" "") "/v1/events")) - -(defn- open-sse-connection - [url token] - (let [^java.net.HttpURLConnection conn (.openConnection (java.net.URL. url))] - (.setRequestMethod conn "GET") - (.setRequestProperty conn "Accept" "text/event-stream") - (when (seq token) - (.setRequestProperty conn "Authorization" (str "Bearer " token))) - (.setDoInput conn true) - (.connect conn) - conn)) - -(defn- wait-for-sse! - [^java.net.HttpURLConnection conn timeout-ms] - (let [event-seen (promise) - reader (future - (try - (with-open [rdr (io/reader (.getInputStream conn))] - (doseq [line (line-seq rdr)] - (when (string/starts-with? line "data:") - (deliver event-seen line) - (reduced nil)))) - (catch Exception _ nil)))] - (try - (let [result (deref event-seen timeout-ms ::timeout)] - (when (= result ::timeout) - (throw (ex-info "No SSE events captured" {:url events-url}))) - result) - (finally - (.disconnect conn) - (future-cancel reader))))) - -(defn- run-smoke-test! - [] - (let [{:keys [exit]} (process/shell {:inherit true} - "bb" "tmp_scripts/db-worker-smoke-test.clj")] - (when-not (zero? exit) - (throw (ex-info "Smoke test failed" {:exit exit}))))) - -(comment - (let [conn (open-sse-connection events-url auth-token)] - (run-smoke-test!) - (wait-for-sse! conn 2000) - (println "SSE smoke test OK"))) From 7697097158ac8c4f973ee3b3a2df408d4f1d4356 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 16:35:03 +0800 Subject: [PATCH 024/375] add 008-logseq-cli-move-command.md --- .../008-logseq-cli-move-command.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/agent-guide/008-logseq-cli-move-command.md diff --git a/docs/agent-guide/008-logseq-cli-move-command.md b/docs/agent-guide/008-logseq-cli-move-command.md new file mode 100644 index 0000000000..281c613800 --- /dev/null +++ b/docs/agent-guide/008-logseq-cli-move-command.md @@ -0,0 +1,92 @@ +# Logseq CLI Move Command Implementation Plan + +Goal: Add a move subcommand to logseq-cli that moves a non-page block and its children under a target block or page with positional control. +Architecture: Extend the existing CLI command table with a new move command that resolves source and target entities via db-worker-node and invokes :thread-api/apply-outliner-ops using :move-blocks. +Architecture: Use existing outliner move semantics for ordering while validating CLI inputs and translating --pos into outliner options. +Tech Stack: ClojureScript, babashka.cli, db-worker-node :thread-api/apply-outliner-ops, Logseq outliner ops. +Related: Builds on docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md. + +## Problem statement + +Users need a CLI way to move a block and its subtree to a new parent or sibling position in a graph without opening the UI. +The current CLI supports add and remove but lacks a move operation even though db-worker-node and outliner already support :move-blocks. +The move command must align with existing CLI patterns, validate non-page sources, and support positioning under a block or page target. + +## Testing Plan + +I will add unit tests for command parsing and validation of move options, including invalid combinations and default --pos behavior. +I will add unit tests for human output formatting of move success messages. +I will add an integration test that creates blocks, moves a block to a target page and block, and verifies the resulting tree via show. +NOTE: I will write all tests before I add any implementation behavior. + +## Command behavior and options + +The move command will be a new inspect and edit verb alongside add and remove. +The command will accept exactly one source selector and exactly one target selector. +The command will move a single block per invocation. +The command will allow a page target via --page-name and a block target via a block selector. +The move position will default to first-child unless --pos is provided. +Source selectors will be --id or --uuid, and target selectors will be --target-id or --target-uuid. + +| Flag | Meaning | Required | Notes | +| --- | --- | --- | --- | +| --id | Source block db/id | Yes | Mutually exclusive with --uuid. | +| --uuid | Source block UUID | Yes | Mutually exclusive with --id. | +| --target-id | Target block db/id | Yes | Mutually exclusive with --target-uuid and --page-name. | +| --target-uuid | Target block UUID | Yes | Mutually exclusive with --target-id and --page-name. | +| --page-name | Target page name | Yes | Only valid when target is a page. | +| --pos | Position relative to target | No | Allowed values: first-child, last-child, sibling. | + +## Implementation Plan + +1. Read @prompts/review.md and capture any relevant review checklist items for CLI commands and db-worker-node usage. +2. Add a new command namespace at src/main/logseq/cli/command/move.cljs with a command entry, spec, and action builder. +3. Add option validation in src/main/logseq/cli/command/move.cljs for allowed --pos values and mutually exclusive selectors. +4. Add source resolution logic in src/main/logseq/cli/command/move.cljs that fetches the source block by id or uuid and rejects page entities. +5. Add target resolution logic in src/main/logseq/cli/command/move.cljs that fetches the target block or page and returns a target db/id. +6. Map --pos to outliner options for :move-blocks and document the mapping in comments for future maintenance. +7. Implement execute-move in src/main/logseq/cli/command/move.cljs that calls :thread-api/apply-outliner-ops with :move-blocks. +8. Wire the new command into src/main/logseq/cli/commands.cljs for parsing, validation, action building, and execution. +9. Update src/main/logseq/cli/command/core.cljs to include move in the Graph Inspect and Edit group in top-level summaries. +10. Update src/main/logseq/cli/main.cljs to include move in the usage command list string. +11. Add human output formatting for move in src/main/logseq/cli/format.cljs and include the relevant context keys. +12. Update src/test/logseq/cli/commands_test.cljs with parse and help coverage for move and validation error cases. +13. Update src/test/logseq/cli/format_test.cljs with a move success formatting test. +14. Update src/test/logseq/cli/integration_test.cljs with a move workflow that asserts the moved block appears under the new target. +15. Update docs/cli/logseq-cli.md to document the new move command, its flags, and examples. +16. Run bb dev:lint-and-test and fix any failures. + +## Edge cases to cover + +Moving a page block should fail with a clear error message and code. +Providing --pos sibling should return a validation error when the target is a page. +Moving a block to itself or into its descendants should be rejected by outliner and surfaced as an error. +Supplying both id and uuid selectors should return a validation error. +Supplying no target selector should return a validation error. + +## Notes on position mapping + +first-child will use :sibling? false and no :bottom? so that the moved block becomes the first child of the target. +last-child will use :bottom? true so outliner places the block after the last child of the target. +sibling will use :sibling? true so outliner places the block immediately after the target block. + +## Testing Details + +I will add command parsing tests that assert move is present in help output and that invalid flag combinations are rejected. +I will add a format test that ensures the human output for move references the source block and target. +I will add an integration test that creates a page, adds blocks, moves a block under a target, and verifies the show tree includes it in the expected position. + +## Implementation Details + +- Add a new command entry in src/main/logseq/cli/command/move.cljs with a spec that includes source and target selectors plus --pos. +- Resolve source and target entities via :thread-api/pull and reject page sources by checking page attributes. +- Translate --pos into outliner options for :move-blocks and pass them through :thread-api/apply-outliner-ops. +- Extend command parsing and execution switches in src/main/logseq/cli/commands.cljs to include :move-block. +- Extend human formatting in src/main/logseq/cli/format.cljs with a concise move success line. +- Update docs/cli/logseq-cli.md to list the new move command in the inspect and edit section and help output. +- Follow @skills/test-driven-development for all tests and implementation work. + +## Question + + +--- From ccc3123eb6d33461d35fef630559e056d453ac67 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 17:06:52 +0800 Subject: [PATCH 025/375] impl 008-logseq-cli-move-command.md --- docs/cli/logseq-cli.md | 3 + src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/move.cljs | 181 ++++++++++++++++++++++ src/main/logseq/cli/commands.cljs | 30 +++- src/main/logseq/cli/format.cljs | 5 + src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 103 +++++++++--- src/test/logseq/cli/format_test.cljs | 12 +- src/test/logseq/cli/integration_test.cljs | 47 ++++++ 9 files changed, 360 insertions(+), 25 deletions(-) create mode 100644 src/main/logseq/cli/command/move.cljs diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 4022aa95f2..b4b074f417 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -67,6 +67,7 @@ Inspect and edit commands: - `add block --blocks [--page ] [--parent ]` - insert blocks via EDN vector - `add block --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file - `add page --page ` - create a page +- `move --id |--uuid --target-id |--target-uuid |--page-name [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) - `remove block --block ` - remove a block and its children - `remove page --page ` - remove a page and its children - `search --text [--type page|block|tag|property|all] [--include-content] [--limit ]` - search across pages, blocks, tags, and properties @@ -83,6 +84,7 @@ Subcommands: list property [options] List properties add block [options] Add blocks add page [options] Create page + move [options] Move block remove block [options] Remove block remove page [options] Remove page search [options] Search graph @@ -116,6 +118,7 @@ node ./static/logseq-cli.js graph create --repo demo node ./static/logseq-cli.js graph export --type edn --output /tmp/demo.edn --repo demo node ./static/logseq-cli.js graph import --type edn --input /tmp/demo.edn --repo demo-import node ./static/logseq-cli.js add block --page TestPage --content "hello world" +node ./static/logseq-cli.js move --uuid --page-name TargetPage node ./static/logseq-cli.js search --text "hello" node ./static/logseq-cli.js show --page-name TestPage --format json --output json node ./static/logseq-cli.js server list diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 7fa0d7f1c6..407f6bf558 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -76,7 +76,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "search" "show"}} + :commands #{"list" "add" "remove" "move" "search" "show"}} {:title "Graph Management" :commands #{"graph" "server"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/move.cljs b/src/main/logseq/cli/command/move.cljs new file mode 100644 index 0000000000..4f57d0bc97 --- /dev/null +++ b/src/main/logseq/cli/command/move.cljs @@ -0,0 +1,181 @@ +(ns logseq.cli.command.move + "Move-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] + [promesa.core :as p])) + +(def ^:private move-spec + {:id {:desc "Source block db/id" + :coerce :long} + :uuid {:desc "Source block UUID"} + :target-id {:desc "Target block db/id" + :coerce :long} + :target-uuid {:desc "Target block UUID"} + :page-name {:desc "Target page name"} + :pos {:desc "Position (first-child, last-child, sibling)"}}) + +(def entries + [(core/command-entry ["move"] :move-block "Move block" move-spec)]) + +(def ^:private move-positions + #{"first-child" "last-child" "sibling"}) + +(defn invalid-options? + [opts] + (let [pos (some-> (:pos opts) string/trim string/lower-case) + source-selectors (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) + target-selectors (filter some? [(:target-id opts) + (:target-uuid opts) + (some-> (:page-name opts) string/trim)])] + (cond + (and (seq pos) (not (contains? move-positions pos))) + (str "invalid pos: " (:pos opts)) + + (> (count source-selectors) 1) + "only one of --id or --uuid is allowed" + + (> (count target-selectors) 1) + "only one of --target-id, --target-uuid, or --page-name is allowed" + + (and (= pos "sibling") (seq (some-> (:page-name opts) string/trim))) + "--pos sibling is only valid for block targets" + + :else + nil))) + +(def ^:private block-selector + [:db/id :block/uuid :block/name :block/title]) + +(defn- fetch-entity-by-uuid + [config repo uuid-str] + (p/let [entity (transport/invoke config :thread-api/pull false + [repo block-selector [:block/uuid (uuid uuid-str)]])] + (if (:db/id entity) + entity + (transport/invoke config :thread-api/pull false + [repo block-selector [:block/uuid uuid-str]])))) + +(defn- ensure-non-page + [entity message code] + (if (:block/name entity) + (throw (ex-info message {:code code})) + entity)) + +(defn- resolve-source + [config repo {:keys [id uuid]}] + (cond + (some? id) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo block-selector id])] + (if (:db/id entity) + (ensure-non-page entity "source must be a non-page block" :invalid-source) + (throw (ex-info "source block not found" {:code :source-not-found})))) + + (seq uuid) + (if-not (common-util/uuid-string? uuid) + (p/rejected (ex-info "source must be a uuid" {:code :invalid-source})) + (p/let [entity (fetch-entity-by-uuid config repo uuid)] + (if (:db/id entity) + (ensure-non-page entity "source must be a non-page block" :invalid-source) + (throw (ex-info "source block not found" {:code :source-not-found}))))) + + :else + (p/rejected (ex-info "source is required" {:code :missing-source})))) + +(defn- resolve-target + [config repo {:keys [target-id target-uuid page-name]}] + (cond + (some? target-id) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo block-selector target-id])] + (if (:db/id entity) + (ensure-non-page entity "target must be a block" :invalid-target) + (throw (ex-info "target block not found" {:code :target-not-found})))) + + (seq target-uuid) + (if-not (common-util/uuid-string? target-uuid) + (p/rejected (ex-info "target must be a uuid" {:code :invalid-target})) + (p/let [entity (fetch-entity-by-uuid config repo target-uuid)] + (if (:db/id entity) + (ensure-non-page entity "target must be a block" :invalid-target) + (throw (ex-info "target block not found" {:code :target-not-found}))))) + + (seq page-name) + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/name :block/title] + [:block/name page-name]])] + (if (:db/id entity) + entity + (throw (ex-info "page not found" {:code :page-not-found})))) + + :else + (p/rejected (ex-info "target is required" {:code :missing-target})))) + +;; Position mapping for move-blocks opts. +(defn- pos->opts + [pos] + (case pos + "last-child" {:sibling? false :bottom? true} + "sibling" {:sibling? true} + {:sibling? false})) + +(defn build-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for move"}} + (let [id (:id options) + uuid (some-> (:uuid options) string/trim) + target-id (:target-id options) + target-uuid (some-> (:target-uuid options) string/trim) + page-name (some-> (:page-name options) string/trim) + pos (some-> (:pos options) string/trim string/lower-case) + source-label (cond + (seq uuid) uuid + (some? id) (str id) + :else nil) + target-label (cond + (seq page-name) (str "page:" page-name) + (seq target-uuid) target-uuid + (some? target-id) (str target-id) + :else nil)] + (cond + (not (or (some? id) (seq uuid))) + {:ok? false + :error {:code :missing-source + :message "source block is required"}} + + (not (or (some? target-id) (seq target-uuid) (seq page-name))) + {:ok? false + :error {:code :missing-target + :message "target is required"}} + + :else + {:ok? true + :action {:type :move-block + :repo repo + :graph (core/repo->graph repo) + :id id + :uuid uuid + :target-id target-id + :target-uuid target-uuid + :page-name page-name + :pos (or pos "first-child") + :source source-label + :target target-label}})))) + +(defn execute-move + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + source (resolve-source cfg (:repo action) action) + target (resolve-target cfg (:repo action) action) + opts (pos->opts (:pos action)) + ops [[:move-blocks [[(:db/id source)] (:db/id target) opts]]] + result (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) ops {}])] + {:status :ok + :data {:result result}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index bff5d14b9a..7c1d6aa3b5 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -6,6 +6,7 @@ [logseq.cli.command.core :as command-core] [logseq.cli.command.graph :as graph-command] [logseq.cli.command.list :as list-command] + [logseq.cli.command.move :as move-command] [logseq.cli.command.remove :as remove-command] [logseq.cli.command.search :as search-command] [logseq.cli.command.server :as server-command] @@ -46,6 +47,13 @@ :message "block or page is required"} :summary summary}) +(defn- missing-source-result + [summary] + {:ok? false + :error {:code :missing-source + :message "source block is required"} + :summary summary}) + (defn- missing-page-name-result [summary] {:ok? false @@ -90,6 +98,7 @@ server-command/entries list-command/entries add-command/entries + move-command/entries remove-command/entries search-command/entries show-command/entries))) @@ -112,7 +121,11 @@ (seq (:blocks opts)) (seq (:blocks-file opts)) has-args?) - show-targets (filter some? [(:id opts) (:uuid opts) (:page-name opts)])] + show-targets (filter some? [(:id opts) (:uuid opts) (:page-name opts)]) + move-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) + move-targets (filter some? [(:target-id opts) + (some-> (:target-uuid opts) string/trim) + (some-> (:page-name opts) string/trim)])] (cond (:help opts) (command-core/help-result cmd-summary) @@ -133,6 +146,15 @@ (and (= command :remove-page) (not (seq (:page opts)))) (missing-target-result summary) + (and (= command :move-block) (move-command/invalid-options? opts)) + (command-core/invalid-options-result summary (move-command/invalid-options? opts)) + + (and (= command :move-block) (empty? move-sources)) + (missing-source-result summary) + + (and (= command :move-block) (empty? move-targets)) + (missing-target-result summary) + (and (= command :show) (empty? show-targets)) (missing-target-result summary) @@ -302,6 +324,9 @@ :add-page (add-command/build-add-page-action options repo) + :move-block + (move-command/build-action options repo) + :remove-block (remove-command/build-remove-block-action options repo) @@ -344,6 +369,7 @@ :list-property (list-command/execute-list-property action config) :add-block (add-command/execute-add-block action config) :add-page (add-command/execute-add-page action config) + :move-block (move-command/execute-move action config) :remove-block (remove-command/execute-remove action config) :remove-page (remove-command/execute-remove action config) :search (search-command/execute-search action config) @@ -358,4 +384,4 @@ :message "unknown action"}}))] (assoc result :command (or (:command action) (:type action)) - :context (select-keys action [:repo :graph :page :block :blocks]))))) + :context (select-keys action [:repo :graph :page :block :blocks :source :target]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 4b872ea8d6..8c8869af92 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -218,6 +218,10 @@ [{:keys [repo block]}] (str "Removed block: " block " (repo: " repo ")")) +(defn- format-move-block + [{:keys [repo source target]}] + (str "Moved block: " source " -> " target " (repo: " repo ")")) + (defn- format-graph-export [{:keys [export-type output]}] (str "Exported " export-type " to " output)) @@ -256,6 +260,7 @@ :add-page (format-add-page context) :remove-page (format-remove-page context) :remove-block (format-remove-block context) + :move-block (format-move-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) :search (format-search-results (:results data)) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index eb5f1ab1cb..e540212a88 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 7d99d921ab..7bbb287748 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -17,6 +17,7 @@ (is (string/includes? summary "list")) (is (string/includes? summary "add")) (is (string/includes? summary "remove")) + (is (string/includes? summary "move")) (is (string/includes? summary "search")) (is (string/includes? summary "show")) (is (string/includes? summary "graph")) @@ -61,6 +62,13 @@ (is (string/includes? summary "remove block")) (is (string/includes? summary "remove page")))) + (testing "move command shows help" + (let [result (commands/parse-args ["move" "--help"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "Usage: logseq move")) + (is (string/includes? summary "Command options:")))) + (testing "server group shows subcommands" (let [result (commands/parse-args ["server"]) summary (:summary result)] @@ -290,7 +298,25 @@ (let [result (commands/parse-args ["remove" "page" "--page" "Home"])] (is (true? (:ok? result))) (is (= :remove-page (:command result))) - (is (= "Home" (get-in result [:options :page])))))) + (is (= "Home" (get-in result [:options :page]))))) + + (testing "move requires source selector" + (let [result (commands/parse-args ["move" "--target-id" "10"])] + (is (false? (:ok? result))) + (is (= :missing-source (get-in result [:error :code]))))) + + (testing "move requires target selector" + (let [result (commands/parse-args ["move" "--id" "1"])] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "move parses with source and target" + (let [result (commands/parse-args ["move" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] + (is (true? (:ok? result))) + (is (= :move-block (:command result))) + (is (= "abc" (get-in result [:options :uuid]))) + (is (= "def" (get-in result [:options :target-uuid]))) + (is (= "last-child" (get-in result [:options :pos])))))) (deftest test-verb-subcommand-parse-search-show (testing "search requires text" @@ -366,6 +392,7 @@ (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] ["remove" "block" "--wat"] + ["move" "--wat"] ["search" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] @@ -377,31 +404,13 @@ (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) -(deftest test-build-action +(deftest test-build-action-graph (testing "graph-list uses list-db" (let [parsed {:ok? true :command :graph-list :options {}} result (commands/build-action parsed {})] (is (true? (:ok? result))) (is (= :graph-list (get-in result [:action :type]))))) - (testing "server list builds action" - (let [parsed {:ok? true :command :server-list :options {}} - result (commands/build-action parsed {})] - (is (true? (:ok? result))) - (is (= :server-list (get-in result [:action :type]))))) - - (testing "server start requires repo" - (let [parsed {:ok? true :command :server-start :options {}} - result (commands/build-action parsed {})] - (is (false? (:ok? result))) - (is (= :missing-repo (get-in result [:error :code]))))) - - (testing "server stop builds action" - (let [parsed {:ok? true :command :server-stop :options {:repo "demo"}} - result (commands/build-action parsed {})] - (is (true? (:ok? result))) - (is (= :server-stop (get-in result [:action :type]))))) - (testing "graph-create requires repo name" (let [parsed {:ok? true :command :graph-create :options {}} result (commands/build-action parsed {})] @@ -434,8 +443,28 @@ :options {:type "edn" :input "import.edn"}} result (commands/build-action parsed {})] (is (false? (:ok? result))) + (is (= :missing-repo (get-in result [:error :code])))))) + +(deftest test-build-action-server + (testing "server list builds action" + (let [parsed {:ok? true :command :server-list :options {}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :server-list (get-in result [:action :type]))))) + + (testing "server start requires repo" + (let [parsed {:ok? true :command :server-start :options {}} + result (commands/build-action parsed {})] + (is (false? (:ok? result))) (is (= :missing-repo (get-in result [:error :code]))))) + (testing "server stop builds action" + (let [parsed {:ok? true :command :server-stop :options {:repo "demo"}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :server-stop (get-in result [:action :type])))))) + +(deftest test-build-action-inspect-edit (testing "list page requires repo" (let [parsed {:ok? true :command :list-page :options {}} result (commands/build-action parsed {})] @@ -478,6 +507,40 @@ (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code])))))) +(deftest test-build-action-move + (testing "move requires source selector" + (let [parsed {:ok? true :command :move-block :options {:target-id 2}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-source (get-in result [:error :code]))))) + + (testing "move requires target selector" + (let [parsed {:ok? true :command :move-block :options {:id 1}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code])))))) + +(deftest test-move-parse-validation + (testing "move rejects multiple source selectors" + (let [result (commands/parse-args ["move" "--id" "1" "--uuid" "abc" "--target-id" "2"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "move rejects multiple target selectors" + (let [result (commands/parse-args ["move" "--id" "1" "--target-id" "2" "--target-uuid" "def"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "move rejects invalid position" + (let [result (commands/parse-args ["move" "--id" "1" "--target-id" "2" "--pos" "middle"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "move rejects sibling pos for page target" + (let [result (commands/parse-args ["move" "--id" "1" "--page-name" "Home" "--pos" "sibling"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) + (deftest test-execute-requires-existing-graph (async done (with-redefs [cli-server/list-graphs (fn [_] []) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 7ab050f1d7..a0c13c830a 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -103,7 +103,17 @@ :page "Home"} :data {:result {:ok true}}} {:output-format nil})] - (is (= "Removed page: Home (repo: demo-repo)" result))))) + (is (= "Removed page: Home (repo: demo-repo)" result)))) + + (testing "move block renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :move-block + :context {:repo "demo-repo" + :source "source-uuid" + :target "target-uuid"} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Moved block: source-uuid -> target-uuid (repo: demo-repo)" result))))) (deftest test-human-output-graph-import-export (testing "graph export renders a succinct success line" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index fde2079428..82e8feb84c 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -30,6 +30,21 @@ [result] (reader/read-string (:output result))) +(defn- node-title + [node] + (or (:block/title node) (:title node))) + +(defn- node-children + [node] + (or (:block/children node) (:children node))) + +(defn- find-block-by-title + [node title] + (when node + (if (= title (node-title node)) + node + (some #(find-block-by-title % title) (node-children node))))) + (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -110,6 +125,38 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-move-block + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "block" "--page" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) + source-show (run-cli ["--repo" "move-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) + source-payload (parse-json-output source-show) + parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") + parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + _ (run-cli ["--repo" "move-graph" "add" "block" "--parent" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) + move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--page-name" "TargetPage"] data-dir cfg-path) + move-payload (parse-json-output move-result) + target-show (run-cli ["--repo" "move-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-payload (parse-json-output target-show) + moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") + child-node (find-block-by-title moved-node "Child Block") + stop-result (run-cli ["server" "stop" "--repo" "move-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status move-payload))) + (is (some? parent-uuid)) + (is (some? moved-node)) + (is (some? child-node)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-output-formats-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From c2793e443f8566c73871f0778e761ad4f8a9f144 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 18:57:47 +0800 Subject: [PATCH 026/375] add 009-cli-add-pos-show-tree-align.md --- .../009-cli-add-pos-show-tree-align.md | 110 ++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 docs/agent-guide/009-cli-add-pos-show-tree-align.md diff --git a/docs/agent-guide/009-cli-add-pos-show-tree-align.md b/docs/agent-guide/009-cli-add-pos-show-tree-align.md new file mode 100644 index 0000000000..e822d4b36d --- /dev/null +++ b/docs/agent-guide/009-cli-add-pos-show-tree-align.md @@ -0,0 +1,110 @@ +# CLI Add Pos And Show Tree Alignment Implementation Plan + +Goal: Add --pos support to logseq-cli add block, rename move --page-name to --target-page-name, and fix show tree alignment when db/id widths differ. + +Architecture: Extend the logseq-cli command layer to parse and validate add block target options and --pos, map it to outliner insert options sent over db-worker-node, and update tree rendering to use a fixed-width id column computed from the tree. + +Tech Stack: ClojureScript, babashka.cli, promesa, db-worker-node thread-api. + +Related: Builds on docs/agent-guide/008-logseq-cli-move-command.md. + +## Problem statement + +The logseq-cli add block command always inserts at the bottom, and it cannot express first-child or sibling insertion positions, and it relies on --page/--parent targets instead of explicit target ids. + +The move command uses --page-name for page targets, which is inconsistent with the target naming used by other move flags. + +The show command renders a text tree with id prefixes, but the glyph column shifts when db/id digit widths differ, making the tree hard to read. + +We need to add a --pos option to add block that mirrors existing move semantics and fix show tree alignment for variable id widths. + +## Testing Plan + +I will add a unit test that parses add block with --target-id/--target-uuid/--target-page-name and --pos and validates invalid pos values, ensuring it is rejected with a clear error. + +I will add a unit test that parses move with --target-page-name and rejects --page-name as an unknown option. + +I will add a unit test for show tree rendering that uses mixed-width db/id values and verifies glyph alignment is consistent. + +I will add an integration test that inserts two blocks with different --pos values using the new target flags and verifies the resulting order via show output or a query in db-worker-node. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation Plan + +1. Read the existing add block command spec and execution path in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs. + +2. Read the existing move --pos implementation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to mirror the allowed values and option mapping, and to scope the rename from --page-name to --target-page-name. + +3. Write a failing unit test that parses add block with --target-id/--target-uuid/--target-page-name and --pos in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. + +4. Write a failing unit test for show tree alignment with mixed-width ids in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. + +5. Run the CLI unit tests to confirm both tests fail for the correct reasons. + +6. Replace add block target options in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs with --target-id, --target-uuid, --target-page-name, and add :pos to the spec so help text includes the option. + +7. Add an invalid-options? helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to validate allowed positions and to reject sibling positioning when the target is a page or when no target is supplied. + +8. Wire add-block invalid option checks into command validation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs. + +9. Update build-add-block-action in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to normalize :pos, resolve --target-* options into a single target selector, and include it in the action payload while keeping the default behavior as last-child. + +10. Update execute-add-block in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to resolve the target selector and translate :pos into insert-blocks options, using the same mapping as move and keeping compatibility with db-worker-node. + +11. Rename move --page-name to --target-page-name in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs and update parsing in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs. + +12. Update any move-related tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to use --target-page-name and to assert --page-name is rejected. + +13. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to compute a fixed id column width from all nodes in the tree and pad id cells consistently for all rows and multiline continuations. + +14. Update the show tree unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to reflect the new alignment behavior. + +15. Write a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that uses db-worker-node to insert blocks with different --pos values and checks the resulting order. + +16. Run the specific unit tests and the new integration test to verify they pass. + +17. Run the full CLI test suite with bb dev:lint-and-test to ensure no regressions. + +## Edge cases + +Add --pos sibling with --target-page-name should be rejected because a page target has no sibling context. + +Add --pos values should be case-insensitive and trimmed, matching move semantics. + +Show tree rendering should keep alignment for nodes that have no db/id or use a placeholder. + +Multiline block titles should continue to render under the glyph column even with mixed-width ids. + +## Testing Details + +Unit tests cover add --pos parsing and show tree alignment with mixed-width ids and multiline titles to validate visible behavior rather than internal helpers. + +Integration tests cover db-worker-node insert ordering by creating a page and inserting blocks with first-child and last-child positions using the new target flags, then asserting order via show output or a query. + +Move command tests cover renaming --page-name to --target-page-name and ensure the legacy flag is rejected. + +## Implementation Details + +- Replace add block target flags with --target-id, --target-uuid, --target-page-name and add :pos to the spec in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs. +- Mirror move position values and mapping to {:sibling? false} and {:sibling? false :bottom? true} and {:sibling? true}. +- Keep default add behavior as last-child when --pos is omitted for backward compatibility. +- Reject --pos sibling when the target is a page or when no target is provided. +- Normalize and validate :pos and target selector in add command parsing to avoid leaking invalid values to db-worker-node. +- Rename move --page-name to --target-page-name in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs and update parsing and help output accordingly. +- Compute max id width in tree->text by traversing root and descendants before rendering lines. +- Build id padding from the max width and use it for both first rows and multiline continuation rows. +- Update tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to assert alignment with mixed-width ids. +- Ensure db-worker-node invocations remain unchanged aside from the extra insert options map. + +## Question + +Should --pos default to last-child to match current add behavior, or should it default to first-child for consistency with move. + +Answer: Default to last-child to preserve current add behavior. + +Should the old --page and --parent flags be removed immediately or supported as deprecated aliases for one release. + +Answer: Remove immediately. + +--- From d48b857d6bd90f908dbe4b06e2d9c9f3a3ce4333 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 19:21:26 +0800 Subject: [PATCH 027/375] impl 009-cli-add-pos-show-tree-align.md --- docs/cli/logseq-cli.md | 12 ++-- src/main/logseq/cli/command/add.cljs | 69 ++++++++++++++++++----- src/main/logseq/cli/command/move.cljs | 18 +++--- src/main/logseq/cli/command/show.cljs | 21 +++++-- src/main/logseq/cli/commands.cljs | 5 +- src/test/logseq/cli/commands_test.cljs | 41 +++++++++++++- src/test/logseq/cli/integration_test.cljs | 40 +++++++++++-- 7 files changed, 163 insertions(+), 43 deletions(-) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index b4b074f417..4caa1601ec 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -63,11 +63,11 @@ Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags - `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties -- `add block --content [--page ] [--parent ]` - add blocks; defaults to today’s journal page if no page is given -- `add block --blocks [--page ] [--parent ]` - insert blocks via EDN vector -- `add block --blocks-file [--page ] [--parent ]` - insert blocks from an EDN file +- `add block --content [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - add blocks; defaults to today’s journal page if no target is given +- `add block --blocks [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector +- `add block --blocks-file [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file - `add page --page ` - create a page -- `move --id |--uuid --target-id |--target-uuid |--page-name [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) +- `move --id |--uuid --target-id |--target-uuid |--target-page-name [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) - `remove block --block ` - remove a block and its children - `remove page --page ` - remove a page and its children - `search --text [--type page|block|tag|property|all] [--include-content] [--limit ]` - search across pages, blocks, tags, and properties @@ -117,8 +117,8 @@ Examples: node ./static/logseq-cli.js graph create --repo demo node ./static/logseq-cli.js graph export --type edn --output /tmp/demo.edn --repo demo node ./static/logseq-cli.js graph import --type edn --input /tmp/demo.edn --repo demo-import -node ./static/logseq-cli.js add block --page TestPage --content "hello world" -node ./static/logseq-cli.js move --uuid --page-name TargetPage +node ./static/logseq-cli.js add block --target-page-name TestPage --content "hello world" +node ./static/logseq-cli.js move --uuid --target-page-name TargetPage node ./static/logseq-cli.js search --text "hello" node ./static/logseq-cli.js show --page-name TestPage --format json --output json node ./static/logseq-cli.js server list diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 4569a1eaf2..f9c31566f7 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -15,8 +15,11 @@ {:content {:desc "Block content for add"} :blocks {:desc "EDN vector of blocks for add"} :blocks-file {:desc "EDN file of blocks for add"} - :page {:desc "Page name"} - :parent {:desc "Parent block UUID for add"}}) + :target-id {:desc "Target block db/id" + :coerce :long} + :target-uuid {:desc "Target block UUID"} + :target-page-name {:desc "Target page name"} + :pos {:desc "Position (first-child, last-child, sibling)"}}) (def ^:private add-page-spec {:page {:desc "Page name"}}) @@ -44,17 +47,50 @@ (transport/invoke config :thread-api/pull false [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]]))))) +(def ^:private add-positions + #{"first-child" "last-child" "sibling"}) + +(defn invalid-options? + [opts] + (let [pos (some-> (:pos opts) string/trim string/lower-case) + target-id (:target-id opts) + target-uuid (some-> (:target-uuid opts) string/trim) + target-page (some-> (:target-page-name opts) string/trim) + target-selectors (filter some? [target-id target-uuid target-page])] + (cond + (and (seq pos) (not (contains? add-positions pos))) + (str "invalid pos: " (:pos opts)) + + (> (count target-selectors) 1) + "only one of --target-id, --target-uuid, or --target-page-name is allowed" + + (and (= pos "sibling") (or (seq target-page) (empty? target-selectors))) + "--pos sibling is only valid for block targets" + + :else + nil))) + (defn- resolve-add-target - [config {:keys [repo page parent]}] - (if (seq parent) - (if-not (common-util/uuid-string? parent) - (p/rejected (ex-info "parent must be a uuid" {:code :invalid-parent})) + [config {:keys [repo target-id target-uuid target-page-name]}] + (cond + (some? target-id) + (p/let [block (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/title] target-id])] + (if-let [id (:db/id block)] + id + (throw (ex-info "target block not found" {:code :target-not-found})))) + + (seq target-uuid) + (if-not (common-util/uuid-string? target-uuid) + (p/rejected (ex-info "target must be a uuid" {:code :invalid-target})) (p/let [block (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/title] [:block/uuid (uuid parent)]])] + [repo [:db/id :block/uuid :block/title] [:block/uuid (uuid target-uuid)]])] (if-let [id (:db/id block)] id - (throw (ex-info "parent block not found" {:code :parent-not-found}))))) - (p/let [page-name (if (seq page) page (today-page-title config repo)) + (throw (ex-info "target block not found" {:code :target-not-found}))))) + + :else + (p/let [page-name (if (seq target-page-name) target-page-name (today-page-title config repo)) page-entity (ensure-page! config repo page-name)] (or (:db/id page-entity) (throw (ex-info "page not found" {:code :page-not-found})))))) @@ -104,8 +140,10 @@ :action {:type :add-block :repo repo :graph (core/repo->graph repo) - :page (:page options) - :parent (:parent options) + :target-id (:target-id options) + :target-uuid (some-> (:target-uuid options) string/trim) + :target-page-name (some-> (:target-page-name options) string/trim) + :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") :blocks (:value vector-result)}})))))) (defn build-add-page-action @@ -129,11 +167,14 @@ [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) target-id (resolve-add-target cfg action) + pos (:pos action) + opts (case pos + "last-child" {:sibling? false :bottom? true} + "sibling" {:sibling? true} + {:sibling? false}) ops [[:insert-blocks [(:blocks action) target-id - {:sibling? false - :bottom? true - :outliner-op :insert-blocks}]]] + (assoc opts :outliner-op :insert-blocks)]]] result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])] {:status :ok :data {:result result}}))) diff --git a/src/main/logseq/cli/command/move.cljs b/src/main/logseq/cli/command/move.cljs index 4f57d0bc97..60453ea61a 100644 --- a/src/main/logseq/cli/command/move.cljs +++ b/src/main/logseq/cli/command/move.cljs @@ -14,7 +14,7 @@ :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} - :page-name {:desc "Target page name"} + :target-page-name {:desc "Target page name"} :pos {:desc "Position (first-child, last-child, sibling)"}}) (def entries @@ -29,7 +29,7 @@ source-selectors (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) target-selectors (filter some? [(:target-id opts) (:target-uuid opts) - (some-> (:page-name opts) string/trim)])] + (some-> (:target-page-name opts) string/trim)])] (cond (and (seq pos) (not (contains? move-positions pos))) (str "invalid pos: " (:pos opts)) @@ -38,9 +38,9 @@ "only one of --id or --uuid is allowed" (> (count target-selectors) 1) - "only one of --target-id, --target-uuid, or --page-name is allowed" + "only one of --target-id, --target-uuid, or --target-page-name is allowed" - (and (= pos "sibling") (seq (some-> (:page-name opts) string/trim))) + (and (= pos "sibling") (seq (some-> (:target-page-name opts) string/trim))) "--pos sibling is only valid for block targets" :else @@ -86,7 +86,7 @@ (p/rejected (ex-info "source is required" {:code :missing-source})))) (defn- resolve-target - [config repo {:keys [target-id target-uuid page-name]}] + [config repo {:keys [target-id target-uuid target-page-name]}] (cond (some? target-id) (p/let [entity (transport/invoke config :thread-api/pull false @@ -103,10 +103,10 @@ (ensure-non-page entity "target must be a block" :invalid-target) (throw (ex-info "target block not found" {:code :target-not-found}))))) - (seq page-name) + (seq target-page-name) (p/let [entity (transport/invoke config :thread-api/pull false [repo [:db/id :block/uuid :block/name :block/title] - [:block/name page-name]])] + [:block/name target-page-name]])] (if (:db/id entity) entity (throw (ex-info "page not found" {:code :page-not-found})))) @@ -132,7 +132,7 @@ uuid (some-> (:uuid options) string/trim) target-id (:target-id options) target-uuid (some-> (:target-uuid options) string/trim) - page-name (some-> (:page-name options) string/trim) + page-name (some-> (:target-page-name options) string/trim) pos (some-> (:pos options) string/trim string/lower-case) source-label (cond (seq uuid) uuid @@ -163,7 +163,7 @@ :uuid uuid :target-id target-id :target-uuid target-uuid - :page-name page-name + :target-page-name page-name :pos (or pos "first-child") :source source-label :target target-label}})))) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 85ed029851..d738bb3a0e 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -119,8 +119,17 @@ (or (:block/title node) (:block/name node) (str (:block/uuid node)))) node-id (fn [node] (or (:db/id node) "-")) - id-padding (fn [node] - (apply str (repeat (inc (count (str (node-id node)))) " "))) + collect-nodes (fn collect-nodes [node] + (if-let [children (:block/children node)] + (into [node] (mapcat collect-nodes children)) + [node])) + nodes (collect-nodes root) + id-width (apply max (map (fn [node] (count (str (node-id node)))) nodes)) + pad-id (fn [node] + (let [id-str (str (node-id node)) + padding (max 0 (- id-width (count id-str)))] + (str id-str (apply str (repeat padding " "))))) + id-padding (apply str (repeat (inc id-width) " ")) split-lines (fn [value] (string/split (or value "") #"\n")) lines (atom []) @@ -134,17 +143,17 @@ rows (split-lines (label child)) first-row (first rows) rest-rows (rest rows) - line (str (node-id child) " " prefix branch first-row)] + line (str (pad-id child) " " prefix branch first-row)] (swap! lines conj line) (doseq [row rest-rows] - (swap! lines conj (str (id-padding child) next-prefix row))) + (swap! lines conj (str id-padding next-prefix row))) (walk child next-prefix)))))] (let [rows (split-lines (label root)) first-row (first rows) rest-rows (rest rows)] - (swap! lines conj (str (node-id root) " " first-row)) + (swap! lines conj (str (pad-id root) " " first-row)) (doseq [row rest-rows] - (swap! lines conj (str (id-padding root) row)))) + (swap! lines conj (str id-padding row)))) (walk root "") (string/join "\n" @lines))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 7c1d6aa3b5..1da9b0e9ff 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -125,7 +125,7 @@ move-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) move-targets (filter some? [(:target-id opts) (some-> (:target-uuid opts) string/trim) - (some-> (:page-name opts) string/trim)])] + (some-> (:target-page-name opts) string/trim)])] (cond (:help opts) (command-core/help-result cmd-summary) @@ -137,6 +137,9 @@ (and (= command :add-block) (not has-content?)) (missing-content-result summary) + (and (= command :add-block) (add-command/invalid-options? opts)) + (command-core/invalid-options-result summary (add-command/invalid-options? opts)) + (and (= command :add-page) (not (seq (:page opts)))) (missing-page-name-result summary) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 7bbb287748..de8a888ee5 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -166,6 +166,23 @@ "4 └── Child B") (tree->text tree-data)))))) +(deftest test-tree->text-aligns-mixed-id-widths + (testing "show tree text aligns glyph column with mixed-width ids" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 7 + :block/title "Root" + :block/children [{:db/id 88 + :block/title "Child A" + :block/children [{:db/id 3 + :block/title "Grand"}]} + {:db/id 1000 + :block/title "Child B"}]}}] + (is (= (str "7 Root\n" + "88 ├── Child A\n" + "3 │ └── Grand\n" + "1000 └── Child B") + (tree->text tree-data)))))) + (deftest test-tree->text-multiline (testing "show tree text renders multiline blocks under glyph column" (let [tree->text #'show-command/tree->text @@ -272,6 +289,23 @@ (is (= :add-block (:command result))) (is (= "hello" (get-in result [:options :content]))))) + (testing "add block parses with target selectors and pos" + (let [result (commands/parse-args ["add" "block" + "--content" "hello" + "--target-uuid" "abc" + "--pos" "first-child"])] + (is (true? (:ok? result))) + (is (= :add-block (:command result))) + (is (= "abc" (get-in result [:options :target-uuid]))) + (is (= "first-child" (get-in result [:options :pos]))))) + + (testing "add block rejects invalid pos" + (let [result (commands/parse-args ["add" "block" + "--content" "hello" + "--pos" "middle"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "add page requires page name" (let [result (commands/parse-args ["add" "page"])] (is (false? (:ok? result))) @@ -537,7 +571,12 @@ (is (= :invalid-options (get-in result [:error :code]))))) (testing "move rejects sibling pos for page target" - (let [result (commands/parse-args ["move" "--id" "1" "--page-name" "Home" "--pos" "sibling"])] + (let [result (commands/parse-args ["move" "--id" "1" "--target-page-name" "Home" "--pos" "sibling"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "move rejects legacy page-name option" + (let [result (commands/parse-args ["move" "--id" "1" "--page-name" "Home"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 82e8feb84c..ef5941fe0d 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -96,7 +96,7 @@ list-tag-payload (parse-json-output list-tag-result) list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--page" "TestPage" "--content" "hello world"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "hello world"] data-dir cfg-path) _ (parse-json-output add-block-result) search-result (run-cli ["--repo" "content-graph" "search" "--text" "hello world"] data-dir cfg-path) search-payload (parse-json-output search-result) @@ -133,13 +133,13 @@ _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "block" "--page" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) source-show (run-cli ["--repo" "move-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) source-payload (parse-json-output source-show) parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) - _ (run-cli ["--repo" "move-graph" "add" "block" "--parent" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) - move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--page-name" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) + move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--target-page-name" "TargetPage"] data-dir cfg-path) move-payload (parse-json-output move-result) target-show (run-cli ["--repo" "move-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) target-payload (parse-json-output target-show) @@ -157,6 +157,34 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-add-block-pos-ordering + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) + parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) + parent-payload (parse-json-output parent-show) + parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") + parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "first-child" "--content" "First"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) + final-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) + final-payload (parse-json-output final-show) + final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") + child-titles (map node-title (node-children final-parent)) + stop-result (run-cli ["server" "stop" "--repo" "add-pos-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? parent-uuid)) + (is (= ["First" "Last"] (vec child-titles))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-output-formats-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -268,7 +296,7 @@ export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) _ (run-cli ["--repo" export-graph "add" "page" "--page" "ExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--page" "ExportPage" "--content" "Export content"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "ExportPage" "--content" "Export content"] data-dir cfg-path) export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "edn" @@ -307,7 +335,7 @@ export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) _ (run-cli ["--repo" export-graph "add" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--page" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "sqlite" From 700b16ebfb4f0342751e99da8b1737c423c16818 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 19:56:36 +0800 Subject: [PATCH 028/375] add 010-logseq-cli-show-linked-references.md --- .../010-logseq-cli-show-linked-references.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/agent-guide/010-logseq-cli-show-linked-references.md diff --git a/docs/agent-guide/010-logseq-cli-show-linked-references.md b/docs/agent-guide/010-logseq-cli-show-linked-references.md new file mode 100644 index 0000000000..aed6232eef --- /dev/null +++ b/docs/agent-guide/010-logseq-cli-show-linked-references.md @@ -0,0 +1,78 @@ +# Logseq CLI Show Linked References Implementation Plan + +Goal: Add task status prefixes to show output and include linked references for the shown block or page. + +Architecture: The CLI show command will fetch marker data with the tree and will build a display label that prefixes the marker before the block content. +Architecture: The CLI show command will also call db-worker-node thread-api/get-block-refs to fetch linked references and append them to text output while returning structured data in JSON and EDN output. +Architecture: Data flow will remain CLI -> db-worker-node -> db-core with no new worker endpoints, reusing existing thread-api functions. + +Tech Stack: ClojureScript, promesa, logseq-cli transport, db-worker-node thread-api, Datascript. + +Related: Builds on docs/agent-guide/009-cli-add-pos-show-tree-align.md. + +## Testing Plan + +I will follow @test-driven-development and write failing tests before any production changes. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text prefixes :block/marker before the block title for TODO and CANCELED blocks. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the marker prefix is present. +I will add an integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page and a referencing block, runs show with --format json, and asserts that linked references are present and include the referencing block uuid and page title. +I will run the new unit tests with bb dev:test -v logseq.cli.commands-test and the new integration test namespace with bb dev:test -v logseq.cli.integration-test to confirm failures, then again to confirm passing. +I will run bb dev:lint-and-test after implementation to ensure no regressions. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Problem statement + +The current show command prints only the block title or page name without the task marker, which hides task status context in CLI usage. +The show command also does not include linked references for the shown block or page, forcing users to query references separately. +We need to enhance the show output to include task status prefixes and linked references while keeping existing formats and db-worker-node integration stable. + +## Implementation Plan + +1. Read /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to identify the tree-block selector, label construction, and output formatting paths. +2. Read /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs to confirm the behavior and return shape of :thread-api/get-block-refs. +3. Read the existing show tree tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to align new expectations with current formatting rules. +4. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that builds a tree with :block/marker values and asserts the marker prefix appears before the block title in tree->text output. +5. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that covers multiline block titles with markers and asserts continuation lines still align under the glyph column. +6. Add a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page, adds a block referencing that page, runs show for the page in JSON, and asserts the response contains linked references with block uuid and page title. +7. Run the two new unit tests and the integration test to confirm failures for the expected reasons. +8. Update the tree-block selector in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to pull :block/marker alongside :block/title and :block/name. +9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :block/marker followed by a space when a marker is present, while falling back to :block/name or :block/uuid when :block/title is missing. +10. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use the new label helper for both root and child nodes so that marker prefixes appear in all output lines. +11. Add a linked references fetch step in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that calls transport/invoke with :thread-api/get-block-refs using the root db/id. +12. Normalize linked references by pulling a minimal selector for each ref block, including :db/id, :block/uuid, :block/title, :block/marker, and {:block/page [:db/id :block/name :block/title :block/uuid]}, so CLI output is predictable and lightweight. +13. Extend the show tree data structure to include :linked-references with a list of normalized blocks and a :count, and ensure this structure is returned for JSON and EDN output paths. +14. For text output, append a Linked References section after the tree that lists each referencing block with its page title and marker-prefixed label, and show a count line when references exist. +15. Update the unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to match the marker-prefixed text output. +16. Update the new integration test assertions to match the final JSON structure for linked references. +17. Run the targeted tests again, then bb dev:lint-and-test, to verify all changes pass. + +## Edge cases + +Blocks without :block/marker should render exactly as before with no extra spacing. +Blocks with :block/title nil should still render using :block/name or :block/uuid with the marker prefix applied only when a title exists. +Linked references can be empty, in which case the Linked References section should be omitted from text output and :linked-references should contain a zero count in JSON and EDN. +Linked reference blocks that are missing a page or title should still render using their uuid fallback. +Show by block id and show by page name should both resolve linked references using the root db/id. + +## Testing Details + +The unit tests exercise tree->text output formatting to ensure marker prefixes appear on the first line and multiline alignment is preserved, which validates CLI-visible behavior. +The integration test uses db-worker-node to create actual referencing blocks and verifies that show output includes linked references in the JSON response without inspecting internal worker details. + +## Implementation Details + +- Pull :block/marker in the tree selector so task status is available for label rendering. +- Build a label helper that prefixes markers without changing existing fallback logic for titles and names. +- Append a Linked References section in text output with a header, count, and marker-prefixed block labels. +- Use :thread-api/get-block-refs for reference discovery and re-pull a minimal selector for stable CLI output. +- Return linked references in JSON and EDN outputs as {:linked-references {:count n :blocks [...]}}. +- Keep all changes inside the CLI show command and avoid new db-worker-node endpoints. + +## Question + +Should the marker prefix use the stored uppercase value (for example CANCELED) or should it be title-cased to match the example with Canceled. +Answer: Use the stored uppercase marker value (for example CANCELED). +Should linked references be grouped by page in text output, or listed as a flat list with page labels. +Answer: Group linked references by page in text output. + +--- From 5d6e46e46d5ac52ad81dcc91b39aaec5b7b305b5 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 21:55:55 +0800 Subject: [PATCH 029/375] impl 010-logseq-cli-show-linked-references.md (1) --- .../010-logseq-cli-show-linked-references.md | 18 +- src/main/logseq/cli/command/show.cljs | 233 +++++++++++++++++- src/test/logseq/cli/commands_test.cljs | 74 ++++++ src/test/logseq/cli/integration_test.cljs | 41 +++ 4 files changed, 351 insertions(+), 15 deletions(-) diff --git a/docs/agent-guide/010-logseq-cli-show-linked-references.md b/docs/agent-guide/010-logseq-cli-show-linked-references.md index aed6232eef..f420b962c2 100644 --- a/docs/agent-guide/010-logseq-cli-show-linked-references.md +++ b/docs/agent-guide/010-logseq-cli-show-linked-references.md @@ -1,9 +1,9 @@ # Logseq CLI Show Linked References Implementation Plan -Goal: Add task status prefixes to show output and include linked references for the shown block or page. +Goal: Add task status prefixes and block tag rendering to show output, replace inline `[[]]` references with `[[]]`, and include linked references for the shown block or page. -Architecture: The CLI show command will fetch marker data with the tree and will build a display label that prefixes the marker before the block content. -Architecture: The CLI show command will also call db-worker-node thread-api/get-block-refs to fetch linked references and append them to text output while returning structured data in JSON and EDN output. +Architecture: The CLI show command will fetch marker data with the tree and will build a display label that prefixes the marker before the block content, replaces inline `[[]]` with `[[]]`, and then appends inline block tags (e.g. `#RTC #Task`) to the content display. +Architecture: The CLI show command will also call db-worker-node thread-api/get-block-refs to fetch linked references and append them to text output in a tree form (db/id in the first column), while returning structured data in JSON and EDN output. Architecture: Data flow will remain CLI -> db-worker-node -> db-core with no new worker endpoints, reusing existing thread-api functions. Tech Stack: ClojureScript, promesa, logseq-cli transport, db-worker-node thread-api, Datascript. @@ -14,6 +14,7 @@ Related: Builds on docs/agent-guide/009-cli-add-pos-show-tree-align.md. I will follow @test-driven-development and write failing tests before any production changes. I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text prefixes :block/marker before the block title for TODO and CANCELED blocks. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text appends :block/tags in `#Tag` format to the rendered block content. I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the marker prefix is present. I will add an integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page and a referencing block, runs show with --format json, and asserts that linked references are present and include the referencing block uuid and page title. I will run the new unit tests with bb dev:test -v logseq.cli.commands-test and the new integration test namespace with bb dev:test -v logseq.cli.integration-test to confirm failures, then again to confirm passing. @@ -36,12 +37,12 @@ We need to enhance the show output to include task status prefixes and linked re 6. Add a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page, adds a block referencing that page, runs show for the page in JSON, and asserts the response contains linked references with block uuid and page title. 7. Run the two new unit tests and the integration test to confirm failures for the expected reasons. 8. Update the tree-block selector in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to pull :block/marker alongside :block/title and :block/name. -9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :block/marker followed by a space when a marker is present, while falling back to :block/name or :block/uuid when :block/title is missing. +9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :block/marker followed by a space when a marker is present, while falling back to :block/name or :block/uuid when :block/title is missing, and appending `#Tag` strings for :block/tags. 10. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use the new label helper for both root and child nodes so that marker prefixes appear in all output lines. 11. Add a linked references fetch step in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that calls transport/invoke with :thread-api/get-block-refs using the root db/id. 12. Normalize linked references by pulling a minimal selector for each ref block, including :db/id, :block/uuid, :block/title, :block/marker, and {:block/page [:db/id :block/name :block/title :block/uuid]}, so CLI output is predictable and lightweight. 13. Extend the show tree data structure to include :linked-references with a list of normalized blocks and a :count, and ensure this structure is returned for JSON and EDN output paths. -14. For text output, append a Linked References section after the tree that lists each referencing block with its page title and marker-prefixed label, and show a count line when references exist. +14. For text output, append a Linked References section after the tree that renders each referencing block in tree form with db/id in the first column (aligned to the glyph column), include the marker-prefixed label, and show a count line when references exist. 15. Update the unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to match the marker-prefixed text output. 16. Update the new integration test assertions to match the final JSON structure for linked references. 17. Run the targeted tests again, then bb dev:lint-and-test, to verify all changes pass. @@ -49,9 +50,12 @@ We need to enhance the show output to include task status prefixes and linked re ## Edge cases Blocks without :block/marker should render exactly as before with no extra spacing. +Blocks with :block/tags should render tag names as `#Tag` appended after the content. +Blocks containing inline `[[]]` references should render those tokens replaced with `[[]]`. Blocks with :block/title nil should still render using :block/name or :block/uuid with the marker prefix applied only when a title exists. Linked references can be empty, in which case the Linked References section should be omitted from text output and :linked-references should contain a zero count in JSON and EDN. Linked reference blocks that are missing a page or title should still render using their uuid fallback. +Linked references should render using the same tree layout rules as the main show tree, including db/id in the first column. Show by block id and show by page name should both resolve linked references using the root db/id. ## Testing Details @@ -62,8 +66,8 @@ The integration test uses db-worker-node to create actual referencing blocks and ## Implementation Details - Pull :block/marker in the tree selector so task status is available for label rendering. -- Build a label helper that prefixes markers without changing existing fallback logic for titles and names. -- Append a Linked References section in text output with a header, count, and marker-prefixed block labels. +- Build a label helper that prefixes markers without changing existing fallback logic for titles and names, replaces inline `[[]]` tokens with `[[]]`, and appends block tags (e.g. `#RTC #Task`) after the content. +- Append a Linked References section in text output with a header, count, and tree-formatted block labels (db/id in the first column). - Use :thread-api/get-block-refs for reference discovery and re-pull a minimal selector for stable CLI output. - Return linked references in JSON and EDN outputs as {:linked-references {:count n :blocks [...]}}. - Keep all changes inside the CLI show command and avoid new db-worker-node endpoints. diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index d738bb3a0e..f37a7e45e9 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -37,7 +37,195 @@ nil))) (def ^:private tree-block-selector - [:db/id :block/uuid :block/title :block/order {:block/parent [:db/id]}]) + [:db/id + :block/uuid + :block/title + :block/marker + :block/order + {:block/parent [:db/id]} + {:block/tags [:db/id :block/name :block/title :block/uuid]}]) + +(def ^:private linked-ref-selector + [:db/id + :block/uuid + :block/title + :block/marker + {:block/tags [:db/id :block/name :block/title :block/uuid]} + {:block/page [:db/id :block/name :block/title :block/uuid]} + {:block/parent [:db/id + :block/name + :block/title + :block/uuid + {:block/page [:db/id :block/name :block/title :block/uuid]}]}]) + +(declare tree->text) + +(def ^:private uuid-ref-pattern #"\[\[([0-9a-fA-F-]{36})\]\]") + +(defn- tag-label + [tag] + (or (:block/title tag) + (:block/name tag) + (some-> (:block/uuid tag) str))) + +(defn- replace-uuid-refs + [value uuid->label] + (if (and (string? value) (seq uuid->label)) + (string/replace value uuid-ref-pattern + (fn [[_ id]] + (if-let [label (get uuid->label (string/lower-case id))] + (str "[[" label "]]") + (str "[[" id "]]")))) + value)) + +(defn- tags->suffix + [tags] + (let [labels (->> tags + (map tag-label) + (remove string/blank?))] + (when (seq labels) + (string/join " " (map #(str "#" %) labels))))) + +(defn- block-label + [node] + (let [title (:block/title node) + marker (:block/marker node) + uuid->label (:uuid->label node) + base (cond + (and title (seq marker)) (str marker " " title) + title title + (:block/name node) (:block/name node) + (:block/uuid node) (some-> (:block/uuid node) str)) + base (replace-uuid-refs base uuid->label) + tags-suffix (tags->suffix (:block/tags node))] + (cond + (and base tags-suffix) (str base " " tags-suffix) + tags-suffix tags-suffix + :else base))) + +(defn- page-label + [block] + (let [page (:block/page block)] + (or (:block/title page) + (:block/name page) + (some-> (:block/uuid page) str) + (some-> (:block/uuid block) str)))) + +(defn- fetch-linked-references + [config repo root-id] + (p/let [refs (transport/invoke config :thread-api/get-block-refs false [repo root-id]) + ref-ids (vec (keep :db/id refs)) + pulled (if (seq ref-ids) + (p/all (map (fn [id] + (transport/invoke config :thread-api/pull false [repo linked-ref-selector id])) + ref-ids)) + [])] + (let [blocks (vec (remove nil? pulled)) + page-id-from (fn [block] + (let [page (:block/page block) + parent (:block/parent block) + parent-page (:block/page parent)] + (or (when (map? page) (:db/id page)) + (when (number? page) page) + (when (map? parent-page) (:db/id parent-page)) + (when (number? parent-page) parent-page) + (when (map? parent) (:db/id parent))))) + page-ids (->> blocks + (keep page-id-from) + distinct + vec)] + (p/let [pages (if (seq page-ids) + (p/all (map (fn [id] + (transport/invoke config :thread-api/pull false + [repo [:db/id :block/name :block/title :block/uuid] id])) + page-ids)) + [])] + (let [page-id->page (zipmap page-ids pages) + blocks (mapv (fn [block] + (let [page (:block/page block) + parent (:block/parent block) + parent-page (:block/page parent) + page-id (page-id-from block) + page (or (when (map? page) + (select-keys page [:db/id :block/name :block/title :block/uuid])) + (when (map? parent-page) + (select-keys parent-page [:db/id :block/name :block/title :block/uuid])) + (get page-id->page page-id) + (when (map? parent) + (select-keys parent [:db/id :block/name :block/title :block/uuid])) + (when (or (:block/title block) (:block/name block) (:block/uuid block)) + (select-keys block [:db/id :block/name :block/title :block/uuid])))] + (cond-> (dissoc block :block/parent) + page (assoc :block/page page)))) + blocks)] + {:count (count blocks) + :blocks blocks}))))) + +(defn- linked-refs->text + [blocks uuid->label] + (let [page-key (fn [block] + (let [page (:block/page block)] + (or (:db/id page) + (:block/uuid page) + (page-label block)))) + page-node (fn [block] + (let [page (:block/page block)] + (cond + (map? page) page + (some? page) {:db/id page} + :else {}))) + groups (->> blocks + (group-by page-key) + (sort-by (fn [[_ page-blocks]] + (page-label (first page-blocks)))))] + (string/join + "\n\n" + (map (fn [[_ page-blocks]] + (let [root (page-node (first page-blocks)) + root (assoc root :block/children (vec page-blocks))] + (tree->text {:root root :uuid->label uuid->label}))) + groups)))) + +(defn- extract-uuid-refs + [value] + (->> (re-seq uuid-ref-pattern (or value "")) + (map second) + (filter common-util/uuid-string?) + distinct)) + +(defn- collect-uuid-refs + [{:keys [root]} linked-refs] + (let [collect-nodes (fn collect-nodes [node] + (if-let [children (:block/children node)] + (into [node] (mapcat collect-nodes children)) + [node])) + nodes (when root (collect-nodes root)) + ref-blocks (:blocks linked-refs) + pages (keep :block/page ref-blocks) + texts (->> (concat nodes ref-blocks pages) + (mapcat (fn [node] (keep node [:block/title :block/name]))) + (remove string/blank?))] + (->> texts + (mapcat extract-uuid-refs) + distinct + vec))) + +(defn- fetch-uuid-labels + [config repo uuid-strings] + (if (seq uuid-strings) + (p/let [blocks (p/all (map (fn [uuid-str] + (transport/invoke config :thread-api/pull false + [repo [:block/uuid :block/title :block/name] + [:block/uuid (uuid uuid-str)]])) + uuid-strings))] + (->> blocks + (remove nil?) + (map (fn [block] + (let [uuid-str (some-> (:block/uuid block) str)] + [(string/lower-case uuid-str) + (or (:block/title block) (:block/name block) uuid-str)]))) + (into {}))) + (p/resolved {}))) (defn- fetch-blocks-for-page [config repo page-id] @@ -69,7 +257,9 @@ (cond (some? id) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] id])] + [repo [:db/id :block/name :block/uuid :block/title :block/marker + {:block/page [:db/id :block/title]} + {:block/tags [:db/id :block/name :block/title :block/uuid]}] id])] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks (:db/id entity) max-depth)] @@ -84,12 +274,16 @@ (if-not (common-util/uuid-string? uuid-str) (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [repo [:db/id :block/name :block/uuid :block/title :block/marker + {:block/page [:db/id :block/title]} + {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid (uuid uuid-str)]]) entity (if (:db/id entity) entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title {:block/page [:db/id :block/title]}] + [repo [:db/id :block/name :block/uuid :block/title :block/marker + {:block/page [:db/id :block/title]} + {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid uuid-str]]))] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) @@ -103,7 +297,9 @@ (seq page-name) (p/let [page-entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/title] [:block/name page-name]])] + [repo [:db/id :block/uuid :block/title :block/marker + {:block/tags [:db/id :block/name :block/title :block/uuid]}] + [:block/name page-name]])] (if-let [page-id (:db/id page-entity)] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks page-id max-depth)] @@ -114,9 +310,9 @@ (p/rejected (ex-info "block or page required" {:code :missing-target}))))) (defn tree->text - [{:keys [root]}] + [{:keys [root uuid->label]}] (let [label (fn [node] - (or (:block/title node) (:block/name node) (str (:block/uuid node)))) + (or (block-label (assoc node :uuid->label uuid->label)) "-")) node-id (fn [node] (or (:db/id node) "-")) collect-nodes (fn collect-nodes [node] @@ -157,6 +353,18 @@ (walk root "") (string/join "\n" @lines))) +(defn- tree->text-with-linked-refs + [{:keys [linked-references uuid->label] :as tree-data}] + (let [tree-text (tree->text tree-data) + refs (:blocks linked-references) + count (:count linked-references)] + (if (seq refs) + (str tree-text + "\n\n" + "Linked References (" count ")\n" + (linked-refs->text refs uuid->label)) + tree-text))) + (defn build-action [options repo] (if-not (seq repo) @@ -182,6 +390,15 @@ [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) tree-data (fetch-tree cfg action) + root-id (get-in tree-data [:root :db/id]) + linked-refs (if root-id + (fetch-linked-references cfg (:repo action) root-id) + {:count 0 :blocks []}) + uuid-refs (collect-uuid-refs tree-data linked-refs) + uuid->label (fetch-uuid-labels cfg (:repo action) uuid-refs) + tree-data (assoc tree-data + :linked-references linked-refs + :uuid->label uuid->label) format (:format action)] (case format "edn" @@ -195,4 +412,4 @@ :output-format :json} {:status :ok - :data {:message (tree->text tree-data)}})))) + :data {:message (tree->text-with-linked-refs tree-data)}})))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index de8a888ee5..4bfa7635a6 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -204,6 +204,80 @@ "175 └── cccc") (tree->text tree-data)))))) +(deftest test-tree->text-prefixes-marker + (testing "show tree text prefixes markers before block titles" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/marker "TODO" + :block/children [{:db/id 2 + :block/title "Child" + :block/marker "CANCELED"}]}}] + (is (= (str "1 TODO Root\n" + "2 └── CANCELED Child") + (tree->text tree-data)))))) + +(deftest test-tree->text-marker-multiline-alignment + (testing "show tree text keeps multiline alignment when marker prefix is present" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 22 + :block/title "line1\nline2" + :block/marker "TODO"}]}}] + (is (= (str "1 Root\n" + "22 └── TODO line1\n" + " line2") + (tree->text tree-data)))))) + +(deftest test-tree->text-linked-references-tree + (testing "show tree text renders linked references as trees with db/id in first column" + (let [tree->text-with-linked-refs #'show-command/tree->text-with-linked-refs + tree-data {:root {:db/id 1 + :block/title "Root"} + :linked-references {:count 2 + :blocks [{:db/id 10 + :block/title "Ref A" + :block/marker "TODO" + :block/page {:db/id 100 + :block/title "Page A"}} + {:db/id 11 + :block/title "Ref B" + :block/page {:db/id 101 + :block/title "Page B"}}]}}] + (is (= (str "1 Root\n" + "\n" + "Linked References (2)\n" + "100 Page A\n" + "10 └── TODO Ref A\n" + "\n" + "101 Page B\n" + "11 └── Ref B") + (tree->text-with-linked-refs tree-data)))))) + +(deftest test-tree->text-appends-tags + (testing "show tree text appends block tags to content" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child" + :block/tags [{:block/title "RTC"} + {:block/name "task"}]}]}}] + (is (= (str "1 Root\n" + "2 └── Child #RTC #task") + (tree->text tree-data)))))) + +(deftest test-tree->text-replaces-uuid-refs + (testing "show tree text replaces inline [[uuid]] with referenced block content" + (let [tree->text #'show-command/tree->text + uuid "11111111-1111-1111-1111-111111111111" + tree-data {:root {:db/id 1 + :block/title (str "See [[" uuid "]]")} + :uuid->label {(string/lower-case uuid) "Target block"}}] + (is (= (str "1 See [[Target block]]") + (tree->text tree-data)))))) + (deftest test-list-subcommand-parse (testing "list page parses" (let [result (commands/parse-args ["list" "page" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index ef5941fe0d..884ba41607 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -286,6 +286,47 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-show-linked-references + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + list-page-result (run-cli ["--repo" "linked-refs-graph" "list" "page" "--expand"] + data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-item (some (fn [item] + (when (= "TargetPage" (or (:block/title item) (:title item))) + item)) + (get-in list-page-payload [:data :items])) + page-id (or (:db/id page-item) (:id page-item)) + blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") + _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" + "--blocks" blocks-edn] data-dir cfg-path) + show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] + data-dir cfg-path) + show-payload (parse-json-output show-result) + linked (get-in show-payload [:data :linked-references]) + ref-block (first (:blocks linked)) + stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status show-payload))) + (is (some? page-id)) + (is (map? linked)) + (is (pos? (:count linked))) + (is (seq (:blocks linked))) + (is (some? ref-block)) + (is (some? (or (:block/uuid ref-block) (:uuid ref-block)))) + (is (some? (or (get-in ref-block [:page :title]) + (get-in ref-block [:page :name])))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-graph-export-import-edn (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-export-edn")] From 33b8a3207dbcaec343d85ddbac5c948843fb09ec Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 20 Jan 2026 22:35:46 +0800 Subject: [PATCH 030/375] impl 010-logseq-cli-show-linked-references.md (2) --- .../010-logseq-cli-show-linked-references.md | 38 ++++++------- src/main/logseq/cli/command/show.cljs | 56 ++++++++++++++----- src/test/logseq/cli/commands_test.cljs | 20 ++++--- src/test/logseq/cli/integration_test.cljs | 44 +++++++++++++++ 4 files changed, 118 insertions(+), 40 deletions(-) diff --git a/docs/agent-guide/010-logseq-cli-show-linked-references.md b/docs/agent-guide/010-logseq-cli-show-linked-references.md index f420b962c2..1b3b788d83 100644 --- a/docs/agent-guide/010-logseq-cli-show-linked-references.md +++ b/docs/agent-guide/010-logseq-cli-show-linked-references.md @@ -2,7 +2,7 @@ Goal: Add task status prefixes and block tag rendering to show output, replace inline `[[]]` references with `[[]]`, and include linked references for the shown block or page. -Architecture: The CLI show command will fetch marker data with the tree and will build a display label that prefixes the marker before the block content, replaces inline `[[]]` with `[[]]`, and then appends inline block tags (e.g. `#RTC #Task`) to the content display. +Architecture: The CLI show command will fetch status data via `:logseq.property/status` with the tree and will build a display label that prefixes the status before the block content, replaces inline `[[]]` with `[[]]`, and then appends inline block tags (e.g. `#RTC #Task`) to the content display. Architecture: The CLI show command will also call db-worker-node thread-api/get-block-refs to fetch linked references and append them to text output in a tree form (db/id in the first column), while returning structured data in JSON and EDN output. Architecture: Data flow will remain CLI -> db-worker-node -> db-core with no new worker endpoints, reusing existing thread-api functions. @@ -13,9 +13,9 @@ Related: Builds on docs/agent-guide/009-cli-add-pos-show-tree-align.md. ## Testing Plan I will follow @test-driven-development and write failing tests before any production changes. -I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text prefixes :block/marker before the block title for TODO and CANCELED blocks. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text prefixes :logseq.property/status before the block title for TODO and CANCELED blocks. I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text appends :block/tags in `#Tag` format to the rendered block content. -I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the marker prefix is present. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the status prefix is present. I will add an integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page and a referencing block, runs show with --format json, and asserts that linked references are present and include the referencing block uuid and page title. I will run the new unit tests with bb dev:test -v logseq.cli.commands-test and the new integration test namespace with bb dev:test -v logseq.cli.integration-test to confirm failures, then again to confirm passing. I will run bb dev:lint-and-test after implementation to ensure no regressions. @@ -23,7 +23,7 @@ NOTE: I will write *all* tests before I add any implementation behavior. ## Problem statement -The current show command prints only the block title or page name without the task marker, which hides task status context in CLI usage. +The current show command prints only the block title or page name without the task status, which hides task status context in CLI usage. The show command also does not include linked references for the shown block or page, forcing users to query references separately. We need to enhance the show output to include task status prefixes and linked references while keeping existing formats and db-worker-node integration stable. @@ -32,27 +32,27 @@ We need to enhance the show output to include task status prefixes and linked re 1. Read /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to identify the tree-block selector, label construction, and output formatting paths. 2. Read /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs to confirm the behavior and return shape of :thread-api/get-block-refs. 3. Read the existing show tree tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to align new expectations with current formatting rules. -4. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that builds a tree with :block/marker values and asserts the marker prefix appears before the block title in tree->text output. -5. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that covers multiline block titles with markers and asserts continuation lines still align under the glyph column. +4. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that builds a tree with :logseq.property/status values and asserts the status prefix appears before the block title in tree->text output. +5. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that covers multiline block titles with statuses and asserts continuation lines still align under the glyph column. 6. Add a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page, adds a block referencing that page, runs show for the page in JSON, and asserts the response contains linked references with block uuid and page title. 7. Run the two new unit tests and the integration test to confirm failures for the expected reasons. -8. Update the tree-block selector in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to pull :block/marker alongside :block/title and :block/name. -9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :block/marker followed by a space when a marker is present, while falling back to :block/name or :block/uuid when :block/title is missing, and appending `#Tag` strings for :block/tags. -10. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use the new label helper for both root and child nodes so that marker prefixes appear in all output lines. +8. Update the tree-block selector in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to pull :logseq.property/status alongside :block/title and :block/name. +9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :logseq.property/status followed by a space when a status is present, while falling back to :block/name or :block/uuid when :block/title is missing, and appending `#Tag` strings for :block/tags. +10. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use the new label helper for both root and child nodes so that status prefixes appear in all output lines. 11. Add a linked references fetch step in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that calls transport/invoke with :thread-api/get-block-refs using the root db/id. -12. Normalize linked references by pulling a minimal selector for each ref block, including :db/id, :block/uuid, :block/title, :block/marker, and {:block/page [:db/id :block/name :block/title :block/uuid]}, so CLI output is predictable and lightweight. +12. Normalize linked references by pulling a minimal selector for each ref block, including :db/id, :block/uuid, :block/title, :logseq.property/status, and {:block/page [:db/id :block/name :block/title :block/uuid]}, so CLI output is predictable and lightweight. 13. Extend the show tree data structure to include :linked-references with a list of normalized blocks and a :count, and ensure this structure is returned for JSON and EDN output paths. -14. For text output, append a Linked References section after the tree that renders each referencing block in tree form with db/id in the first column (aligned to the glyph column), include the marker-prefixed label, and show a count line when references exist. -15. Update the unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to match the marker-prefixed text output. +14. For text output, append a Linked References section after the tree that renders each referencing block in tree form with db/id in the first column (aligned to the glyph column), include the status-prefixed label, and show a count line when references exist. +15. Update the unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to match the status-prefixed text output. 16. Update the new integration test assertions to match the final JSON structure for linked references. 17. Run the targeted tests again, then bb dev:lint-and-test, to verify all changes pass. ## Edge cases -Blocks without :block/marker should render exactly as before with no extra spacing. +Blocks without :logseq.property/status should render exactly as before with no extra spacing. Blocks with :block/tags should render tag names as `#Tag` appended after the content. Blocks containing inline `[[]]` references should render those tokens replaced with `[[]]`. -Blocks with :block/title nil should still render using :block/name or :block/uuid with the marker prefix applied only when a title exists. +Blocks with :block/title nil should still render using :block/name or :block/uuid with the status prefix applied only when a title exists. Linked references can be empty, in which case the Linked References section should be omitted from text output and :linked-references should contain a zero count in JSON and EDN. Linked reference blocks that are missing a page or title should still render using their uuid fallback. Linked references should render using the same tree layout rules as the main show tree, including db/id in the first column. @@ -60,13 +60,13 @@ Show by block id and show by page name should both resolve linked references usi ## Testing Details -The unit tests exercise tree->text output formatting to ensure marker prefixes appear on the first line and multiline alignment is preserved, which validates CLI-visible behavior. +The unit tests exercise tree->text output formatting to ensure status prefixes appear on the first line and multiline alignment is preserved, which validates CLI-visible behavior. The integration test uses db-worker-node to create actual referencing blocks and verifies that show output includes linked references in the JSON response without inspecting internal worker details. ## Implementation Details -- Pull :block/marker in the tree selector so task status is available for label rendering. -- Build a label helper that prefixes markers without changing existing fallback logic for titles and names, replaces inline `[[]]` tokens with `[[]]`, and appends block tags (e.g. `#RTC #Task`) after the content. +- Pull :logseq.property/status in the tree selector so task status is available for label rendering. +- Build a label helper that prefixes status values without changing existing fallback logic for titles and names, replaces inline `[[]]` tokens with `[[]]`, and appends block tags (e.g. `#RTC #Task`) after the content. - Append a Linked References section in text output with a header, count, and tree-formatted block labels (db/id in the first column). - Use :thread-api/get-block-refs for reference discovery and re-pull a minimal selector for stable CLI output. - Return linked references in JSON and EDN outputs as {:linked-references {:count n :blocks [...]}}. @@ -74,8 +74,8 @@ The integration test uses db-worker-node to create actual referencing blocks and ## Question -Should the marker prefix use the stored uppercase value (for example CANCELED) or should it be title-cased to match the example with Canceled. -Answer: Use the stored uppercase marker value (for example CANCELED). +Should the status prefix use the stored uppercase value (for example CANCELED) or should it be title-cased to match the example with Canceled. +Answer: Use the stored uppercase status value (for example CANCELED). Should linked references be grouped by page in text output, or listed as a flat list with page labels. Answer: Group linked references by page in text output. diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index f37a7e45e9..430280dc83 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -40,7 +40,7 @@ [:db/id :block/uuid :block/title - :block/marker + {:logseq.property/status [:db/ident :block/name :block/title]} :block/order {:block/parent [:db/id]} {:block/tags [:db/id :block/name :block/title :block/uuid]}]) @@ -49,7 +49,7 @@ [:db/id :block/uuid :block/title - :block/marker + {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]} {:block/page [:db/id :block/name :block/title :block/uuid]} {:block/parent [:db/id @@ -86,13 +86,32 @@ (when (seq labels) (string/join " " (map #(str "#" %) labels))))) +(defn- status-from-ident + [ident] + (let [name* (name ident) + parts (string/split name* #"\.") + status (or (last parts) name*)] + (string/upper-case status))) + +(defn- status-label + [node] + (let [status (:logseq.property/status node)] + (cond + (string? status) (when (seq status) status) + (keyword? status) (status-from-ident status) + (map? status) (or (:block/title status) + (:block/name status) + (when-let [ident (:db/ident status)] + (status-from-ident ident))) + :else nil))) + (defn- block-label [node] (let [title (:block/title node) - marker (:block/marker node) + status (status-label node) uuid->label (:uuid->label node) base (cond - (and title (seq marker)) (str marker " " title) + (and title (seq status)) (str status " " title) title title (:block/name node) (:block/name node) (:block/uuid node) (some-> (:block/uuid node) str)) @@ -121,15 +140,22 @@ ref-ids)) [])] (let [blocks (vec (remove nil? pulled)) + page-lookup-key (fn [value] + (cond + (map? value) (or (:db/id value) + (when-let [uuid (:block/uuid value)] + [:block/uuid uuid])) + (number? value) value + (uuid? value) [:block/uuid value] + (and (string? value) (common-util/uuid-string? value)) [:block/uuid (uuid value)] + :else nil)) page-id-from (fn [block] (let [page (:block/page block) parent (:block/parent block) parent-page (:block/page parent)] - (or (when (map? page) (:db/id page)) - (when (number? page) page) - (when (map? parent-page) (:db/id parent-page)) - (when (number? parent-page) parent-page) - (when (map? parent) (:db/id parent))))) + (or (page-lookup-key page) + (page-lookup-key parent-page) + (page-lookup-key parent)))) page-ids (->> blocks (keep page-id-from) distinct @@ -257,7 +283,8 @@ (cond (some? id) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title :block/marker + [repo [:db/id :block/name :block/uuid :block/title + {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] id])] (if-let [page-id (get-in entity [:block/page :db/id])] @@ -274,14 +301,16 @@ (if-not (common-util/uuid-string? uuid-str) (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title :block/marker + [repo [:db/id :block/name :block/uuid :block/title + {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid (uuid uuid-str)]]) entity (if (:db/id entity) entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title :block/marker + [repo [:db/id :block/name :block/uuid :block/title + {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid uuid-str]]))] @@ -297,7 +326,8 @@ (seq page-name) (p/let [page-entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/title :block/marker + [repo [:db/id :block/uuid :block/title + {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/name page-name]])] (if-let [page-id (:db/id page-entity)] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 4bfa7635a6..8c983635b8 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -204,27 +204,30 @@ "175 └── cccc") (tree->text tree-data)))))) -(deftest test-tree->text-prefixes-marker - (testing "show tree text prefixes markers before block titles" +(deftest test-tree->text-prefixes-status + (testing "show tree text prefixes status before block titles" (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 1 :block/title "Root" - :block/marker "TODO" + :logseq.property/status {:db/ident :logseq.property/status.todo + :block/title "TODO"} :block/children [{:db/id 2 :block/title "Child" - :block/marker "CANCELED"}]}}] + :logseq.property/status {:db/ident :logseq.property/status.canceled + :block/title "CANCELED"}}]}}] (is (= (str "1 TODO Root\n" "2 └── CANCELED Child") (tree->text tree-data)))))) -(deftest test-tree->text-marker-multiline-alignment - (testing "show tree text keeps multiline alignment when marker prefix is present" +(deftest test-tree->text-status-multiline-alignment + (testing "show tree text keeps multiline alignment when status prefix is present" (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 1 :block/title "Root" :block/children [{:db/id 22 :block/title "line1\nline2" - :block/marker "TODO"}]}}] + :logseq.property/status {:db/ident :logseq.property/status.todo + :block/title "TODO"}}]}}] (is (= (str "1 Root\n" "22 └── TODO line1\n" " line2") @@ -238,7 +241,8 @@ :linked-references {:count 2 :blocks [{:db/id 10 :block/title "Ref A" - :block/marker "TODO" + :logseq.property/status {:db/ident :logseq.property/status.todo + :block/title "TODO"} :block/page {:db/id 100 :block/title "Page A"}} {:db/id 11 diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 884ba41607..600ef15eee 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -125,6 +125,50 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-show-linked-references-json + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + target-show-before (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-before-payload (parse-json-output target-show-before) + target-uuid (or (get-in target-before-payload [:data :root :block/uuid]) + (get-in target-before-payload [:data :root :uuid])) + ref-content (str "See [[" target-uuid "]]") + _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) + source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) + source-payload (parse-json-output source-show) + ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-content) + ref-uuid (or (:block/uuid ref-node) (:uuid ref-node)) + target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-payload (parse-json-output target-show) + linked-refs (get-in target-payload [:data :linked-references]) + linked-blocks (:blocks linked-refs) + linked-uuids (set (map (fn [block] + (or (:block/uuid block) (:uuid block))) + linked-blocks)) + linked-page-titles (set (keep (fn [block] + (or (get-in block [:block/page :block/title]) + (get-in block [:block/page :block/name]) + (get-in block [:page :title]) + (get-in block [:page :name]))) + linked-blocks)) + stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? target-uuid)) + (is (= "ok" (:status target-payload))) + (is (some? ref-uuid)) + (is (contains? linked-uuids ref-uuid)) + (is (contains? linked-page-titles "SourcePage")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-move-block (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] From 7a9d0181ec9301d568941c9d3cae9b5cae8a2fc0 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 21 Jan 2026 22:33:16 +0800 Subject: [PATCH 031/375] 011-logseq-cli-search-optimization.md --- .../agent-guide/002-logseq-cli-subcommands.md | 4 +- .../004-logseq-cli-verb-subcommands.md | 11 +- .../011-logseq-cli-search-optimization.md | 111 +++++++++ docs/cli/logseq-cli.md | 9 +- src/main/logseq/cli/command/core.cljs | 17 +- src/main/logseq/cli/command/search.cljs | 211 +++++++++++++----- src/main/logseq/cli/command/show.cljs | 38 +++- src/main/logseq/cli/commands.cljs | 2 +- src/main/logseq/cli/format.cljs | 11 +- src/test/logseq/cli/commands_test.cljs | 72 +++++- src/test/logseq/cli/format_test.cljs | 54 +++-- src/test/logseq/cli/integration_test.cljs | 60 ++++- 12 files changed, 487 insertions(+), 113 deletions(-) create mode 100644 docs/agent-guide/011-logseq-cli-search-optimization.md diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index 5824f5c475..836a34ab16 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -43,7 +43,7 @@ The CLI will expose these subcommands and shared output controls. | graph info | Graph metadata | human, json, edn | Replaces graph-info | | block add | Add blocks | human, json, edn | Replaces add | | block remove | Remove block or page | human, json, edn | Replaces remove | -| block search | Search blocks | human, json, edn | Replaces search | +| search | Search graph | human, json, edn | Replaces search | | block tree | Show tree | human, json, edn | Replaces tree | The plan assumes a single global output flag that defaults to human, and each subcommand may also accept it. @@ -74,7 +74,7 @@ Each subcommand uses a nested path and its own options. | graph info | none | --repo GRAPH, --output | Shows metadata, defaults to config repo if omitted. | | block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. | | block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. | -| block search | none | --text TEXT, --limit N, --output | Search text is required. | +| search | QUERY | --type page|block|tag|property|all, --tag NAME, --case-sensitive, --sort updated-at|created-at, --order asc|desc, --output | Search text is positional and required. Human output columns: ID (db/id), TITLE. Block reference UUIDs in text are resolved recursively up to 10 levels. | | block tree | none | --block UUID, --page PAGE, --format FORMAT, --output | One of block or page is required, and format controls tree rendering. | ## Plan diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md index dd21e48368..e2eb899a03 100644 --- a/docs/agent-guide/004-logseq-cli-verb-subcommands.md +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -103,16 +103,15 @@ List block is removed to avoid overlap with search. ## Search options detail -Search has no subcommands and searches across pages, blocks, tags, and properties by default. +Search has no subcommands and searches across pages, blocks, tags, and properties by default. Human output columns are `ID` (db/id) and `TITLE`. +Search and show outputs resolve block reference UUIDs in text. Nested references are resolved recursively up to 10 levels (e.g., `[[]]` → `[[some text [[]]]]`, then `` is also replaced). | Option | Purpose | Notes | | --- | --- | --- | -| --text QUERY | Search text | Required unless positional args are used. | +| QUERY | Search text | Required positional argument. | | --type page|block|tag|property|all | Restrict types | Default is all. | | --tag NAME | Restrict to a specific tag | Tag is a class page, e.g. Page, Asset, Task. | -| --limit N | Limit results | Apply after merging type results. | | --case-sensitive | Case sensitive search | Default is case-insensitive. | -| --include-content | Search block content, not just title | Requires query expansion. | | --sort updated-at|created-at | Sort results | Default is relevance or stable order. | | --order asc|desc | Sort direction | Defaults to desc for time sorts. | @@ -133,7 +132,7 @@ Show has no subcommands and returns the block tree for a page or block. 1. Review current CLI command parsing and action routing in src/main/logseq/cli/commands.cljs to map block group behavior to verb-first commands. 2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and show. 3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, and list property. -4. Add failing unit tests that assert search defaults to all types and respects --type and --include-content options. +4. Add failing unit tests that assert search defaults to all types and respects --type and positional text. 5. Add failing unit tests that assert show accepts --page-name, --uuid, or --id and rejects missing targets. 6. Run bb dev:test -v logseq.cli.commands-test/test-parse-args and confirm failures are about the new verbs and options. 7. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. @@ -142,7 +141,7 @@ Show has no subcommands and returns the block tree for a page or block. 10. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags. 11. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties. 12. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs. -13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --include-content. +13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --order. 14. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. 15. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. 16. Run bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search and confirm failures before implementation. diff --git a/docs/agent-guide/011-logseq-cli-search-optimization.md b/docs/agent-guide/011-logseq-cli-search-optimization.md new file mode 100644 index 0000000000..921b9812a3 --- /dev/null +++ b/docs/agent-guide/011-logseq-cli-search-optimization.md @@ -0,0 +1,111 @@ +# Logseq CLI Search Optimization Implementation Plan + +Goal: Simplify the search command arguments and ensure search covers blocks, pages, tags, and properties by default. + +Architecture: The CLI parses args with babashka.cli, builds a search action, and queries db-worker-node through thread-api endpoints for pages, blocks, tags, and properties. +Architecture: The change stays in the CLI layer and relies on existing thread-api methods in db-worker-node for data access. + +Tech Stack: ClojureScript, babashka.cli, Datascript queries, db-worker-node thread-api. + +Related: Relates to docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md. + +## Problem statement + +The current search command requires the --text option or positional args and joins all positional args into the search string. + +The current search documentation and tests are still written around the --text flag and do not describe default type behavior. + +We need to remove the --text option, use the first positional string as the search text, and ensure searches cover block titles, page names, tag names, and property names with --type defaulting to all. + +We also need to remove the --include-content option and any :block/content usage from CLI search because :block/content is no longer present in db-graph. + +We also need to remove the --limit option because it currently only trims output and does not reduce query work. + +## Testing Plan + +I will add unit tests for CLI parsing that assert the search text is taken from the first positional argument and that --type defaults to all when omitted. + +I will add integration tests for the CLI search command that use the positional search text and verify results include at least one matching item from each type when the graph contains data. + +I will add a formatting test that validates the missing search text hint no longer references --text. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope + +This plan updates CLI search option parsing and action building for search types. + +This plan removes --include-content from CLI search options and eliminates :block/content usage from CLI search code and tests. + +This plan updates user-facing docs that describe the search command usage. + +## Non-goals + +This plan does not change db-worker-node thread-api method signatures or introduce new server endpoints. + +This plan does not alter vector search or inference-worker behavior. + +## Expected CLI behavior + +| Scenario | Input | Behavior | +| --- | --- | --- | +| Basic search | logseq-cli search "hello" | Uses "hello" as search text and searches pages, blocks, tags, and properties. | +| Type filter | logseq-cli search "hello" --type page | Searches only pages. | +| Missing text | logseq-cli search | Returns missing-search-text with a hint that positional text is required. | +| Block titles | logseq-cli search "todo" | Matches block titles only, not :block/content. | + +## Implementation Plan + +1. Read @test-driven-development and follow TDD for every behavior change in this plan. +2. Update CLI parsing tests in `src/test/logseq/cli/commands_test.cljs` to use positional search text and verify default types are all when --type is omitted. +3. Run `bb dev:test -v logseq.cli.commands-test` and confirm the new tests fail for the expected reasons. +4. Update integration tests in `src/test/logseq/cli/integration_test.cljs` to call search without --text and to assert results still return ok with data. +5. Run `bb dev:test -v logseq.cli.integration-test` and confirm the new tests fail for the expected reasons. +6. Update formatting expectations in `src/test/logseq/cli/format_test.cljs` if missing-search-text hints change. +7. Run `bb dev:test -v logseq.cli.format-test` and confirm the new tests fail for the expected reasons. +8. Remove the :text, :include-content, and :limit options from search spec in `src/main/logseq/cli/command/search.cljs` and update help text accordingly. +9. Update `src/main/logseq/cli/commands.cljs` to require positional search text and to stop referencing :text in missing-search-text logic. +10. Update `src/main/logseq/cli/command/search.cljs` build-action to use the first positional argument as text and ignore any additional positional args for search text. +11. Remove :block/content references from `src/main/logseq/cli/command/search.cljs` so block searches use block title fields only. +12. Update `src/main/logseq/cli/format.cljs` to remove the hint that references --text. +13. Update docs in `docs/cli/logseq-cli.md`, `docs/agent-guide/004-logseq-cli-verb-subcommands.md`, and `docs/agent-guide/002-logseq-cli-subcommands.md` to remove --text, --include-content, and --limit and document the new positional argument behavior and default --type behavior. +14. Update CLI tests in `src/test/logseq/cli/commands_test.cljs`, `src/test/logseq/cli/integration_test.cljs`, and `src/test/logseq/cli/format_test.cljs` to remove --include-content, --limit, and any :block/content expectations. +15. Run `bb dev:lint-and-test` and confirm all linters and unit tests pass. + +## Edge cases and validation + +Multiple positional arguments should not be concatenated into a single search string unless explicitly required by design. + +Search text containing spaces should still work when the shell passes it as a single quoted argument. + +When --type is provided, only the requested type set should be searched and defaults should not override the filter. + +Search with --tag should continue to filter block searches and should not filter page, tag, or property results unless explicitly required. + +The CLI must not attempt to query :block/content because the attribute is absent in db-graph. + +## Testing Details + +The CLI command tests will assert that a positional search term maps to the action :text and that missing text errors are raised when no positional arguments exist. + +The integration tests will execute the CLI against a sample graph, verify the command exits with ok, and confirm search results are a vector for each type requested. + +The formatting tests will assert the error hint no longer suggests the deprecated --text option. + +The CLI tests will assert that no --include-content or --limit option exists and that searches do not rely on :block/content. + +## Implementation Details + +- Remove :text, :include-content, and :limit from the search option spec and adjust help text generation to avoid advertising those options. +- Update missing-search-text validation to rely only on positional args. +- Use only the first positional argument as the search text to match the new spec. +- Confirm default search types flow through normalize-search-types when --type is absent. +- Remove :block/content usage from query-blocks and any related logic. +- Update docs and examples to show quoted positional search text. +- Ensure error hints reference positional text rather than options. + +## Question + +No open questions. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 4caa1601ec..32188fae58 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -70,7 +70,7 @@ Inspect and edit commands: - `move --id |--uuid --target-id |--target-uuid |--target-page-name [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) - `remove block --block ` - remove a block and its children - `remove page --page ` - remove a page and its children -- `search --text [--type page|block|tag|property|all] [--include-content] [--limit ]` - search across pages, blocks, tags, and properties +- `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) - `show --page-name [--format text|json|edn] [--level ]` - show page tree - `show --uuid [--format text|json|edn] [--level ]` - show block tree - `show --id [--format text|json|edn] [--level ]` - show block tree by db/id @@ -87,7 +87,7 @@ Subcommands: move [options] Move block remove block [options] Remove block remove page [options] Remove page - search [options] Search graph + search [options] Search graph show [options] Show tree ``` @@ -97,7 +97,8 @@ Options grouping: Output formats: - Global `--output ` (also accepted per subcommand) - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. -- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. - `show` human output prints the `:db/id` as the first column followed by a tree: ``` @@ -119,7 +120,7 @@ node ./static/logseq-cli.js graph export --type edn --output /tmp/demo.edn --rep node ./static/logseq-cli.js graph import --type edn --input /tmp/demo.edn --repo demo-import node ./static/logseq-cli.js add block --target-page-name TestPage --content "hello world" node ./static/logseq-cli.js move --uuid --target-page-name TargetPage -node ./static/logseq-cli.js search --text "hello" +node ./static/logseq-cli.js search "hello" node ./static/logseq-cli.js show --page-name TestPage --format json --output json node ./static/logseq-cli.js server list ``` diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 407f6bf558..58d9d9fa75 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -41,13 +41,21 @@ :opts opts :args args})})) +(defn- command-usage + [cmds spec] + (let [base (string/join " " cmds) + positional (when (= cmds ["search"]) "") + has-options? (seq spec)] + (cond-> base + positional (str " " positional) + has-options? (str " [options]")))) + (defn- format-commands [table] (let [rows (->> table (filter (comp seq :cmds)) (map (fn [{:keys [cmds desc spec]}] - (let [command (str (string/join " " cmds) - (when (seq spec) " [options]"))] + (let [command (command-usage cmds spec)] {:command command :desc desc})))) width (apply max 0 (map (comp count :command) rows))] @@ -98,7 +106,7 @@ [{:keys [cmds spec]}] (let [command-spec (apply dissoc spec (keys global-spec*))] (string/join "\n" - [(str "Usage: logseq " (string/join " " cmds) " [options]") + [(str "Usage: logseq " (command-usage cmds spec)) "" "Global options:" (cli/format-opts {:spec global-spec*}) @@ -210,7 +218,6 @@ (graph->repo graph)))) (defn pick-graph - [options command-args config] + [options _command-args config] (or (:repo options) - (first command-args) (:repo config))) diff --git a/src/main/logseq/cli/command/search.cljs b/src/main/logseq/cli/command/search.cljs index 074422324a..c7cff1f65f 100644 --- a/src/main/logseq/cli/command/search.cljs +++ b/src/main/logseq/cli/command/search.cljs @@ -1,21 +1,18 @@ (ns logseq.cli.command.search "Search-related CLI commands." - (:require [clojure.string :as string] + (:require [clojure.set :as set] + [clojure.string :as string] [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] [promesa.core :as p])) (def ^:private search-spec - {:text {:desc "Search text"} - :type {:desc "Search types (page, block, tag, property, all)"} + {:type {:desc "Search types (page, block, tag, property, all)"} :tag {:desc "Restrict to a specific tag"} - :limit {:desc "Limit results" - :coerce :long} :case-sensitive {:desc "Case sensitive search" :coerce :boolean} - :include-content {:desc "Search block content" - :coerce :boolean} :sort {:desc "Sort field (updated-at, created-at)"} :order {:desc "Sort order (asc, desc)"}}) @@ -25,6 +22,9 @@ (def ^:private search-types #{"page" "block" "tag" "property" "all"}) +(def ^:private uuid-ref-pattern #"\[\[([0-9a-fA-F-]{36})\]\]") +(def ^:private uuid-ref-max-depth 10) + (defn invalid-options? [opts] (let [type (:type opts) @@ -49,17 +49,15 @@ {:ok? false :error {:code :missing-repo :message "repo is required for search"}} - (let [text (or (:text options) (string/join " " args))] + (let [text (some-> (first args) string/trim)] (if (seq text) {:ok? true :action {:type :search :repo repo :text text - :search-type (:type options) + :search-type (or (:type options) "all") :tag (:tag options) - :limit (:limit options) :case-sensitive (:case-sensitive options) - :include-content (:include-content options) :sort (:sort options) :order (:order options)}} {:ok? false @@ -86,68 +84,170 @@ [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?title) ?q)]]) + [(clojure.string/includes? ?name ?q)]]) q* (if case-sensitive? text (string/lower-case text))] (transport/invoke cfg :thread-api/q false [repo [query q*]]))) -#_{:clj-kondo/ignore [:aliased-namespace-symbol]} (defn- query-blocks - [cfg repo text case-sensitive? tag include-content?] - (let [has-tag? (seq tag) - content-attr (if include-content? :block/content :block/title) + [cfg repo text case-sensitive? tag] + (let [q* (if case-sensitive? text (string/lower-case text)) + tag-name (some-> tag string/lower-case) query (cond - (and case-sensitive? has-tag?) - `[:find ?e ?value ?uuid ?updated ?created + (and case-sensitive? (seq tag-name)) + '[:find ?e ?title ?uuid ?updated ?created :in $ ?q ?tag-name :where - [?tag :block/name ?tag-name] - [?e :block/tags ?tag] - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] + [?e :block/title ?title] + [?e :block/page ?page] [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?value ?q)]] + [(clojure.string/includes? ?title ?q)] + [?tag :block/name ?tag-name] + [?e :block/tags ?tag]] case-sensitive? - `[:find ?e ?value ?uuid ?updated ?created + '[:find ?e ?title ?uuid ?updated ?created :in $ ?q :where - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] + [?e :block/title ?title] + [?e :block/page ?page] [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?value ?q)]] + [(clojure.string/includes? ?title ?q)]] - has-tag? - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q ?tag-name + (seq tag-name) + '[:find ?e ?title ?uuid ?updated ?created + :in $ ?tag-name :where - [?tag :block/name ?tag-name] - [?e :block/tags ?tag] - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] + [?e :block/title ?title] + [?e :block/page ?page] [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]] + [?tag :block/name ?tag-name] + [?e :block/tags ?tag]] :else - `[:find ?e ?value ?uuid ?updated ?created - :in $ ?q + '[:find ?e ?title ?uuid ?updated ?created :where - [?e ~content-attr ?value] - [(missing? $ ?e :block/name)] + [?e :block/title ?title] + [?e :block/page ?page] [?e :block/uuid ?uuid] [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? (clojure.string/lower-case ?value) ?q)]]) - q* (if case-sensitive? text (string/lower-case text)) - tag-name (some-> tag string/lower-case)] - (if has-tag? - (transport/invoke cfg :thread-api/q false [repo [query q* tag-name]]) - (transport/invoke cfg :thread-api/q false [repo [query q*]])))) + [(get-else $ ?e :block/created-at 0) ?created]]) + query-args (cond + (and case-sensitive? (seq tag-name)) + [repo [query q* tag-name]] + + case-sensitive? + [repo [query q*]] + + (seq tag-name) + [repo [query tag-name]] + + :else + [repo [query]]) + matches-text? (fn [title] + (when (string? title) + (if case-sensitive? + (string/includes? title q*) + (string/includes? (string/lower-case title) q*))))] + (-> (p/let [rows (transport/invoke cfg :thread-api/q false query-args)] + (->> (or rows []) + (filter (fn [[_ title _ _ _]] + (matches-text? title))) + (mapv (fn [[id title uuid updated created]] + {:type "block" + :db/id id + :content title + :uuid (str uuid) + :updated-at updated + :created-at created})))) + (p/catch (fn [_] + []))))) + +(defn- replace-uuid-refs-once + [value uuid->label] + (if (and (string? value) (seq uuid->label)) + (string/replace value uuid-ref-pattern + (fn [[_ id]] + (if-let [label (get uuid->label (string/lower-case id))] + (str "[[" label "]]") + (str "[[" id "]]")))) + value)) + +(defn- replace-uuid-refs + [value uuid->label] + (loop [current value + remaining uuid-ref-max-depth] + (if (or (not (string? current)) (zero? remaining) (empty? uuid->label)) + current + (let [next (replace-uuid-refs-once current uuid->label)] + (if (= next current) + current + (recur next (dec remaining))))))) + +(defn- extract-uuid-refs + [value] + (->> (re-seq uuid-ref-pattern (or value "")) + (map second) + (filter common-util/uuid-string?) + (map string/lower-case) + distinct)) + +(defn- collect-uuid-refs + [results] + (->> results + (mapcat (fn [item] (keep item [:title :content]))) + (remove string/blank?) + (mapcat extract-uuid-refs) + distinct + vec)) + +(defn- fetch-uuid-labels + [config repo uuid-strings] + (if (seq uuid-strings) + (p/let [blocks (p/all (map (fn [uuid-str] + (transport/invoke config :thread-api/pull false + [repo [:block/uuid :block/title :block/name] + [:block/uuid (uuid uuid-str)]])) + uuid-strings))] + (->> blocks + (remove nil?) + (map (fn [block] + (let [uuid-str (some-> (:block/uuid block) str)] + [(string/lower-case uuid-str) + (or (:block/title block) (:block/name block) uuid-str)]))) + (into {}))) + (p/resolved {}))) + +(defn- fetch-uuid-labels-recursive + [config repo uuid-strings] + (p/loop [pending (set (map string/lower-case uuid-strings)) + seen #{} + labels {} + remaining uuid-ref-max-depth] + (if (or (empty? pending) (zero? remaining)) + labels + (p/let [fetched (fetch-uuid-labels config repo pending) + next-labels (merge labels fetched) + next-seen (into seen (keys fetched)) + nested-refs (->> (vals fetched) + (mapcat extract-uuid-refs) + (remove next-seen) + set) + next-pending (set/difference nested-refs next-seen)] + (p/recur next-pending next-seen next-labels (dec remaining)))))) + +(defn- resolve-uuid-refs-in-results + [results uuid->label] + (mapv (fn [item] + (cond-> item + (:title item) (update :title replace-uuid-refs uuid->label) + (:content item) (update :content replace-uuid-refs uuid->label))) + (or results []))) (defn- normalize-search-types [type] @@ -183,17 +283,8 @@ :updated-at updated :created-at created}) rows))) - include-content? (boolean (:include-content action)) block-results (when (some #{:block} types) - (p/let [rows (query-blocks cfg (:repo action) text case-sensitive? tag include-content?)] - (mapv (fn [[id content uuid updated created]] - {:type "block" - :db/id id - :content content - :uuid (str uuid) - :updated-at updated - :created-at created}) - rows))) + (query-blocks cfg (:repo action) text case-sensitive? tag)) tag-results (when (some #{:tag} types) (p/let [items (transport/invoke cfg :thread-api/api-list-tags false [(:repo action) {:expand true :include-built-in true}]) @@ -206,6 +297,7 @@ (string/includes? (string/lower-case title) q*))))) (mapv (fn [item] {:type "tag" + :db/id (:db/id item) :title (:block/title item) :uuid (:block/uuid item)}))))) property-results (when (some #{:property} types) @@ -220,6 +312,7 @@ (string/includes? (string/lower-case title) q*))))) (mapv (fn [item] {:type "property" + :db/id (:db/id item) :title (:block/title item) :uuid (:block/uuid item)}))))) results (->> (concat (or page-results []) @@ -235,6 +328,8 @@ (cond-> (= order "desc") reverse) vec)) results) - limited (if (some? (:limit action)) (vec (take (:limit action) sorted)) sorted)] + uuid-refs (collect-uuid-refs sorted) + uuid->label (fetch-uuid-labels-recursive cfg (:repo action) uuid-refs) + resolved (resolve-uuid-refs-in-results sorted uuid->label)] {:status :ok - :data {:results limited}}))) + :data {:results resolved}}))) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 430280dc83..91e47dcfb4 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -61,6 +61,7 @@ (declare tree->text) (def ^:private uuid-ref-pattern #"\[\[([0-9a-fA-F-]{36})\]\]") +(def ^:private uuid-ref-max-depth 10) (defn- tag-label [tag] @@ -68,7 +69,7 @@ (:block/name tag) (some-> (:block/uuid tag) str))) -(defn- replace-uuid-refs +(defn- replace-uuid-refs-once [value uuid->label] (if (and (string? value) (seq uuid->label)) (string/replace value uuid-ref-pattern @@ -78,6 +79,17 @@ (str "[[" id "]]")))) value)) +(defn- replace-uuid-refs + [value uuid->label] + (loop [current value + remaining uuid-ref-max-depth] + (if (or (not (string? current)) (zero? remaining) (empty? uuid->label)) + current + (let [next (replace-uuid-refs-once current uuid->label)] + (if (= next current) + current + (recur next (dec remaining))))))) + (defn- tags->suffix [tags] (let [labels (->> tags @@ -122,6 +134,29 @@ tags-suffix tags-suffix :else base))) +(defn- resolve-uuid-refs-in-node + [node uuid->label] + (cond-> node + (:block/title node) (update :block/title replace-uuid-refs uuid->label) + (:block/name node) (update :block/name replace-uuid-refs uuid->label) + (:block/children node) (update :block/children (fn [children] + (mapv #(resolve-uuid-refs-in-node % uuid->label) children))) + (:block/page node) (update :block/page (fn [page] + (if (map? page) + (resolve-uuid-refs-in-node page uuid->label) + page))) + (:block/tags node) (update :block/tags (fn [tags] + (mapv #(resolve-uuid-refs-in-node % uuid->label) tags))))) + +(defn- resolve-uuid-refs-in-tree-data + [{:keys [linked-references] :as tree-data} uuid->label] + (let [resolve-node #(resolve-uuid-refs-in-node % uuid->label)] + (cond-> (update tree-data :root resolve-node) + (seq (:blocks linked-references)) + (update :linked-references + (fn [refs] + (update refs :blocks #(mapv resolve-node %))))))) + (defn- page-label [block] (let [page (:block/page block)] @@ -429,6 +464,7 @@ tree-data (assoc tree-data :linked-references linked-refs :uuid->label uuid->label) + tree-data (resolve-uuid-refs-in-tree-data tree-data uuid->label) format (:format action)] (case format "edn" diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 1da9b0e9ff..ca0598bb73 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -164,7 +164,7 @@ (and (= command :show) (> (count show-targets) 1)) (command-core/invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") - (and (= command :search) (not (or (seq (:text opts)) has-args?))) + (and (= command :search) (not has-args?)) (missing-search-result summary) (and (#{:list-page :list-tag :list-property} command) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 8c8869af92..13b1cddc99 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -71,7 +71,7 @@ :missing-graph "Use --repo " :missing-repo "Use --repo " :missing-content "Use --content or pass content as args" - :missing-search-text "Provide search text or --text" + :missing-search-text "Provide search text as a positional argument" nil)) (defn- format-error @@ -166,13 +166,10 @@ (defn- format-search-results [results] (format-counted-table - ["TYPE" "TITLE/CONTENT" "UUID" "UPDATED-AT" "CREATED-AT"] + ["ID" "TITLE"] (mapv (fn [item] - [(:type item) - (or (:title item) (:content item)) - (:uuid item) - (:updated-at item) - (:created-at item)]) + [(:db/id item) + (or (:title item) (:content item))]) (or results [])))) (defn- format-graph-info diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 8c983635b8..19a3005944 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -273,15 +273,44 @@ (tree->text tree-data)))))) (deftest test-tree->text-replaces-uuid-refs - (testing "show tree text replaces inline [[uuid]] with referenced block content" + (testing "show tree text replaces inline [[uuid]] with referenced block content recursively" (let [tree->text #'show-command/tree->text uuid "11111111-1111-1111-1111-111111111111" + nested "22222222-2222-2222-2222-222222222222" tree-data {:root {:db/id 1 :block/title (str "See [[" uuid "]]")} - :uuid->label {(string/lower-case uuid) "Target block"}}] - (is (= (str "1 See [[Target block]]") + :uuid->label {(string/lower-case uuid) (str "Target [[" nested "]]") + (string/lower-case nested) "Inner"}}] + (is (= (str "1 See [[Target [[Inner]]]]") (tree->text tree-data)))))) +(deftest test-tree->text-uuid-ref-recursion-limit + (testing "show tree text limits uuid ref replacement depth" + (let [tree->text #'show-command/tree->text + uuids ["00000000-0000-0000-0000-000000000001" + "00000000-0000-0000-0000-000000000002" + "00000000-0000-0000-0000-000000000003" + "00000000-0000-0000-0000-000000000004" + "00000000-0000-0000-0000-000000000005" + "00000000-0000-0000-0000-000000000006" + "00000000-0000-0000-0000-000000000007" + "00000000-0000-0000-0000-000000000008" + "00000000-0000-0000-0000-000000000009" + "00000000-0000-0000-0000-000000000010" + "00000000-0000-0000-0000-000000000011"] + uuid->label (into {} + (map-indexed (fn [idx id] + (let [label (if (< idx 10) + (str "L" (inc idx) " [[" (nth uuids (inc idx)) "]]") + (str "L" (inc idx)))] + [(string/lower-case id) label])) + uuids)) + tree-data {:root {:db/id 1 + :block/title (str "Root [[" (first uuids) "]]")} + :uuid->label uuid->label} + result (tree->text tree-data)] + (is (string/includes? result (str "[[" (nth uuids 10) "]]")))))) + (deftest test-list-subcommand-parse (testing "list page parses" (let [result (commands/parse-args ["list" "page" @@ -437,10 +466,10 @@ (is (= :missing-search-text (get-in result [:error :code]))))) (testing "search parses with text" - (let [result (commands/parse-args ["search" "--text" "hello"])] + (let [result (commands/parse-args ["search" "hello"])] (is (true? (:ok? result))) (is (= :search (:command result))) - (is (= "hello" (get-in result [:options :text]))))) + (is (= ["hello"] (:args result))))) (testing "show requires target" (let [result (commands/parse-args ["show"])] @@ -454,6 +483,11 @@ (is (= "Home" (get-in result [:options :page-name])))))) (deftest test-verb-subcommand-parse-graph-import-export + (testing "graph create requires --repo even with positional args" + (let [result (commands/parse-args ["graph" "create" "demo"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + (testing "graph export parses with type and output" (let [result (commands/parse-args ["graph" "export" "--type" "edn" @@ -511,8 +545,16 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) + (testing "search rejects deprecated flags" + (doseq [args [["search" "--limit" "10" "hello"] + ["search" "--include-content" "hello"] + ["search" "--text" "hello"]]] + (let [result (commands/parse-args args)] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) + (testing "verb subcommands accept output option" - (let [result (commands/parse-args ["search" "--text" "hello" "--output" "json"])] + (let [result (commands/parse-args ["search" "--output" "json" "hello"])] (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) @@ -613,6 +655,24 @@ (is (false? (:ok? result))) (is (= :missing-search-text (get-in result [:error :code]))))) + (testing "search defaults to all types" + (let [parsed {:ok? true :command :search :options {} :args ["hello"]} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= "all" (get-in result [:action :search-type]))))) + + (testing "search uses config repo and ignores positional text for repo" + (let [parsed {:ok? true :command :search :options {} :args ["hello"]} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= "logseq_db_demo" (get-in result [:action :repo]))))) + + (testing "search uses first positional argument" + (let [parsed {:ok? true :command :search :options {} :args ["hello" "world"]} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= "hello" (get-in result [:action :text]))))) + (testing "show requires target" (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index a0c13c830a..49e1ba19b1 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -164,27 +164,31 @@ (let [result (format/format-result {:status :ok :command :search :data {:results [{:type "page" - :title "Alpha" - :uuid "u1" - :updated-at 3 - :created-at 1} - {:type "block" - :content "Note" - :uuid "u2" - :updated-at 4 - :created-at 2} - {:type "tag" - :title "Taggy" - :uuid "u3"} - {:type "property" - :title "Prop" - :uuid "u4"}]}} + :db/id 101 + :title "Alpha" + :uuid "u1" + :updated-at 3 + :created-at 1} + {:type "block" + :db/id 102 + :content "Note" + :uuid "u2" + :updated-at 4 + :created-at 2} + {:type "tag" + :db/id 103 + :title "Taggy" + :uuid "u3"} + {:type "property" + :db/id 104 + :title "Prop" + :uuid "u4"}]}} {:output-format nil})] - (is (= (str "TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT\n" - "page Alpha u1 3 1\n" - "block Note u2 4 2\n" - "tag Taggy u3 - -\n" - "property Prop u4 - -\n" + (is (= (str "ID TITLE\n" + "101 Alpha\n" + "102 Note\n" + "103 Taggy\n" + "104 Prop\n" "Count: 4") result)))) @@ -204,4 +208,14 @@ {:output-format nil})] (is (= (str "Error (missing-graph): graph name is required\n" "Hint: Use --repo ") + result)))) + + (testing "missing search text hints use positional argument" + (let [result (format/format-result {:status :error + :command :search + :error {:code :missing-search-text + :message "search text is required"}} + {:output-format nil})] + (is (= (str "Error (missing-search-text): search text is required\n" + "Hint: Provide search text as a positional argument") result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 600ef15eee..9bfbe4cf97 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -34,6 +34,10 @@ [node] (or (:block/title node) (:title node))) +(defn- node-uuid + [node] + (or (:block/uuid node) (:uuid node))) + (defn- node-children [node] (or (:block/children node) (:children node))) @@ -45,6 +49,7 @@ node (some #(find-block-by-title % title) (node-children node))))) + (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -96,9 +101,10 @@ list-tag-payload (parse-json-output list-tag-result) list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "hello world"] data-dir cfg-path) - _ (parse-json-output add-block-result) - search-result (run-cli ["--repo" "content-graph" "search" "--text" "hello world"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + search-result (run-cli ["--repo" "content-graph" "search" "t"] data-dir cfg-path) search-payload (parse-json-output search-result) show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) show-payload (parse-json-output show-result) @@ -108,6 +114,7 @@ stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) + (is (= "ok" (:status add-block-payload))) (is (= "ok" (:status list-page-payload))) (is (vector? (get-in list-page-payload [:data :items]))) (is (= "ok" (:status list-tag-payload))) @@ -116,6 +123,11 @@ (is (vector? (get-in list-property-payload [:data :items]))) (is (= "ok" (:status search-payload))) (is (vector? (get-in search-payload [:data :results]))) + (let [types (set (map :type (get-in search-payload [:data :results])))] + (is (contains? types "page")) + (is (contains? types "block")) + (is (contains? types "tag")) + (is (contains? types "property"))) (is (= "ok" (:status show-payload))) (is (contains? (get-in show-payload [:data :root]) :uuid)) (is (= "ok" (:status remove-page-payload))) @@ -125,6 +137,48 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-show-search-resolve-nested-uuid-refs + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) + show-inner (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-inner-payload (parse-json-output show-inner) + inner-node (find-block-by-title (get-in show-inner-payload [:data :root]) "Inner") + inner-uuid (node-uuid inner-node) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + "--content" (str "See [[" inner-uuid "]]")] data-dir cfg-path) + show-middle (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-middle-payload (parse-json-output show-middle) + middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) "See [[Inner]]") + middle-uuid (node-uuid middle-node) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) + show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-outer-payload (parse-json-output show-outer) + outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") + search-result (run-cli ["--repo" "nested-refs" "search" "Outer"] data-dir cfg-path) + search-payload (parse-json-output search-result) + search-item (some (fn [item] + (when (and (string? (:content item)) + (string/includes? (:content item) "Outer")) + item)) + (get-in search-payload [:data :results])) + stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? inner-uuid)) + (is (some? middle-uuid)) + (is (some? outer-node)) + (is (= "Outer [[See [[Inner]]]]" (:content search-item))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-show-linked-references-json (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] From 176854356be3d269d9a1b0ab0d4142b4191959a9 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 21 Jan 2026 22:39:21 +0800 Subject: [PATCH 032/375] 011-logseq-cli-search-optimization.md (2) --- docs/cli/logseq-cli.md | 2 +- src/main/logseq/cli/format.cljs | 23 ++++++++++++++++++----- src/test/logseq/cli/format_test.cljs | 5 +++-- 3 files changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 32188fae58..53245724d6 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -97,7 +97,7 @@ Options grouping: Output formats: - Global `--output ` (also accepted per subcommand) - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. -- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. - `show` human output prints the `:db/id` as the first column followed by a tree: diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 13b1cddc99..957f5c15de 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -40,22 +40,35 @@ (defn- render-table [headers rows] - (let [normalized-rows (mapv (fn [row] - (mapv normalize-cell row)) + (let [split-lines (fn [value] + (string/split (normalize-cell value) #"\n" -1)) + normalized-rows (mapv (fn [row] + (mapv split-lines row)) rows) trim-right (fn [value] (string/replace value #"\s+$" "")) widths (mapv (fn [idx header] - (apply max (count header) - (map #(count (nth % idx)) normalized-rows))) + (reduce max + (count header) + (mapcat #(map count (nth % idx)) normalized-rows))) (range (count headers)) headers) render-row (fn [row] (->> (map pad-right row widths) (string/join " ") (trim-right))) + render-multiline-row (fn [row] + (let [line-count (reduce max 1 (map count row))] + (mapv (fn [line-idx] + (->> (map-indexed (fn [col-idx lines] + (pad-right (get lines line-idx "") + (nth widths col-idx))) + row) + (string/join " ") + (trim-right))) + (range line-count)))) lines (cons (render-row headers) - (map render-row normalized-rows))] + (mapcat render-multiline-row normalized-rows))] (string/join "\n" lines))) (defn- format-counted-table diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 49e1ba19b1..657b9810d7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -171,7 +171,7 @@ :created-at 1} {:type "block" :db/id 102 - :content "Note" + :content "Note line 1\nNote line 2" :uuid "u2" :updated-at 4 :created-at 2} @@ -186,7 +186,8 @@ {:output-format nil})] (is (= (str "ID TITLE\n" "101 Alpha\n" - "102 Note\n" + "102 Note line 1\n" + " Note line 2\n" "103 Taggy\n" "104 Prop\n" "Count: 4") From 342ae3417db2b2e0d46a9cdc89b89a41b4cfeb6e Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 22 Jan 2026 20:06:49 +0800 Subject: [PATCH 033/375] 012-logseq-cli-graph-storage.md --- .../003-db-worker-node-cli-orchestration.md | 4 +- .../012-logseq-cli-graph-storage.md | 77 +++++++++++++++++++ .../task--db-worker-nodejs-compatible.md | 2 +- src/main/frontend/worker/db_core.cljs | 8 +- src/main/frontend/worker/db_worker_node.cljs | 2 +- .../frontend/worker/db_worker_node_lock.cljs | 5 +- src/main/frontend/worker/platform/node.cljs | 23 +++--- src/main/frontend/worker_common/util.cljc | 15 ++++ src/main/logseq/cli/command/graph.cljs | 3 +- src/main/logseq/cli/config.cljs | 2 +- src/main/logseq/cli/format.cljs | 19 ++++- src/main/logseq/cli/server.cljs | 17 ++-- .../frontend/worker/db_worker_node_test.cljs | 17 +--- .../worker/worker_common_util_test.cljs | 20 +++++ src/test/logseq/cli/commands_test.cljs | 13 ++++ src/test/logseq/cli/format_test.cljs | 2 +- 16 files changed, 178 insertions(+), 51 deletions(-) create mode 100644 docs/agent-guide/012-logseq-cli-graph-storage.md create mode 100644 src/test/frontend/worker/worker_common_util_test.cljs diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index 6303688cd9..841fcbfe5a 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -44,7 +44,7 @@ Key changes: - Reject `thread-api/create-or-open-db`, `thread-api/unsafe-unlink-db`, etc. when repo differs. - Return 409/400 with `:repo-mismatch` error shape. - **Lock file**: - - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-/db-worker.lock`). + - Location: inside repo dir (e.g. `~/.logseq/cli-graphs//db-worker.lock`). - Content: JSON `{repo, pid, host, port, startedAt}`. - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”. - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM. @@ -108,7 +108,7 @@ Implementation notes: - Answer: No --repo needed, using 'out-of-band access to data-dir' way 2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). - lockfile name:`db-worker.lock`, - - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-/db-worker.lock`). + - Location: inside repo dir (e.g. `~/.logseq/cli-graphs//db-worker.lock`). - only consider linux/macos for now 3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle. 4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI? diff --git a/docs/agent-guide/012-logseq-cli-graph-storage.md b/docs/agent-guide/012-logseq-cli-graph-storage.md new file mode 100644 index 0000000000..d210dfc21f --- /dev/null +++ b/docs/agent-guide/012-logseq-cli-graph-storage.md @@ -0,0 +1,77 @@ +# Logseq CLI Graph Storage Plan + +## Context +logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logseq/db-worker/` and use per-graph directories named like `.logseq-pool-/` (with partial encoding). This plan captures the non-functional updates requested: + +1) Rename `~/.logseq/db-worker/` to `~/.logseq/cli-graphs/` for CLI-managed graphs. +2) Rename per-graph directory format from `.logseq-pool-/` to `/`. +3) Ensure graph names that are not valid directory names are encoded, and decoding is symmetric when reading/listing. + +## Goals +- Move CLI graph storage to `~/.logseq/cli-graphs` by default. +- Use a clean per-graph directory name equal to the (encoded) graph name, without `.` prefix or `logseq-pool-` prefix. +- Provide a reversible encode/decode for graph names so list/read operations reconstruct the original graph name. +- CLI commands and outputs should hide the internal `logseq_db_` prefix; user-facing graph names strip that db prefix. +- Maintain db-worker-node functionality (locks/logs/kv-store) with the new paths. + +## Non-goals +- Changing Electron or browser-based db graph storage (`~/logseq/graphs`) or OPFS behavior. +- Changing db schema or sqlite storage format. +- Changing db-worker-node API semantics. + +## Current Behavior (Key References) +- CLI default data dir is `~/.logseq/db-worker`: `src/main/logseq/cli/config.cljs`. +- db-worker-node default data dir is `~/.logseq/db-worker`: `src/main/frontend/worker/db_worker_node.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. +- Per-graph directory currently `.logseq-pool-`: + - `frontend.worker-common.util/get-pool-name` returns `logseq-pool-`: `src/main/frontend/worker_common/util.cljc`. + - `repo-dir` uses `"." + pool-name` in CLI server, db-worker-node lock, and node platform: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. +- Current graph decoding in list operations reverses only `+3A+` and `++` (file-based graphs); other characters are not reversible: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/platform/node.cljs`. + +## Proposed Approach +### 1) New default data dir +- Change default data dir for CLI and db-worker-node from `~/.logseq/db-worker` to `~/.logseq/cli-graphs`. +- Update help text and any user-facing docs mentioning the old default. + +### 2) New per-graph directory naming +- Replace `.logseq-pool-/` with `/`. +- Remove the leading dot and `logseq-pool-` prefix entirely for CLI-managed graphs. + +### 3) Reversible graph name encoding +- Introduce a shared encode/decode pair for graph directory names that is bijective for all graph names. +- The encoding must avoid path separators and other invalid characters (esp. `/`, `\`, `:` on Windows). +- Suggested approach (reversible and simple): + - Encode: apply `encodeURIComponent` to the graph name, then replace `%` with a safe delimiter (e.g. `~`) to keep filenames readable and avoid `%` edge cases. + - Decode: reverse the delimiter replacement, then `decodeURIComponent`. +- Provide helper functions in a shared place (e.g. `frontend.worker-common.util` or a new shared CLI/worker helper) so CLI server, db-worker-node lock, and node platform list all use the same encode/decode logic. + +## Implementation Steps +1) Add encode/decode helpers + - Add new helpers for reversible graph name <-> directory name. + - Update `get-pool-name` or replace its usage for CLI/db-worker-node paths. + - Files: `src/main/frontend/worker_common/util.cljc`, potentially `deps/cli/src/logseq/cli/common/graph.cljs`. + +2) Update data dir defaults + - Change defaults to `~/.logseq/cli-graphs` in: + - `src/main/logseq/cli/config.cljs` + - `src/main/logseq/cli/server.cljs` + - `src/main/frontend/worker/db_worker_node_lock.cljs` + - `src/main/frontend/worker/db_worker_node.cljs` (help text) + - `src/main/frontend/worker/platform/node.cljs` + - Update any CLI docs/tests that reference `db-worker` as default. + +3) Update repo-dir/path derivation + - Replace `"." + pool-name` usage with new `` directory naming. + - Update list-graphs and list-servers to decode from new directory names. + - Files: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. + +4) Tests & verification + - Update CLI integration tests that construct temp dirs named `db-worker*` to match new defaults or explicitly pass `--data-dir`. + - Update db-worker-node tests to use new naming and to validate encode/decode. + - Ensure `bb dev:lint-and-test` passes. + +## Open Questions +- The new encoding is CLI and db-worker-node only (no Electron changes). + +## Rollout Notes +- This is a filesystem layout change. Include release notes and ensure users can override via `--data-dir`. +- Provide a one-time warning if old layout is detected and not migrated. diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index fc80681804..8789fc82f8 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -129,7 +129,7 @@ The db-worker should be runnable as a standalone process for Node.js environment - Provide a CLI entry (example: `bin/logseq-db-worker` or `node dist/db-worker-node.js`). - CLI flags (suggested): - Binds to localhost on a random port and records it in the repo lock file. - - `--data-dir` (path for sqlite files, required or default to `~/.logseq/db-worker`) + - `--data-dir` (path for sqlite files, required or default to `~/.logseq/cli-graphs`) - `--repo` (optional: auto-open a repo on boot) - `--rtc-ws-url` (optional) - `--log-level` (default `info`) diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index d6ed898222..21feec84fe 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -68,6 +68,12 @@ [] (= :node (platform/env-flag (platform/current) :runtime))) +(defn- storage-pool-name + [graph] + (if (node-runtime?) + (worker-util/encode-graph-dir-name graph) + (worker-util/get-pool-name graph))) + (defn- get-storage-pool [graph] (if (node-runtime?) @@ -91,7 +97,7 @@ (when-not @*publishing? (or (get-storage-pool graph) (p/let [storage (platform/storage (platform/current)) - ^js pool ((:install-opfs-pool storage) @*sqlite (worker-util/get-pool-name graph))] + ^js pool ((:install-opfs-pool storage) @*sqlite (storage-pool-name graph))] (remember-storage-pool! graph pool) pool)))) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 368b5fe188..cf9233450f 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -241,7 +241,7 @@ (defn- show-help! [] (println "db-worker-node options:") - (println " --data-dir (default ~/.logseq/db-worker)") + (println " --data-dir (default ~/.logseq/cli-graphs)") (println " --repo (required)") (println " --rtc-ws-url (optional)") (println " --log-level (default info)") diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index be180dc468..d272bb4b08 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -16,12 +16,11 @@ (defn resolve-data-dir [data-dir] - (expand-home (or data-dir "~/.logseq/db-worker"))) + (expand-home (or data-dir "~/.logseq/cli-graphs"))) (defn repo-dir [data-dir repo] - (let [pool-name (worker-util/get-pool-name repo)] - (node-path/join data-dir (str "." pool-name)))) + (node-path/join data-dir (worker-util/encode-graph-dir-name repo))) (defn lock-path [data-dir repo] diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index afa84644cf..8296b42d31 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -30,7 +30,7 @@ (defn- repo-dir [data-dir pool-name] - (node-path/join data-dir (str "." pool-name))) + (node-path/join data-dir pool-name)) (defn- pool-path [^js pool path] @@ -52,25 +52,20 @@ (defn- list-graphs [data-dir] - (let [dir? #(and % (.isDirectory %)) - db-dir-prefix ".logseq-pool-"] + (let [dir? #(and % (.isDirectory %))] (p/let [entries (fs/readdir data-dir #js {:withFileTypes true}) db-dirs (->> entries - (filter dir?) - (filter (fn [dirent] - (string/starts-with? (.-name dirent) db-dir-prefix)))) + (filter dir?)) graph-names (map (fn [dirent] - (-> (.-name dirent) - (string/replace-first db-dir-prefix "") - ;; TODO: DRY - (string/replace "+3A+" ":") - (string/replace "++" "/"))) + (worker-util/decode-graph-dir-name (.-name dirent))) db-dirs)] - (vec graph-names)))) + (->> graph-names + (filter some?) + (vec))))) (defn- db-exists? [data-dir graph] - (p/let [pool-name (worker-util/get-pool-name graph) + (p/let [pool-name (worker-util/encode-graph-dir-name graph) db-path (node-path/join (repo-dir data-dir pool-name) "db.sqlite")] (-> (fs/stat db-path) (p/then (fn [_] true)) @@ -191,7 +186,7 @@ (defn node-platform [{:keys [data-dir event-fn]}] - (let [data-dir (expand-home (or data-dir "~/.logseq/db-worker")) + (let [data-dir (expand-home (or data-dir "~/.logseq/cli-graphs")) kv (kv-store data-dir)] (p/do! (ensure-dir! data-dir) diff --git a/src/main/frontend/worker_common/util.cljc b/src/main/frontend/worker_common/util.cljc index 3184ec506d..ec9dab4695 100644 --- a/src/main/frontend/worker_common/util.cljc +++ b/src/main/frontend/worker_common/util.cljc @@ -31,6 +31,21 @@ (when-let [worker (or port js/self)] (.postMessage worker (ldb/write-transit-str [type data])))) + (defn encode-graph-dir-name + [graph-name] + (let [encoded (js/encodeURIComponent (or graph-name ""))] + (-> encoded + (string/replace "~" "%7E") + (string/replace "%" "~")))) + + (defn decode-graph-dir-name + [dir-name] + (when (some? dir-name) + (try + (js/decodeURIComponent (string/replace dir-name "~" "%")) + (catch :default _ + nil)))) + (defn get-pool-name [graph-name] (str "logseq-pool-" (common-sqlite/sanitize-db-name graph-name))) diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 8a23a08848..ccec6d6e9c 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -135,7 +135,8 @@ (defn execute-graph-list [_action config] - (let [graphs (cli-server/list-graphs config)] + (let [graphs (->> (cli-server/list-graphs config) + (mapv core/repo->graph))] {:status :ok :data {:graphs graphs}})) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index ae03dfc039..4293ffebb5 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -80,7 +80,7 @@ (let [defaults {:timeout-ms 10000 :retries 0 :output-format nil - :data-dir "~/.logseq/db-worker" + :data-dir "~/.logseq/cli-graphs" :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 957f5c15de..bebffd374a 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -1,7 +1,8 @@ (ns logseq.cli.format "Formatting helpers for CLI output." (:require [clojure.string :as string] - [clojure.walk :as walk])) + [clojure.walk :as walk] + [logseq.cli.command.core :as command-core])) (defn- normalize-json [value] @@ -290,9 +291,23 @@ (= status :ok) (assoc :data data) (= status :error) (assoc :error error)))) +(defn- normalize-graph-result + [result] + (walk/postwalk + (fn [entry] + (if (map? entry) + (cond-> entry + (string? (:repo entry)) (update :repo command-core/repo->graph) + (string? (:graph entry)) (update :graph command-core/repo->graph) + (seq (:graphs entry)) (update :graphs (fn [graphs] + (mapv command-core/repo->graph graphs)))) + entry)) + result)) + (defn format-result [result {:keys [output-format] :as opts}] - (let [format (cond + (let [result (normalize-graph-result result) + format (cond (= output-format :edn) :edn (= output-format :json) :json :else :human)] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index f835fcbc60..0099685b24 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -6,6 +6,7 @@ ["os" :as os] ["path" :as node-path] [clojure.string :as string] + [logseq.cli.command.core :as command-core] [frontend.worker-common.util :as worker-util] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -18,12 +19,11 @@ (defn resolve-data-dir [config] - (expand-home (or (:data-dir config) "~/.logseq/db-worker"))) + (expand-home (or (:data-dir config) "~/.logseq/cli-graphs"))) (defn- repo-dir [data-dir repo] - (let [pool-name (worker-util/get-pool-name repo)] - (node-path/join data-dir (str "." pool-name)))) + (node-path/join data-dir (worker-util/encode-graph-dir-name repo))) (defn lock-path [data-dir repo] @@ -286,17 +286,12 @@ (defn list-graphs [config] (let [data-dir (resolve-data-dir config) - db-dir-prefix ".logseq-pool-" entries (when (fs/existsSync data-dir) (fs/readdirSync data-dir #js {:withFileTypes true}))] (->> entries (filter #(.isDirectory ^js %)) (map (fn [^js dirent] - (.-name dirent))) - (filter #(string/starts-with? % db-dir-prefix)) - (map (fn [dir-name] - (-> dir-name - (string/replace-first db-dir-prefix "") - (string/replace "+3A+" ":") - (string/replace "++" "/")))) + (worker-util/decode-graph-dir-name (.-name dirent)))) + (filter some?) + (map command-core/repo->graph) (vec)))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 677d60dc03..cb5668f273 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -1,13 +1,11 @@ (ns frontend.worker.db-worker-node-test (:require ["http" :as http] [cljs.test :refer [async deftest is]] - [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker-common.util :as worker-util] [frontend.worker.db-worker-node :as db-worker-node] [goog.object :as gobj] [logseq.db :as ldb] - [logseq.db.sqlite.util :as sqlite-util] [promesa.core :as p] ["fs" :as fs] ["path" :as node-path])) @@ -75,8 +73,7 @@ (defn- lock-path [data-dir repo] - (let [pool-name (worker-util/get-pool-name repo) - repo-dir (node-path/join data-dir (str "." pool-name))] + (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo))] (node-path/join repo-dir "db-worker.lock"))) (defn- pad2 @@ -93,8 +90,7 @@ (defn- log-path [data-dir repo] - (let [pool-name (worker-util/get-pool-name repo) - repo-dir (node-path/join data-dir (str "." pool-name)) + (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo)) date-str (yyyymmdd (js/Date.))] (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) @@ -144,8 +140,7 @@ (let [enforce-log-retention! #'db-worker-node/enforce-log-retention! data-dir (node-helper/create-tmp-dir "db-worker-log-retention") repo (str "logseq_db_log_retention_" (subs (str (random-uuid)) 0 8)) - pool-name (worker-util/get-pool-name repo) - repo-dir (node-path/join data-dir (str "." pool-name)) + repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo)) days ["20240101" "20240102" "20240103" "20240104" "20240105" "20240106" "20240107" "20240108" "20240109"] make-log (fn [day] @@ -222,11 +217,7 @@ dbs (invoke host port "thread-api/list-db" []) _ (do (println "[db-worker-node-test] list-db" dbs) - (let [prefix sqlite-util/db-version-prefix - expected-name (if (string/starts-with? repo prefix) - (subs repo (count prefix)) - repo)] - (is (some #(= expected-name (:name %)) dbs)))) + (is (some #(= repo (:name %)) dbs))) lock-file (lock-path data-dir repo) _ (is (fs/existsSync lock-file)) lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) diff --git a/src/test/frontend/worker/worker_common_util_test.cljs b/src/test/frontend/worker/worker_common_util_test.cljs new file mode 100644 index 0000000000..2b7ca2ac45 --- /dev/null +++ b/src/test/frontend/worker/worker_common_util_test.cljs @@ -0,0 +1,20 @@ +(ns frontend.worker.worker-common-util-test + (:require [cljs.test :refer [deftest is]] + [clojure.string :as string] + [frontend.worker-common.util :as worker-util])) + +(deftest encode-decode-graph-dir-name-roundtrip + (let [names ["Demo" + "foo/bar" + "a:b" + "space name" + "100% legit" + "til~de" + "mix/ed:chars%~"] + encoded (map worker-util/encode-graph-dir-name names)] + (doseq [[name enc] (map vector names encoded)] + (is (= name (worker-util/decode-graph-dir-name enc)))) + (doseq [enc encoded] + (is (not (string/includes? enc "/"))) + (is (not (string/includes? enc "\\"))))) + (is (nil? (worker-util/decode-graph-dir-name nil)))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 19a3005944..9ec04f5935 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -871,3 +871,16 @@ (set! transport/read-input orig-read-input) (set! transport/invoke orig-invoke) (done))))))) + +(deftest test-execute-graph-list-strips-db-prefix + (async done + (let [orig-list-graphs cli-server/list-graphs] + (set! cli-server/list-graphs (fn [_] ["logseq_db_demo" "logseq_db_other"])) + (-> (p/let [result (commands/execute {:type :graph-list} {})] + (is (= :ok (:status result))) + (is (= ["demo" "other"] (get-in result [:data :graphs])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (done))))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 657b9810d7..10d50c4d46 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -150,7 +150,7 @@ (testing "server status includes repo, status, host, port" (let [result (format/format-result {:status :ok :command :server-status - :data {:repo "demo-repo" + :data {:repo "logseq_db_demo-repo" :status :ready :host "127.0.0.1" :port 1234}} From 3793920c02817d94c5544150686923b585373aff Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 22 Jan 2026 21:09:44 +0800 Subject: [PATCH 034/375] feat(cli): add bin wrapper --- .gitignore | 2 ++ bb.edn | 2 +- dist/logseq.js | 6 ++++++ docs/agent-guide/001-logseq-cli.md | 2 +- .../003-db-worker-node-cli-orchestration.md | 2 +- docs/cli/logseq-cli.md | 18 +++++++++--------- package.json | 5 +++-- shadow-cljs.edn | 4 ++-- 8 files changed, 25 insertions(+), 16 deletions(-) create mode 100755 dist/logseq.js diff --git a/.gitignore b/.gitignore index 44c09b6fab..336b78f00d 100644 --- a/.gitignore +++ b/.gitignore @@ -83,3 +83,5 @@ clj-e2e/e2e-dump .projectile deps/db-sync/data *.map +/dist/db-worker-node.js +/dist/logseq-cli.js diff --git a/bb.edn b/bb.edn index 3bea2155f3..052f6f815f 100644 --- a/bb.edn +++ b/bb.edn @@ -186,7 +186,7 @@ {:doc "Compile and start db-worker-node (pass-through args forwarded to node)" :task (do (shell "clojure" "-M:cljs" "compile" "db-worker-node") - (apply shell "node" "./static/db-worker-node.js" *command-line-args*))} + (apply shell "node" "./dist/db-worker-node.js" *command-line-args*))} lint:dev logseq.tasks.dev.lint/dev diff --git a/dist/logseq.js b/dist/logseq.js new file mode 100755 index 0000000000..4b2b0f7cf5 --- /dev/null +++ b/dist/logseq.js @@ -0,0 +1,6 @@ +#!/usr/bin/env node +"use strict"; + +const path = require("path"); + +require(path.resolve(__dirname, "./logseq-cli.js")); diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md index aca73ea487..287a63994b 100644 --- a/docs/agent-guide/001-logseq-cli.md +++ b/docs/agent-guide/001-logseq-cli.md @@ -12,7 +12,7 @@ Related: Relates to docs/agent-guide/task--basic-logseq-cli.md and docs/agent-gu ## Problem statement We need a new Logseq CLI that is independent of any existing CLI code in the repo. -The CLI must run in Node.js, be written in ClojureScript, and connect to the db-worker-node server started from static/db-worker-node.js. +The CLI must run in Node.js, be written in ClojureScript, and connect to the db-worker-node server started from dist/db-worker-node.js. The CLI should provide a stable interface for scripting and troubleshooting, and it should be easy to extend with new commands. ## Testing Plan diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index 841fcbfe5a..a3ea3cf7e5 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -65,7 +65,7 @@ Key changes: 2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`. 3. If healthy, reuse existing server; build the connection URL from localhost and the lock file port. 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. - 5. Spawn via `child_process.spawn`: `node ./static/db-worker-node.js --repo --data-dir <...>`. + 5. Spawn via `child_process.spawn`: `node ./dist/db-worker-node.js --repo --data-dir <...>`. 6. Resolve actual port from the lock file written by db-worker-node. - **Connection URL**: derived from the repo lock file; host is always localhost and the port is always discovered from the lock file. diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 53245724d6..14c4e9ff6a 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -15,7 +15,7 @@ clojure -M:cljs compile logseq-cli ## Run the CLI ```bash -node ./static/logseq-cli.js graph list +node ./dist/logseq.js graph list If installed globally, run: @@ -115,12 +115,12 @@ id8 └── b8 Examples: ```bash -node ./static/logseq-cli.js graph create --repo demo -node ./static/logseq-cli.js graph export --type edn --output /tmp/demo.edn --repo demo -node ./static/logseq-cli.js graph import --type edn --input /tmp/demo.edn --repo demo-import -node ./static/logseq-cli.js add block --target-page-name TestPage --content "hello world" -node ./static/logseq-cli.js move --uuid --target-page-name TargetPage -node ./static/logseq-cli.js search "hello" -node ./static/logseq-cli.js show --page-name TestPage --format json --output json -node ./static/logseq-cli.js server list +node ./dist/logseq.js graph create --repo demo +node ./dist/logseq.js graph export --type edn --output /tmp/demo.edn --repo demo +node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --repo demo-import +node ./dist/logseq.js add block --target-page-name TestPage --content "hello world" +node ./dist/logseq.js move --uuid --target-page-name TargetPage +node ./dist/logseq.js search "hello" +node ./dist/logseq.js show --page-name TestPage --format json --output json +node ./dist/logseq.js server list ``` diff --git a/package.json b/package.json index 4186c8f5c2..2b6dbb0fa4 100644 --- a/package.json +++ b/package.json @@ -4,10 +4,11 @@ "private": true, "main": "static/electron.js", "bin": { - "logseq": "static/logseq-cli.js" + "logseq": "dist/logseq.js" }, "files": [ - "static/db-worker-node.js" + "dist/db-worker-node.js", + "dist/logseq-cli.js" ], "engines": { "node": ">=22.20.0" diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 181e2622bf..2ce8251b37 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -88,7 +88,7 @@ :loader-mode :eval}} :db-worker-node {:target :node-script - :output-to "static/db-worker-node.js" + :output-to "dist/db-worker-node.js" :main frontend.worker.db-worker-node/main :compiler-options {:infer-externs :auto :source-map true @@ -98,7 +98,7 @@ :redef false}}} :logseq-cli {:target :node-script - :output-to "static/logseq-cli.js" + :output-to "dist/logseq-cli.js" :main logseq.cli.main/main :compiler-options {:infer-externs :auto :source-map true From 311eb68687cc9eb418c8005c085375bd9b3abcb3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 22 Jan 2026 23:03:09 +0800 Subject: [PATCH 035/375] add 013-logseq-cli-datascript-query.md --- .../013-logseq-cli-datascript-query.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 docs/agent-guide/013-logseq-cli-datascript-query.md diff --git a/docs/agent-guide/013-logseq-cli-datascript-query.md b/docs/agent-guide/013-logseq-cli-datascript-query.md new file mode 100644 index 0000000000..97250ab360 --- /dev/null +++ b/docs/agent-guide/013-logseq-cli-datascript-query.md @@ -0,0 +1,69 @@ +# Logseq CLI Datascript Query Implementation Plan + +Goal: Add a logseq-cli query subcommand that runs a Datascript query via db-worker-node and returns a list of block IDs. +Architecture: The CLI will parse a query form from arguments, call db-worker-node using the existing /v1/invoke transport with :thread-api/q, and normalize results into a stable list of block IDs. +Architecture: No new db-worker-node HTTP endpoints are required because :thread-api/q already exists in the worker thread API. +Tech Stack: ClojureScript, babashka.cli, Datascript, db-worker-node HTTP transport. +Related: Relates to docs/agent-guide/012-logseq-cli-graph-storage.md. + +## Problem statement + +The current logseq-cli does not expose a query subcommand for running Datascript queries against a graph. +Users need a CLI command that executes a Datascript query and returns only the matching block IDs for scripting and downstream tooling. +The solution should follow the existing logseq-cli and db-worker-node invocation patterns so it works with the current daemon and transport. + +## Testing Plan + +I will add an integration test that creates a graph, inserts blocks, runs the new query subcommand, and asserts that the returned IDs match the expected block IDs. +I will add a unit test that validates query argument parsing, including invalid EDN, missing query text, and optional inputs parsing. +I will add a unit test that verifies result normalization from the raw query result to a vector of block IDs. +I will follow @test-driven-development and write the failing tests before implementing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +Create a new command module at src/main/logseq/cli/command/query.cljs that defines the query command spec and action builder. +Use --query for the Datascript query form and --inputs for the optional EDN inputs vector, and parse them with cljs.reader/read-string or logseq.common.util/safe-read-string with error handling. +Return a structured action map that includes :type :query, :repo, :query, and :inputs to keep execution isolated from parsing. +Register the new command in src/main/logseq/cli/commands.cljs and include it in the command table so help output includes query. +Update src/main/logseq/cli/main.cljs usage text to include query in the command list. +Implement execution in src/main/logseq/cli/command/query.cljs using logseq.cli.transport/invoke with :thread-api/q and args [repo [query & inputs]]. +Normalize the raw Datascript result into a vector of block IDs by accepting numbers or entities with :db/id, and raise an error when no IDs can be derived. +Ensure output is stable by sorting numeric IDs ascending and removing duplicates before returning. +Add formatting in src/main/logseq/cli/format.cljs for :query to render a single-column table of IDs in human output and a vector in json or edn output. +Add command-level validation in src/main/logseq/cli/commands.cljs to return a missing-query error when no query is supplied. +Update src/test/logseq/cli/commands_test.cljs to expect query in help output and to validate parse errors for missing query or invalid input. +Add a CLI integration test in src/test/logseq/cli/integration_test.cljs that uses run-cli to execute query and verifies IDs in JSON output. +Confirm that no db-worker-node changes are required by verifying that :thread-api/q continues to accept the same argument shape in src/main/frontend/worker/db_core.cljs. + +## Edge cases + +A query string that cannot be read as EDN should return a clear invalid-options error and a non-zero exit code. +A query that returns no results should return an empty ID list with a successful status. +A query that returns non-entity values should error if no block IDs can be extracted from the result set. +Queries with :in parameters should work when --inputs supplies the matching values in order. + +## Testing Details + +The integration test will create a graph, add known blocks, run a query that finds those blocks, and verify that the output vector contains their db/id values and only those values. +The unit tests will assert that parsing rejects invalid EDN for --inputs, that a missing query produces a :missing-query error, that result normalization handles tuples and entity maps into a flat vector of IDs, and that it errors when no IDs can be extracted. + +## Implementation Details + +- Add a new command module at src/main/logseq/cli/command/query.cljs. +- Add command entries in src/main/logseq/cli/commands.cljs. +- Add output formatting in src/main/logseq/cli/format.cljs. +- Update usage text in src/main/logseq/cli/main.cljs. +- Use transport/invoke with :thread-api/q and [repo [query & inputs]]. +- Normalize results into unique, sorted numeric IDs. +- Keep all argument parsing and validation inside query command module using --query and --inputs. +- Keep db-worker-node changes to zero unless a new worker API is required. + +## Question + +Use --query and --inputs options for the query subcommand. +Output should be sorted and de-duplicated for scripting stability. +Command should error when no block IDs can be extracted. + +--- From 6f3d0741950628b6bac2dc698088bc4f53010ad0 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 23 Jan 2026 11:30:41 +0800 Subject: [PATCH 036/375] impl 013-logseq-cli-datascript-query.md (1) --- .gitignore | 2 - bb.edn | 2 +- dist/db-worker-node.js | 6 + dist/logseq.js | 2 +- .../003-db-worker-node-cli-orchestration.md | 2 +- .../013-logseq-cli-datascript-query.md | 25 +- .../task--db-worker-nodejs-compatible.md | 2 +- package.json | 5 +- shadow-cljs.edn | 4 +- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/query.cljs | 66 ++ src/main/logseq/cli/command/show.cljs | 11 +- src/main/logseq/cli/commands.cljs | 18 +- src/main/logseq/cli/format.cljs | 8 +- src/main/logseq/cli/main.cljs | 2 +- src/main/logseq/cli/server.cljs | 6 +- .../frontend/worker/db_worker_node_test.cljs | 2 +- src/test/logseq/cli/command/query_test.cljs | 32 + src/test/logseq/cli/commands_test.cljs | 17 + src/test/logseq/cli/format_test.cljs | 8 + src/test/logseq/cli/integration_test.cljs | 867 +++++++++--------- src/test/logseq/cli/server_test.cljs | 5 +- 22 files changed, 640 insertions(+), 454 deletions(-) create mode 100755 dist/db-worker-node.js create mode 100644 src/main/logseq/cli/command/query.cljs create mode 100644 src/test/logseq/cli/command/query_test.cljs diff --git a/.gitignore b/.gitignore index 336b78f00d..44c09b6fab 100644 --- a/.gitignore +++ b/.gitignore @@ -83,5 +83,3 @@ clj-e2e/e2e-dump .projectile deps/db-sync/data *.map -/dist/db-worker-node.js -/dist/logseq-cli.js diff --git a/bb.edn b/bb.edn index 052f6f815f..3bea2155f3 100644 --- a/bb.edn +++ b/bb.edn @@ -186,7 +186,7 @@ {:doc "Compile and start db-worker-node (pass-through args forwarded to node)" :task (do (shell "clojure" "-M:cljs" "compile" "db-worker-node") - (apply shell "node" "./dist/db-worker-node.js" *command-line-args*))} + (apply shell "node" "./static/db-worker-node.js" *command-line-args*))} lint:dev logseq.tasks.dev.lint/dev diff --git a/dist/db-worker-node.js b/dist/db-worker-node.js new file mode 100755 index 0000000000..4dabd0bf17 --- /dev/null +++ b/dist/db-worker-node.js @@ -0,0 +1,6 @@ +#!/usr/bin/env node +"use strict"; + +const path = require("path"); + +require(path.resolve(__dirname, "../static/db-worker-node.js")); diff --git a/dist/logseq.js b/dist/logseq.js index 4b2b0f7cf5..28b93e441c 100755 --- a/dist/logseq.js +++ b/dist/logseq.js @@ -3,4 +3,4 @@ const path = require("path"); -require(path.resolve(__dirname, "./logseq-cli.js")); +require(path.resolve(__dirname, "../static/logseq-cli.js")); diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index a3ea3cf7e5..e699f5bc84 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -65,7 +65,7 @@ Key changes: 2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`. 3. If healthy, reuse existing server; build the connection URL from localhost and the lock file port. 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. - 5. Spawn via `child_process.spawn`: `node ./dist/db-worker-node.js --repo --data-dir <...>`. + 5. Spawn via `child_process.spawn`: `./dist/db-worker-node.js --repo --data-dir <...>`. 6. Resolve actual port from the lock file written by db-worker-node. - **Connection URL**: derived from the repo lock file; host is always localhost and the port is always discovered from the lock file. diff --git a/docs/agent-guide/013-logseq-cli-datascript-query.md b/docs/agent-guide/013-logseq-cli-datascript-query.md index 97250ab360..2bd4bc730b 100644 --- a/docs/agent-guide/013-logseq-cli-datascript-query.md +++ b/docs/agent-guide/013-logseq-cli-datascript-query.md @@ -1,7 +1,7 @@ # Logseq CLI Datascript Query Implementation Plan -Goal: Add a logseq-cli query subcommand that runs a Datascript query via db-worker-node and returns a list of block IDs. -Architecture: The CLI will parse a query form from arguments, call db-worker-node using the existing /v1/invoke transport with :thread-api/q, and normalize results into a stable list of block IDs. +Goal: Add a logseq-cli query subcommand that runs a Datascript query via db-worker-node and returns the raw datascript-query result shape. +Architecture: The CLI will parse a query form from arguments, call db-worker-node using the existing /v1/invoke transport with :thread-api/q, and return whatever datascript-query returns without normalization. Architecture: No new db-worker-node HTTP endpoints are required because :thread-api/q already exists in the worker thread API. Tech Stack: ClojureScript, babashka.cli, Datascript, db-worker-node HTTP transport. Related: Relates to docs/agent-guide/012-logseq-cli-graph-storage.md. @@ -9,14 +9,14 @@ Related: Relates to docs/agent-guide/012-logseq-cli-graph-storage.md. ## Problem statement The current logseq-cli does not expose a query subcommand for running Datascript queries against a graph. -Users need a CLI command that executes a Datascript query and returns only the matching block IDs for scripting and downstream tooling. +Users need a CLI command that executes a Datascript query and returns the same result shape as datascript-query for scripting and downstream tooling. The solution should follow the existing logseq-cli and db-worker-node invocation patterns so it works with the current daemon and transport. ## Testing Plan I will add an integration test that creates a graph, inserts blocks, runs the new query subcommand, and asserts that the returned IDs match the expected block IDs. I will add a unit test that validates query argument parsing, including invalid EDN, missing query text, and optional inputs parsing. -I will add a unit test that verifies result normalization from the raw query result to a vector of block IDs. +I will add a unit test that verifies the query command returns the same shape as datascript-query without transformation. I will follow @test-driven-development and write the failing tests before implementing behavior. NOTE: I will write *all* tests before I add any implementation behavior. @@ -29,9 +29,8 @@ Return a structured action map that includes :type :query, :repo, :query, and :i Register the new command in src/main/logseq/cli/commands.cljs and include it in the command table so help output includes query. Update src/main/logseq/cli/main.cljs usage text to include query in the command list. Implement execution in src/main/logseq/cli/command/query.cljs using logseq.cli.transport/invoke with :thread-api/q and args [repo [query & inputs]]. -Normalize the raw Datascript result into a vector of block IDs by accepting numbers or entities with :db/id, and raise an error when no IDs can be derived. -Ensure output is stable by sorting numeric IDs ascending and removing duplicates before returning. -Add formatting in src/main/logseq/cli/format.cljs for :query to render a single-column table of IDs in human output and a vector in json or edn output. +Return the raw Datascript result as-is, matching datascript-query output across human, json, and edn formats. +Add formatting in src/main/logseq/cli/format.cljs for :query to render the raw result in human output and pass-through for json or edn output. Add command-level validation in src/main/logseq/cli/commands.cljs to return a missing-query error when no query is supplied. Update src/test/logseq/cli/commands_test.cljs to expect query in help output and to validate parse errors for missing query or invalid input. Add a CLI integration test in src/test/logseq/cli/integration_test.cljs that uses run-cli to execute query and verifies IDs in JSON output. @@ -40,14 +39,13 @@ Confirm that no db-worker-node changes are required by verifying that :thread-ap ## Edge cases A query string that cannot be read as EDN should return a clear invalid-options error and a non-zero exit code. -A query that returns no results should return an empty ID list with a successful status. -A query that returns non-entity values should error if no block IDs can be extracted from the result set. +A query that returns no results should return an empty result with a successful status. Queries with :in parameters should work when --inputs supplies the matching values in order. ## Testing Details -The integration test will create a graph, add known blocks, run a query that finds those blocks, and verify that the output vector contains their db/id values and only those values. -The unit tests will assert that parsing rejects invalid EDN for --inputs, that a missing query produces a :missing-query error, that result normalization handles tuples and entity maps into a flat vector of IDs, and that it errors when no IDs can be extracted. +The integration test will create a graph, add known blocks, run a query that finds those blocks, and verify that the output matches the datascript-query result shape. +The unit tests will assert that parsing rejects invalid EDN for --inputs, that a missing query produces a :missing-query error, and that query execution returns raw results unchanged. ## Implementation Details @@ -56,14 +54,13 @@ The unit tests will assert that parsing rejects invalid EDN for --inputs, that a - Add output formatting in src/main/logseq/cli/format.cljs. - Update usage text in src/main/logseq/cli/main.cljs. - Use transport/invoke with :thread-api/q and [repo [query & inputs]]. -- Normalize results into unique, sorted numeric IDs. +- Return datascript-query results without transformation. - Keep all argument parsing and validation inside query command module using --query and --inputs. - Keep db-worker-node changes to zero unless a new worker API is required. ## Question Use --query and --inputs options for the query subcommand. -Output should be sorted and de-duplicated for scripting stability. -Command should error when no block IDs can be extracted. +Output should mirror datascript-query for scripting stability. --- diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index 8789fc82f8..bef1870ebb 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -126,7 +126,7 @@ Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as The db-worker should be runnable as a standalone process for Node.js environments. ### Entry Point -- Provide a CLI entry (example: `bin/logseq-db-worker` or `node dist/db-worker-node.js`). +- Provide a CLI entry (example: `bin/logseq-db-worker` or `./dist/db-worker-node.js`). - CLI flags (suggested): - Binds to localhost on a random port and records it in the repo lock file. - `--data-dir` (path for sqlite files, required or default to `~/.logseq/cli-graphs`) diff --git a/package.json b/package.json index 2b6dbb0fa4..4293f8118a 100644 --- a/package.json +++ b/package.json @@ -7,8 +7,9 @@ "logseq": "dist/logseq.js" }, "files": [ - "dist/db-worker-node.js", - "dist/logseq-cli.js" + "dist/logseq.js", + "static/db-worker-node.js", + "static/logseq-cli.js" ], "engines": { "node": ">=22.20.0" diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 2ce8251b37..181e2622bf 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -88,7 +88,7 @@ :loader-mode :eval}} :db-worker-node {:target :node-script - :output-to "dist/db-worker-node.js" + :output-to "static/db-worker-node.js" :main frontend.worker.db-worker-node/main :compiler-options {:infer-externs :auto :source-map true @@ -98,7 +98,7 @@ :redef false}}} :logseq-cli {:target :node-script - :output-to "dist/logseq-cli.js" + :output-to "static/logseq-cli.js" :main logseq.cli.main/main :compiler-options {:infer-externs :auto :source-map true diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 58d9d9fa75..a84b3884ef 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -84,7 +84,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "move" "search" "show"}} + :commands #{"list" "add" "remove" "move" "search" "query" "show"}} {:title "Graph Management" :commands #{"graph" "server"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs new file mode 100644 index 0000000000..42ef0f7fc0 --- /dev/null +++ b/src/main/logseq/cli/command/query.cljs @@ -0,0 +1,66 @@ +(ns logseq.cli.command.query + "Query-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] + [promesa.core :as p])) + +(def ^:private query-spec + {:query {:desc "Datascript query EDN"} + :inputs {:desc "EDN vector of query inputs"}}) + +(def entries + [(core/command-entry ["query"] :query "Run a Datascript query" query-spec)]) + +(defn- parse-edn + [label value] + (let [parsed (common-util/safe-read-string {:log-error? false} value)] + (if (nil? parsed) + {:ok? false + :error {:code :invalid-options + :message (str "invalid " label " edn")}} + {:ok? true :value parsed}))) + +(defn build-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for query"}} + (let [query-text (some-> (:query options) string/trim)] + (if-not (seq query-text) + {:ok? false + :error {:code :missing-query + :message "query is required"}} + (let [query-result (parse-edn "query" query-text)] + (if-not (:ok? query-result) + query-result + (let [inputs-text (some-> (:inputs options) string/trim) + inputs-result (when (seq inputs-text) + (parse-edn "inputs" inputs-text))] + (cond + (and inputs-result (not (:ok? inputs-result))) + inputs-result + + (and inputs-result (not (vector? (:value inputs-result)))) + {:ok? false + :error {:code :invalid-options + :message "inputs must be a vector"}} + + :else + {:ok? true + :action {:type :query + :repo repo + :graph (core/repo->graph repo) + :query (:value query-result) + :inputs (or (:value inputs-result) [])}})))))))) + +(defn execute-query + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + args (into [(:query action)] (:inputs action)) + results (transport/invoke cfg :thread-api/q false [(:repo action) args])] + {:status :ok + :data {:result results}}))) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 91e47dcfb4..2a82cea3cc 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -40,6 +40,7 @@ [:db/id :block/uuid :block/title + :block/content {:logseq.property/status [:db/ident :block/name :block/title]} :block/order {:block/parent [:db/id]} @@ -49,6 +50,7 @@ [:db/id :block/uuid :block/title + :block/content {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]} {:block/page [:db/id :block/name :block/title :block/uuid]} @@ -120,11 +122,13 @@ (defn- block-label [node] (let [title (:block/title node) + content (:block/content node) status (status-label node) uuid->label (:uuid->label node) + text (or title content) base (cond - (and title (seq status)) (str status " " title) - title title + (and text (seq status)) (str status " " text) + text text (:block/name node) (:block/name node) (:block/uuid node) (some-> (:block/uuid node) str)) base (replace-uuid-refs base uuid->label) @@ -138,6 +142,7 @@ [node uuid->label] (cond-> node (:block/title node) (update :block/title replace-uuid-refs uuid->label) + (:block/content node) (update :block/content replace-uuid-refs uuid->label) (:block/name node) (update :block/name replace-uuid-refs uuid->label) (:block/children node) (update :block/children (fn [children] (mapv #(resolve-uuid-refs-in-node % uuid->label) children))) @@ -264,7 +269,7 @@ ref-blocks (:blocks linked-refs) pages (keep :block/page ref-blocks) texts (->> (concat nodes ref-blocks pages) - (mapcat (fn [node] (keep node [:block/title :block/name]))) + (mapcat (fn [node] (keep node [:block/title :block/name :block/content]))) (remove string/blank?))] (->> texts (mapcat extract-uuid-refs) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index ca0598bb73..a1f9b79af0 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -7,6 +7,7 @@ [logseq.cli.command.graph :as graph-command] [logseq.cli.command.list :as list-command] [logseq.cli.command.move :as move-command] + [logseq.cli.command.query :as query-command] [logseq.cli.command.remove :as remove-command] [logseq.cli.command.search :as search-command] [logseq.cli.command.server :as server-command] @@ -89,6 +90,13 @@ :message "search text is required"} :summary summary}) +(defn- missing-query-result + [summary] + {:ok? false + :error {:code :missing-query + :message "query is required"} + :summary summary}) + ;; Error helpers are in logseq.cli.command.core. ;; Command-specific validation and entries are in subcommand namespaces. @@ -101,6 +109,7 @@ move-command/entries remove-command/entries search-command/entries + query-command/entries show-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -110,7 +119,7 @@ (string/join " " (cond-> (vec dispatch) wrong-input (conj wrong-input)))) -(defn- finalize-command +(defn- ^:large-vars/cleanup-todo finalize-command [summary {:keys [command opts args cmds spec]}] (let [opts (command-core/normalize-opts opts) args (vec args) @@ -167,6 +176,9 @@ (and (= command :search) (not has-args?)) (missing-search-result summary) + (and (= command :query) (not (seq (some-> (:query opts) string/trim)))) + (missing-query-result summary) + (and (#{:list-page :list-tag :list-property} command) (list-command/invalid-options? command opts)) (command-core/invalid-options-result summary (list-command/invalid-options? command opts)) @@ -339,6 +351,9 @@ :search (search-command/build-action options args repo) + :query + (query-command/build-action options repo) + :show (show-command/build-action options repo) @@ -376,6 +391,7 @@ :remove-block (remove-command/execute-remove action config) :remove-page (remove-command/execute-remove action config) :search (search-command/execute-search action config) + :query (query-command/execute-query action config) :show (show-command/execute-show action config) :server-list (server-command/execute-list action config) :server-status (server-command/execute-status action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index bebffd374a..988d983a83 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -86,6 +86,7 @@ :missing-repo "Use --repo " :missing-content "Use --content or pass content as args" :missing-search-text "Provide search text as a positional argument" + :missing-query "Use --query " nil)) (defn- format-error @@ -184,7 +185,11 @@ (mapv (fn [item] [(:db/id item) (or (:title item) (:content item))]) - (or results [])))) + (or results [])))) + +(defn- format-query-results + [result] + (pr-str result)) (defn- format-graph-info [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]} now-ms] @@ -275,6 +280,7 @@ :graph-export (format-graph-export context) :graph-import (format-graph-import context) :search (format-search-results (:results data)) + :query (format-query-results (:result data)) :show (or (:message data) (pr-str data)) (if (and (map? data) (contains? data :message)) (:message data) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index e540212a88..1af322508c 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, query, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 0099685b24..dfadd11647 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -165,9 +165,9 @@ (defn- spawn-server! [{:keys [repo data-dir]}] - (let [script (node-path/join js/__dirname "db-worker-node.js") - args #js [script "--repo" repo "--data-dir" data-dir] - child (.spawn child-process "node" args #js {:detached true + (let [script (node-path/join js/__dirname "../dist/db-worker-node.js") + args #js ["--repo" repo "--data-dir" data-dir] + child (.spawn child-process script args #js {:detached true :stdio "ignore"})] (.unref child) child)) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index cb5668f273..6ccc40a4bb 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -165,7 +165,7 @@ (deftest db-worker-node-parse-args-ignores-host-and-port (let [parse-args #'db-worker-node/parse-args - result (parse-args #js ["node" "db-worker-node.js" + result (parse-args #js ["node" "dist/db-worker-node.js" "--host" "0.0.0.0" "--port" "1234" "--repo" "logseq_db_parse_args" diff --git a/src/test/logseq/cli/command/query_test.cljs b/src/test/logseq/cli/command/query_test.cljs new file mode 100644 index 0000000000..54a032f547 --- /dev/null +++ b/src/test/logseq/cli/command/query_test.cljs @@ -0,0 +1,32 @@ +(ns logseq.cli.command.query-test + (:require [cljs.test :refer [deftest is testing]] + [clojure.string :as string] + [logseq.cli.command.query :as query-command])) + +(deftest test-build-action-parses-query + (testing "query parses query and inputs" + (let [result (query-command/build-action {:query "[:find ?e :in $ ?title :where [?e :block/title ?title]]" + :inputs "[\"Hello\"]"} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= :query (get-in result [:action :type]))) + (is (= "logseq_db_demo" (get-in result [:action :repo]))) + (is (= '[:find ?e :in $ ?title :where [?e :block/title ?title]] + (get-in result [:action :query]))) + (is (= ["Hello"] (get-in result [:action :inputs])))))) + +(deftest test-build-action-invalid-edn + (testing "invalid query edn returns invalid-options" + (let [result (query-command/build-action {:query "[:find ?e"} + "logseq_db_demo")] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))) + (is (string/includes? (get-in result [:error :message]) "query")))) + + (testing "invalid inputs edn returns invalid-options" + (let [result (query-command/build-action {:query "[:find ?e :where [?e :block/title \"Hello\"]]" + :inputs "[\"Hello"} + "logseq_db_demo")] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))) + (is (string/includes? (get-in result [:error :message]) "inputs"))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 9ec04f5935..c6196177ea 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -19,6 +19,7 @@ (is (string/includes? summary "remove")) (is (string/includes? summary "move")) (is (string/includes? summary "search")) + (is (string/includes? summary "query")) (is (string/includes? summary "show")) (is (string/includes? summary "graph")) (is (string/includes? summary "server")))) @@ -482,6 +483,22 @@ (is (= :show (:command result))) (is (= "Home" (get-in result [:options :page-name])))))) +(deftest test-verb-subcommand-parse-query + (testing "query requires query option" + (let [result (commands/parse-args ["query"])] + (is (false? (:ok? result))) + (is (= :missing-query (get-in result [:error :code]))))) + + (testing "query parses with query and inputs" + (let [result (commands/parse-args ["query" + "--query" "[:find ?e :where [?e :block/title \"Hello\"]]" + "--inputs" "[\"Hello\"]"])] + (is (true? (:ok? result))) + (is (= :query (:command result))) + (is (= "[:find ?e :where [?e :block/title \"Hello\"]]" + (get-in result [:options :query]))) + (is (= "[\"Hello\"]" (get-in result [:options :inputs])))))) + (deftest test-verb-subcommand-parse-graph-import-export (testing "graph create requires --repo even with positional args" (let [result (commands/parse-args ["graph" "create" "demo"])] diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 10d50c4d46..b82ca0b0b7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -200,6 +200,14 @@ {:output-format nil})] (is (= "Line 1\nLine 2" result))))) +(deftest test-human-output-query + (testing "query renders raw result" + (let [result (format/format-result {:status :ok + :command :query + :data {:result [[1] [2] [3]]}} + {:output-format nil})] + (is (= "[[1] [2] [3]]" result))))) + (deftest test-human-output-error-formatting (testing "errors include code and hint when available" (let [result (format/format-result {:status :error diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9bfbe4cf97..c1a1f8dab8 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1,12 +1,12 @@ (ns logseq.cli.integration-test - (:require [cljs.reader :as reader] + (:require ["fs" :as fs] + ["path" :as node-path] + [cljs.reader :as reader] [cljs.test :refer [deftest is async]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] [logseq.cli.main :as cli-main] - [promesa.core :as p] - ["fs" :as fs] - ["path" :as node-path])) + [promesa.core :as p])) (defn- run-cli [args data-dir cfg-path] @@ -32,7 +32,7 @@ (defn- node-title [node] - (or (:block/title node) (:title node))) + (or (:block/title node) (:block/content node) (:title node) (:content node))) (defn- node-uuid [node] @@ -49,456 +49,491 @@ node (some #(find-block-by-title % title) (node-children node))))) - (deftest test-cli-graph-list (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - result (run-cli ["graph" "list"] data-dir cfg-path) - payload (parse-json-output result)] - (is (= 0 (:exit-code result))) - (is (= "ok" (:status payload))) - (is (contains? payload :data)) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + result (run-cli ["graph" "list"] data-dir cfg-path) + payload (parse-json-output result)] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (contains? payload :data)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-graph-create-and-info (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "demo-graph"] data-dir cfg-path) - create-payload (parse-json-output create-result) - info-result (run-cli ["graph" "info"] data-dir cfg-path) - info-payload (parse-json-output info-result) - stop-result (run-cli ["server" "stop" "--repo" "demo-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code create-result))) - (is (= "ok" (:status create-payload))) - (is (= 0 (:exit-code info-result))) - (is (= "ok" (:status info-payload))) - (is (= "demo-graph" (get-in info-payload [:data :graph]))) - (is (= 0 (:exit-code stop-result))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--repo" "demo-graph"] data-dir cfg-path) + create-payload (parse-json-output create-result) + info-result (run-cli ["graph" "info"] data-dir cfg-path) + info-payload (parse-json-output info-result) + stop-result (run-cli ["server" "stop" "--repo" "demo-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code create-result))) + (is (= "ok" (:status create-payload))) + (is (= 0 (:exit-code info-result))) + (is (= "ok" (:status info-payload))) + (is (= "demo-graph" (get-in info-payload [:data :graph]))) + (is (= 0 (:exit-code stop-result))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-list-add-search-show-remove (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) - add-page-result (run-cli ["--repo" "content-graph" "add" "page" "--page" "TestPage"] data-dir cfg-path) - add-page-payload (parse-json-output add-page-result) - list-page-result (run-cli ["--repo" "content-graph" "list" "page"] data-dir cfg-path) - list-page-payload (parse-json-output list-page-result) - list-tag-result (run-cli ["--repo" "content-graph" "list" "tag"] data-dir cfg-path) - list-tag-payload (parse-json-output list-tag-result) - list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) - list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) - add-block-payload (parse-json-output add-block-result) - _ (p/delay 100) - search-result (run-cli ["--repo" "content-graph" "search" "t"] data-dir cfg-path) - search-payload (parse-json-output search-result) - show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) - show-payload (parse-json-output show-result) - remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--page" "TestPage"] data-dir cfg-path) - remove-page-payload (parse-json-output remove-page-result) - stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code add-page-result))) - (is (= "ok" (:status add-page-payload))) - (is (= "ok" (:status add-block-payload))) - (is (= "ok" (:status list-page-payload))) - (is (vector? (get-in list-page-payload [:data :items]))) - (is (= "ok" (:status list-tag-payload))) - (is (vector? (get-in list-tag-payload [:data :items]))) - (is (= "ok" (:status list-property-payload))) - (is (vector? (get-in list-property-payload [:data :items]))) - (is (= "ok" (:status search-payload))) - (is (vector? (get-in search-payload [:data :results]))) - (let [types (set (map :type (get-in search-payload [:data :results])))] - (is (contains? types "page")) - (is (contains? types "block")) - (is (contains? types "tag")) - (is (contains? types "property"))) - (is (= "ok" (:status show-payload))) - (is (contains? (get-in show-payload [:data :root]) :uuid)) - (is (= "ok" (:status remove-page-payload))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) + add-page-result (run-cli ["--repo" "content-graph" "add" "page" "--page" "TestPage"] data-dir cfg-path) + add-page-payload (parse-json-output add-page-result) + list-page-result (run-cli ["--repo" "content-graph" "list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + list-tag-result (run-cli ["--repo" "content-graph" "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) + list-property-payload (parse-json-output list-property-result) + add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + search-result (run-cli ["--repo" "content-graph" "search" "t"] data-dir cfg-path) + search-payload (parse-json-output search-result) + show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) + show-payload (parse-json-output show-result) + remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--page" "TestPage"] data-dir cfg-path) + remove-page-payload (parse-json-output remove-page-result) + stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (= "ok" (:status add-block-payload))) + (is (= "ok" (:status list-page-payload))) + (is (vector? (get-in list-page-payload [:data :items]))) + (is (= "ok" (:status list-tag-payload))) + (is (vector? (get-in list-tag-payload [:data :items]))) + (is (= "ok" (:status list-property-payload))) + (is (vector? (get-in list-property-payload [:data :items]))) + (is (= "ok" (:status search-payload))) + (is (vector? (get-in search-payload [:data :results]))) + (let [types (set (map :type (get-in search-payload [:data :results])))] + (is (contains? types "page")) + (is (contains? types "block")) + (is (contains? types "tag")) + (is (contains? types "property"))) + (is (= "ok" (:status show-payload))) + (is (contains? (get-in show-payload [:data :root]) :uuid)) + (is (= "ok" (:status remove-page-payload))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-query + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-query") + query-text "[:find ?e :in $ ?title :where [?e :block/title ?title]]"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--repo" "query-graph"] data-dir cfg-path) + create-payload (parse-json-output create-result) + _ (run-cli ["--repo" "query-graph" "add" "page" "--page" "QueryPage"] data-dir cfg-path) + _ (run-cli ["--repo" "query-graph" "add" "block" "--target-page-name" "QueryPage" "--content" "Query block"] data-dir cfg-path) + _ (run-cli ["--repo" "query-graph" "add" "block" "--target-page-name" "QueryPage" "--content" "Query block"] data-dir cfg-path) + _ (p/delay 100) + query-result (run-cli ["--repo" "query-graph" + "query" + "--query" query-text + "--inputs" "[\"Query block\"]"] + data-dir cfg-path) + query-payload (parse-json-output query-result) + result (get-in query-payload [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" "query-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status create-payload))) + (is (= 0 (:exit-code query-result))) + (is (= "ok" (:status query-payload))) + (is (vector? result)) + (is (= 2 (count result))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-show-search-resolve-nested-uuid-refs (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) - show-inner (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) - show-inner-payload (parse-json-output show-inner) - inner-node (find-block-by-title (get-in show-inner-payload [:data :root]) "Inner") - inner-uuid (node-uuid inner-node) - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" - "--content" (str "See [[" inner-uuid "]]")] data-dir cfg-path) - show-middle (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) - show-middle-payload (parse-json-output show-middle) - middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) "See [[Inner]]") - middle-uuid (node-uuid middle-node) - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" - "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) - show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) - show-outer-payload (parse-json-output show-outer) - outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") - search-result (run-cli ["--repo" "nested-refs" "search" "Outer"] data-dir cfg-path) - search-payload (parse-json-output search-result) - search-item (some (fn [item] - (when (and (string? (:content item)) - (string/includes? (:content item) "Outer")) - item)) - (get-in search-payload [:data :results])) - stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (some? inner-uuid)) - (is (some? middle-uuid)) - (is (some? outer-node)) - (is (= "Outer [[See [[Inner]]]]" (:content search-item))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) + show-inner (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-inner-payload (parse-json-output show-inner) + inner-node (find-block-by-title (get-in show-inner-payload [:data :root]) "Inner") + inner-uuid (node-uuid inner-node) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + "--content" (str "See [[" inner-uuid "]]")] data-dir cfg-path) + show-middle (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-middle-payload (parse-json-output show-middle) + middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) "See [[Inner]]") + middle-uuid (node-uuid middle-node) + _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) + show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-outer-payload (parse-json-output show-outer) + outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") + search-result (run-cli ["--repo" "nested-refs" "search" "Outer"] data-dir cfg-path) + search-payload (parse-json-output search-result) + search-item (some (fn [item] + (when (and (string? (:content item)) + (string/includes? (:content item) "Outer")) + item)) + (get-in search-payload [:data :results])) + stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? inner-uuid)) + (is (some? middle-uuid)) + (is (some? outer-node)) + (is (= "Outer [[See [[Inner]]]]" (:content search-item))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-show-linked-references-json (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - target-show-before (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) - target-before-payload (parse-json-output target-show-before) - target-uuid (or (get-in target-before-payload [:data :root :block/uuid]) - (get-in target-before-payload [:data :root :uuid])) - ref-content (str "See [[" target-uuid "]]") - _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) - source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) - source-payload (parse-json-output source-show) - ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-content) - ref-uuid (or (:block/uuid ref-node) (:uuid ref-node)) - target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) - target-payload (parse-json-output target-show) - linked-refs (get-in target-payload [:data :linked-references]) - linked-blocks (:blocks linked-refs) - linked-uuids (set (map (fn [block] - (or (:block/uuid block) (:uuid block))) - linked-blocks)) - linked-page-titles (set (keep (fn [block] - (or (get-in block [:block/page :block/title]) - (get-in block [:block/page :block/name]) - (get-in block [:page :title]) - (get-in block [:page :name]))) - linked-blocks)) - stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (some? target-uuid)) - (is (= "ok" (:status target-payload))) - (is (some? ref-uuid)) - (is (contains? linked-uuids ref-uuid)) - (is (contains? linked-page-titles "SourcePage")) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + target-show-before (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-before-payload (parse-json-output target-show-before) + target-uuid (or (get-in target-before-payload [:data :root :block/uuid]) + (get-in target-before-payload [:data :root :uuid])) + target-title (or (get-in target-before-payload [:data :root :block/title]) + (get-in target-before-payload [:data :root :block/name]) + "TargetPage") + ref-content (str "See [[" target-uuid "]]") + ref-title (str "See [[" target-title "]]") + _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) + source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) + source-payload (parse-json-output source-show) + ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-title) + ref-uuid (or (:block/uuid ref-node) (:uuid ref-node)) + target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-payload (parse-json-output target-show) + linked-refs (get-in target-payload [:data :linked-references]) + linked-blocks (:blocks linked-refs) + linked-uuids (set (map (fn [block] + (or (:block/uuid block) (:uuid block))) + linked-blocks)) + linked-page-titles (set (keep (fn [block] + (or (get-in block [:block/page :block/title]) + (get-in block [:block/page :block/name]) + (get-in block [:page :title]) + (get-in block [:page :name]))) + linked-blocks)) + stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? target-uuid)) + (is (= "ok" (:status target-payload))) + (is (some? ref-uuid)) + (is (contains? linked-uuids ref-uuid)) + (is (contains? linked-page-titles "SourcePage")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-move-block (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) - source-show (run-cli ["--repo" "move-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) - source-payload (parse-json-output source-show) - parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") - parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) - _ (run-cli ["--repo" "move-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) - move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--target-page-name" "TargetPage"] data-dir cfg-path) - move-payload (parse-json-output move-result) - target-show (run-cli ["--repo" "move-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) - target-payload (parse-json-output target-show) - moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") - child-node (find-block-by-title moved-node "Child Block") - stop-result (run-cli ["server" "stop" "--repo" "move-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= "ok" (:status move-payload))) - (is (some? parent-uuid)) - (is (some? moved-node)) - (is (some? child-node)) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) + source-show (run-cli ["--repo" "move-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) + source-payload (parse-json-output source-show) + parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") + parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + _ (run-cli ["--repo" "move-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) + move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--target-page-name" "TargetPage"] data-dir cfg-path) + move-payload (parse-json-output move-result) + target-show (run-cli ["--repo" "move-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + target-payload (parse-json-output target-show) + moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") + child-node (find-block-by-title moved-node "Child Block") + stop-result (run-cli ["server" "stop" "--repo" "move-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status move-payload))) + (is (some? parent-uuid)) + (is (some? moved-node)) + (is (some? child-node)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-add-block-pos-ordering (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) - parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) - parent-payload (parse-json-output parent-show) - parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") - parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "first-child" "--content" "First"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) - final-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) - final-payload (parse-json-output final-show) - final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") - child-titles (map node-title (node-children final-parent)) - stop-result (run-cli ["server" "stop" "--repo" "add-pos-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (some? parent-uuid)) - (is (= ["First" "Last"] (vec child-titles))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) + parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) + parent-payload (parse-json-output parent-show) + parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") + parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "first-child" "--content" "First"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) + final-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) + final-payload (parse-json-output final-show) + final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") + child-titles (map node-title (node-children final-parent)) + stop-result (run-cli ["server" "stop" "--repo" "add-pos-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (some? parent-uuid)) + (is (= ["First" "Last"] (vec child-titles))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-output-formats-graph-list (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - json-result (run-cli ["graph" "list" "--output" "json"] data-dir cfg-path) - json-payload (parse-json-output json-result) - edn-result (run-cli ["graph" "list" "--output" "edn"] data-dir cfg-path) - edn-payload (parse-edn-output edn-result) - human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path)] - (is (= 0 (:exit-code json-result))) - (is (= "ok" (:status json-payload))) - (is (= 0 (:exit-code edn-result))) - (is (= :ok (:status edn-payload))) - (is (not (string/starts-with? (:output human-result) "{:status"))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + json-result (run-cli ["graph" "list" "--output" "json"] data-dir cfg-path) + json-payload (parse-json-output json-result) + edn-result (run-cli ["graph" "list" "--output" "edn"] data-dir cfg-path) + edn-payload (parse-edn-output edn-result) + human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path)] + (is (= 0 (:exit-code json-result))) + (is (= "ok" (:status json-payload))) + (is (= 0 (:exit-code edn-result))) + (is (= :ok (:status edn-payload))) + (is (not (string/starts-with? (:output human-result) "{:status"))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-list-outputs-include-id (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "list-id-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) - list-page-result (run-cli ["list" "page"] data-dir cfg-path) - list-page-payload (parse-json-output list-page-result) - list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) - list-tag-payload (parse-json-output list-tag-result) - list-property-result (run-cli ["list" "property"] data-dir cfg-path) - list-property-payload (parse-json-output list-property-result) - stop-result (run-cli ["server" "stop" "--repo" "list-id-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= "ok" (:status list-page-payload))) - (is (every? #(contains? % :id) (get-in list-page-payload [:data :items]))) - (is (= "ok" (:status list-tag-payload))) - (is (every? #(contains? % :id) (get-in list-tag-payload [:data :items]))) - (is (= "ok" (:status list-property-payload))) - (is (every? #(contains? % :id) (get-in list-property-payload [:data :items]))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "list-id-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + list-property-result (run-cli ["list" "property"] data-dir cfg-path) + list-property-payload (parse-json-output list-property-result) + stop-result (run-cli ["server" "stop" "--repo" "list-id-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status list-page-payload))) + (is (every? #(contains? % :id) (get-in list-page-payload [:data :items]))) + (is (= "ok" (:status list-tag-payload))) + (is (every? #(contains? % :id) (get-in list-tag-payload [:data :items]))) + (is (= "ok" (:status list-property-payload))) + (is (every? #(contains? % :id) (get-in list-property-payload [:data :items]))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-list-page-human-output (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "human-list-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) - list-page-result (run-cli ["list" "page" "--output" "human"] data-dir cfg-path) - output (:output list-page-result)] - (is (= 0 (:exit-code list-page-result))) - (is (string/includes? output "TITLE")) - (is (string/includes? output "TestPage")) - (is (string/includes? output "Count:")) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "human-list-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page" "--output" "human"] data-dir cfg-path) + output (:output list-page-result)] + (is (= 0 (:exit-code list-page-result))) + (is (string/includes? output "TITLE")) + (is (string/includes? output "TestPage")) + (is (string/includes? output "Count:")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-show-page-block-by-id-and-uuid (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "show-page-block-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) - list-page-result (run-cli ["list" "page" "--expand"] data-dir cfg-path) - list-page-payload (parse-json-output list-page-result) - page-item (some (fn [item] - (when (= "TestPage" (or (:block/title item) (:title item))) - item)) - (get-in list-page-payload [:data :items])) - page-id (or (:db/id page-item) (:id page-item)) - page-uuid (or (:block/uuid page-item) (:uuid page-item)) - show-by-id-result (run-cli ["show" "--id" (str page-id) "--format" "json"] data-dir cfg-path) - show-by-id-payload (parse-json-output show-by-id-result) - show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) "--format" "json"] data-dir cfg-path) - show-by-uuid-payload (parse-json-output show-by-uuid-result) - stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= "ok" (:status list-page-payload))) - (is (some? page-item)) - (is (some? page-id)) - (is (some? page-uuid)) - (is (= "ok" (:status show-by-id-payload))) - (is (= (str page-uuid) (str (or (get-in show-by-id-payload [:data :root :uuid]) - (get-in show-by-id-payload [:data :root :block/uuid]))))) - (is (= "ok" (:status show-by-uuid-payload))) - (is (= (str page-uuid) (str (or (get-in show-by-uuid-payload [:data :root :uuid]) - (get-in show-by-uuid-payload [:data :root :block/uuid]))))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "show-page-block-graph"] data-dir cfg-path) + _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + list-page-result (run-cli ["list" "page" "--expand"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-item (some (fn [item] + (when (= "TestPage" (or (:block/title item) (:title item))) + item)) + (get-in list-page-payload [:data :items])) + page-id (or (:db/id page-item) (:id page-item)) + page-uuid (or (:block/uuid page-item) (:uuid page-item)) + show-by-id-result (run-cli ["show" "--id" (str page-id) "--format" "json"] data-dir cfg-path) + show-by-id-payload (parse-json-output show-by-id-result) + show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) "--format" "json"] data-dir cfg-path) + show-by-uuid-payload (parse-json-output show-by-uuid-result) + stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status list-page-payload))) + (is (some? page-item)) + (is (some? page-id)) + (is (some? page-uuid)) + (is (= "ok" (:status show-by-id-payload))) + (is (= (str page-uuid) (str (or (get-in show-by-id-payload [:data :root :uuid]) + (get-in show-by-id-payload [:data :root :block/uuid]))))) + (is (= "ok" (:status show-by-uuid-payload))) + (is (= (str page-uuid) (str (or (get-in show-by-uuid-payload [:data :root :uuid]) + (get-in show-by-uuid-payload [:data :root :block/uuid]))))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-show-linked-references (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - list-page-result (run-cli ["--repo" "linked-refs-graph" "list" "page" "--expand"] + (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + list-page-result (run-cli ["--repo" "linked-refs-graph" "list" "page" "--expand"] + data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-item (some (fn [item] + (when (= "TargetPage" (or (:block/title item) (:title item))) + item)) + (get-in list-page-payload [:data :items])) + page-id (or (:db/id page-item) (:id page-item)) + blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") + _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" + "--blocks" blocks-edn] data-dir cfg-path) + show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) - list-page-payload (parse-json-output list-page-result) - page-item (some (fn [item] - (when (= "TargetPage" (or (:block/title item) (:title item))) - item)) - (get-in list-page-payload [:data :items])) - page-id (or (:db/id page-item) (:id page-item)) - blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") - _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" - "--blocks" blocks-edn] data-dir cfg-path) - show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] - data-dir cfg-path) - show-payload (parse-json-output show-result) - linked (get-in show-payload [:data :linked-references]) - ref-block (first (:blocks linked)) - stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= "ok" (:status show-payload))) - (is (some? page-id)) - (is (map? linked)) - (is (pos? (:count linked))) - (is (seq (:blocks linked))) - (is (some? ref-block)) - (is (some? (or (:block/uuid ref-block) (:uuid ref-block)))) - (is (some? (or (get-in ref-block [:page :title]) - (get-in ref-block [:page :name])))) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + show-payload (parse-json-output show-result) + linked (get-in show-payload [:data :linked-references]) + ref-block (first (:blocks linked)) + stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status show-payload))) + (is (some? page-id)) + (is (map? linked)) + (is (pos? (:count linked))) + (is (seq (:blocks linked))) + (is (some? ref-block)) + (is (some? (or (:block/uuid ref-block) (:uuid ref-block)))) + (is (some? (or (get-in ref-block [:page :title]) + (get-in ref-block [:page :name])))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-graph-export-import-edn (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-export-edn")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - export-graph "export-edn-graph" - import-graph "import-edn-graph" - export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") - _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "page" "--page" "ExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "ExportPage" "--content" "Export content"] data-dir cfg-path) - export-result (run-cli ["--repo" export-graph - "graph" "export" - "--type" "edn" - "--output" export-path] data-dir cfg-path) - export-payload (parse-json-output export-result) - _ (run-cli ["--repo" import-graph - "graph" "import" - "--type" "edn" - "--input" export-path] data-dir cfg-path) - list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) - list-payload (parse-json-output list-result) - stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) - stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] - (is (= 0 (:exit-code export-result))) - (is (= "ok" (:status export-payload))) - (is (fs/existsSync export-path)) - (is (pos? (.-size (fs/statSync export-path)))) - (is (= "ok" (:status list-payload))) - (is (some (fn [item] - (= "ExportPage" (or (:title item) (:block/title item)))) - (get-in list-payload [:data :items]))) - (is (= 0 (:exit-code stop-export))) - (is (= 0 (:exit-code stop-import))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-export-edn")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + export-graph "export-edn-graph" + import-graph "import-edn-graph" + export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") + _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "page" "--page" "ExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "ExportPage" "--content" "Export content"] data-dir cfg-path) + export-result (run-cli ["--repo" export-graph + "graph" "export" + "--type" "edn" + "--output" export-path] data-dir cfg-path) + export-payload (parse-json-output export-result) + _ (run-cli ["--repo" import-graph + "graph" "import" + "--type" "edn" + "--input" export-path] data-dir cfg-path) + list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-payload (parse-json-output list-result) + stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + (is (= 0 (:exit-code export-result))) + (is (= "ok" (:status export-payload))) + (is (fs/existsSync export-path)) + (is (pos? (.-size (fs/statSync export-path)))) + (is (= "ok" (:status list-payload))) + (is (some (fn [item] + (= "ExportPage" (or (:title item) (:block/title item)))) + (get-in list-payload [:data :items]))) + (is (= 0 (:exit-code stop-export))) + (is (= 0 (:exit-code stop-import))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-graph-export-import-sqlite (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-export-sqlite")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - export-graph "export-sqlite-graph" - import-graph "import-sqlite-graph" - export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") - _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) - export-result (run-cli ["--repo" export-graph - "graph" "export" - "--type" "sqlite" - "--output" export-path] data-dir cfg-path) - export-payload (parse-json-output export-result) - _ (run-cli ["--repo" import-graph - "graph" "import" - "--type" "sqlite" - "--input" export-path] data-dir cfg-path) - list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) - list-payload (parse-json-output list-result) - stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) - stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] - (is (= 0 (:exit-code export-result))) - (is (= "ok" (:status export-payload))) - (is (fs/existsSync export-path)) - (is (pos? (.-size (fs/statSync export-path)))) - (is (= "ok" (:status list-payload))) - (is (some (fn [item] - (= "SQLiteExportPage" (or (:title item) (:block/title item)))) - (get-in list-payload [:data :items]))) - (is (= 0 (:exit-code stop-export))) - (is (= 0 (:exit-code stop-import))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-export-sqlite")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + export-graph "export-sqlite-graph" + import-graph "import-sqlite-graph" + export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") + _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) + export-result (run-cli ["--repo" export-graph + "graph" "export" + "--type" "sqlite" + "--output" export-path] data-dir cfg-path) + export-payload (parse-json-output export-result) + _ (run-cli ["--repo" import-graph + "graph" "import" + "--type" "sqlite" + "--input" export-path] data-dir cfg-path) + list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-payload (parse-json-output list-result) + stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + (is (= 0 (:exit-code export-result))) + (is (= "ok" (:status export-payload))) + (is (fs/existsSync export-path)) + (is (pos? (.-size (fs/statSync export-path)))) + (is (= "ok" (:status list-payload))) + (is (some (fn [item] + (= "SQLiteExportPage" (or (:title item) (:block/title item)))) + (get-in list-payload [:data :items]))) + (is (= 0 (:exit-code stop-export))) + (is (= 0 (:exit-code stop-import))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index a995cfea6c..dd0482d5ac 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -23,9 +23,8 @@ (.chdir js/process "/") (spawn-server! {:repo "logseq_db_spawn_test" :data-dir "/tmp/logseq-db-worker"}) - (is (= "node" (:cmd @captured))) - (is (= (node-path/join js/__dirname "db-worker-node.js") - (first (:args @captured)))) + (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") + (:cmd @captured))) (is (some #{"--repo"} (:args @captured))) (is (some #{"--data-dir"} (:args @captured))) (is (not-any? #{"--host" "--port"} (:args @captured))) From fd1dd6d63ed94c06acc3a4e8620c224d9ea509b3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 23 Jan 2026 18:08:55 +0800 Subject: [PATCH 037/375] impl 013-logseq-cli-datascript-query.md (2) --- .../013-logseq-cli-datascript-query.md | 91 +++++++++ src/main/logseq/cli/command/add.cljs | 106 +++++++++-- src/main/logseq/cli/command/query.cljs | 177 +++++++++++++++++- src/main/logseq/cli/commands.cljs | 12 +- src/main/logseq/cli/format.cljs | 13 ++ src/main/logseq/cli/main.cljs | 9 +- src/test/logseq/cli/command/query_test.cljs | 109 ++++++++++- src/test/logseq/cli/commands_test.cljs | 17 +- src/test/logseq/cli/format_test.cljs | 14 ++ src/test/logseq/cli/integration_test.cljs | 69 +++++++ 10 files changed, 582 insertions(+), 35 deletions(-) diff --git a/docs/agent-guide/013-logseq-cli-datascript-query.md b/docs/agent-guide/013-logseq-cli-datascript-query.md index 2bd4bc730b..9fb2903dc3 100644 --- a/docs/agent-guide/013-logseq-cli-datascript-query.md +++ b/docs/agent-guide/013-logseq-cli-datascript-query.md @@ -57,7 +57,98 @@ The unit tests will assert that parsing rejects invalid EDN for --inputs, that a - Return datascript-query results without transformation. - Keep all argument parsing and validation inside query command module using --query and --inputs. - Keep db-worker-node changes to zero unless a new worker API is required. +- Add `custom-queries` to cli.edn for storing pre-defined Datascript queries that the CLI can list and run by name. +- Add built-in queries that appear in the query list alongside custom queries. +- Optional inputs should support default values in cli.edn, and built-in queries should ship with reasonable defaults for their optional inputs (required inputs can omit defaults). + - `block-search` (search-title) + ``` + [:find [?e ...] + :in $ ?search-title + :where + [?e :block/title ?title] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title)]] + ``` + - `task-search` (search-status, ?search-title, ?recent-days) + ``` + ;; Modify this query so search-title and recent-days are optional parameters. + ;; ?now-ms is injected by the CLI so users don't need to pass it (and should be hidden in query list output). + ;; Example: logseq query --name task-search --inputs '["doing"]' + [:find [?e ...] + :in $ ?search-status ?search-title ?recent-days ?now-ms + :where + [?e :block/title ?title] + [?e :logseq.property/status ?s] + [?s :db/ident ?status-ident] + [(= ?status-ident ?search-status)] + [(clojure.string/lower-case ?title) ?title-lower-case] + (or-join [?search-title ?title-lower-case] + [(nil? ?search-title)] + [(clojure.string/blank? ?search-title)] + (and [(clojure.string/lower-case ?search-title) ?search-title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title-lower-case)])) + [(get-else $ ?e :block/updated-at 0) ?updated-at] + (or + [(nil? ?recent-days)] + (and [(number? ?recent-days)] + [(<= ?recent-days 0)]) + (and [(number? ?recent-days)] + [(>= ?updated-at (- ?now-ms (* ?recent-days 86400000)))]) )] + ``` +## cli.edn query shape + +Represent queries as a map keyed by query name. Keep the query form as EDN data (not a string) so it can be read directly. Optional fields like `:doc` and `:inputs` are included to help with listing and UX. `:inputs` should allow optional inputs to declare default values that are used when the CLI caller omits them. Internal inputs like `?now-ms` should be hidden from `query list` output. + +Suggested `:inputs` shapes: +- `["search-status" "?search-title" "?recent-days"]` (legacy string-only form) +- `[{:name "search-status"} {:name "?search-title" :default nil} {:name "?recent-days" :default nil}]` (explicit defaults) + +Example: +``` +{:custom-queries + {"block-search" + {:doc "Find blocks by title substring (case-insensitive)." + :inputs ["search-title"] + :query [:find [?e ...] + :in $ ?search-title + :where + [?e :block/title ?title] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title)]]} + + "task-search" + {:doc "Find tasks by status, optional title substring, optional recent-days." + :inputs [{:name "search-status"} + {:name "?search-title" :default nil} + {:name "?recent-days" :default nil} + ;; ?now-ms is internal; CLI fills it with current ms and query list should hide it. + {:name "?now-ms" :default :now-ms}] + :query [:find [?e ...] + :in $ ?search-status ?search-title ?recent-days ?now-ms + :where + [?e :block/title ?title] + [?e :logseq.property/status ?s] + [?s :db/ident ?status-ident] + [(= ?status-ident ?search-status)] + [(clojure.string/lower-case ?title) ?title-lower-case] + (or-join [?search-title ?title-lower-case] + [(nil? ?search-title)] + [(clojure.string/blank? ?search-title)] + (and [(clojure.string/lower-case ?search-title) ?search-title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title-lower-case)])) + [(get-else $ ?e :block/updated-at 0) ?updated-at] + (or + [(nil? ?recent-days)] + (and [(number? ?recent-days)] + [(<= ?recent-days 0)]) + (and [(number? ?recent-days)] + [(>= ?updated-at (- ?now-ms (* ?recent-days 86400000)))]) )]}}} +``` + +Notes: +- Built-in queries live in code but should be merged into the same map shape when listing or resolving by name. +- `:inputs` is optional metadata for CLI help. It does not affect execution. ## Question Use --query and --inputs options for the query subcommand. diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index f9c31566f7..105188dd8d 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -9,6 +9,7 @@ [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [logseq.common.util.date-time :as date-time-util] + [logseq.common.uuid :as common-uuid] [promesa.core :as p])) (def ^:private content-add-spec @@ -19,7 +20,8 @@ :coerce :long} :target-uuid {:desc "Target block UUID"} :target-page-name {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling)"}}) + :pos {:desc "Position (first-child, last-child, sibling)"} + :status {:desc "Task status (todo, doing, done, etc.)"}}) (def ^:private add-page-spec {:page {:desc "Page name"}}) @@ -50,6 +52,57 @@ (def ^:private add-positions #{"first-child" "last-child" "sibling"}) +(def ^:private status-aliases + {"todo" :logseq.property/status.todo + "doing" :logseq.property/status.doing + "done" :logseq.property/status.done + "now" :logseq.property/status.doing + "later" :logseq.property/status.todo + "wait" :logseq.property/status.backlog + "waiting" :logseq.property/status.backlog + "backlog" :logseq.property/status.backlog + "canceled" :logseq.property/status.canceled + "cancelled" :logseq.property/status.canceled + "in-review" :logseq.property/status.in-review + "in_review" :logseq.property/status.in-review + "inreview" :logseq.property/status.in-review + "in-progress" :logseq.property/status.doing + "in progress" :logseq.property/status.doing + "inprogress" :logseq.property/status.doing}) + +(defn- normalize-status + [value] + (let [text (some-> value string/trim) + parsed (when (and (seq text) (string/starts-with? text ":")) + (common-util/safe-read-string {:log-error? false} text)) + normalized (cond + (qualified-keyword? parsed) + parsed + + (keyword? parsed) + (get status-aliases (name parsed)) + + (seq text) + (get status-aliases (string/lower-case text)) + + :else nil)] + normalized)) + +(defn- ensure-block-uuids + [blocks] + (mapv (fn [block] + (let [current (:block/uuid block)] + (cond + (some? current) + (update block :block/uuid (fn [value] + (if (and (string? value) (common-util/uuid-string? value)) + (uuid value) + value))) + + :else + (assoc block :block/uuid (common-uuid/gen-uuid))))) + blocks)) + (defn invalid-options? [opts] (let [pos (some-> (:pos opts) string/trim string/lower-case) @@ -130,21 +183,34 @@ {:ok? false :error {:code :missing-repo :message "repo is required for add"}} - (let [blocks-result (read-blocks options args)] + (let [blocks-result (read-blocks options args) + status-text (some-> (:status options) string/trim) + status (when (seq status-text) (normalize-status status-text))] + (cond + (and (seq status-text) (nil? status)) + {:ok? false + :error {:code :invalid-options + :message (str "invalid status: " status-text)}} + + :else (if-not (:ok? blocks-result) blocks-result (let [vector-result (ensure-blocks (:value blocks-result))] (if-not (:ok? vector-result) vector-result - {:ok? true - :action {:type :add-block - :repo repo - :graph (core/repo->graph repo) - :target-id (:target-id options) - :target-uuid (some-> (:target-uuid options) string/trim) - :target-page-name (some-> (:target-page-name options) string/trim) - :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") - :blocks (:value vector-result)}})))))) + (let [blocks (cond-> (:value vector-result) + status + ensure-block-uuids)] + {:ok? true + :action {:type :add-block + :repo repo + :graph (core/repo->graph repo) + :target-id (:target-id options) + :target-uuid (some-> (:target-uuid options) string/trim) + :target-page-name (some-> (:target-page-name options) string/trim) + :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") + :status status + :blocks blocks}})))))))) (defn build-add-page-action [options repo] @@ -167,17 +233,31 @@ [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) target-id (resolve-add-target cfg action) + status (:status action) pos (:pos action) opts (case pos "last-child" {:sibling? false :bottom? true} "sibling" {:sibling? true} {:sibling? false}) + opts (cond-> opts + status + (assoc :keep-uuid? true)) ops [[:insert-blocks [(:blocks action) target-id (assoc opts :outliner-op :insert-blocks)]]] - result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])] + _ (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) + _ (when status + (let [block-ids (->> (:blocks action) + (map :block/uuid) + (remove nil?) + vec)] + (when (seq block-ids) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:batch-set-property [block-ids :logseq.property/status status {}]]] + {}]))))] {:status :ok - :data {:result result}}))) + :data {:result nil}}))) (defn execute-add-page [action config] diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index 42ef0f7fc0..6b1a8c2fc1 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -9,10 +9,54 @@ (def ^:private query-spec {:query {:desc "Datascript query EDN"} + :name {:desc "Query name from cli.edn custom-queries or built-ins"} :inputs {:desc "EDN vector of query inputs"}}) +(def ^:private query-list-spec + {}) + (def entries - [(core/command-entry ["query"] :query "Run a Datascript query" query-spec)]) + [(core/command-entry ["query"] :query "Run a Datascript query" query-spec) + (core/command-entry ["query" "list"] :query-list "List available queries" query-list-spec)]) + +(def ^:private built-in-query-specs + {"block-search" + {:doc "Find blocks by title substring (case-insensitive)." + :inputs ["search-title"] + :query '[:find [?e ...] + :in $ ?search-title + :where + [?e :block/title ?title] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(clojure.string/lower-case ?search-title) ?search-title-lower-case] + [(clojure.string/includes? ?title-lower-case ?search-title-lower-case)]]} + + "task-search" + {:doc "Find tasks by status, optional title substring, optional recent-days." + :inputs [{:name "search-status"} + {:name "?search-title" :default ""} + {:name "?recent-days" :default 0} + {:name "?now-ms" :default :now-ms}] + :query '[:find [?e ...] + :in $ ?search-status ?search-title ?recent-days ?now-ms + :where + [?e :block/title ?title] + [?e :logseq.property/status ?status] + [?status :db/ident ?status-ident] + [(= ?status-ident ?search-status)] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(str ?search-title) ?search-title-string] + [(clojure.string/lower-case ?search-title-string) ?search-title-lower-case] + [(clojure.string/includes? ?title-lower-case ?search-title-lower-case)] + [(get-else $ ?e :block/updated-at 0) ?updated-at] + (or-join [?recent-days ?updated-at ?now-ms ?days-ago] + (and [(nil? ?recent-days)] + [(identity 0) ?days-ago]) + (and [(<= ?recent-days 0)] + [(identity 0) ?days-ago]) + (and [(* ?recent-days 86400000) ?recent-days-ms] + [(- ?now-ms ?recent-days-ms) ?days-ago] + [(>= ?updated-at ?days-ago)]))]}}) (defn- parse-edn [label value] @@ -23,27 +67,134 @@ :message (str "invalid " label " edn")}} {:ok? true :value parsed}))) +(defn- normalize-query-name + [name] + (when (some? name) + (let [raw (if (keyword? name) (name name) (str name)) + text (string/trim raw)] + (when (seq text) text)))) + +(defn- normalize-query-entry + [name source spec] + (let [spec (cond + (vector? spec) {:query spec} + (map? spec) spec + :else nil) + query (:query spec) + name (normalize-query-name name)] + (when (and name query) + (cond-> (assoc spec :name name :source source) + (nil? (:inputs spec)) (assoc :inputs []))))) + +(defn- hide-internal-inputs + [entry] + (let [inputs (vec (remove (fn [input] + (let [name (cond + (string? (:name input)) (:name input) + (keyword? (:name input)) (name (:name input)) + :else nil)] + (= "?now-ms" name))) + (or (:inputs entry) [])))] + (assoc entry :inputs inputs))) + +(defn list-queries + [config] + (let [built-ins (mapv (fn [[name spec]] + (normalize-query-entry name :built-in spec)) + built-in-query-specs) + custom-queries (or (:custom-queries config) {}) + customs (mapv (fn [[name spec]] + (normalize-query-entry name :custom spec)) + custom-queries) + merged (reduce (fn [acc entry] + (if entry + (assoc acc (:name entry) entry) + acc)) + {} + (concat built-ins customs))] + (->> (vals merged) + (sort-by :name) + vec))) + +(defn- find-query + [config name] + (some #(when (= name (:name %)) %) (list-queries config))) + +(defn- optional-input? + [input] + (let [name (cond + (string? input) input + (keyword? input) (name input) + (map? input) (some-> (:name input) str) + :else nil)] + (and (string? name) (string/starts-with? name "?")))) + +(defn- input-default + [input] + (when (map? input) + (if (contains? input :default) + (let [value (:default input)] + (if (= :now-ms value) + (js/Date.now) + value)) + nil))) + +(defn- normalize-named-inputs + [entry inputs] + (let [spec-inputs (or (:inputs entry) []) + required-count (count (remove optional-input? spec-inputs)) + inputs (or inputs [])] + (if (< (count inputs) required-count) + {:ok? false + :error {:code :invalid-options + :message "inputs missing required values"}} + {:ok? true + :value (if (< (count inputs) (count spec-inputs)) + (let [missing (subvec (vec spec-inputs) (count inputs))] + (into (vec inputs) (map input-default missing))) + inputs)}))) + (defn build-action - [options repo] + [options repo config] (if-not (seq repo) {:ok? false :error {:code :missing-repo :message "repo is required for query"}} - (let [query-text (some-> (:query options) string/trim)] - (if-not (seq query-text) + (let [query-text (some-> (:query options) string/trim) + query-name (normalize-query-name (:name options))] + (cond + (and (seq query-text) (seq query-name)) + {:ok? false + :error {:code :invalid-options + :message "use either --query or --name, not both"}} + + (and (not (seq query-text)) (not (seq query-name))) {:ok? false :error {:code :missing-query :message "query is required"}} - (let [query-result (parse-edn "query" query-text)] + + :else + (let [query-result (if (seq query-text) + (parse-edn "query" query-text) + (if-let [entry (find-query config query-name)] + {:ok? true :value (:query entry) :entry entry} + {:ok? false + :error {:code :unknown-query + :message (str "unknown query: " query-name)}}))] (if-not (:ok? query-result) query-result (let [inputs-text (some-> (:inputs options) string/trim) inputs-result (when (seq inputs-text) - (parse-edn "inputs" inputs-text))] + (parse-edn "inputs" inputs-text)) + named-inputs (when-let [entry (:entry query-result)] + (normalize-named-inputs entry (or (:value inputs-result) [])))] (cond (and inputs-result (not (:ok? inputs-result))) inputs-result + (and named-inputs (not (:ok? named-inputs))) + named-inputs + (and inputs-result (not (vector? (:value inputs-result)))) {:ok? false :error {:code :invalid-options @@ -55,7 +206,14 @@ :repo repo :graph (core/repo->graph repo) :query (:value query-result) - :inputs (or (:value inputs-result) [])}})))))))) + :inputs (or (:value named-inputs) + (:value inputs-result) + [])}})))))))) + +(defn build-list-action + [_options _repo] + {:ok? true + :action {:type :query-list}}) (defn execute-query [action config] @@ -64,3 +222,8 @@ results (transport/invoke cfg :thread-api/q false [(:repo action) args])] {:status :ok :data {:result results}}))) + +(defn execute-query-list + [_action config] + (p/resolved {:status :ok + :data {:queries (mapv hide-internal-inputs (list-queries config))}})) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index a1f9b79af0..208b749d14 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -176,7 +176,9 @@ (and (= command :search) (not has-args?)) (missing-search-result summary) - (and (= command :query) (not (seq (some-> (:query opts) string/trim)))) + (and (= command :query) + (not (seq (some-> (:query opts) string/trim))) + (not (seq (some-> (:name opts) string/trim)))) (missing-query-result summary) (and (#{:list-page :list-tag :list-property} command) @@ -237,7 +239,7 @@ :error {:code :missing-command :message "missing command"} :summary summary}) - (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "remove"} (first args))) + (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "remove" "query"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) (try (let [result (cli/dispatch table args {:spec global-spec})] @@ -352,7 +354,10 @@ (search-command/build-action options args repo) :query - (query-command/build-action options repo) + (query-command/build-action options repo config) + + :query-list + (query-command/build-list-action options repo) :show (show-command/build-action options repo) @@ -392,6 +397,7 @@ :remove-page (remove-command/execute-remove action config) :search (search-command/execute-search action config) :query (query-command/execute-query action config) + :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) :server-list (server-command/execute-list action config) :server-status (server-command/execute-status action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 988d983a83..d773b198f8 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -87,6 +87,7 @@ :missing-content "Use --content or pass content as args" :missing-search-text "Provide search text as a positional argument" :missing-query "Use --query " + :unknown-query "Use `logseq query list` to see available queries" nil)) (defn- format-error @@ -191,6 +192,17 @@ [result] (pr-str result)) +(defn- format-query-list + [queries] + (format-counted-table + ["NAME" "INPUTS" "SOURCE" "DOC"] + (mapv (fn [{:keys [name inputs source doc]}] + [name + (if (seq inputs) (string/join ", " inputs) "-") + (clojure.core/name (or source :custom)) + (or doc "-")]) + (or queries [])))) + (defn- format-graph-info [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]} now-ms] (string/join "\n" @@ -281,6 +293,7 @@ :graph-import (format-graph-import context) :search (format-search-results (:results data)) :query (format-query-results (:result data)) + :query-list (format-query-list (:queries data)) :show (or (:message data) (pr-str data)) (if (and (map? data) (contains? data :message)) (:message data) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 1af322508c..dcba210339 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, query, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) @@ -47,8 +47,8 @@ (-> (commands/execute (:action action-result) cfg) (p/then (fn [result] (let [opts (cond-> cfg - (:output-format result) - (assoc :output-format (:output-format result)))] + (:output-format result) + (assoc :output-format (:output-format result)))] {:exit-code 0 :output (format/format-result result opts)}))) (p/catch (fn [error] @@ -74,4 +74,5 @@ (p/then (fn [{:keys [exit-code output]}] (when (seq output) (println output)) - (.exit js/process exit-code))))) + (when-not (zero? exit-code) + (.exit js/process exit-code)))))) diff --git a/src/test/logseq/cli/command/query_test.cljs b/src/test/logseq/cli/command/query_test.cljs index 54a032f547..a6becea914 100644 --- a/src/test/logseq/cli/command/query_test.cljs +++ b/src/test/logseq/cli/command/query_test.cljs @@ -7,7 +7,8 @@ (testing "query parses query and inputs" (let [result (query-command/build-action {:query "[:find ?e :in $ ?title :where [?e :block/title ?title]]" :inputs "[\"Hello\"]"} - "logseq_db_demo")] + "logseq_db_demo" + {})] (is (true? (:ok? result))) (is (= :query (get-in result [:action :type]))) (is (= "logseq_db_demo" (get-in result [:action :repo]))) @@ -18,7 +19,8 @@ (deftest test-build-action-invalid-edn (testing "invalid query edn returns invalid-options" (let [result (query-command/build-action {:query "[:find ?e"} - "logseq_db_demo")] + "logseq_db_demo" + {})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))) (is (string/includes? (get-in result [:error :message]) "query")))) @@ -26,7 +28,108 @@ (testing "invalid inputs edn returns invalid-options" (let [result (query-command/build-action {:query "[:find ?e :where [?e :block/title \"Hello\"]]" :inputs "[\"Hello"} - "logseq_db_demo")] + "logseq_db_demo" + {})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))) (is (string/includes? (get-in result [:error :message]) "inputs"))))) + +(deftest test-build-action-named-query + (testing "named query resolves from config" + (let [config {:custom-queries {"my-query" + {:doc "Custom query" + :inputs ["title"] + :query '[:find ?e + :in $ ?title + :where + [?e :block/title ?title]]}}} + result (query-command/build-action {:name "my-query" + :inputs "[\"Alpha\"]"} + "logseq_db_demo" + config)] + (is (true? (:ok? result))) + (is (= '[:find ?e :in $ ?title :where [?e :block/title ?title]] + (get-in result [:action :query]))) + (is (= ["Alpha"] (get-in result [:action :inputs]))))) + + (testing "unknown named query returns unknown-query error" + (let [result (query-command/build-action {:name "missing"} + "logseq_db_demo" + {:custom-queries {}})] + (is (false? (:ok? result))) + (is (= :unknown-query (get-in result [:error :code]))))) + + (testing "rejects both name and query" + (let [result (query-command/build-action {:name "my-query" + :query "[:find ?e :where [?e :block/title ?title]]"} + "logseq_db_demo" + {:custom-queries {"my-query" {:query '[:find ?e]}}})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "optional inputs are padded with nils" + (let [config {:custom-queries {"task-search" + {:inputs ["search-status" "?search-title" "?recent-days"] + :query '[:find [?e ...] + :in $ ?search-status ?search-title ?recent-days + :where + [?e :block/title ?title]]}}} + result (query-command/build-action {:name "task-search" + :inputs "[\"doing\"]"} + "logseq_db_demo" + config)] + (is (true? (:ok? result))) + (is (= ["doing" nil nil] (get-in result [:action :inputs]))))) + + (testing "missing required inputs returns invalid-options" + (let [config {:custom-queries {"task-search" + {:inputs ["search-status" "?search-title"] + :query '[:find [?e ...] + :in $ ?search-status ?search-title + :where + [?e :block/title ?title]]}}} + result (query-command/build-action {:name "task-search" + :inputs "[]"} + "logseq_db_demo" + config)] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "optional inputs can define defaults in cli.edn" + (let [config {:custom-queries {"task-search" + {:inputs [{:name "search-status"} + {:name "?search-title" :default "fallback-title"} + {:name "?recent-days" :default 7}] + :query '[:find [?e ...] + :in $ ?search-status ?search-title ?recent-days + :where + [?e :block/title ?title]]}}} + result (query-command/build-action {:name "task-search" + :inputs "[\"doing\"]"} + "logseq_db_demo" + config)] + (is (true? (:ok? result))) + (is (= ["doing" "fallback-title" 7] (get-in result [:action :inputs]))))) + + (testing "built-in task-search uses defaults for optional inputs" + (let [result (query-command/build-action {:name "task-search" + :inputs "[\"doing\"]"} + "logseq_db_demo" + {})] + (is (true? (:ok? result))) + (let [inputs (get-in result [:action :inputs])] + (is (= ["doing" "" 0] (subvec inputs 0 3))) + (is (number? (nth inputs 3))))))) + +(deftest test-query-list-merges-built-in-and-custom + (testing "built-in and custom queries are both listed" + (let [queries (query-command/list-queries {:custom-queries {"custom-q" {:query '[:find ?e]}}}) + names (set (map :name queries))] + (is (contains? names "block-search")) + (is (contains? names "task-search")) + (is (contains? names "custom-q")))) + + (testing "custom query overrides built-in name" + (let [queries (query-command/list-queries {:custom-queries {"block-search" {:query '[:find ?e]}}}) + block-search (first (filter #(= "block-search" (:name %)) queries))] + (is (= :custom (:source block-search)))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index c6196177ea..e8533c24b1 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -75,7 +75,14 @@ summary (:summary result)] (is (true? (:help? result))) (is (string/includes? summary "server list")) - (is (string/includes? summary "server start"))))) + (is (string/includes? summary "server start")))) + + (testing "query group shows subcommands" + (let [result (commands/parse-args ["query"]) + summary (:summary result)] + (is (true? (:help? result))) + (is (string/includes? summary "query list")) + (is (string/includes? summary "query [options]"))))) (deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" @@ -112,7 +119,7 @@ (testing "rejects legacy commands" (doseq [command ["graph-list" "graph-create" "graph-switch" "graph-remove" "graph-validate" "graph-info" "block" "tree" - "ping" "status" "query" "export"]] + "ping" "status" "export"]] (let [result (commands/parse-args [command])] (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) @@ -484,10 +491,10 @@ (is (= "Home" (get-in result [:options :page-name])))))) (deftest test-verb-subcommand-parse-query - (testing "query requires query option" + (testing "query shows group help" (let [result (commands/parse-args ["query"])] - (is (false? (:ok? result))) - (is (= :missing-query (get-in result [:error :code]))))) + (is (true? (:help? result))) + (is (string/includes? (:summary result) "query list")))) (testing "query parses with query and inputs" (let [result (commands/parse-args ["query" diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index b82ca0b0b7..4e630e11ac 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -208,6 +208,20 @@ {:output-format nil})] (is (= "[[1] [2] [3]]" result))))) +(deftest test-human-output-query-list + (testing "query list renders a table with count" + (let [result (format/format-result {:status :ok + :command :query-list + :data {:queries [{:name "block-search" + :inputs ["search-title"] + :doc "Find blocks" + :source :built-in}]}} + {:output-format nil})] + (is (= (str "NAME INPUTS SOURCE DOC\n" + "block-search search-title built-in Find blocks\n" + "Count: 1") + result))))) + (deftest test-human-output-error-formatting (testing "errors include code and hint when available" (let [result (format/format-result {:status :error diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index c1a1f8dab8..c7ecb41c06 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -168,6 +168,75 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-query-task-search + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-task-query")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--repo" "task-query-graph"] data-dir cfg-path) + create-payload (parse-json-output create-result) + _ (run-cli ["--repo" "task-query-graph" "add" "page" "--page" "Tasks"] data-dir cfg-path) + _ (run-cli ["--repo" "task-query-graph" + "add" "block" + "--target-page-name" "Tasks" + "--content" "Task one" + "--status" "doing"] + data-dir cfg-path) + _ (run-cli ["--repo" "task-query-graph" + "add" "block" + "--target-page-name" "Tasks" + "--content" "Task two" + "--status" "doing"] + data-dir cfg-path) + _ (run-cli ["--repo" "task-query-graph" + "add" "block" + "--target-page-name" "Tasks" + "--content" "Task three" + "--status" "todo"] + data-dir cfg-path) + _ (p/delay 100) + list-result (run-cli ["query" "list"] data-dir cfg-path) + list-payload (parse-json-output list-result) + task-entry (some (fn [entry] + (when (= "task-search" (:name entry)) entry)) + (get-in list-payload [:data :queries])) + query-result (run-cli ["--repo" "task-query-graph" + "query" + "--name" "task-search" + "--inputs" "[:logseq.property/status.doing]"] + data-dir cfg-path) + query-payload (parse-json-output query-result) + query-nil-result (run-cli ["--repo" "task-query-graph" + "query" + "--name" "task-search" + "--inputs" "[:logseq.property/status.doing nil 1]"] + data-dir cfg-path) + query-nil-payload (parse-json-output query-nil-result) + _ (prn :xxxx query-payload) + result (get-in query-payload [:data :result]) + nil-result (get-in query-nil-payload [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" "task-query-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status create-payload))) + (is (= "ok" (:status list-payload))) + (is (= [{:name "search-status"} + {:name "?search-title" :default ""} + {:name "?recent-days" :default 0}] + (:inputs task-entry))) + (is (= 0 (:exit-code query-result))) + (is (= "ok" (:status query-payload))) + (is (vector? result)) + (is (= 2 (count result))) + (is (= 0 (:exit-code query-nil-result))) + (is (= "ok" (:status query-nil-payload))) + (is (vector? nil-result)) + (is (= 2 (count nil-result))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-show-search-resolve-nested-uuid-refs (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] From 138519daedcf1fee5afc917f06378ce038ead5d3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 23 Jan 2026 22:20:06 +0800 Subject: [PATCH 038/375] 014-logseq-cli-show-multi-id.md (1) --- .../014-logseq-cli-show-multi-id.md | 63 ++++ docs/cli/logseq-cli.md | 2 + src/main/logseq/cli/command/show.cljs | 192 +++++++++-- src/main/logseq/cli/format.cljs | 13 +- src/test/logseq/cli/commands_test.cljs | 298 ++++++++++-------- src/test/logseq/cli/format_test.cljs | 2 +- src/test/logseq/cli/integration_test.cljs | 125 ++++++++ 7 files changed, 517 insertions(+), 178 deletions(-) create mode 100644 docs/agent-guide/014-logseq-cli-show-multi-id.md diff --git a/docs/agent-guide/014-logseq-cli-show-multi-id.md b/docs/agent-guide/014-logseq-cli-show-multi-id.md new file mode 100644 index 0000000000..a17ae363e9 --- /dev/null +++ b/docs/agent-guide/014-logseq-cli-show-multi-id.md @@ -0,0 +1,63 @@ +# Logseq CLI Show Multi-ID Implementation Plan + +Goal: Extend the logseq-cli show command so `--id` accepts one or more block ids, and when passed as `[ ...]`, it displays all blocks separated by a clear delimiter. +Architecture: Keep existing CLI parsing and db-worker-node transport, but allow `--id` to accept an EDN vector of ids; execute one fetch per id or a single multi-id fetch if supported by the worker thread API. +Tech Stack: ClojureScript, babashka.cli, db-worker-node HTTP transport, Logseq block formatting. +Related: Builds on existing show command behavior and db-worker-node thread API usage. + +## Problem statement + +The current `logseq show --id ` only supports a single block id, which makes it cumbersome to inspect multiple blocks from scripts. +We need `--id` to accept `[ ...]` and print each corresponding block, separated by a reasonable visual delimiter. +This should align with existing logseq-cli and db-worker-node patterns and preserve existing single-id behavior. +This also enables a pipeline workflow such as: `logseq query --name task-search --inputs '["todo"]' | xargs logseq show -id`. + +## Note on `logseq query` output + +`logseq query` output handling: +1. Validate it is valid EDN. +2. Replace all spaces with commas. + +## Testing Plan + +I will add unit tests for show argument parsing to accept a vector of ids and to reject invalid EDN in `--id`. +I will add an integration test that runs `logseq show --id '["id1" "id2"]'` and asserts that both blocks are present in output with the delimiter between them. +I will keep existing single-id tests intact and ensure no regressions in JSON/EDN output modes. +I will follow @test-driven-development and write the failing tests before implementing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +Locate the show command definition and parsing in `src/main/logseq/cli/command/show.cljs` (or equivalent) and identify current `--id` parsing behavior. +Update argument parsing so `--id` accepts either a single id (string) or an EDN vector of ids; parse via `cljs.reader/read-string` or `logseq.common.util/safe-read-string` with clear errors on invalid EDN. +Normalize the parsed value into a vector of ids, preserving the current single-id behavior by wrapping the string in a vector. +Update the show execution path to iterate through ids and fetch each block via existing db-worker-node transport; if a batch API exists, prefer it but keep compatibility with current API. +Add output formatting logic to insert a delimiter between each block in human output (for example `\n-----\n` or `\n====\n`), and keep JSON/EDN output structured as a vector of blocks instead of concatenated text. +Update command help/usage in `src/main/logseq/cli/main.cljs` and `src/main/logseq/cli/commands.cljs` to document vector form, including an example. +Add unit tests in `src/test/logseq/cli/commands_test.cljs` or a show-specific test file to validate parsing and error cases. +Add an integration test in `src/test/logseq/cli/integration_test.cljs` to verify multi-id output with delimiter and correct block ordering. +Confirm db-worker-node thread API endpoints used by show (likely in `src/main/frontend/worker/`) do not need changes; if they do, add a minimal batch fetch method and corresponding tests. + +## Edge cases + +`--id` contains invalid EDN (e.g., `[` without closing bracket) should return a clear invalid-options error and non-zero exit. +Mixed types in the id vector (e.g., numbers, maps) should either coerce to strings or be rejected with a clear error; prefer rejection to avoid surprising behavior. +Missing blocks (id not found) should return a clear message per block while still printing other valid blocks. +Output delimiter should not appear before the first block or after the last block. + +## Testing Details + +Unit tests should cover parsing of a single id, a vector of ids, and invalid EDN. +Integration tests should create two blocks, fetch them by ids, and verify both are present in order with the delimiter separating them in human output. +JSON/EDN outputs should be a vector of block structures matching current single-id output shape. + +## Implementation Details + +- Update `--id` parsing to accept EDN vectors of ids while preserving single-id strings. +- Normalize `id` input to `ids` vector for downstream handling. +- Loop fetches through existing db-worker-node transport, or add a batch fetch endpoint only if necessary. +- Insert a delimiter between blocks in human output; keep machine-readable outputs as structured vectors. +- Update help text and tests accordingly. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 14c4e9ff6a..52fd6a6f7b 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -71,6 +71,7 @@ Inspect and edit commands: - `remove block --block ` - remove a block and its children - `remove page --page ` - remove a page and its children - `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) +- `query --query [--inputs ]` - run a Datascript query against the graph - `show --page-name [--format text|json|edn] [--level ]` - show page tree - `show --uuid [--format text|json|edn] [--level ]` - show block tree - `show --id [--format text|json|edn] [--level ]` - show block tree by db/id @@ -98,6 +99,7 @@ Output formats: - Global `--output ` (also accepted per subcommand) - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. - Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. - `show` human output prints the `:db/id` as the first column followed by a tree: diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 2a82cea3cc..62cb0744aa 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -8,8 +8,7 @@ [promesa.core :as p])) (def ^:private show-spec - {:id {:desc "Block db/id" - :coerce :long} + {:id {:desc "Block db/id or EDN vector of ids"} :uuid {:desc "Block UUID"} :page-name {:desc "Page name"} :level {:desc "Limit tree depth" @@ -22,10 +21,58 @@ (def ^:private show-formats #{"text" "json" "edn"}) +(def ^:private multi-id-delimiter "\n================================================================\n") + +(defn- valid-id? + [value] + (and (number? value) (integer? value))) + +(defn- parse-id-option + [value] + (let [invalid (fn [message] + {:ok? false :message message})] + (cond + (nil? value) + {:ok? true :value nil :multi? false} + + (vector? value) + (cond + (empty? value) (invalid "id vector must contain at least one id") + (every? valid-id? value) {:ok? true :value (vec value) :multi? true} + :else (invalid "id vector must contain only integers")) + + (valid-id? value) + {:ok? true :value [value] :multi? false} + + (string? value) + (let [text (string/trim value)] + (cond + (string/blank? text) + (invalid "id is required") + + (string/starts-with? text "[") + (let [parsed (common-util/safe-read-string {:log-error? false} text)] + (cond + (nil? parsed) (invalid "invalid id edn") + (not (vector? parsed)) (invalid "id must be a vector") + (empty? parsed) (invalid "id vector must contain at least one id") + (every? valid-id? parsed) {:ok? true :value (vec parsed) :multi? true} + :else (invalid "id vector must contain only integers"))) + + (re-matches #"-?\\d+" text) + {:ok? true :value [(js/parseInt text 10)] :multi? false} + + :else + (invalid "id must be a number or vector of numbers"))) + + :else + (invalid "id must be a number or vector of numbers")))) + (defn invalid-options? [opts] (let [format (:format opts) - level (:level opts)] + level (:level opts) + id-result (parse-id-option (:id opts))] (cond (and (seq format) (not (contains? show-formats (string/lower-case format)))) (str "invalid format: " format) @@ -33,6 +80,9 @@ (and (some? level) (< level 1)) "level must be >= 1" + (and (some? (:id opts)) (not (:ok? id-result))) + (:message id-result) + :else nil))) @@ -442,45 +492,119 @@ :error {:code :missing-repo :message "repo is required for show"}} (let [format (some-> (:format options) string/lower-case) + id-result (parse-id-option (:id options)) + ids (:value id-result) + multi-id? (:multi? id-result) targets (filter some? [(:id options) (:uuid options) (:page-name options)])] (if (empty? targets) {:ok? false :error {:code :missing-target :message "block or page is required"}} - {:ok? true - :action {:type :show - :repo repo - :id (:id options) - :uuid (:uuid options) - :page-name (:page-name options) - :level (:level options) - :format format}})))) + (if (and (some? (:id options)) (not (:ok? id-result))) + {:ok? false + :error {:code :invalid-options + :message (:message id-result)}} + {:ok? true + :action {:type :show + :repo repo + :id (when (and (seq ids) (not multi-id?)) (first ids)) + :ids ids + :multi-id? multi-id? + :uuid (:uuid options) + :page-name (:page-name options) + :level (:level options) + :format format}}))))) + +(defn- build-tree-data + [config action] + (p/let [tree-data (fetch-tree config action) + root-id (get-in tree-data [:root :db/id]) + linked-refs (if root-id + (fetch-linked-references config (:repo action) root-id) + {:count 0 :blocks []}) + uuid-refs (collect-uuid-refs tree-data linked-refs) + uuid->label (fetch-uuid-labels config (:repo action) uuid-refs) + tree-data (assoc tree-data + :linked-references linked-refs + :uuid->label uuid->label) + tree-data (resolve-uuid-refs-in-tree-data tree-data uuid->label)] + tree-data)) + +(defn- multi-id-error-message + [id error] + (let [data (ex-data error) + code (:code data) + message (or (:message data) (.-message error) (str error))] + (if (= code :block-not-found) + (str "Block " id " not found") + (str "Block " id ": " message)))) + +(defn- multi-id-error-entry + [id error] + (let [data (ex-data error) + code (:code data) + message (multi-id-error-message id error) + error-map (cond-> {:message message} + code (assoc :code code))] + {:id id + :error error-map})) (defn execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - tree-data (fetch-tree cfg action) - root-id (get-in tree-data [:root :db/id]) - linked-refs (if root-id - (fetch-linked-references cfg (:repo action) root-id) - {:count 0 :blocks []}) - uuid-refs (collect-uuid-refs tree-data linked-refs) - uuid->label (fetch-uuid-labels cfg (:repo action) uuid-refs) - tree-data (assoc tree-data - :linked-references linked-refs - :uuid->label uuid->label) - tree-data (resolve-uuid-refs-in-tree-data tree-data uuid->label) - format (:format action)] - (case format - "edn" - {:status :ok - :data tree-data - :output-format :edn} + format (:format action) + ids (:ids action) + multi-id? (:multi-id? action)] + (if (and (seq ids) multi-id?) + (p/let [results (p/all (map (fn [id] + (-> (build-tree-data cfg (assoc action :id id)) + (p/then (fn [tree-data] + {:ok? true + :id id + :tree tree-data})) + (p/catch (fn [error] + {:ok? false + :id id + :error error})))) + ids)) + payload (case format + "edn" + {:status :ok + :data (mapv (fn [{:keys [ok? tree id error]}] + (if ok? + tree + (multi-id-error-entry id error))) + results) + :output-format :edn} - "json" - {:status :ok - :data tree-data - :output-format :json} + "json" + {:status :ok + :data (mapv (fn [{:keys [ok? tree id error]}] + (if ok? + tree + (multi-id-error-entry id error))) + results) + :output-format :json} - {:status :ok - :data {:message (tree->text-with-linked-refs tree-data)}})))) + {:status :ok + :data {:message (string/join multi-id-delimiter + (map (fn [{:keys [ok? tree id error]}] + (if ok? + (tree->text-with-linked-refs tree) + (multi-id-error-message id error))) + results))}})] + payload) + (p/let [tree-data (build-tree-data cfg action)] + (case format + "edn" + {:status :ok + :data tree-data + :output-format :edn} + + "json" + {:status :ok + :data tree-data + :output-format :json} + + {:status :ok + :data {:message (tree->text-with-linked-refs tree-data)}})))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index d773b198f8..58da921d43 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -2,7 +2,8 @@ "Formatting helpers for CLI output." (:require [clojure.string :as string] [clojure.walk :as walk] - [logseq.cli.command.core :as command-core])) + [logseq.cli.command.core :as command-core] + [logseq.common.util :as common-util])) (defn- normalize-json [value] @@ -190,7 +191,12 @@ (defn- format-query-results [result] - (pr-str result)) + (let [edn-str (pr-str result) + parsed (common-util/safe-read-string {:log-error? false} edn-str) + valid? (or (some? parsed) (= "nil" (string/trim edn-str)))] + (if valid? + (string/replace edn-str " " ",") + edn-str))) (defn- format-query-list [queries] @@ -255,7 +261,8 @@ (str "Exported " export-type " to " output)) (defn- format-graph-import - [{:keys [import-type input]}] + [{:keys [import-type input] :as xxx}] + (prn :xxx xxx) (str "Imported " import-type " from " input)) (defn- format-graph-action diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index e8533c24b1..1109511945 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -488,7 +488,18 @@ (let [result (commands/parse-args ["show" "--page-name" "Home"])] (is (true? (:ok? result))) (is (= :show (:command result))) - (is (= "Home" (get-in result [:options :page-name])))))) + (is (= "Home" (get-in result [:options :page-name]))))) + + (testing "show parses with id vector" + (let [result (commands/parse-args ["show" "--id" "[1 2]"])] + (is (true? (:ok? result))) + (is (= :show (:command result))) + (is (= "[1 2]" (get-in result [:options :id]))))) + + (testing "show rejects invalid id edn" + (let [result (commands/parse-args ["show" "--id" "[1"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-query (testing "query shows group help" @@ -701,7 +712,14 @@ (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code])))))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "show normalizes id vector in build action" + (let [parsed {:ok? true :command :show :options {:id "[1 2]"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :show (get-in result [:action :type]))) + (is (= [1 2] (get-in result [:action :ids])))))) (deftest test-build-action-move (testing "move requires source selector" @@ -761,150 +779,150 @@ (deftest test-execute-graph-import-rejects-existing-graph (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server!] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] - (throw (ex-info "should not start server" {})))) - (-> (p/let [result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :allow-missing-graph true} - {})] - (is (= :error (:status result))) - (is (= :graph-exists (get-in result [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (done))))))) + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server!] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))) + (-> (p/let [result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :allow-missing-graph true} + {})] + (is (= :error (:status result))) + (is (= :graph-exists (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (done))))))) (deftest test-execute-graph-export (async done - (let [invoke-calls (atom []) - write-calls (atom []) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - orig-write-output transport/write-output] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [config _] - (assoc config :base-url "http://127.0.0.1:9999"))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (if (= method :thread-api/export-db-base64) - "c3FsaXRl" - {:exported true}))) - (set! transport/write-output (fn [opts] - (swap! write-calls conj opts))) - (-> (p/let [edn-result (commands/execute {:type :graph-export - :repo "logseq_db_demo" - :graph "demo" - :export-type "edn" - :output "/tmp/export.edn" - :allow-missing-graph true} - {}) - sqlite-result (commands/execute {:type :graph-export - :repo "logseq_db_demo" - :graph "demo" - :export-type "sqlite" - :output "/tmp/export.sqlite" - :allow-missing-graph true} - {})] - (is (= :ok (:status edn-result))) - (is (= :ok (:status sqlite-result))) - (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] - [:thread-api/export-db-base64 true ["logseq_db_demo"]]] - @invoke-calls)) - (is (= 2 (count @write-calls))) - (let [[edn-write sqlite-write] @write-calls] - (is (= {:format :edn :path "/tmp/export.edn" :data {:exported true}} - edn-write)) - (is (= :sqlite (:format sqlite-write))) - (is (= "/tmp/export.sqlite" (:path sqlite-write))) - (is (= "sqlite" (.toString (:data sqlite-write) "utf8"))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (set! transport/write-output orig-write-output) - (done))))))) + (let [invoke-calls (atom []) + write-calls (atom []) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + orig-write-output transport/write-output] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [config _] + (assoc config :base-url "http://127.0.0.1:9999"))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (if (= method :thread-api/export-db-base64) + "c3FsaXRl" + {:exported true}))) + (set! transport/write-output (fn [opts] + (swap! write-calls conj opts))) + (-> (p/let [edn-result (commands/execute {:type :graph-export + :repo "logseq_db_demo" + :graph "demo" + :export-type "edn" + :output "/tmp/export.edn" + :allow-missing-graph true} + {}) + sqlite-result (commands/execute {:type :graph-export + :repo "logseq_db_demo" + :graph "demo" + :export-type "sqlite" + :output "/tmp/export.sqlite" + :allow-missing-graph true} + {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] + [:thread-api/export-db-base64 true ["logseq_db_demo"]]] + @invoke-calls)) + (is (= 2 (count @write-calls))) + (let [[edn-write sqlite-write] @write-calls] + (is (= {:format :edn :path "/tmp/export.edn" :data {:exported true}} + edn-write)) + (is (= :sqlite (:format sqlite-write))) + (is (= "/tmp/export.sqlite" (:path sqlite-write))) + (is (= "sqlite" (.toString (:data sqlite-write) "utf8"))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (set! transport/write-output orig-write-output) + (done))))))) (deftest test-execute-graph-import (async done - (let [invoke-calls (atom []) - read-calls (atom []) - stop-calls (atom []) - restart-calls (atom []) - orig-list-graphs cli-server/list-graphs - orig-stop-server! cli-server/stop-server! - orig-restart-server! cli-server/restart-server! - orig-ensure-server! cli-server/ensure-server! - orig-read-input transport/read-input - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] [])) - (set! cli-server/stop-server! (fn [_ repo] - (swap! stop-calls conj repo) - (p/resolved {:ok? true}))) - (set! cli-server/restart-server! (fn [_ repo] - (swap! restart-calls conj repo) - (p/resolved {:ok? true}))) - (set! cli-server/ensure-server! (fn [config _] - (assoc config :base-url "http://127.0.0.1:9999"))) - (set! transport/read-input (fn [{:keys [format path]}] - (swap! read-calls conj [format path]) - (if (= format :edn) - {:page "Import Page"} - (js/Buffer.from "sqlite" "utf8")))) - (set! transport/invoke (fn [_ method _ args] - (swap! invoke-calls conj [method args]) - {:ok true})) - (-> (p/let [edn-result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :graph "demo" - :import-type "edn" - :input "/tmp/import.edn" - :allow-missing-graph true} - {}) - sqlite-result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :graph "demo" - :import-type "sqlite" - :input "/tmp/import.sqlite" - :allow-missing-graph true} - {})] - (is (= :ok (:status edn-result))) - (is (= :ok (:status sqlite-result))) - (is (= [[:edn "/tmp/import.edn"] - [:sqlite "/tmp/import.sqlite"]] - @read-calls)) - (is (= [[:thread-api/import-edn ["logseq_db_demo" {:page "Import Page"}]] - [:thread-api/import-db-base64 ["logseq_db_demo" "c3FsaXRl"]]] - @invoke-calls)) - (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) - (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/stop-server! orig-stop-server!) - (set! cli-server/restart-server! orig-restart-server!) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/read-input orig-read-input) - (set! transport/invoke orig-invoke) - (done))))))) + (let [invoke-calls (atom []) + read-calls (atom []) + stop-calls (atom []) + restart-calls (atom []) + orig-list-graphs cli-server/list-graphs + orig-stop-server! cli-server/stop-server! + orig-restart-server! cli-server/restart-server! + orig-ensure-server! cli-server/ensure-server! + orig-read-input transport/read-input + orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] [])) + (set! cli-server/stop-server! (fn [_ repo] + (swap! stop-calls conj repo) + (p/resolved {:ok? true}))) + (set! cli-server/restart-server! (fn [_ repo] + (swap! restart-calls conj repo) + (p/resolved {:ok? true}))) + (set! cli-server/ensure-server! (fn [config _] + (assoc config :base-url "http://127.0.0.1:9999"))) + (set! transport/read-input (fn [{:keys [format path]}] + (swap! read-calls conj [format path]) + (if (= format :edn) + {:page "Import Page"} + (js/Buffer.from "sqlite" "utf8")))) + (set! transport/invoke (fn [_ method _ args] + (swap! invoke-calls conj [method args]) + {:ok true})) + (-> (p/let [edn-result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :graph "demo" + :import-type "edn" + :input "/tmp/import.edn" + :allow-missing-graph true} + {}) + sqlite-result (commands/execute {:type :graph-import + :repo "logseq_db_demo" + :graph "demo" + :import-type "sqlite" + :input "/tmp/import.sqlite" + :allow-missing-graph true} + {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= [[:edn "/tmp/import.edn"] + [:sqlite "/tmp/import.sqlite"]] + @read-calls)) + (is (= [[:thread-api/import-edn ["logseq_db_demo" {:page "Import Page"}]] + [:thread-api/import-db-base64 ["logseq_db_demo" "c3FsaXRl"]]] + @invoke-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/stop-server! orig-stop-server!) + (set! cli-server/restart-server! orig-restart-server!) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/read-input orig-read-input) + (set! transport/invoke orig-invoke) + (done))))))) (deftest test-execute-graph-list-strips-db-prefix (async done - (let [orig-list-graphs cli-server/list-graphs] - (set! cli-server/list-graphs (fn [_] ["logseq_db_demo" "logseq_db_other"])) - (-> (p/let [result (commands/execute {:type :graph-list} {})] - (is (= :ok (:status result))) - (is (= ["demo" "other"] (get-in result [:data :graphs])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (done))))))) + (let [orig-list-graphs cli-server/list-graphs] + (set! cli-server/list-graphs (fn [_] ["logseq_db_demo" "logseq_db_other"])) + (-> (p/let [result (commands/execute {:type :graph-list} {})] + (is (= :ok (:status result))) + (is (= ["demo" "other"] (get-in result [:data :graphs])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (done))))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 4e630e11ac..277837bcae 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -206,7 +206,7 @@ :command :query :data {:result [[1] [2] [3]]}} {:output-format nil})] - (is (= "[[1] [2] [3]]" result))))) + (is (= "[[1],[2],[3]]" result))))) (deftest test-human-output-query-list (testing "query list renders a table with count" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index c7ecb41c06..bc5c60fd00 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -488,6 +488,131 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-show-multi-id + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "show-multi-id-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "show-multi-id-graph" "add" "page" "--page" "MultiPage"] + data-dir cfg-path) + _ (run-cli ["--repo" "show-multi-id-graph" "add" "block" + "--target-page-name" "MultiPage" + "--content" "Multi show one"] + data-dir cfg-path) + _ (run-cli ["--repo" "show-multi-id-graph" "add" "block" + "--target-page-name" "MultiPage" + "--content" "Multi show two"] + data-dir cfg-path) + _ (p/delay 100) + query-text "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" + query-one-result (run-cli ["--repo" "show-multi-id-graph" "query" + "--query" query-text + "--inputs" (pr-str ["Multi show one"])] + data-dir cfg-path) + query-one-payload (parse-json-output query-one-result) + block-one-id (get-in query-one-payload [:data :result]) + query-two-result (run-cli ["--repo" "show-multi-id-graph" "query" + "--query" query-text + "--inputs" (pr-str ["Multi show two"])] + data-dir cfg-path) + query-two-payload (parse-json-output query-two-result) + block-two-id (get-in query-two-payload [:data :result]) + ids-edn (str "[" block-one-id " " block-two-id "]") + show-text-result (run-cli ["--repo" "show-multi-id-graph" "show" + "--id" ids-edn + "--format" "text" + "--output" "human"] + data-dir cfg-path) + output (:output show-text-result) + idx-one (string/index-of output "Multi show one") + idx-two (string/index-of output "Multi show two") + idx-delim (string/index-of output "-----") + show-json-result (run-cli ["--repo" "show-multi-id-graph" "show" + "--id" ids-edn + "--format" "json"] + data-dir cfg-path) + show-json-payload (parse-json-output show-json-result) + show-data (:data show-json-payload) + root-titles (set (map (comp node-title :root) show-data)) + stop-result (run-cli ["server" "stop" "--repo" "show-multi-id-graph"] + data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code query-one-result))) + (is (= "ok" (:status query-one-payload))) + (is (= 0 (:exit-code query-two-result))) + (is (= "ok" (:status query-two-payload))) + (is (some? block-one-id)) + (is (some? block-two-id)) + (is (= 0 (:exit-code show-text-result))) + (is (string/includes? output "Multi show one")) + (is (string/includes? output "Multi show two")) + (is (some? idx-delim)) + (is (< idx-one idx-delim idx-two)) + (is (= 0 (:exit-code show-json-result))) + (is (= "ok" (:status show-json-payload))) + (is (vector? show-data)) + (is (= 2 (count show-data))) + (is (contains? root-titles "Multi show one")) + (is (contains? root-titles "Multi show two")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-query-human-output-pipes-to-show + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-query-pipe")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "query-pipe-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "query-pipe-graph" "add" "page" "--page" "PipePage"] + data-dir cfg-path) + _ (run-cli ["--repo" "query-pipe-graph" "add" "block" + "--target-page-name" "PipePage" + "--content" "Pipe One"] + data-dir cfg-path) + _ (run-cli ["--repo" "query-pipe-graph" "add" "block" + "--target-page-name" "PipePage" + "--content" "Pipe Two"] + data-dir cfg-path) + _ (p/delay 100) + query-text (str "[:find [?e ...]" + " :in $ ?q" + " :where" + " [?e :block/title ?title]" + " [(clojure.string/includes? ?title ?q)]]") + query-result (run-cli ["--repo" "query-pipe-graph" + "--output" "human" + "query" + "--query" query-text + "--inputs" (pr-str ["Pipe"])] + data-dir cfg-path) + ids-edn (string/trim (:output query-result)) + show-json-result (run-cli ["--repo" "query-pipe-graph" "show" + "--id" ids-edn + "--format" "json"] + data-dir cfg-path) + show-json-payload (parse-json-output show-json-result) + show-data (:data show-json-payload) + root-titles (set (map (comp node-title :root) show-data)) + stop-result (run-cli ["server" "stop" "--repo" "query-pipe-graph"] + data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code query-result))) + (is (seq ids-edn)) + (is (= 0 (:exit-code show-json-result))) + (is (= "ok" (:status show-json-payload))) + (is (vector? show-data)) + (is (contains? root-titles "Pipe One")) + (is (contains? root-titles "Pipe Two")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-show-linked-references (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] From 5dc1026a444b2ad8f59f6303b53c7b6f9c539dda Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 23 Jan 2026 23:45:52 +0800 Subject: [PATCH 039/375] 014-logseq-cli-show-multi-id.md (2) --- .../014-logseq-cli-show-multi-id.md | 8 + src/main/logseq/cli/command/core.cljs | 4 +- src/main/logseq/cli/command/query.cljs | 36 +- src/main/logseq/cli/command/search.cljs | 335 ------------------ src/main/logseq/cli/commands.cljs | 19 - src/main/logseq/cli/format.cljs | 11 - src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 54 +-- src/test/logseq/cli/format_test.cljs | 45 +-- src/test/logseq/cli/integration_test.cljs | 29 +- 10 files changed, 48 insertions(+), 495 deletions(-) delete mode 100644 src/main/logseq/cli/command/search.cljs diff --git a/docs/agent-guide/014-logseq-cli-show-multi-id.md b/docs/agent-guide/014-logseq-cli-show-multi-id.md index a17ae363e9..7b0587a855 100644 --- a/docs/agent-guide/014-logseq-cli-show-multi-id.md +++ b/docs/agent-guide/014-logseq-cli-show-multi-id.md @@ -12,12 +12,20 @@ We need `--id` to accept `[ ...]` and print each corresponding block, This should align with existing logseq-cli and db-worker-node patterns and preserve existing single-id behavior. This also enables a pipeline workflow such as: `logseq query --name task-search --inputs '["todo"]' | xargs logseq show -id`. +## Note on removing `search` + +Remove the `search` subcommand. No migration or compatibility work is required. + ## Note on `logseq query` output `logseq query` output handling: 1. Validate it is valid EDN. 2. Replace all spaces with commas. +## Note on `logseq query task-search` inputs + +The first `task-search` input `status` should be a string like `"todo"` or `"doing"`, not `:logseq.property/status.todo`. + ## Testing Plan I will add unit tests for show argument parsing to accept a vector of ids and to reject invalid EDN in `--id`. diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index a84b3884ef..cd43cd74e2 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -44,10 +44,8 @@ (defn- command-usage [cmds spec] (let [base (string/join " " cmds) - positional (when (= cmds ["search"]) "") has-options? (seq spec)] (cond-> base - positional (str " " positional) has-options? (str " [options]")))) (defn- format-commands @@ -84,7 +82,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "move" "search" "query" "show"}} + :commands #{"list" "add" "remove" "move" "query" "show"}} {:title "Graph Management" :commands #{"graph" "server"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index 6b1a8c2fc1..97a601314c 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -54,7 +54,7 @@ [(identity 0) ?days-ago]) (and [(<= ?recent-days 0)] [(identity 0) ?days-ago]) - (and [(* ?recent-days 86400000) ?recent-days-ms] + (and [(* ?recent-days 86400000) ?recent-days-ms] [(- ?now-ms ?recent-days-ms) ?days-ago] [(>= ?updated-at ?days-ago)]))]}}) @@ -154,6 +154,21 @@ (into (vec inputs) (map input-default missing))) inputs)}))) +(defn- normalize-task-search-inputs + [entry inputs] + (if (and entry (= "task-search" (:name entry)) (seq inputs)) + (let [status (first inputs) + normalized (cond + (keyword? status) status + (string? status) (let [text (string/trim status)] + (if (seq text) + (keyword "logseq.property" + (str "status." (string/lower-case text))) + status)) + :else status)] + (assoc (vec inputs) 0 normalized)) + inputs)) + (defn build-action [options repo config] (if-not (seq repo) @@ -201,14 +216,17 @@ :message "inputs must be a vector"}} :else - {:ok? true - :action {:type :query - :repo repo - :graph (core/repo->graph repo) - :query (:value query-result) - :inputs (or (:value named-inputs) - (:value inputs-result) - [])}})))))))) + (let [inputs (normalize-task-search-inputs + (:entry query-result) + (or (:value named-inputs) + (:value inputs-result) + []))] + {:ok? true + :action {:type :query + :repo repo + :graph (core/repo->graph repo) + :query (:value query-result) + :inputs inputs}}))))))))) (defn build-list-action [_options _repo] diff --git a/src/main/logseq/cli/command/search.cljs b/src/main/logseq/cli/command/search.cljs deleted file mode 100644 index c7cff1f65f..0000000000 --- a/src/main/logseq/cli/command/search.cljs +++ /dev/null @@ -1,335 +0,0 @@ -(ns logseq.cli.command.search - "Search-related CLI commands." - (:require [clojure.set :as set] - [clojure.string :as string] - [logseq.cli.command.core :as core] - [logseq.cli.server :as cli-server] - [logseq.cli.transport :as transport] - [logseq.common.util :as common-util] - [promesa.core :as p])) - -(def ^:private search-spec - {:type {:desc "Search types (page, block, tag, property, all)"} - :tag {:desc "Restrict to a specific tag"} - :case-sensitive {:desc "Case sensitive search" - :coerce :boolean} - :sort {:desc "Sort field (updated-at, created-at)"} - :order {:desc "Sort order (asc, desc)"}}) - -(def entries - [(core/command-entry ["search"] :search "Search graph" search-spec)]) - -(def ^:private search-types - #{"page" "block" "tag" "property" "all"}) - -(def ^:private uuid-ref-pattern #"\[\[([0-9a-fA-F-]{36})\]\]") -(def ^:private uuid-ref-max-depth 10) - -(defn invalid-options? - [opts] - (let [type (:type opts) - order (:order opts) - sort-field (:sort opts)] - (cond - (and (seq type) (not (contains? search-types type))) - (str "invalid type: " type) - - (and (seq sort-field) (not (#{"updated-at" "created-at"} sort-field))) - (str "invalid sort field: " sort-field) - - (and (seq order) (not (#{"asc" "desc"} order))) - (str "invalid order: " order) - - :else - nil))) - -(defn build-action - [options args repo] - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for search"}} - (let [text (some-> (first args) string/trim)] - (if (seq text) - {:ok? true - :action {:type :search - :repo repo - :text text - :search-type (or (:type options) "all") - :tag (:tag options) - :case-sensitive (:case-sensitive options) - :sort (:sort options) - :order (:order options)}} - {:ok? false - :error {:code :missing-search-text - :message "search text is required"}})))) - -(defn- query-pages - [cfg repo text case-sensitive?] - (let [query (if case-sensitive? - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q - :where - [?e :block/name ?name] - [?e :block/title ?title] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?title ?q)]] - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q - :where - [?e :block/name ?name] - [?e :block/title ?title] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?name ?q)]]) - q* (if case-sensitive? text (string/lower-case text))] - (transport/invoke cfg :thread-api/q false [repo [query q*]]))) - -(defn- query-blocks - [cfg repo text case-sensitive? tag] - (let [q* (if case-sensitive? text (string/lower-case text)) - tag-name (some-> tag string/lower-case) - query (cond - (and case-sensitive? (seq tag-name)) - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q ?tag-name - :where - [?e :block/title ?title] - [?e :block/page ?page] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?title ?q)] - [?tag :block/name ?tag-name] - [?e :block/tags ?tag]] - - case-sensitive? - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?q - :where - [?e :block/title ?title] - [?e :block/page ?page] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [(clojure.string/includes? ?title ?q)]] - - (seq tag-name) - '[:find ?e ?title ?uuid ?updated ?created - :in $ ?tag-name - :where - [?e :block/title ?title] - [?e :block/page ?page] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created] - [?tag :block/name ?tag-name] - [?e :block/tags ?tag]] - - :else - '[:find ?e ?title ?uuid ?updated ?created - :where - [?e :block/title ?title] - [?e :block/page ?page] - [?e :block/uuid ?uuid] - [(get-else $ ?e :block/updated-at 0) ?updated] - [(get-else $ ?e :block/created-at 0) ?created]]) - query-args (cond - (and case-sensitive? (seq tag-name)) - [repo [query q* tag-name]] - - case-sensitive? - [repo [query q*]] - - (seq tag-name) - [repo [query tag-name]] - - :else - [repo [query]]) - matches-text? (fn [title] - (when (string? title) - (if case-sensitive? - (string/includes? title q*) - (string/includes? (string/lower-case title) q*))))] - (-> (p/let [rows (transport/invoke cfg :thread-api/q false query-args)] - (->> (or rows []) - (filter (fn [[_ title _ _ _]] - (matches-text? title))) - (mapv (fn [[id title uuid updated created]] - {:type "block" - :db/id id - :content title - :uuid (str uuid) - :updated-at updated - :created-at created})))) - (p/catch (fn [_] - []))))) - -(defn- replace-uuid-refs-once - [value uuid->label] - (if (and (string? value) (seq uuid->label)) - (string/replace value uuid-ref-pattern - (fn [[_ id]] - (if-let [label (get uuid->label (string/lower-case id))] - (str "[[" label "]]") - (str "[[" id "]]")))) - value)) - -(defn- replace-uuid-refs - [value uuid->label] - (loop [current value - remaining uuid-ref-max-depth] - (if (or (not (string? current)) (zero? remaining) (empty? uuid->label)) - current - (let [next (replace-uuid-refs-once current uuid->label)] - (if (= next current) - current - (recur next (dec remaining))))))) - -(defn- extract-uuid-refs - [value] - (->> (re-seq uuid-ref-pattern (or value "")) - (map second) - (filter common-util/uuid-string?) - (map string/lower-case) - distinct)) - -(defn- collect-uuid-refs - [results] - (->> results - (mapcat (fn [item] (keep item [:title :content]))) - (remove string/blank?) - (mapcat extract-uuid-refs) - distinct - vec)) - -(defn- fetch-uuid-labels - [config repo uuid-strings] - (if (seq uuid-strings) - (p/let [blocks (p/all (map (fn [uuid-str] - (transport/invoke config :thread-api/pull false - [repo [:block/uuid :block/title :block/name] - [:block/uuid (uuid uuid-str)]])) - uuid-strings))] - (->> blocks - (remove nil?) - (map (fn [block] - (let [uuid-str (some-> (:block/uuid block) str)] - [(string/lower-case uuid-str) - (or (:block/title block) (:block/name block) uuid-str)]))) - (into {}))) - (p/resolved {}))) - -(defn- fetch-uuid-labels-recursive - [config repo uuid-strings] - (p/loop [pending (set (map string/lower-case uuid-strings)) - seen #{} - labels {} - remaining uuid-ref-max-depth] - (if (or (empty? pending) (zero? remaining)) - labels - (p/let [fetched (fetch-uuid-labels config repo pending) - next-labels (merge labels fetched) - next-seen (into seen (keys fetched)) - nested-refs (->> (vals fetched) - (mapcat extract-uuid-refs) - (remove next-seen) - set) - next-pending (set/difference nested-refs next-seen)] - (p/recur next-pending next-seen next-labels (dec remaining)))))) - -(defn- resolve-uuid-refs-in-results - [results uuid->label] - (mapv (fn [item] - (cond-> item - (:title item) (update :title replace-uuid-refs uuid->label) - (:content item) (update :content replace-uuid-refs uuid->label))) - (or results []))) - -(defn- normalize-search-types - [type] - (let [type (or type "all")] - (case type - "page" [:page] - "block" [:block] - "tag" [:tag] - "property" [:property] - [:page :block :tag :property]))) - -(defn- search-sort-key - [item sort-field] - (case sort-field - "updated-at" (:updated-at item) - "created-at" (:created-at item) - nil)) - -(defn execute-search - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - types (normalize-search-types (:search-type action)) - case-sensitive? (boolean (:case-sensitive action)) - text (:text action) - tag (:tag action) - page-results (when (some #{:page} types) - (p/let [rows (query-pages cfg (:repo action) text case-sensitive?)] - (mapv (fn [[id title uuid updated created]] - {:type "page" - :db/id id - :title title - :uuid (str uuid) - :updated-at updated - :created-at created}) - rows))) - block-results (when (some #{:block} types) - (query-blocks cfg (:repo action) text case-sensitive? tag)) - tag-results (when (some #{:tag} types) - (p/let [items (transport/invoke cfg :thread-api/api-list-tags false - [(:repo action) {:expand true :include-built-in true}]) - q* (if case-sensitive? text (string/lower-case text))] - (->> items - (filter (fn [item] - (let [title (:block/title item)] - (if case-sensitive? - (string/includes? title q*) - (string/includes? (string/lower-case title) q*))))) - (mapv (fn [item] - {:type "tag" - :db/id (:db/id item) - :title (:block/title item) - :uuid (:block/uuid item)}))))) - property-results (when (some #{:property} types) - (p/let [items (transport/invoke cfg :thread-api/api-list-properties false - [(:repo action) {:expand true :include-built-in true}]) - q* (if case-sensitive? text (string/lower-case text))] - (->> items - (filter (fn [item] - (let [title (:block/title item)] - (if case-sensitive? - (string/includes? title q*) - (string/includes? (string/lower-case title) q*))))) - (mapv (fn [item] - {:type "property" - :db/id (:db/id item) - :title (:block/title item) - :uuid (:block/uuid item)}))))) - results (->> (concat (or page-results []) - (or block-results []) - (or tag-results []) - (or property-results [])) - (distinct) - vec) - sorted (if-let [sort-field (:sort action)] - (let [order (or (:order action) "desc")] - (->> results - (sort-by #(search-sort-key % sort-field)) - (cond-> (= order "desc") reverse) - vec)) - results) - uuid-refs (collect-uuid-refs sorted) - uuid->label (fetch-uuid-labels-recursive cfg (:repo action) uuid-refs) - resolved (resolve-uuid-refs-in-results sorted uuid->label)] - {:status :ok - :data {:results resolved}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 208b749d14..83df14e17c 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -9,7 +9,6 @@ [logseq.cli.command.move :as move-command] [logseq.cli.command.query :as query-command] [logseq.cli.command.remove :as remove-command] - [logseq.cli.command.search :as search-command] [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] [logseq.cli.server :as cli-server] @@ -83,13 +82,6 @@ :message "output is required"} :summary summary}) -(defn- missing-search-result - [summary] - {:ok? false - :error {:code :missing-search-text - :message "search text is required"} - :summary summary}) - (defn- missing-query-result [summary] {:ok? false @@ -108,7 +100,6 @@ add-command/entries move-command/entries remove-command/entries - search-command/entries query-command/entries show-command/entries))) @@ -173,9 +164,6 @@ (and (= command :show) (> (count show-targets) 1)) (command-core/invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") - (and (= command :search) (not has-args?)) - (missing-search-result summary) - (and (= command :query) (not (seq (some-> (:query opts) string/trim))) (not (seq (some-> (:name opts) string/trim)))) @@ -188,9 +176,6 @@ (and (= command :show) (show-command/invalid-options? opts)) (command-core/invalid-options-result summary (show-command/invalid-options? opts)) - (and (= command :search) (search-command/invalid-options? opts)) - (command-core/invalid-options-result summary (search-command/invalid-options? opts)) - (and (= command :graph-export) (not (seq (graph-command/normalize-import-export-type (:type opts))))) (missing-type-result summary) @@ -350,9 +335,6 @@ :remove-page (remove-command/build-remove-page-action options repo) - :search - (search-command/build-action options args repo) - :query (query-command/build-action options repo config) @@ -395,7 +377,6 @@ :move-block (move-command/execute-move action config) :remove-block (remove-command/execute-remove action config) :remove-page (remove-command/execute-remove action config) - :search (search-command/execute-search action config) :query (query-command/execute-query action config) :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 58da921d43..9e6ac77d8f 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -86,7 +86,6 @@ :missing-graph "Use --repo " :missing-repo "Use --repo " :missing-content "Use --content or pass content as args" - :missing-search-text "Provide search text as a positional argument" :missing-query "Use --query " :unknown-query "Use `logseq query list` to see available queries" nil)) @@ -180,15 +179,6 @@ (:pid server)]) (or servers [])))) -(defn- format-search-results - [results] - (format-counted-table - ["ID" "TITLE"] - (mapv (fn [item] - [(:db/id item) - (or (:title item) (:content item))]) - (or results [])))) - (defn- format-query-results [result] (let [edn-str (pr-str result) @@ -298,7 +288,6 @@ :move-block (format-move-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) - :search (format-search-results (:results data)) :query (format-query-results (:result data)) :query-list (format-query-list (:queries data)) :show (or (:message data) (pr-str data)) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index dcba210339..c7ba6a0ec5 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -12,7 +12,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, search, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 1109511945..42dbf3fb9f 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -18,7 +18,6 @@ (is (string/includes? summary "add")) (is (string/includes? summary "remove")) (is (string/includes? summary "move")) - (is (string/includes? summary "search")) (is (string/includes? summary "query")) (is (string/includes? summary "show")) (is (string/includes? summary "graph")) @@ -467,18 +466,7 @@ (is (= "def" (get-in result [:options :target-uuid]))) (is (= "last-child" (get-in result [:options :pos])))))) -(deftest test-verb-subcommand-parse-search-show - (testing "search requires text" - (let [result (commands/parse-args ["search"])] - (is (false? (:ok? result))) - (is (= :missing-search-text (get-in result [:error :code]))))) - - (testing "search parses with text" - (let [result (commands/parse-args ["search" "hello"])] - (is (true? (:ok? result))) - (is (= :search (:command result))) - (is (= ["hello"] (:args result))))) - +(deftest test-verb-subcommand-parse-show (testing "show requires target" (let [result (commands/parse-args ["show"])] (is (false? (:ok? result))) @@ -574,22 +562,13 @@ ["add" "block" "--wat"] ["remove" "block" "--wat"] ["move" "--wat"] - ["search" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) - (testing "search rejects deprecated flags" - (doseq [args [["search" "--limit" "10" "hello"] - ["search" "--include-content" "hello"] - ["search" "--text" "hello"]]] - (let [result (commands/parse-args args)] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code])))))) - (testing "verb subcommands accept output option" - (let [result (commands/parse-args ["search" "--output" "json" "hello"])] + (let [result (commands/parse-args ["show" "--output" "json" "--page-name" "Home"])] (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) @@ -684,30 +663,6 @@ (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "search requires text" - (let [parsed {:ok? true :command :search :options {}} - result (commands/build-action parsed {:repo "demo"})] - (is (false? (:ok? result))) - (is (= :missing-search-text (get-in result [:error :code]))))) - - (testing "search defaults to all types" - (let [parsed {:ok? true :command :search :options {} :args ["hello"]} - result (commands/build-action parsed {:repo "demo"})] - (is (true? (:ok? result))) - (is (= "all" (get-in result [:action :search-type]))))) - - (testing "search uses config repo and ignores positional text for repo" - (let [parsed {:ok? true :command :search :options {} :args ["hello"]} - result (commands/build-action parsed {:repo "demo"})] - (is (true? (:ok? result))) - (is (= "logseq_db_demo" (get-in result [:action :repo]))))) - - (testing "search uses first positional argument" - (let [parsed {:ok? true :command :search :options {} :args ["hello" "world"]} - result (commands/build-action parsed {:repo "demo"})] - (is (true? (:ok? result))) - (is (= "hello" (get-in result [:action :text]))))) - (testing "show requires target" (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] @@ -765,9 +720,8 @@ (with-redefs [cli-server/list-graphs (fn [_] []) cli-server/ensure-server! (fn [_ _] (throw (ex-info "should not start server" {})))] - (-> (p/let [result (commands/execute {:type :search - :repo "logseq_db_missing" - :text "hello"} + (-> (p/let [result (commands/execute {:type :list-page + :repo "logseq_db_missing"} {})] (is (= :error (:status result))) (is (= :graph-not-exists (get-in result [:error :code]))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 277837bcae..76f29de945 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -159,40 +159,7 @@ "Host: 127.0.0.1 Port: 1234") result))))) -(deftest test-human-output-search-and-show - (testing "search renders a table with count" - (let [result (format/format-result {:status :ok - :command :search - :data {:results [{:type "page" - :db/id 101 - :title "Alpha" - :uuid "u1" - :updated-at 3 - :created-at 1} - {:type "block" - :db/id 102 - :content "Note line 1\nNote line 2" - :uuid "u2" - :updated-at 4 - :created-at 2} - {:type "tag" - :db/id 103 - :title "Taggy" - :uuid "u3"} - {:type "property" - :db/id 104 - :title "Prop" - :uuid "u4"}]}} - {:output-format nil})] - (is (= (str "ID TITLE\n" - "101 Alpha\n" - "102 Note line 1\n" - " Note line 2\n" - "103 Taggy\n" - "104 Prop\n" - "Count: 4") - result)))) - +(deftest test-human-output-show (testing "show renders text payloads directly" (let [result (format/format-result {:status :ok :command :show @@ -231,14 +198,4 @@ {:output-format nil})] (is (= (str "Error (missing-graph): graph name is required\n" "Hint: Use --repo ") - result)))) - - (testing "missing search text hints use positional argument" - (let [result (format/format-result {:status :error - :command :search - :error {:code :missing-search-text - :message "search text is required"}} - {:output-format nil})] - (is (= (str "Error (missing-search-text): search text is required\n" - "Hint: Provide search text as a positional argument") result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index bc5c60fd00..6f02d1fff8 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -86,7 +86,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-list-add-search-show-remove +(deftest test-cli-list-add-show-remove (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -103,8 +103,6 @@ add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - search-result (run-cli ["--repo" "content-graph" "search" "t"] data-dir cfg-path) - search-payload (parse-json-output search-result) show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) show-payload (parse-json-output show-result) remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--page" "TestPage"] data-dir cfg-path) @@ -120,13 +118,6 @@ (is (vector? (get-in list-tag-payload [:data :items]))) (is (= "ok" (:status list-property-payload))) (is (vector? (get-in list-property-payload [:data :items]))) - (is (= "ok" (:status search-payload))) - (is (vector? (get-in search-payload [:data :results]))) - (let [types (set (map :type (get-in search-payload [:data :results])))] - (is (contains? types "page")) - (is (contains? types "block")) - (is (contains? types "tag")) - (is (contains? types "property"))) (is (= "ok" (:status show-payload))) (is (contains? (get-in show-payload [:data :root]) :uuid)) (is (= "ok" (:status remove-page-payload))) @@ -203,16 +194,15 @@ query-result (run-cli ["--repo" "task-query-graph" "query" "--name" "task-search" - "--inputs" "[:logseq.property/status.doing]"] + "--inputs" "[\"doing\"]"] data-dir cfg-path) query-payload (parse-json-output query-result) query-nil-result (run-cli ["--repo" "task-query-graph" "query" "--name" "task-search" - "--inputs" "[:logseq.property/status.doing nil 1]"] + "--inputs" "[\"doing\" nil 1]"] data-dir cfg-path) query-nil-payload (parse-json-output query-nil-result) - _ (prn :xxxx query-payload) result (get-in query-payload [:data :result]) nil-result (get-in query-nil-payload [:data :result]) stop-result (run-cli ["server" "stop" "--repo" "task-query-graph"] data-dir cfg-path) @@ -237,7 +227,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-search-resolve-nested-uuid-refs +(deftest test-cli-show-resolve-nested-uuid-refs (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -260,19 +250,12 @@ show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) show-outer-payload (parse-json-output show-outer) outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") - search-result (run-cli ["--repo" "nested-refs" "search" "Outer"] data-dir cfg-path) - search-payload (parse-json-output search-result) - search-item (some (fn [item] - (when (and (string? (:content item)) - (string/includes? (:content item) "Outer")) - item)) - (get-in search-payload [:data :results])) stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (some? inner-uuid)) (is (some? middle-uuid)) (is (some? outer-node)) - (is (= "Outer [[See [[Inner]]]]" (:content search-item))) + (is (= "Outer [[See [[Inner]]]]" (node-title outer-node))) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] @@ -527,7 +510,7 @@ output (:output show-text-result) idx-one (string/index-of output "Multi show one") idx-two (string/index-of output "Multi show two") - idx-delim (string/index-of output "-----") + idx-delim (string/index-of output "================================================================") show-json-result (run-cli ["--repo" "show-multi-id-graph" "show" "--id" ids-edn "--format" "json"] From 2b1d9a5ed9076ec931ddd31cee266004e7e12420 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 24 Jan 2026 16:44:41 +0800 Subject: [PATCH 040/375] 015-logseq-cli-db-worker-node-housekeeping.md --- docs/agent-guide/001-logseq-cli.md | 2 +- .../agent-guide/002-logseq-cli-subcommands.md | 3 +- ...-logseq-cli-db-worker-node-housekeeping.md | 64 +++++++++++ .../task--db-worker-nodejs-compatible.md | 2 - docs/cli/logseq-cli.md | 16 ++- shadow-cljs.edn | 5 +- src/dev-cljs/shadow/hooks.clj | 26 +++++ src/main/frontend/worker/db_worker_node.cljs | 105 +++++++----------- src/main/logseq/cli/command/core.cljs | 5 +- src/main/logseq/cli/commands.cljs | 4 +- src/main/logseq/cli/config.cljs | 11 +- src/main/logseq/cli/main.cljs | 5 + src/main/logseq/cli/transport.cljs | 53 +++------ src/main/logseq/cli/version.cljs | 10 ++ .../frontend/worker/db_worker_node_test.cljs | 14 +++ src/test/logseq/cli/commands_test.cljs | 2 + src/test/logseq/cli/config_test.cljs | 28 +++-- src/test/logseq/cli/main_test.cljs | 16 +++ src/test/logseq/cli/transport_test.cljs | 69 ++++++++---- 19 files changed, 289 insertions(+), 151 deletions(-) create mode 100644 docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md create mode 100644 src/main/logseq/cli/version.cljs create mode 100644 src/test/logseq/cli/main_test.cljs diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md index 287a63994b..ae40f8e606 100644 --- a/docs/agent-guide/001-logseq-cli.md +++ b/docs/agent-guide/001-logseq-cli.md @@ -19,7 +19,7 @@ The CLI should provide a stable interface for scripting and troubleshooting, and I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple graph/content request. I will add unit tests for command parsing, configuration precedence, and error formatting. -I will add unit tests for the client transport layer to ensure timeouts and retries behave correctly. +I will add unit tests for the client transport layer to ensure timeouts behave correctly. I will add unit tests for new graph/content commands (parsing, validation, and request mapping). I will add integration tests for graph lifecycle commands and content commands against a real db-worker-node. I will follow @test-driven-development for all behavior changes. diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index 836a34ab16..eb9b1fbbe8 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -55,11 +55,10 @@ Global options apply to all subcommands and are parsed before subcommand options | Option | Purpose | Notes | | --- | --- | --- | | --help | Show help | Available at top level and per subcommand. | +| --version | Show version | Prints build time and revision. | | --config PATH | Config file path | Defaults to ~/.logseq/cli.edn. | -| --auth-token TOKEN | Auth token | Sent as header. | | --repo REPO | Graph name | Used as current repo. | | --timeout-ms MS | Request timeout | Integer milliseconds. | -| --retries N | Retry count | Integer count. | | --output FORMAT | Output format | One of human, json, edn. | Each subcommand uses a nested path and its own options. diff --git a/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md b/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md new file mode 100644 index 0000000000..8efa52b043 --- /dev/null +++ b/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md @@ -0,0 +1,64 @@ +# Logseq CLI and db-worker-node Housekeeping Implementation Plan + +Goal: Remove the --retries and --auth-token options from logseq-cli and db-worker-node, and add a --version option that prints build time and commit. + +Architecture: Keep option parsing centralized in logseq.cli.command.core and frontend.worker.db-worker-node, and move build metadata into a dedicated ClojureScript namespace injected via shadow-cljs closure defines. +Architecture: Ensure logseq-cli prints version info without needing a running db-worker-node, and db-worker-node no longer gates endpoints on auth tokens. + +Tech Stack: ClojureScript, babashka.cli, shadow-cljs, Node.js. + +Related: Relates to docs/agent-guide/001-logseq-cli.md. +Related: Relates to docs/agent-guide/002-logseq-cli-subcommands.md. +Related: Relates to docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Problem statement + +The current logseq-cli and db-worker-node expose --retries and --auth-token options that are no longer desired, and the CLI lacks a version command that prints build time and commit. +The cleanup should remove these options without compatibility shims and introduce a clear version output backed by build metadata. + +## Testing Plan + +I will add a unit test for logseq-cli parsing that asserts --version short-circuits command execution and prints build metadata fields. +I will update the config and transport tests to remove retries and auth token expectations while still validating timeout behavior. +I will add a db-worker-node CLI test that verifies the help output no longer mentions --auth-token and that args parsing ignores the removed flag. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +1. Review current option definitions and call sites for retries and auth-token in src/main/logseq/cli/command/core.cljs, src/main/logseq/cli/config.cljs, src/main/logseq/cli/transport.cljs, and src/main/frontend/worker/db_worker_node.cljs. +2. Update src/main/logseq/cli/command/core.cljs to remove :auth-token and :retries from the global spec and option summary output. +3. Update src/main/logseq/cli/config.cljs to remove env parsing and defaults for auth-token and retries, and to stop persisting those keys in config files. +4. Update src/main/logseq/cli/transport.cljs to drop auth header handling and retry loops, and adjust invoke to pass only method, url, body, and timeout values. +5. Update any logseq-cli action builders or server helpers that still read :auth-token or :retries from config, and delete those plumbing paths if present. +6. Update src/main/frontend/worker/db_worker_node.cljs to remove --auth-token parsing, remove authorization checks on endpoints, and delete auth-token from daemon options and help output. +7. Add a new namespace such as src/main/logseq/cli/version.cljs that defines BUILD_TIME and REVISION via goog-define with safe defaults, and exposes a formatter for the CLI. +8. Add a global --version flag in logseq-cli by extending the global spec and by adding a short-circuit path in src/main/logseq/cli/main.cljs or src/main/logseq/cli/commands.cljs that prints build time and commit without requiring a command. +9. Add closure defines for the CLI build in shadow-cljs.edn under :logseq-cli and for the node test build under :test so tests can assert deterministic build metadata values. +10. Update build entry points that compile logseq-cli, such as package.json scripts or any CI workflow that calls clojure -M:cljs compile logseq-cli, to export LOGSEQ_BUILD_TIME and LOGSEQ_REVISION for the defines. +11. Update docs/cli/logseq-cli.md to remove auth-token and retries from the configuration list and to document --version output format with build time and commit. +12. Scan for remaining user-facing mentions of --auth-token or --retries in docs and README files, and update or remove them where appropriate. +13. Run unit tests for CLI and db-worker-node using bb dev:test for the modified namespaces, and run bb dev:lint-and-test if time allows. +14. Follow @prompts/review.md and @skills/test-driven-development throughout implementation and verification. + +## Testing Details + +The CLI tests will assert that --version returns a non-empty output containing build time and commit keys and that it exits successfully without requiring a subcommand. +The transport tests will still cover timeout behavior but will no longer assert retries behavior or auth header inclusion. +The db-worker-node tests will validate updated help output and ensure that argument parsing still recognizes required flags after removing --auth-token. + +## Implementation Details + +- Remove :auth-token and :retries from global CLI option specs and summaries in src/main/logseq/cli/command/core.cljs. +- Remove env parsing and defaults for auth-token and retries in src/main/logseq/cli/config.cljs. +- Simplify HTTP request and invoke logic in src/main/logseq/cli/transport.cljs to remove retries and auth headers. +- Remove auth-token CLI parsing and authorization gating in src/main/frontend/worker/db_worker_node.cljs. +- Add build metadata defines in a new CLI version namespace and wire --version output through logseq-cli entrypoints. +- Add closure defines for LOGSEQ_BUILD_TIME and LOGSEQ_REVISION in shadow-cljs.edn for :logseq-cli and :test builds. +- Update scripts or CI to populate LOGSEQ_BUILD_TIME and LOGSEQ_REVISION at compile time. +- Update docs/cli/logseq-cli.md and any other user-facing documentation to reflect the new option set. + +## Question + +Answer: remove all auth support entirely, including env vars and header checks. + +--- diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index bef1870ebb..499fc04c0f 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -133,7 +133,6 @@ The db-worker should be runnable as a standalone process for Node.js environment - `--repo` (optional: auto-open a repo on boot) - `--rtc-ws-url` (optional) - `--log-level` (default `info`) - - `--auth-token` (optional; bearer token for HTTP) ### Lifecycle 1. Initialize platform adapter (Node). @@ -166,7 +165,6 @@ Event delivery options: - Alternatively, provide `WS /v1/events` with the same payload format. ### Security -- If `--auth-token` is provided, require `Authorization: Bearer ` for all endpoints except `healthz` and `readyz`. - Bind to localhost by default. ## Notes on Compatibility Gaps diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 52fd6a6f7b..904d631415 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -5,9 +5,13 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to ## Build the CLI ```bash +LOGSEQ_BUILD_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \ +LOGSEQ_REVISION=$(git rev-parse --short HEAD) \ clojure -M:cljs compile logseq-cli ``` +If `LOGSEQ_BUILD_TIME` or `LOGSEQ_REVISION` are not provided, the CLI prints defaults in `--version` output. + ## db-worker-node lifecycle `logseq` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. @@ -16,24 +20,22 @@ clojure -M:cljs compile logseq-cli ```bash node ./dist/logseq.js graph list +``` If installed globally, run: ```bash logseq graph list ``` -``` ## Configuration Optional configuration file: `~/.logseq/cli.edn` Supported keys include: -- `:auth-token` - `:repo` - `:data-dir` - `:timeout-ms` -- `:retries` - `:output-format` (use `:json` or `:edn` for scripting) CLI flags take precedence over environment variables, which take precedence over the config file. @@ -95,6 +97,14 @@ Subcommands: Options grouping: - Help output separates **Global options** (apply to all commands) and **Command options** (command-specific flags). +Version output: +- `logseq --version` prints: + +``` +Build time: +Revision: +``` + Output formats: - Global `--output ` (also accepted per subcommand) - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 181e2622bf..2d5dc7210a 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -100,6 +100,7 @@ :logseq-cli {:target :node-script :output-to "static/logseq-cli.js" :main logseq.cli.main/main + :build-hooks [(shadow.hooks/logseq-cli-metadata-hook "--long --always --dirty")] :compiler-options {:infer-externs :auto :source-map true :externs ["datascript/externs.js" @@ -196,7 +197,9 @@ :test {:target :node-test :output-to "static/tests.js" :closure-defines {frontend.util/NODETEST true - logseq.shui.util/NODETEST true} + logseq.shui.util/NODETEST true + logseq.cli.version/BUILD_TIME "test-build-time" + logseq.cli.version/REVISION "test-revision"} :devtools {:enabled false} ;; disable :static-fns to allow for with-redefs and repl development :compiler-options {:static-fns false} diff --git a/src/dev-cljs/shadow/hooks.clj b/src/dev-cljs/shadow/hooks.clj index 60b4cd34f4..67920abde6 100644 --- a/src/dev-cljs/shadow/hooks.clj +++ b/src/dev-cljs/shadow/hooks.clj @@ -24,3 +24,29 @@ (assoc defines-in-config 'frontend.config/REVISION revision)) (assoc-in [:compiler-options :closure-defines] (assoc defines-in-options 'frontend.config/REVISION revision))))) + +(defn- env-or + [key fallback] + (or (System/getenv key) fallback)) + +(defn- iso-now + [] + (.format java.time.format.DateTimeFormatter/ISO_INSTANT (java.time.Instant/now))) + +(defn logseq-cli-metadata-hook + {:shadow.build/stage :configure} + [build-state & args] + (let [defines-in-config (get-in build-state [:shadow.build/config :closure-defines]) + defines-in-options (get-in build-state [:compiler-options :closure-defines]) + revision (env-or "LOGSEQ_REVISION" (or (exec "git" "describe" args) "dev")) + build-time (env-or "LOGSEQ_BUILD_TIME" (iso-now))] + (prn ::logseq-cli-metadata-hook {:revision revision :build-time build-time}) + (-> build-state + (assoc-in [:shadow.build/config :closure-defines] + (assoc defines-in-config + 'logseq.cli.version/REVISION revision + 'logseq.cli.version/BUILD_TIME build-time)) + (assoc-in [:compiler-options :closure-defines] + (assoc defines-in-options + 'logseq.cli.version/REVISION revision + 'logseq.cli.version/BUILD_TIME build-time))))) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index cf9233450f..36775f6121 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -3,12 +3,10 @@ (:require ["fs" :as fs] ["http" :as http] ["path" :as node-path] - [clojure.string :as string] [frontend.worker.db-core :as db-core] [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.platform.node :as platform-node] [frontend.worker.state :as worker-state] - [goog.object :as gobj] [lambdaisland.glogi :as log] [logseq.db :as ldb] [promesa.core :as p])) @@ -39,13 +37,6 @@ (resolve (.toString buf "utf8"))))) (.on req "error" reject))))) -(defn- authorized? - [^js req auth-token] - (if (string/blank? auth-token) - true - (let [auth (gobj/get (.-headers req) "authorization")] - (= auth (str "Bearer " auth-token))))) - (defn- parse-args [argv] (loop [args (vec (drop 2 argv)) @@ -58,7 +49,6 @@ "--repo" (recur remaining (assoc opts :repo value)) "--rtc-ws-url" (recur remaining (assoc opts :rtc-ws-url value)) "--log-level" (recur remaining (assoc opts :log-level value)) - "--auth-token" (recur remaining (assoc opts :auth-token value)) "--help" (recur remaining (assoc opts :help? true)) (recur remaining opts)))))) @@ -171,7 +161,7 @@ {:method qkw}))))) (defn- make-server - [proxy {:keys [auth-token bound-repo stop-fn]}] + [proxy {:keys [bound-repo stop-fn]}] (http/createServer (fn [^js req ^js res] (let [url (.-url req) @@ -186,54 +176,48 @@ (send-text! res 503 "not-ready")) (= url "/v1/events") - (if (authorized? req auth-token) - (sse-handler req res) - (send-text! res 401 "unauthorized")) + (sse-handler req res) (= url "/v1/invoke") - (if (authorized? req auth-token) - (if (= method "POST") - (-> (p/let [body (clj payload :keywordize-keys true) - method-kw (normalize-method-kw method) - method-str (normalize-method-str method) - direct-pass? (boolean directPass) - args' (if direct-pass? - args - (or argsTransit args)) - args-for-validation (if direct-pass? - args' - (if (string? args') - (ldb/read-transit-str args') - args'))] - (if-let [{:keys [status error]} (repo-error method-kw args-for-validation bound-repo)] - (send-json! res status {:ok false :error error}) - (p/let [result ( (p/let [body (clj payload :keywordize-keys true) + method-kw (normalize-method-kw method) + method-str (normalize-method-str method) + direct-pass? (boolean directPass) + args' (if direct-pass? + args + (or argsTransit args)) + args-for-validation (if direct-pass? + args' + (if (string? args') + (ldb/read-transit-str args') + args'))] + (if-let [{:keys [status error]} (repo-error method-kw args-for-validation bound-repo)] + (send-json! res status {:ok false :error error}) + (p/let [result ( (required)") (println " --rtc-ws-url (optional)") (println " --log-level (default info)") - (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)") - (println " --auth-token (optional)")) + (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)")) (defn- pad2 [value] @@ -312,7 +295,7 @@ file-path)) (defn start-daemon! - [{:keys [data-dir repo rtc-ws-url auth-token log-level]}] + [{:keys [data-dir repo rtc-ws-url log-level]}] (let [host "127.0.0.1" port 0] (if-not (seq repo) @@ -336,8 +319,7 @@ method-str (normalize-method-str method-kw)] (result summary data) :else - (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args)))))))))))) + (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args))))))))))))) ;; Repo/graph helpers live in logseq.cli.command.core. diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 4293ffebb5..2f3ddf2e99 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -45,7 +45,9 @@ [{:keys [config-path]} updates] (let [path (or config-path (default-config-path)) current (or (read-config-file path) {}) - next (merge current updates)] + filtered-current (dissoc current :auth-token :retries) + filtered-updates (dissoc updates :auth-token :retries) + next (merge filtered-current filtered-updates)] (ensure-config-dir! path) (.writeFileSync fs path (pr-str next)) next)) @@ -54,9 +56,6 @@ [] (let [env (.-env js/process)] (cond-> {} - (seq (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) - (assoc :auth-token (gobj/get env "LOGSEQ_DB_WORKER_AUTH_TOKEN")) - (seq (gobj/get env "LOGSEQ_CLI_REPO")) (assoc :repo (gobj/get env "LOGSEQ_CLI_REPO")) @@ -66,9 +65,6 @@ (seq (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS")) (assoc :timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS"))) - (seq (gobj/get env "LOGSEQ_CLI_RETRIES")) - (assoc :retries (parse-int (gobj/get env "LOGSEQ_CLI_RETRIES"))) - (seq (gobj/get env "LOGSEQ_CLI_OUTPUT")) (assoc :output-format (parse-output-format (gobj/get env "LOGSEQ_CLI_OUTPUT"))) @@ -78,7 +74,6 @@ (defn resolve-config [opts] (let [defaults {:timeout-ms 10000 - :retries 0 :output-format nil :data-dir "~/.logseq/cli-graphs" :config-path (default-config-path)} diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index c7ba6a0ec5..3dfac8d76b 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -5,6 +5,7 @@ [logseq.cli.commands :as commands] [logseq.cli.config :as config] [logseq.cli.format :as format] + [logseq.cli.version :as version] [promesa.core :as p])) (defn- usage @@ -33,6 +34,10 @@ :command (:command parsed)} {})}) + (= :version (:command parsed)) + (p/resolved {:exit-code 0 + :output (version/format-version)}) + :else (let [cfg (config/resolve-config (:options parsed)) action-result (commands/build-action parsed cfg)] diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index fc39591fa5..519b4937e0 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -16,11 +16,9 @@ http)) (defn- base-headers - [auth-token] - (cond-> {"Content-Type" "application/json" - "Accept" "application/json"} - (seq auth-token) - (assoc "Authorization" (str "Bearer " auth-token)))) + [] + {"Content-Type" "application/json" + "Accept" "application/json"}) (defn- = (or status 0) 500))))) - (defn request - [{:keys [method url headers body timeout-ms retries] - :or {retries 0}}] - (letfn [(attempt-request [attempt] - (-> (p/let [response (js payload))] (p/let [{:keys [body]} (request {:method "POST" :url url - :headers (base-headers auth-token) + :headers (base-headers) :body body - :timeout-ms timeout-ms - :retries retries}) + :timeout-ms timeout-ms}) {:keys [result resultTransit]} (js->clj (js/JSON.parse body) :keywordize-keys true)] (if direct-pass? result diff --git a/src/main/logseq/cli/version.cljs b/src/main/logseq/cli/version.cljs new file mode 100644 index 0000000000..19bf4fb6e4 --- /dev/null +++ b/src/main/logseq/cli/version.cljs @@ -0,0 +1,10 @@ +(ns logseq.cli.version + "Build metadata for logseq-cli.") + +(goog-define BUILD_TIME "unknown") +(goog-define REVISION "dev") + +(defn format-version + [] + (str "Build time: " BUILD_TIME "\n" + "Revision: " REVISION)) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 6ccc40a4bb..4b61784e98 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -1,6 +1,7 @@ (ns frontend.worker.db-worker-node-test (:require ["http" :as http] [cljs.test :refer [async deftest is]] + [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker-common.util :as worker-util] [frontend.worker.db-worker-node :as db-worker-node] @@ -175,6 +176,19 @@ (is (= "logseq_db_parse_args" (:repo result))) (is (= "/tmp/db-worker" (:data-dir result))))) +(deftest db-worker-node-parse-args-ignores-auth-token + (let [parse-args #'db-worker-node/parse-args + result (parse-args #js ["node" "dist/db-worker-node.js" + "--auth-token" "secret" + "--data-dir" "/tmp/db-worker"])] + (is (nil? (:auth-token result))) + (is (= "/tmp/db-worker" (:data-dir result))))) + +(deftest db-worker-node-help-omits-auth-token + (let [show-help! #'db-worker-node/show-help! + output (with-out-str (show-help!))] + (is (not (string/includes? output "--auth-token"))))) + (deftest db-worker-node-repo-error-handles-keyword-methods (let [repo-error #'db-worker-node/repo-error bound-repo "logseq_db_bound"] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 42dbf3fb9f..323e2ebbe3 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -12,6 +12,8 @@ (let [result (commands/parse-args ["--help"]) summary (:summary result)] (is (true? (:help? result))) + (is (not (string/includes? summary "--auth-token"))) + (is (not (string/includes? summary "--retries"))) (is (string/includes? summary "Graph Inspect and Edit")) (is (string/includes? summary "Graph Management")) (is (string/includes? summary "list")) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index c408e4bd8f..251de45e5e 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -23,32 +23,26 @@ (let [dir (node-helper/create-tmp-dir) cfg-path (node-path/join dir "cli.edn") _ (fs/writeFileSync cfg-path - (str "{:auth-token \"file-token\" " - ":repo \"file-repo\" " + (str "{:repo \"file-repo\" " ":data-dir \"file-data\" " ":timeout-ms 111 " - ":retries 1 " ":output-format :edn}")) - env {"LOGSEQ_DB_WORKER_AUTH_TOKEN" "env-token" - "LOGSEQ_CLI_REPO" "env-repo" + env {"LOGSEQ_CLI_REPO" "env-repo" "LOGSEQ_CLI_DATA_DIR" "env-data" "LOGSEQ_CLI_TIMEOUT_MS" "222" - "LOGSEQ_CLI_RETRIES" "2" "LOGSEQ_CLI_OUTPUT" "json"} opts {:config-path cfg-path - :auth-token "cli-token" :repo "cli-repo" :data-dir "cli-data" :timeout-ms 333 - :retries 3 :output-format :human} result (with-env env #(config/resolve-config opts))] (is (= cfg-path (:config-path result))) - (is (= "cli-token" (:auth-token result))) (is (= "cli-repo" (:repo result))) (is (= "cli-data" (:data-dir result))) (is (= 333 (:timeout-ms result))) - (is (= 3 (:retries result))) + (is (nil? (:auth-token result))) + (is (nil? (:retries result))) (is (= :human (:output-format result))))) (deftest test-env-overrides-file @@ -91,3 +85,17 @@ contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] (is (= "new" (:repo parsed))))) + +(deftest test-update-config-strips-removed-options + (let [dir (node-helper/create-tmp-dir "cli") + cfg-path (node-path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:repo \"old\"}") + _ (config/update-config! {:config-path cfg-path} + {:repo "new" + :auth-token "secret" + :retries 2}) + contents (.toString (fs/readFileSync cfg-path) "utf8") + parsed (reader/read-string contents)] + (is (= "new" (:repo parsed))) + (is (not (contains? parsed :auth-token))) + (is (not (contains? parsed :retries))))) diff --git a/src/test/logseq/cli/main_test.cljs b/src/test/logseq/cli/main_test.cljs new file mode 100644 index 0000000000..3fb098bf88 --- /dev/null +++ b/src/test/logseq/cli/main_test.cljs @@ -0,0 +1,16 @@ +(ns logseq.cli.main-test + (:require [cljs.test :refer [async deftest is]] + [clojure.string :as string] + [logseq.cli.main :as cli-main] + [promesa.core :as p])) + +(deftest test-version-output + (async done + (-> (p/let [result (cli-main/run! ["--version"] {:exit? false})] + (is (= 0 (:exit-code result))) + (is (string/includes? (:output result) "Build time: test-build-time")) + (is (string/includes? (:output result) "Revision: test-revision")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done)))))) diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index 5cb0f553a0..83297b55db 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -1,7 +1,7 @@ (ns logseq.cli.transport-test (:require [cljs.test :refer [deftest is async testing]] - [promesa.core :as p] - [logseq.cli.transport :as transport])) + [logseq.cli.transport :as transport] + [promesa.core :as p])) (def ^:private fs (js/require "fs")) (def ^:private os (js/require "os")) @@ -29,7 +29,7 @@ (resolve {:url (str "http://127.0.0.1:" port) :stop! stop!})))))))) -(deftest test-request-retries +(deftest test-request-does-not-retry (async done (let [calls (atom 0)] (-> (p/let [{:keys [url stop!]} (start-server @@ -41,15 +41,17 @@ (.end res "boom")) (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) - (.end res "ok")))))) - response (transport/request {:method "GET" - :url (str url "/retry") - :retries 1 - :timeout-ms 1000})] - (is (= 200 (:status response))) - (is (= 2 @calls)) - (p/let [_ (stop!)] - (done))) + (.end res "ok"))))))] + (p/catch + (transport/request {:method "GET" + :url (str url "/retry") + :timeout-ms 1000}) + (fn [e] + (is (= :http-error (-> (ex-data e) :code))) + (is (= 500 (-> (ex-data e) :status))))) + (is (= 1 @calls)) + (p/let [_ (stop!)] true)) + (p/then (fn [_] (done))) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -62,12 +64,11 @@ (p/catch (transport/request {:method "GET" :url (str url "/hang") - :timeout-ms 10 - :retries 0}) + :timeout-ms 10}) (fn [e] (is (= :timeout (-> (ex-data e) :code))) - (p/let [_ (stop!)] - (done))))) + (p/let [_ (stop!)] true)))) + (p/then (fn [_] (done))) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done)))))) @@ -84,13 +85,35 @@ payload (js/JSON.parse (.toString buf "utf8"))] (reset! received (js->clj payload :keywordize-keys true)) (.writeHead res 200 #js {"Content-Type" "application/json"}) - (.end res (js/JSON.stringify #js {:result "ok"})))))))) - result (transport/invoke {:base-url url} :thread-api/pull true ["repo" [:block/title]])] - (is (= "ok" result)) - (is (= "thread-api/pull" (:method @received))) - (is (= true (:directPass @received))) - (p/let [_ (stop!)] - (done))) + (.end res (js/JSON.stringify #js {:result "ok"}))))))))] + (p/let [result (transport/invoke {:base-url url} :thread-api/pull true ["repo" [:block/title]])] + (is (= "ok" result)) + (is (= "thread-api/pull" (:method @received))) + (is (= true (:directPass @received))) + (p/let [_ (stop!)] true))) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-invoke-does-not-send-auth-header + (async done + (let [auth-header (atom :unset)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [^js req ^js res] + (let [headers (.-headers req)] + (reset! auth-header (aget headers "authorization"))) + (.writeHead res 200 #js {"Content-Type" "application/json"}) + (.end res (js/JSON.stringify #js {:result "ok"}))))] + (p/let [result (transport/invoke {:base-url url + :auth-token "secret"} + :thread-api/pull + true + ["repo" [:block/title]])] + (is (= "ok" result)) + (is (nil? @auth-header)) + (p/let [_ (stop!)] true))) + (p/then (fn [_] (done))) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) From 9edbe1be3fca6b23748272fc04da6c073209d44a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 24 Jan 2026 20:20:53 +0800 Subject: [PATCH 041/375] 016-recent-updated-query.md --- docs/agent-guide/016-recent-updated-query.md | 89 ++++++++++ src/main/logseq/cli/command/query.cljs | 45 ++++- src/main/logseq/cli/command/show.cljs | 26 +++ src/test/logseq/cli/integration_test.cljs | 163 +++++++++++++++++++ 4 files changed, 315 insertions(+), 8 deletions(-) create mode 100644 docs/agent-guide/016-recent-updated-query.md diff --git a/docs/agent-guide/016-recent-updated-query.md b/docs/agent-guide/016-recent-updated-query.md new file mode 100644 index 0000000000..427625ae51 --- /dev/null +++ b/docs/agent-guide/016-recent-updated-query.md @@ -0,0 +1,89 @@ +# Recent Updated Query Implementation Plan + +Goal: Add a built-in CLI query named recent-updated that filters entities updated within a configurable recent-days window. + +Architecture: The logseq CLI will expose a new built-in query spec in the query command module and pass inputs to db-worker-node via thread-api/q with no db-worker-node changes. +The query will rely on :block/updated-at and a now-ms input to compute a rolling cutoff in milliseconds. + +Tech Stack: ClojureScript, Datascript queries, logseq CLI, db-worker-node thread-api. + +Related: Builds on 015-logseq-cli-db-worker-node-housekeeping.md. + +## Problem statement + +Users need a built-in CLI query to list recently updated content without crafting ad hoc datalog each time. +The new recent-updated query should accept a recent-days parameter and integrate with existing logseq CLI query listing and execution paths. +The implementation must align with db-worker-node query execution and hide internal inputs from query list output. + +## Testing Plan + +I will add an integration test that creates a temporary graph, adds blocks, and runs the recent-updated query with a small recent-days window, asserting only recently updated entities are returned. +I will add a second integration test case in the same test to cover recent-days values of nil and non-positive numbers, confirming the CLI returns a clear invalid input error. +I will add a CLI query list assertion that recent-updated appears with the correct input spec and defaults, excluding the internal now-ms input from list output. +I will add a CLI show -id test to ensure duplicate trees are filtered when ids include parent/child relationships. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation Plan + +1. Write the failing integration test for recent-updated in `src/test/logseq/cli/integration_test.cljs` that uses `run-cli` to create a graph, insert blocks, and query by name, asserting the result is a vector of db/id values. +2. Write the failing assertion that `query list` returns a recent-updated entry with an inputs vector that only shows recent-days. +3. Run the single integration test to confirm failures are due to missing built-in query behavior. +4. Add a new built-in query spec entry in `src/main/logseq/cli/command/query.cljs` with name "recent-updated" and inputs [{:name "?recent-days" :default 0} {:name "?now-ms" :default :now-ms}]. +5. Define the query to find entities that have :block/updated-at and apply an or-join to bypass filtering when recent-days is nil or <= 0. +6. Ensure the query result shape matches other built-ins and returns a vector of entity ids. +7. Run the integration test again and confirm it now passes. +8. Refactor the built-in query spec and test setup for clarity if necessary while keeping behavior unchanged. +9. Run the CLI integration test suite or the focused test case to ensure no regressions. +10. Update the show -id output path to drop contained trees when multiple ids overlap. + +## Edge Cases and Considerations + +- recent-days must be greater than 0, and nil or non-positive values should be rejected with a clear CLI error. +- Blocks without :block/updated-at should be excluded consistently and not cause query errors. +- The query should return both pages and blocks. +- The query should exclude built-in entities by default. + +## Testing Details + +The tests will exercise the CLI path end to end by invoking db-worker-node and asserting the query results contain only the expected entities based on updated-at timestamps. +The tests will validate both the query list metadata and the data returned from the query invocation. + +## Implementation Details + +- Add a new entry to built-in-query-specs in `src/main/logseq/cli/command/query.cljs`. +- Use :block/updated-at and a computed cutoff (now-ms minus recent-days in milliseconds) inside the datalog query, and enforce recent-days > 0 in CLI input validation. +- Keep the inputs list consistent with existing built-in queries and rely on hide-internal-inputs to remove ?now-ms from list output. +- Reuse the same optional input handling logic as task-search for recent-days defaults. +- Ensure the query name is normalized and discoverable via `logseq query list`. +- Return a vector of db/id values with no ordering guarantees, matching task-search behavior. +- Keep db-worker-node unchanged since it already supports arbitrary datascript queries via thread-api/q. + +## Question + +recent-updated must return both pages and blocks, and exclude built-in entities by default. +recent-days must be greater than 0, and nil or non-positive values should be treated as invalid input. +Results must be returned as an unordered vector of db/id values, and users should use logseq show to view content and apply sorting. + +## Show -id duplicate filtering note + +When `logseq show -id` is given multiple ids, the output can include duplicate trees if some ids are children of others. +Filter out smaller (contained) trees from the result, so only the largest parent tree is kept. + +Example: + +```text +logseq show -id [7830,7831,7832,7833] +7830 Jan 23rd, 2026 #Journal +7831 ├── asdfasdfasdf +7832 │ └── yyyy +7833 └── [[xxxx]] yyy +================================================================ +7831 asdfasdfasdf +7832 └── yyyy +================================================================ +7832 yyyy +================================================================ +7833 [[xxxx]] yyy +``` + +--- diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index 97a601314c..ebf61d89eb 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -56,7 +56,20 @@ [(identity 0) ?days-ago]) (and [(* ?recent-days 86400000) ?recent-days-ms] [(- ?now-ms ?recent-days-ms) ?days-ago] - [(>= ?updated-at ?days-ago)]))]}}) + [(>= ?updated-at ?days-ago)]))]} + + "recent-updated" + {:doc "Find entities updated within recent-days." + :inputs [{:name "recent-days"} + {:name "?now-ms" :default :now-ms}] + :query '[:find [?e ...] + :in $ ?recent-days ?now-ms + :where + [?e :block/updated-at ?updated-at] + [(missing? $ ?e :logseq.property/built-in?)] + [(* ?recent-days 86400000) ?recent-days-ms] + [(- ?now-ms ?recent-days-ms) ?days-ago] + [(>= ?updated-at ?days-ago)]]}}) (defn- parse-edn [label value] @@ -169,6 +182,19 @@ (assoc (vec inputs) 0 normalized)) inputs)) +(defn- validate-recent-updated-inputs + [entry inputs] + (if (and entry + (= "recent-updated" (:name entry)) + (= :built-in (:source entry))) + (let [recent-days (first inputs)] + (if (and (integer? recent-days) (pos? recent-days)) + {:ok? true :value inputs} + {:ok? false + :error {:code :invalid-options + :message "recent-days must be a positive integer"}})) + {:ok? true :value inputs})) + (defn build-action [options repo config] (if-not (seq repo) @@ -220,13 +246,16 @@ (:entry query-result) (or (:value named-inputs) (:value inputs-result) - []))] - {:ok? true - :action {:type :query - :repo repo - :graph (core/repo->graph repo) - :query (:value query-result) - :inputs inputs}}))))))))) + [])) + validated (validate-recent-updated-inputs (:entry query-result) inputs)] + (if-not (:ok? validated) + validated + {:ok? true + :action {:type :query + :repo repo + :graph (core/repo->graph repo) + :query (:value query-result) + :inputs (:value validated)}})))))))))) (defn build-list-action [_options _repo] diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 62cb0744aa..240567239e 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -549,6 +549,19 @@ {:id id :error error-map})) +(defn- collect-tree-ids + [root] + (letfn [(walk [node acc] + (let [acc (cond-> acc + (:db/id node) (conj (:db/id node)))] + (reduce (fn [memo child] + (walk child memo)) + acc + (or (:block/children node) []))))] + (if root + (walk root #{}) + #{}))) + (defn execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -567,6 +580,19 @@ :id id :error error})))) ids)) + ok-results (filter :ok? results) + id->tree-ids (into {} + (map (fn [{:keys [id tree]}] + [id (collect-tree-ids (:root tree))])) + ok-results) + contained? (fn [id] + (some (fn [[other-id tree-ids]] + (and (not= other-id id) + (contains? tree-ids id))) + id->tree-ids)) + results (vec (remove (fn [{:keys [ok? id]}] + (and ok? (contained? id))) + results)) payload (case format "edn" {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 6f02d1fff8..5657cd3024 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -227,6 +227,104 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-query-recent-updated + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-recent-updated")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "recent-updated-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "recent-updated-graph" "add" "page" "--page" "RecentPage"] data-dir cfg-path) + _ (run-cli ["--repo" "recent-updated-graph" "add" "block" + "--target-page-name" "RecentPage" + "--content" "Recent block"] + data-dir cfg-path) + _ (p/delay 100) + list-page-result (run-cli ["--repo" "recent-updated-graph" "list" "page" "--expand"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-item (some (fn [item] + (when (= "RecentPage" (or (:block/title item) (:title item))) + item)) + (get-in list-page-payload [:data :items])) + page-id (or (:db/id page-item) (:id page-item)) + show-result (run-cli ["--repo" "recent-updated-graph" + "show" + "--page-name" "RecentPage" + "--format" "json"] + data-dir cfg-path) + show-payload (parse-json-output show-result) + show-root (get-in show-payload [:data :root]) + block-node (find-block-by-title show-root "Recent block") + block-id (or (:db/id block-node) (:id block-node)) + list-result (run-cli ["query" "list"] data-dir cfg-path) + list-payload (parse-json-output list-result) + recent-entry (some (fn [entry] + (when (= "recent-updated" (:name entry)) entry)) + (get-in list-payload [:data :queries])) + now-ms (js/Date.now) + query-result (run-cli ["--repo" "recent-updated-graph" + "query" + "--name" "recent-updated" + "--inputs" (pr-str [1 now-ms])] + data-dir cfg-path) + query-payload (parse-json-output query-result) + result (get-in query-payload [:data :result]) + future-now-ms (+ now-ms (* 10 86400000)) + future-query-result (run-cli ["--repo" "recent-updated-graph" + "query" + "--name" "recent-updated" + "--inputs" (pr-str [1 future-now-ms])] + data-dir cfg-path) + future-query-payload (parse-json-output future-query-result) + future-result (get-in future-query-payload [:data :result]) + zero-result (run-cli ["--repo" "recent-updated-graph" + "query" + "--name" "recent-updated" + "--inputs" "[0]"] + data-dir cfg-path) + zero-payload (parse-json-output zero-result) + nil-result (run-cli ["--repo" "recent-updated-graph" + "query" + "--name" "recent-updated" + "--inputs" "[nil]"] + data-dir cfg-path) + nil-payload (parse-json-output nil-result) + neg-result (run-cli ["--repo" "recent-updated-graph" + "query" + "--name" "recent-updated" + "--inputs" "[-1]"] + data-dir cfg-path) + neg-payload (parse-json-output neg-result) + stop-result (run-cli ["server" "stop" "--repo" "recent-updated-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status list-page-payload))) + (is (some? page-id)) + (is (some? block-id)) + (is (= "ok" (:status list-payload))) + (is (= [{:name "recent-days"}] (:inputs recent-entry))) + (is (= 0 (:exit-code query-result))) + (is (= "ok" (:status query-payload))) + (is (vector? result)) + (is (contains? (set result) page-id)) + (is (contains? (set result) block-id)) + (is (= 0 (:exit-code future-query-result))) + (is (= "ok" (:status future-query-payload))) + (is (vector? future-result)) + (is (empty? future-result)) + (is (= 1 (:exit-code zero-result))) + (is (= "error" (:status zero-payload))) + (is (= "invalid-options" (get-in zero-payload [:error :code]))) + (is (= 1 (:exit-code nil-result))) + (is (= "error" (:status nil-payload))) + (is (= "invalid-options" (get-in nil-payload [:error :code]))) + (is (= 1 (:exit-code neg-result))) + (is (= "error" (:status neg-payload))) + (is (= "invalid-options" (get-in neg-payload [:error :code]))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-show-resolve-nested-uuid-refs (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] @@ -538,6 +636,71 @@ (is (= 2 (count show-data))) (is (contains? root-titles "Multi show one")) (is (contains? root-titles "Multi show two")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-show-multi-id-filters-contained + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id-contained")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "show-multi-id-contained-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "page" "--page" "ParentPage"] + data-dir cfg-path) + _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" + "--target-page-name" "ParentPage" + "--content" "Parent Block"] + data-dir cfg-path) + parent-query (run-cli ["--repo" "show-multi-id-contained-graph" "query" + "--query" "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" + "--inputs" (pr-str ["Parent Block"])] + data-dir cfg-path) + parent-payload (parse-json-output parent-query) + parent-id (get-in parent-payload [:data :result]) + show-parent (run-cli ["--repo" "show-multi-id-contained-graph" + "show" + "--page-name" "ParentPage" + "--format" "json"] + data-dir cfg-path) + show-parent-payload (parse-json-output show-parent) + parent-node (find-block-by-title (get-in show-parent-payload [:data :root]) "Parent Block") + parent-uuid (node-uuid parent-node) + _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" + "--target-uuid" (str parent-uuid) + "--content" "Child Block"] + data-dir cfg-path) + _ (p/delay 100) + show-children (run-cli ["--repo" "show-multi-id-contained-graph" + "show" + "--page-name" "ParentPage" + "--format" "json"] + data-dir cfg-path) + show-children-payload (parse-json-output show-children) + child-node (find-block-by-title (get-in show-children-payload [:data :root]) "Child Block") + child-id (or (:db/id child-node) (:id child-node)) + ids-edn (str "[" parent-id " " child-id "]") + show-json-result (run-cli ["--repo" "show-multi-id-contained-graph" "show" + "--id" ids-edn + "--format" "json"] + data-dir cfg-path) + show-json-payload (parse-json-output show-json-result) + show-data (:data show-json-payload) + root-titles (set (map (comp node-title :root) show-data)) + stop-result (run-cli ["server" "stop" "--repo" "show-multi-id-contained-graph"] + data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code parent-query))) + (is (some? parent-id)) + (is (some? parent-uuid)) + (is (some? child-id)) + (is (= 0 (:exit-code show-json-result))) + (is (= "ok" (:status show-json-payload))) + (is (vector? show-data)) + (is (= 1 (count show-data))) + (is (contains? root-titles "Parent Block")) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From 83bb0b2da62a6c769ff165dfa4a373201cf73213 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 25 Jan 2026 16:12:28 +0800 Subject: [PATCH 042/375] 017-logseq-cli-db-worker-node-housekeeping-2.md --- ...ogseq-cli-db-worker-node-housekeeping-2.md | 97 ++++++++++++ docs/cli/logseq-cli.md | 20 ++- src/main/logseq/cli/command/id.cljs | 49 ++++++ src/main/logseq/cli/command/move.cljs | 18 +-- src/main/logseq/cli/command/remove.cljs | 142 +++++++++++++----- src/main/logseq/cli/command/show.cljs | 88 +++-------- src/main/logseq/cli/commands.cljs | 35 +++-- src/main/logseq/cli/format.cljs | 18 +-- src/main/logseq/cli/main.cljs | 2 +- src/test/logseq/cli/commands_test.cljs | 100 +++++++++--- 10 files changed, 393 insertions(+), 176 deletions(-) create mode 100644 docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md create mode 100644 src/main/logseq/cli/command/id.cljs diff --git a/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md new file mode 100644 index 0000000000..c77feba71b --- /dev/null +++ b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md @@ -0,0 +1,97 @@ +# Logseq CLI Housekeeping 2 Implementation Plan + +Goal: Simplify CLI options for show, move, and remove while keeping db-worker-node behavior unchanged. + +Architecture: The changes are limited to CLI option parsing, action building, and output formatting in the CLI layer. +The db-worker-node API calls remain the same, but we will verify expected input shapes for delete and move operations. +We will centralize shared id parsing so show and remove stay consistent. + +Tech Stack: ClojureScript, babashka.cli, promesa, Logseq CLI, db-worker-node thread-api. + +Related: Builds on docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md and relates to docs/agent-guide/014-logseq-cli-show-multi-id.md. + +## Testing Plan + +I will follow @test-driven-development by writing failing tests for each new CLI behavior before changing implementation. +I will add unit tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs for parsing and validation of the new remove options and renamed flags. +I will add unit tests for show option parsing to accept --page and reject --page-name and --format, while preserving global --output handling. +I will add unit tests for move option parsing to accept --target-page and reject --target-page-name. +I will add unit tests for remove parsing to accept --id, --uuid, and --page, and to reject multiple selectors or missing target. +I will add unit tests for remove parsing to accept multi-id vectors and to reject invalid or empty vectors. +I will run the CLI test namespace and confirm the new tests fail before any implementation changes. +I will rerun the CLI tests after each behavioral change to confirm they pass. + +Command to run tests is shown below. + +```bash +bb dev:test -v logseq.cli.commands-test +``` + +Expected test output is described below. + +The output should include zero failures and zero errors for logseq.cli.commands-test. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Problem statement + +The CLI currently exposes overlapping option names and per-command format flags that conflict with the global output option. +The remove command is split into remove block and remove page, which makes scripting awkward and inconsistent with show and move selectors. +The move and show commands use page-name flags that should be renamed for clarity and consistency across commands. +We need a small, coordinated change that updates CLI parsing, action building, and documentation without changing db-worker-node APIs. + +## Plan + +1. Review current CLI option specs and validation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs, and /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to confirm existing behavior and data shapes. +2. Review db-worker-node call sites for delete and move operations by searching for :delete-blocks and :delete-page usages to confirm expected argument shapes in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs and related call sites. +3. Add failing parsing and validation tests for the unified remove command and renamed flags in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. +4. Add failing tests that assert legacy flags and subcommands are rejected in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. +5. Run the CLI test namespace and record the failing cases using the test command in the testing plan. +6. Extract the show id parsing logic into a shared helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/id.cljs or a similar shared namespace, and update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use it. +7. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs to define a single remove spec with --id, --uuid, and --page, and to build actions based on which selector is present. +8. Update remove execution to support single and multiple id deletion while preserving page deletion behavior, and ensure returned data matches existing format expectations. +9. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to rename --page-name to --page and remove the --format option and related validation. +10. Update show execution to use the resolved output format from config instead of a command-specific flag, while preserving human-readable output for the default format. +11. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to rename --target-page-name to --target-page and adjust validation and target resolution accordingly. +12. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs to reflect the new remove command, show selector, and move target flag in validation, action building, and help routing. +13. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs if the new remove action type changes the command name used for human formatting. +14. Update CLI usage text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/main.cljs to remove the remove block and remove page references. +15. Update CLI documentation and examples in /Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md to use --page, --target-page, and the unified remove command without --format. +16. Rerun the CLI test namespace and confirm all tests pass. +17. Run bb dev:lint-and-test if time permits and confirm there are no regressions in other CLI tests. +18. Review the changes against @prompts/review.md and ensure the updated flags are reflected everywhere in docs and tests. + +## Edge cases + +Removing blocks with a vector of ids that contains non-integers should produce a clear invalid options error. +Removing blocks with an empty id vector should produce a clear invalid options error. +Removing with both --id and --uuid should fail validation with a single-selector error. +Removing with both --page and a block selector should fail validation with a single-selector error. +Showing with --page should still reject invalid level values and missing targets. +Showing with --output json or edn should return structured data rather than human text for both single and multi-id cases. +Moving with --target-page should still reject --pos sibling. +Legacy flags like --page-name, --target-page-name, and show --format should be rejected by the parser with invalid-options errors. + +## Testing Details + +I will focus tests on CLI behavior by asserting parse-args results, invalid option errors, and build-action normalization for ids and selectors. +I will avoid mock-only tests and instead assert actual validation behavior and action shapes that drive CLI execution. + +## Implementation Details + +- Consolidate remove command parsing around a single spec and selector validation. +- Share id parsing between show and remove to keep behavior identical. +- Keep db-worker-node API calls unchanged and only adjust CLI argument shapes. +- Use config output-format in show execution to decide between human text and structured data. +- Remove show-specific format validation and option from the CLI help output. +- Rename show and move page flags and update all associated validation logic. +- Update CLI documentation and examples to match the new flags and remove subcommands. +- Update CLI help routing so logseq remove behaves like a command, not a group. + +## Decisions + +- Remove `--page-name` and `--target-page-name` entirely (no aliases or warnings). +- For remove with multiple ids, continue with best-effort deletion (do not fail on the first missing id). +- For remove ids, allow only the show-style vector and single value formats (no repeated `--id` flags). + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 904d631415..2ea1e7c08e 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -69,14 +69,13 @@ Inspect and edit commands: - `add block --blocks [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector - `add block --blocks-file [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file - `add page --page ` - create a page -- `move --id |--uuid --target-id |--target-uuid |--target-page-name [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) -- `remove block --block ` - remove a block and its children -- `remove page --page ` - remove a page and its children +- `move --id |--uuid --target-id |--target-uuid |--target-page [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) +- `remove --id |--uuid |--page ` - remove blocks (by db/id or UUID) or pages - `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) - `query --query [--inputs ]` - run a Datascript query against the graph -- `show --page-name [--format text|json|edn] [--level ]` - show page tree -- `show --uuid [--format text|json|edn] [--level ]` - show block tree -- `show --id [--format text|json|edn] [--level ]` - show block tree by db/id +- `show --page [--level ]` - show page tree +- `show --uuid [--level ]` - show block tree +- `show --id [--level ]` - show block tree by db/id Help output: @@ -88,8 +87,7 @@ Subcommands: add block [options] Add blocks add page [options] Create page move [options] Move block - remove block [options] Remove block - remove page [options] Remove page + remove [options] Remove block or page search [options] Search graph show [options] Show tree ``` @@ -106,7 +104,7 @@ Revision: ``` Output formats: -- Global `--output ` (also accepted per subcommand) +- Global `--output ` applies to all commands - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. @@ -131,8 +129,8 @@ node ./dist/logseq.js graph create --repo demo node ./dist/logseq.js graph export --type edn --output /tmp/demo.edn --repo demo node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --repo demo-import node ./dist/logseq.js add block --target-page-name TestPage --content "hello world" -node ./dist/logseq.js move --uuid --target-page-name TargetPage +node ./dist/logseq.js move --uuid --target-page TargetPage node ./dist/logseq.js search "hello" -node ./dist/logseq.js show --page-name TestPage --format json --output json +node ./dist/logseq.js show --page TestPage --output json node ./dist/logseq.js server list ``` diff --git a/src/main/logseq/cli/command/id.cljs b/src/main/logseq/cli/command/id.cljs new file mode 100644 index 0000000000..164ad37fec --- /dev/null +++ b/src/main/logseq/cli/command/id.cljs @@ -0,0 +1,49 @@ +(ns logseq.cli.command.id + "Shared id parsing helpers for CLI commands." + (:require [clojure.string :as string] + [logseq.common.util :as common-util])) + +(defn valid-id? + [value] + (and (number? value) (integer? value))) + +(defn parse-id-option + [value] + (let [invalid (fn [message] + {:ok? false :message message})] + (cond + (nil? value) + {:ok? true :value nil :multi? false} + + (vector? value) + (cond + (empty? value) (invalid "id vector must contain at least one id") + (every? valid-id? value) {:ok? true :value (vec value) :multi? true} + :else (invalid "id vector must contain only integers")) + + (valid-id? value) + {:ok? true :value [value] :multi? false} + + (string? value) + (let [text (string/trim value)] + (cond + (string/blank? text) + (invalid "id is required") + + (string/starts-with? text "[") + (let [parsed (common-util/safe-read-string {:log-error? false} text)] + (cond + (nil? parsed) (invalid "invalid id edn") + (not (vector? parsed)) (invalid "id must be a vector") + (empty? parsed) (invalid "id vector must contain at least one id") + (every? valid-id? parsed) {:ok? true :value (vec parsed) :multi? true} + :else (invalid "id vector must contain only integers"))) + + (re-matches #"-?\\d+" text) + {:ok? true :value [(js/parseInt text 10)] :multi? false} + + :else + (invalid "id must be a number or vector of numbers"))) + + :else + (invalid "id must be a number or vector of numbers")))) diff --git a/src/main/logseq/cli/command/move.cljs b/src/main/logseq/cli/command/move.cljs index 60453ea61a..b4cba44a05 100644 --- a/src/main/logseq/cli/command/move.cljs +++ b/src/main/logseq/cli/command/move.cljs @@ -14,7 +14,7 @@ :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} - :target-page-name {:desc "Target page name"} + :target-page {:desc "Target page name"} :pos {:desc "Position (first-child, last-child, sibling)"}}) (def entries @@ -29,7 +29,7 @@ source-selectors (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) target-selectors (filter some? [(:target-id opts) (:target-uuid opts) - (some-> (:target-page-name opts) string/trim)])] + (some-> (:target-page opts) string/trim)])] (cond (and (seq pos) (not (contains? move-positions pos))) (str "invalid pos: " (:pos opts)) @@ -38,9 +38,9 @@ "only one of --id or --uuid is allowed" (> (count target-selectors) 1) - "only one of --target-id, --target-uuid, or --target-page-name is allowed" + "only one of --target-id, --target-uuid, or --target-page is allowed" - (and (= pos "sibling") (seq (some-> (:target-page-name opts) string/trim))) + (and (= pos "sibling") (seq (some-> (:target-page opts) string/trim))) "--pos sibling is only valid for block targets" :else @@ -86,7 +86,7 @@ (p/rejected (ex-info "source is required" {:code :missing-source})))) (defn- resolve-target - [config repo {:keys [target-id target-uuid target-page-name]}] + [config repo {:keys [target-id target-uuid target-page]}] (cond (some? target-id) (p/let [entity (transport/invoke config :thread-api/pull false @@ -103,10 +103,10 @@ (ensure-non-page entity "target must be a block" :invalid-target) (throw (ex-info "target block not found" {:code :target-not-found}))))) - (seq target-page-name) + (seq target-page) (p/let [entity (transport/invoke config :thread-api/pull false [repo [:db/id :block/uuid :block/name :block/title] - [:block/name target-page-name]])] + [:block/name target-page]])] (if (:db/id entity) entity (throw (ex-info "page not found" {:code :page-not-found})))) @@ -132,7 +132,7 @@ uuid (some-> (:uuid options) string/trim) target-id (:target-id options) target-uuid (some-> (:target-uuid options) string/trim) - page-name (some-> (:target-page-name options) string/trim) + page-name (some-> (:target-page options) string/trim) pos (some-> (:pos options) string/trim string/lower-case) source-label (cond (seq uuid) uuid @@ -163,7 +163,7 @@ :uuid uuid :target-id target-id :target-uuid target-uuid - :target-page-name page-name + :target-page page-name :pos (or pos "first-child") :source source-label :target target-label}})))) diff --git a/src/main/logseq/cli/command/remove.cljs b/src/main/logseq/cli/command/remove.cljs index c2ec7df163..3bbc3d82c3 100644 --- a/src/main/logseq/cli/command/remove.cljs +++ b/src/main/logseq/cli/command/remove.cljs @@ -2,32 +2,93 @@ "Remove-related CLI commands." (:require [clojure.string :as string] [logseq.cli.command.core :as core] + [logseq.cli.command.id :as id-command] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [promesa.core :as p])) -(def ^:private remove-block-spec - {:block {:desc "Block UUID"}}) - -(def ^:private remove-page-spec - {:page {:desc "Page name"}}) +(def ^:private remove-spec + {:id {:desc "Block db/id or EDN vector of ids"} + :uuid {:desc "Block UUID"} + :page {:desc "Page name"}}) (def entries - [(core/command-entry ["remove" "block"] :remove-block "Remove block" remove-block-spec) - (core/command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec)]) + [(core/command-entry ["remove"] :remove "Remove blocks or pages" remove-spec)]) + +(defn invalid-options? + [opts] + (let [id-result (id-command/parse-id-option (:id opts))] + (cond + (and (some? (:id opts)) (not (:ok? id-result))) + (:message id-result) + + :else + nil))) + +(def ^:private block-id-selector + [:db/id :block/uuid]) + +(defn- fetch-block-by-id + [config repo id] + (transport/invoke config :thread-api/pull false + [repo block-id-selector id])) + +(defn- fetch-block-by-uuid + [config repo uuid-str] + (p/let [entity (transport/invoke config :thread-api/pull false + [repo block-id-selector [:block/uuid (uuid uuid-str)]])] + (if (:db/id entity) + entity + (transport/invoke config :thread-api/pull false + [repo block-id-selector [:block/uuid uuid-str]])))) + +(defn- delete-block-ids + [config repo ids] + (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:delete-blocks [ids {}]]] {}])) + +(defn- remove-block-id + [config repo id] + (p/let [entity (fetch-block-by-id config repo id)] + (if (:db/id entity) + (delete-block-ids config repo [id]) + (throw (ex-info "block not found" {:code :block-not-found}))))) + +(defn- remove-block-ids-best-effort + [config repo ids] + (p/let [entities (p/all (map (fn [id] + (fetch-block-by-id config repo id)) + ids)) + id-entities (map vector ids entities) + existing-ids (vec (keep (fn [[id entity]] + (when (:db/id entity) id)) + id-entities)) + missing-ids (vec (keep (fn [[id entity]] + (when-not (:db/id entity) id)) + id-entities)) + result (if (seq existing-ids) + (delete-block-ids config repo existing-ids) + nil)] + {:deleted-ids existing-ids + :missing-ids missing-ids + :result result})) (defn- perform-remove - [config {:keys [repo block page]}] + [config {:keys [repo ids multi-id? uuid page]}] (cond - (seq block) - (if-not (common-util/uuid-string? block) + (and (seq ids) multi-id?) + (remove-block-ids-best-effort config repo ids) + + (seq ids) + (remove-block-id config repo (first ids)) + + (seq uuid) + (if-not (common-util/uuid-string? uuid) (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) - (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid] [:block/uuid (uuid block)]])] + (p/let [entity (fetch-block-by-uuid config repo uuid)] (if-let [id (:db/id entity)] - (transport/invoke config :thread-api/apply-outliner-ops false - [repo [[:delete-blocks [[id] {}]]] {}]) + (delete-block-ids config repo [id]) (throw (ex-info "block not found" {:code :block-not-found}))))) (seq page) @@ -41,41 +102,48 @@ :else (p/rejected (ex-info "block or page required" {:code :missing-target})))) -(defn build-remove-block-action +(defn build-action [options repo] (if-not (seq repo) {:ok? false :error {:code :missing-repo :message "repo is required for remove"}} - (let [block (some-> (:block options) string/trim)] - (if (seq block) - {:ok? true - :action {:type :remove-block - :repo repo - :block block}} + (let [id-result (id-command/parse-id-option (:id options)) + ids (:value id-result) + multi-id? (:multi? id-result) + uuid (some-> (:uuid options) string/trim) + page (some-> (:page options) string/trim) + selectors (filter some? [(:id options) uuid page])] + (cond + (empty? selectors) {:ok? false :error {:code :missing-target - :message "block is required"}})))) + :message "block or page is required"}} -(defn build-remove-page-action - [options repo] - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for remove"}} - (let [page (some-> (:page options) string/trim)] - (if (seq page) - {:ok? true - :action {:type :remove-page - :repo repo - :page page}} + (> (count selectors) 1) {:ok? false - :error {:code :missing-target - :message "page is required"}})))) + :error {:code :invalid-options + :message "only one of --id, --uuid, or --page is allowed"}} + + (and (some? (:id options)) (not (:ok? id-result))) + {:ok? false + :error {:code :invalid-options + :message (:message id-result)}} + + :else + {:ok? true + :action {:type :remove + :repo repo + :id (when (and (seq ids) (not multi-id?)) (first ids)) + :ids ids + :multi-id? multi-id? + :uuid uuid + :page page}})))) (defn execute-remove [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) result (perform-remove cfg action)] {:status :ok - :data {:result result}}))) + :data (cond-> {:result result} + (map? result) (merge (dissoc result :result)))}))) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 240567239e..a996f0ade3 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.command.show "Show-related CLI commands." (:require [clojure.string :as string] + [logseq.cli.command.id :as id-command] [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] @@ -10,73 +11,20 @@ (def ^:private show-spec {:id {:desc "Block db/id or EDN vector of ids"} :uuid {:desc "Block UUID"} - :page-name {:desc "Page name"} + :page {:desc "Page name"} :level {:desc "Limit tree depth" - :coerce :long} - :format {:desc "Output format (text, json, edn)"}}) + :coerce :long}}) (def entries [(core/command-entry ["show"] :show "Show tree" show-spec)]) -(def ^:private show-formats - #{"text" "json" "edn"}) - (def ^:private multi-id-delimiter "\n================================================================\n") -(defn- valid-id? - [value] - (and (number? value) (integer? value))) - -(defn- parse-id-option - [value] - (let [invalid (fn [message] - {:ok? false :message message})] - (cond - (nil? value) - {:ok? true :value nil :multi? false} - - (vector? value) - (cond - (empty? value) (invalid "id vector must contain at least one id") - (every? valid-id? value) {:ok? true :value (vec value) :multi? true} - :else (invalid "id vector must contain only integers")) - - (valid-id? value) - {:ok? true :value [value] :multi? false} - - (string? value) - (let [text (string/trim value)] - (cond - (string/blank? text) - (invalid "id is required") - - (string/starts-with? text "[") - (let [parsed (common-util/safe-read-string {:log-error? false} text)] - (cond - (nil? parsed) (invalid "invalid id edn") - (not (vector? parsed)) (invalid "id must be a vector") - (empty? parsed) (invalid "id vector must contain at least one id") - (every? valid-id? parsed) {:ok? true :value (vec parsed) :multi? true} - :else (invalid "id vector must contain only integers"))) - - (re-matches #"-?\\d+" text) - {:ok? true :value [(js/parseInt text 10)] :multi? false} - - :else - (invalid "id must be a number or vector of numbers"))) - - :else - (invalid "id must be a number or vector of numbers")))) - (defn invalid-options? [opts] - (let [format (:format opts) - level (:level opts) - id-result (parse-id-option (:id opts))] + (let [level (:level opts) + id-result (id-command/parse-id-option (:id opts))] (cond - (and (seq format) (not (contains? show-formats (string/lower-case format)))) - (str "invalid format: " format) - (and (some? level) (< level 1)) "level must be >= 1" @@ -367,7 +315,7 @@ (build root-id 1))) (defn- fetch-tree - [config {:keys [repo id page-name level] :as opts}] + [config {:keys [repo id page level] :as opts}] (let [max-depth (or level 10) uuid-str (:uuid opts)] (cond @@ -414,12 +362,12 @@ {:root (assoc entity :block/children children)}) (throw (ex-info "block not found" {:code :block-not-found})))))) - (seq page-name) + (seq page) (p/let [page-entity (transport/invoke config :thread-api/pull false [repo [:db/id :block/uuid :block/title {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] - [:block/name page-name]])] + [:block/name page]])] (if-let [page-id (:db/id page-entity)] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks page-id max-depth)] @@ -491,11 +439,10 @@ {:ok? false :error {:code :missing-repo :message "repo is required for show"}} - (let [format (some-> (:format options) string/lower-case) - id-result (parse-id-option (:id options)) + (let [id-result (id-command/parse-id-option (:id options)) ids (:value id-result) multi-id? (:multi? id-result) - targets (filter some? [(:id options) (:uuid options) (:page-name options)])] + targets (filter some? [(:id options) (:uuid options) (:page options)])] (if (empty? targets) {:ok? false :error {:code :missing-target @@ -511,9 +458,8 @@ :ids ids :multi-id? multi-id? :uuid (:uuid options) - :page-name (:page-name options) - :level (:level options) - :format format}}))))) + :page (:page options) + :level (:level options)}}))))) (defn- build-tree-data [config action] @@ -565,7 +511,7 @@ (defn execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - format (:format action) + format (:output-format config) ids (:ids action) multi-id? (:multi-id? action)] (if (and (seq ids) multi-id?) @@ -594,7 +540,7 @@ (and ok? (contained? id))) results)) payload (case format - "edn" + :edn {:status :ok :data (mapv (fn [{:keys [ok? tree id error]}] (if ok? @@ -603,7 +549,7 @@ results) :output-format :edn} - "json" + :json {:status :ok :data (mapv (fn [{:keys [ok? tree id error]}] (if ok? @@ -622,12 +568,12 @@ payload) (p/let [tree-data (build-tree-data cfg action)] (case format - "edn" + :edn {:status :ok :data tree-data :output-format :edn} - "json" + :json {:status :ok :data tree-data :output-format :json} diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 762ce8f4e1..0bedcd411f 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -121,11 +121,14 @@ (seq (:blocks opts)) (seq (:blocks-file opts)) has-args?) - show-targets (filter some? [(:id opts) (:uuid opts) (:page-name opts)]) + show-targets (filter some? [(:id opts) (:uuid opts) (:page opts)]) + remove-targets (filter some? [(:id opts) + (some-> (:uuid opts) string/trim) + (some-> (:page opts) string/trim)]) move-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) move-targets (filter some? [(:target-id opts) (some-> (:target-uuid opts) string/trim) - (some-> (:target-page-name opts) string/trim)])] + (some-> (:target-page opts) string/trim)])] (cond (:help opts) (command-core/help-result cmd-summary) @@ -143,11 +146,14 @@ (and (= command :add-page) (not (seq (:page opts)))) (missing-page-name-result summary) - (and (= command :remove-block) (not (seq (:block opts)))) + (and (= command :remove) (seq args)) + (command-core/invalid-options-result summary "remove does not accept subcommands") + + (and (= command :remove) (empty? remove-targets)) (missing-target-result summary) - (and (= command :remove-page) (not (seq (:page opts)))) - (missing-target-result summary) + (and (= command :remove) (> (count remove-targets) 1)) + (command-core/invalid-options-result summary "only one of --id, --uuid, or --page is allowed") (and (= command :move-block) (move-command/invalid-options? opts)) (command-core/invalid-options-result summary (move-command/invalid-options? opts)) @@ -162,7 +168,7 @@ (missing-target-result summary) (and (= command :show) (> (count show-targets) 1)) - (command-core/invalid-options-result summary "only one of --id, --uuid, or --page-name is allowed") + (command-core/invalid-options-result summary "only one of --id, --uuid, or --page is allowed") (and (= command :query) (not (seq (some-> (:query opts) string/trim))) @@ -173,6 +179,9 @@ (list-command/invalid-options? command opts)) (command-core/invalid-options-result summary (list-command/invalid-options? command opts)) + (and (= command :remove) (remove-command/invalid-options? opts)) + (command-core/invalid-options-result summary (remove-command/invalid-options? opts)) + (and (= command :show) (show-command/invalid-options? opts)) (command-core/invalid-options-result summary (show-command/invalid-options? opts)) @@ -226,7 +235,7 @@ :error {:code :missing-command :message "missing command"} :summary summary}) - (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "remove" "query"} (first args))) + (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "query"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) (try (let [result (cli/dispatch table args {:spec global-spec})] @@ -331,11 +340,8 @@ :move-block (move-command/build-action options repo) - :remove-block - (remove-command/build-remove-block-action options repo) - - :remove-page - (remove-command/build-remove-page-action options repo) + :remove + (remove-command/build-action options repo) :query (query-command/build-action options repo config) @@ -377,8 +383,7 @@ :add-block (add-command/execute-add-block action config) :add-page (add-command/execute-add-page action config) :move-block (move-command/execute-move action config) - :remove-block (remove-command/execute-remove action config) - :remove-page (remove-command/execute-remove action config) + :remove (remove-command/execute-remove action config) :query (query-command/execute-query action config) :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) @@ -392,4 +397,4 @@ :message "unknown action"}}))] (assoc result :command (or (:command action) (:type action)) - :context (select-keys action [:repo :graph :page :block :blocks :source :target]))))) + :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks :source :target]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 9e6ac77d8f..4806bb3974 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -234,13 +234,14 @@ [{:keys [repo page]}] (str "Added page: " page " (repo: " repo ")")) -(defn- format-remove-page - [{:keys [repo page]}] - (str "Removed page: " page " (repo: " repo ")")) - -(defn- format-remove-block - [{:keys [repo block]}] - (str "Removed block: " block " (repo: " repo ")")) +(defn- format-remove + [{:keys [repo page uuid id ids]}] + (cond + (seq page) (str "Removed page: " page " (repo: " repo ")") + (seq uuid) (str "Removed block: " uuid " (repo: " repo ")") + (seq ids) (str "Removed blocks: " (count ids) " (repo: " repo ")") + (some? id) (str "Removed block: " id " (repo: " repo ")") + :else (str "Removed item (repo: " repo ")"))) (defn- format-move-block [{:keys [repo source target]}] @@ -283,8 +284,7 @@ (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) :add-block (format-add-block context) :add-page (format-add-page context) - :remove-page (format-remove-page context) - :remove-block (format-remove-block context) + :remove (format-remove context) :move-block (format-move-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 3dfac8d76b..903ecc4362 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -13,7 +13,7 @@ (string/join "\n" ["logseq [options]" "" - "Commands: list page, list tag, list property, add block, add page, move, remove block, remove page, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" + "Commands: list page, list tag, list property, add block, add page, move, remove, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 323e2ebbe3..d067d3ba8e 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -57,12 +57,12 @@ (is (string/includes? summary "add block")) (is (string/includes? summary "add page")))) - (testing "remove group shows subcommands" - (let [result (commands/parse-args ["remove"]) + (testing "remove command shows help" + (let [result (commands/parse-args ["remove" "--help"]) summary (:summary result)] (is (true? (:help? result))) - (is (string/includes? summary "remove block")) - (is (string/includes? summary "remove page")))) + (is (string/includes? summary "Usage: logseq remove")) + (is (string/includes? summary "Command options:")))) (testing "move command shows help" (let [result (commands/parse-args ["move" "--help"]) @@ -145,6 +145,14 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) +(deftest test-parse-args-rejects-legacy-remove-subcommands + (testing "rejects legacy remove subcommands" + (doseq [args [["remove" "block"] + ["remove" "page"]]] + (let [result (commands/parse-args args)] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))))) + (deftest test-parse-args-rejects-graph-option (testing "rejects legacy --graph option" (let [result (commands/parse-args ["--graph" "demo" "graph" "list"])] @@ -433,23 +441,44 @@ (is (= :add-page (:command result))) (is (= "Home" (get-in result [:options :page]))))) - (testing "remove block requires target" - (let [result (commands/parse-args ["remove" "block"])] + (testing "remove requires target" + (let [result (commands/parse-args ["remove"])] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "remove block parses with block" - (let [result (commands/parse-args ["remove" "block" "--block" "demo"])] + (testing "remove parses with id" + (let [result (commands/parse-args ["remove" "--id" "10"])] (is (true? (:ok? result))) - (is (= :remove-block (:command result))) - (is (= "demo" (get-in result [:options :block]))))) + (is (= :remove (:command result))) + (is (= 10 (get-in result [:options :id]))))) - (testing "remove page parses with page" - (let [result (commands/parse-args ["remove" "page" "--page" "Home"])] + (testing "remove parses with uuid" + (let [result (commands/parse-args ["remove" "--uuid" "abc"])] (is (true? (:ok? result))) - (is (= :remove-page (:command result))) + (is (= :remove (:command result))) + (is (= "abc" (get-in result [:options :uuid]))))) + + (testing "remove parses with page" + (let [result (commands/parse-args ["remove" "--page" "Home"])] + (is (true? (:ok? result))) + (is (= :remove (:command result))) (is (= "Home" (get-in result [:options :page]))))) + (testing "remove rejects multiple selectors" + (let [result (commands/parse-args ["remove" "--id" "1" "--page" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "remove rejects empty id vector" + (let [result (commands/parse-args ["remove" "--id" "[]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "remove rejects invalid id vector" + (let [result (commands/parse-args ["remove" "--id" "[1 \"no\"]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "move requires source selector" (let [result (commands/parse-args ["move" "--target-id" "10"])] (is (false? (:ok? result))) @@ -468,17 +497,25 @@ (is (= "def" (get-in result [:options :target-uuid]))) (is (= "last-child" (get-in result [:options :pos])))))) +(deftest test-verb-subcommand-parse-move-target-page + (testing "move parses with target page" + (let [result (commands/parse-args ["move" "--id" "1" "--target-page" "Home"])] + (is (true? (:ok? result))) + (is (= :move-block (:command result))) + (is (= 1 (get-in result [:options :id]))) + (is (= "Home" (get-in result [:options :target-page])))))) + (deftest test-verb-subcommand-parse-show (testing "show requires target" (let [result (commands/parse-args ["show"])] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "show parses with page name" - (let [result (commands/parse-args ["show" "--page-name" "Home"])] + (testing "show parses with page" + (let [result (commands/parse-args ["show" "--page" "Home"])] (is (true? (:ok? result))) (is (= :show (:command result))) - (is (= "Home" (get-in result [:options :page-name]))))) + (is (= "Home" (get-in result [:options :page]))))) (testing "show parses with id vector" (let [result (commands/parse-args ["show" "--id" "[1 2]"])] @@ -491,6 +528,16 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) + (testing "show rejects legacy page-name option" + (let [result (commands/parse-args ["show" "--page-name" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "show rejects format option" + (let [result (commands/parse-args ["show" "--format" "json" "--page" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (deftest test-verb-subcommand-parse-query (testing "query shows group help" (let [result (commands/parse-args ["query"])] @@ -562,7 +609,7 @@ (testing "verb subcommands reject unknown flags" (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] - ["remove" "block" "--wat"] + ["remove" "--wat"] ["move" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] @@ -570,7 +617,7 @@ (is (= :invalid-options (get-in result [:error :code])))))) (testing "verb subcommands accept output option" - (let [result (commands/parse-args ["show" "--output" "json" "--page-name" "Home"])] + (let [result (commands/parse-args ["show" "--output" "json" "--page" "Home"])] (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) @@ -659,12 +706,19 @@ (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) - (testing "remove block requires target" - (let [parsed {:ok? true :command :remove-block :options {}} + (testing "remove requires target" + (let [parsed {:ok? true :command :remove :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) + (testing "remove normalizes id vector in build action" + (let [parsed {:ok? true :command :remove :options {:id "[1 2]"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :remove (get-in result [:action :type]))) + (is (= [1 2] (get-in result [:action :ids]))))) + (testing "show requires target" (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] @@ -708,12 +762,12 @@ (is (= :invalid-options (get-in result [:error :code]))))) (testing "move rejects sibling pos for page target" - (let [result (commands/parse-args ["move" "--id" "1" "--target-page-name" "Home" "--pos" "sibling"])] + (let [result (commands/parse-args ["move" "--id" "1" "--target-page" "Home" "--pos" "sibling"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "move rejects legacy page-name option" - (let [result (commands/parse-args ["move" "--id" "1" "--page-name" "Home"])] + (testing "move rejects legacy target-page-name option" + (let [result (commands/parse-args ["move" "--id" "1" "--target-page-name" "Home"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) From 12de72064ee6926cf985919a7ea5df0d0a51df0a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 25 Jan 2026 16:50:02 +0800 Subject: [PATCH 043/375] update code according to remove file-based PR --- deps/cli/src/logseq/cli/common/mcp/tools.cljs | 8 ------ src/main/frontend/worker/db_core.cljs | 26 +++++++------------ 2 files changed, 9 insertions(+), 25 deletions(-) diff --git a/deps/cli/src/logseq/cli/common/mcp/tools.cljs b/deps/cli/src/logseq/cli/common/mcp/tools.cljs index c9c84b9200..8b75fff3d5 100644 --- a/deps/cli/src/logseq/cli/common/mcp/tools.cljs +++ b/deps/cli/src/logseq/cli/common/mcp/tools.cljs @@ -16,11 +16,6 @@ [malli.core :as m] [malli.error :as me])) -(defn- ensure-db-graph - [db] - (when-not (ldb/db-based-graph? db) - (throw (ex-info "This tool must be called on a DB graph" {})))) - (defn- minimal-list-item [e] (cond-> {:db/id (:db/id e) @@ -32,7 +27,6 @@ (defn list-properties "Main fn for ListProperties tool" [db {:keys [expand include-built-in] :as options}] - (ensure-db-graph db) (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] (->> (d/datoms db :avet :block/tags :logseq.class/Property) (map #(d/entity db (:e %))) @@ -57,7 +51,6 @@ (defn list-tags "Main fn for ListTags tool" [db {:keys [expand include-built-in] :as options}] - (ensure-db-graph db) (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] (->> (d/datoms db :avet :block/tags :logseq.class/Tag) (map #(d/entity db (:e %))) @@ -123,7 +116,6 @@ (defn list-pages "Main fn for ListPages tool" [db {:keys [expand include-hidden include-journal journal-only created-after updated-after] :as options}] - (ensure-db-graph db) (let [include-hidden? (boolean include-hidden) include-journal? (if (contains? options :include-journal) include-journal true) journal-only? (boolean journal-only) diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 21feec84fe..911024e033 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -38,7 +38,6 @@ [logseq.common.util :as common-util] [logseq.db :as ldb] [logseq.db.common.entity-plus :as entity-plus] - [logseq.db.common.entity-util :as common-entity-util] [logseq.db.common.initial-data :as common-initial-data] [logseq.db.common.order :as db-order] [logseq.db.common.reference :as db-reference] @@ -54,7 +53,9 @@ [logseq.outliner.op :as outliner-op] [me.tonsky.persistent-sorted-set :as set :refer [BTSet]] [missionary.core :as m] - [promesa.core :as p])) + [promesa.core :as p] + [logseq.db.frontend.schema :as db-schema] + [logseq.db.frontend.entity-util :as entity-util])) (defonce *sqlite worker-state/*sqlite) (defonce *sqlite-conns worker-state/*sqlite-conns) @@ -301,12 +302,9 @@ (when-not @*publishing? (common-sqlite/create-kvs-table! client-ops-db)) (rtc-debug-log/create-tables! debug-log-db) (search/create-tables-and-triggers! search-db) - (ldb/register-transact-pipeline-fn! - (fn [tx-report] - (worker-pipeline/transact-pipeline repo tx-report))) - (let [schema (ldb/get-schema repo) - conn (common-sqlite/get-storage-conn storage schema) - _ (db-fix/check-and-fix-schema! repo conn) + (ldb/register-transact-pipeline-fn! worker-pipeline/transact-pipeline) + (let [conn (common-sqlite/get-storage-conn storage db-schema/schema) + _ (db-fix/check-and-fix-schema! conn) _ (when datoms (let [eid->datoms (group-by :e datoms) {properties true non-properties false} (group-by @@ -526,8 +524,7 @@ tx-data) _ (when context (worker-state/set-context! context)) tx-meta' (cond-> tx-meta - (and (not (:whiteboard/transact? tx-meta)) - (not (:rtc-download-graph? tx-meta))) ; delay writes to the disk + (not (:rtc-download-graph? tx-meta)) ; delay writes to the disk (assoc :skip-store? true) true @@ -691,15 +688,10 @@ (when-let [conn (worker-state/get-datascript-conn repo)] (worker-export/get-debug-datoms conn))) -(def-thread-api :thread-api/export-get-all-pages - [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-export/get-all-pages repo @conn))) - (def-thread-api :thread-api/export-get-all-page->content [repo options] (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-export/get-all-page->content repo @conn options))) + (worker-export/get-all-page->content @conn options))) (def-thread-api :thread-api/validate-db [repo] @@ -742,7 +734,7 @@ [repo class-id] (let [db @(worker-state/get-datascript-conn repo)] (->> (db-class/get-class-objects db class-id) - (map common-entity-util/entity->map)))) + (map entity-util/entity->map)))) (def-thread-api :thread-api/get-property-values [repo {:keys [property-ident] :as option}] From 2779a41ccc4335cc554320fa12677afcc933f55d Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 26 Jan 2026 23:46:11 +0800 Subject: [PATCH 044/375] 018-logseq-cli-add-tags-builtin-properties.md --- ...-logseq-cli-add-tags-builtin-properties.md | 196 +++++++ src/main/logseq/cli/command/add.cljs | 486 +++++++++++++++++- src/test/logseq/cli/commands_test.cljs | 59 +++ src/test/logseq/cli/integration_test.cljs | 100 ++++ 4 files changed, 819 insertions(+), 22 deletions(-) create mode 100644 docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md diff --git a/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md new file mode 100644 index 0000000000..44d617761f --- /dev/null +++ b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md @@ -0,0 +1,196 @@ +# Logseq CLI Add Tag And Built-in Property Support Implementation Plan + +Goal: Extend logseq-cli add block and add page to accept tags and built-in properties with correct type handling. + +Architecture: Parse tag and property options in the CLI, resolve them via db-worker-node, and apply them using existing outliner ops. +Architecture: Use built-in property definitions and property type rules to coerce values before invoking :batch-set-property or :create-page. + +Tech Stack: ClojureScript, babashka.cli, Datascript, db-worker-node, outliner ops. + +Related: Relates to docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md. + +## Problem statement + +Logseq CLI currently supports add block and add page with content and basic status support. +Users need to set tags and built-in properties at creation time from CLI, but built-in properties have multiple value types and validation rules. +The implementation must align with the built-in property definitions and property type system so that values are stored and validated correctly in db-worker-node. + +## Testing Plan + +I will add unit tests that parse new CLI options for tags and properties and validate error cases for invalid property names and invalid type values. +I will add integration tests that run add block and add page with tags and built-in properties and assert the resulting data in the graph. +I will add tests that cover ref-type properties like :logseq.property/deadline and :block/tags to ensure resolution behavior is correct. +I will add tests that cover scalar properties like :logseq.property/publishing-public? and :logseq.property/heading with proper type coercion. +I will verify that invalid types cause CLI errors before any outliner ops are sent. +I will add tests that missing tags in --add-tag fail with a clear error and do not create tags. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior summary + +Add block and add page live in src/main/logseq/cli/command/add.cljs. +Add block supports content, blocks, blocks-file, target selection, position, and status, and it sets status via :batch-set-property. +Add page uses :create-page with an empty options map and does not apply tags or properties. +Built-in properties and their schema are defined in deps/db/src/logseq/db/frontend/property.cljs and type rules in deps/db/src/logseq/db/frontend/property/type.cljs. + +## Requirements + +Add block supports setting tags on all inserted blocks. +Add page supports setting tags on the created page. +Add block supports setting built-in properties with correct type coercion. +Add page supports setting built-in properties with correct type coercion. +CLI rejects non built-in properties for these new options. +CLI rejects built-in properties that are not public and provides a clear error message. +CLI rejects adding non public tags to blocks. +CLI rejects combining --blocks or --blocks-file with --add-property, --remove-property, --add-tag, or --remove-tag. + +## Non goals + +Do not change db-worker-node HTTP APIs or add new endpoints. +Do not change outliner property validation logic. +Do not add support for user properties in this change. + +## Built-in property type considerations + +Built-in property configuration lives in deps/db/src/logseq/db/frontend/property.cljs. +Built-in property types and validation live in deps/db/src/logseq/db/frontend/property/type.cljs. +Property types include user types (:default, :number, :date, :datetime, :checkbox, :url, :node) and internal types (:string, :keyword, :map, :coll, :any, :entity, :class, :page, :property, :raw-number). +Some built-in properties are not public and must not be user-settable via CLI. + +### Value coercion table + +| Property type | Expected CLI input | Resolution behavior | +| --- | --- | --- | +| :checkbox | boolean or "true"/"false" | Coerce to boolean and send directly. | +| :number | number or numeric string | Coerce to number and send directly. | +| :raw-number | number | Send directly and reject non numeric. | +| :datetime | ISO string | Convert to epoch ms number before sending. | +| :date | date string or journal page name | Resolve to journal page entity id. | +| :default | string or EDN | If string, pass as text value and let outliner create property value block. | +| :url | string | Validate with db-property-type/url? or allow macro urls and send as string. | +| :string | string | Send directly. | +| :keyword | keyword or string | Coerce string to keyword if safe. | +| :map | EDN map | Send directly. | +| :coll | EDN vector or list | Send directly. | +| :entity | block uuid or db/id | Resolve to entity id with :thread-api/pull or reject. | +| :page | page name or uuid | Resolve to page entity id, create page if missing. | +| :class | tag name or uuid | Resolve to class entity id and fail if missing. | +| :property | property name or keyword | Resolve to property entity id using built-in properties map. | +| :node | block uuid or page name | Resolve to entity id or allow node text block if required. | +| :any | EDN value | Send directly. | + +## Data flow overview + +CLI input is parsed by babashka.cli and normalized in src/main/logseq/cli/command/add.cljs. +CLI resolves tags and property values via db-worker-node using :thread-api/pull and existing outliner ops. +CLI applies properties using :batch-set-property for blocks and :create-page options for pages. + +ASCII architecture sketch. + +CLI add command + -> parse options + -> resolve tags and properties + -> db-worker-node :thread-api/apply-outliner-ops + -> outliner ops set properties and tags + +## Design decisions + +Tags are applied through :block/tags because it is the built-in tags property and is validated by outliner validation. +Page creation uses :create-page with :tags and :properties options to avoid separate post-create transactions. +Block creation uses existing insert blocks then batch-set-property for tags and each built-in property to keep behavior consistent with status handling. +Tags are always written to :block/tags and never to :logseq.property/page-tags. +Add block rejects any combination of --blocks or --blocks-file with --add-property, --remove-property, --add-tag, or --remove-tag. +Properties provided via CLI apply to all inserted blocks unless the input blocks already include explicit values in their block maps. +Properties provided via CLI override existing values on the newly created blocks to avoid ambiguity. + +## Implementation plan + +### 1. CLI option design and parsing + +Add new options to content-add-spec in src/main/logseq/cli/command/add.cljs for tags and properties. +Add new options to add-page-spec in src/main/logseq/cli/command/add.cljs for tags and properties. +Use a single EDN map option like --properties for multiple properties and a repeated option like --property for key value pairs if needed. +Use a single EDN vector option like --tags and a repeated option like --tag for convenience. +Update command summary and help output in src/main/logseq/cli/command/core.cljs if needed to reflect new options. + +### 2. Tag resolution helpers + +Add a helper in src/main/logseq/cli/command/add.cljs to normalize tag inputs into a vector of tag names or uuids. +Add a helper that validates each tag exists and fails fast when a tag is missing. +Use :thread-api/pull with lookup refs to resolve existing tag pages by name or uuid. +Return a vector of tag entity ids or uuids suitable for outliner ops. + +### 3. Built-in property resolution helpers + +Add a helper in src/main/logseq/cli/command/add.cljs to parse --properties EDN and validate keys against logseq.db.frontend.property/built-in-properties. +Reject keys not present in built-in-properties or not public based on :schema :public?. +Add a helper to get property type from built-in-properties and then coerce values using rules in deps/db/src/logseq/db/frontend/property/type.cljs. +Add a helper to resolve ref values into entity ids via :thread-api/pull for :page, :class, :property, :entity, and :node. +Add a helper to resolve :date to a journal page, creating it when missing if that is consistent with UI behavior. +Do not create tags implicitly when missing for --add-tag. + +### 4. Add page execution changes + +Extend build-add-page-action to carry tags and properties in the action context. +Modify execute-add-page in src/main/logseq/cli/command/add.cljs to pass :tags and :properties into the :create-page op options map. +Ensure property values are coerced before being sent so outliner validation passes. + +### 5. Add block execution changes + +Extend build-add-block-action to carry tags and properties in the action context. +After insert blocks and status application, apply tags via :batch-set-property with :block/tags and the resolved tag ids. +Apply each built-in property via :batch-set-property for the newly created block uuids. +Keep :keep-uuid? behavior for status so tags and properties can reference inserted block ids. + +### 6. CLI formatting and errors + +Update error messages in src/main/logseq/cli/commands.cljs to include new invalid option errors. +Add error formatting in src/main/logseq/cli/format.cljs if needed to show applied tags or properties in human output. +Ensure JSON output includes any new context if the CLI returns it. + +### 7. Tests and fixtures + +Add unit tests in src/test/logseq/cli/commands_test.cljs for option parsing and error handling. +Add integration tests in src/test/logseq/cli/integration_test.cljs that create blocks and pages with tags and built-in properties. +Add tests for at least one ref property (e.g. :logseq.property/deadline) and one scalar property (e.g. :logseq.property/publishing-public?). +Add tests for tag creation when tag pages do not exist. + +## Edge cases + +Tags that collide with private or built-in non-tag classes should be rejected by validation and surfaced to the CLI user. +Missing tags in --add-tag should produce a clear missing tag error without creating new tag pages. +Properties with closed values like :logseq.property/status should accept keyword idents as well as string labels where supported. +Date properties must resolve to journal pages or fail with a clear error if parsing is invalid. +Properties with cardinality many should accept vectors and sets and maintain ordering when required. +Inline tags or page namespaces should not be created implicitly without validation of allowed characters. + +## Resolved decisions + +CLI must not allow setting non public built-in properties, even with a force option. +Tags are applied via :block/tags and not :logseq.property/page-tags. +Add block must reject --blocks or --blocks-file when combined with --add-property, --remove-property, --add-tag, or --remove-tag. +Datetime values are provided as ISO strings. + +## Testing Details + +I will add CLI tests that run add page and add block end to end and assert the actual persisted properties and tags using list or show commands. +I will add tests that confirm invalid inputs fail fast and do not produce partial writes. +I will add tests that assert correct ref resolution for tag pages and journal pages. +I will ensure tests cover behavior rather than internal data structures and follow @test-driven-development. + +## Implementation Details + +- Update src/main/logseq/cli/command/add.cljs with new option parsing and action fields. +- Add tag and property normalization helpers in src/main/logseq/cli/command/add.cljs. +- Use deps/db/src/logseq/db/frontend/property.cljs to validate built-in property keys. +- Use deps/db/src/logseq/db/frontend/property/type.cljs to coerce values by type. +- Use :thread-api/pull in src/main/logseq/cli/command/add.cljs to resolve pages, tags, properties, and blocks. +- Pass tags and properties into :create-page ops in src/main/logseq/cli/command/add.cljs. +- Apply :batch-set-property for :block/tags and built-in properties in src/main/logseq/cli/command/add.cljs. +- Update src/test/logseq/cli/commands_test.cljs with parsing validation tests. +- Update src/test/logseq/cli/integration_test.cljs with behavior tests for tags and built-in properties. + +## Question + +No open questions. + +--- diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 105188dd8d..46d6438308 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -10,12 +10,16 @@ [logseq.common.util :as common-util] [logseq.common.util.date-time :as date-time-util] [logseq.common.uuid :as common-uuid] + [logseq.db.frontend.property :as db-property] + [logseq.db.frontend.property.type :as db-property-type] [promesa.core :as p])) (def ^:private content-add-spec {:content {:desc "Block content for add"} :blocks {:desc "EDN vector of blocks for add"} :blocks-file {:desc "EDN file of blocks for add"} + :tags {:desc "EDN vector of tags"} + :properties {:desc "EDN map of built-in properties"} :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} @@ -24,7 +28,9 @@ :status {:desc "Task status (todo, doing, done, etc.)"}}) (def ^:private add-page-spec - {:page {:desc "Page name"}}) + {:page {:desc "Page name"} + :tags {:desc "EDN vector of tags"} + :properties {:desc "EDN map of built-in properties"}}) (def entries [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) @@ -103,13 +109,249 @@ (assoc block :block/uuid (common-uuid/gen-uuid))))) blocks)) +(defn- invalid-options-result + [message] + {:ok? false + :error {:code :invalid-options + :message message}}) + +(defn- parse-edn-option + [value] + (when (seq value) + (common-util/safe-read-string {:log-error? false} value))) + +(defn- keyword->name + [value] + (subs (str value) 1)) + +(defn- normalize-tag-value + [value] + (cond + (uuid? value) value + (and (string? value) (common-util/uuid-string? (string/trim value))) + (uuid (string/trim value)) + (keyword? value) (keyword->name value) + (string? value) (let [text (-> value string/trim (string/replace #"^#+" ""))] + (when (seq text) text)) + :else nil)) + +(defn- parse-tags-option + [value] + (if-not (seq value) + {:ok? true :value nil} + (let [parsed (parse-edn-option value)] + (cond + (nil? parsed) + (invalid-options-result "tags must be valid EDN vector") + + (not (vector? parsed)) + (invalid-options-result "tags must be a vector") + + (empty? parsed) + (invalid-options-result "tags must be a non-empty vector") + + :else + (let [tags (mapv normalize-tag-value parsed)] + (if (some nil? tags) + (invalid-options-result "tags must be strings, keywords, or uuids") + {:ok? true :value tags})))))) + +(defn- normalize-property-key + [value] + (cond + (keyword? value) value + (string? value) + (let [text (string/trim value)] + (cond + (string/blank? text) nil + (common-util/valid-edn-keyword? text) + (common-util/safe-read-string {:log-error? false} text) + :else (keyword text))) + :else nil)) + +(defn- parse-boolean-value + [value] + (cond + (or (true? value) (false? value)) {:ok? true :value value} + (string? value) (let [text (string/lower-case (string/trim value))] + (cond + (= text "true") {:ok? true :value true} + (= text "false") {:ok? true :value false} + :else {:ok? false})) + :else {:ok? false})) + +(defn- parse-number-value + [value] + (cond + (number? value) {:ok? true :value value} + (string? value) (let [parsed (js/parseFloat value)] + (if (js/isNaN parsed) + {:ok? false} + {:ok? true :value parsed})) + :else {:ok? false})) + +(defn- parse-datetime-value + [value] + (cond + (number? value) {:ok? true :value value} + (string? value) (let [date (js/Date. value) + ms (.getTime date)] + (if (js/isNaN ms) + {:ok? false} + {:ok? true :value ms})) + :else {:ok? false})) + +(defn- parse-keyword-value + [value] + (cond + (keyword? value) {:ok? true :value value} + (string? value) (let [text (string/trim value)] + (if (string/blank? text) + {:ok? false} + (if (common-util/valid-edn-keyword? text) + (let [parsed (common-util/safe-read-string {:log-error? false} text)] + (if (keyword? parsed) + {:ok? true :value parsed} + {:ok? false})) + {:ok? true :value (keyword text)}))) + :else {:ok? false})) + +(defn- coerce-property-value-basic + [property value] + (let [type (get-in property [:schema :type] :default)] + (cond + (nil? value) + {:ok? false :message "property value must not be nil"} + + (= type :checkbox) + (let [{:keys [ok? value]} (parse-boolean-value value)] + (if ok? + {:ok? true :value value} + {:ok? false :message "checkbox property expects true or false"})) + + (= type :number) + (let [{:keys [ok? value]} (parse-number-value value)] + (if ok? + {:ok? true :value value} + {:ok? false :message "number property expects a numeric value"})) + + (= type :raw-number) + (if (number? value) + {:ok? true :value value} + {:ok? false :message "raw-number property expects a number"}) + + (= type :datetime) + (let [{:keys [ok? value]} (parse-datetime-value value)] + (if ok? + {:ok? true :value value} + {:ok? false :message "datetime property expects an ISO date string"})) + + (= type :keyword) + (let [{:keys [ok? value]} (parse-keyword-value value)] + (if ok? + {:ok? true :value value} + {:ok? false :message "keyword property expects a keyword"})) + + (= type :string) + (if (string? value) + {:ok? true :value value} + {:ok? false :message "string property expects a string"}) + + (= type :map) + (if (map? value) + {:ok? true :value value} + {:ok? false :message "map property expects a map"}) + + (= type :coll) + (if (and (coll? value) (not (string? value))) + {:ok? true :value (vec value)} + {:ok? false :message "coll property expects a collection"}) + + (= type :url) + (if (and (string? value) (or (db-property-type/url? value) (db-property-type/macro-url? value))) + {:ok? true :value value} + {:ok? false :message "url property expects a valid url"}) + + (= type :date) + (if (string? value) + {:ok? true :value value} + {:ok? false :message "date property expects a date string"}) + + (contains? #{:entity :page :class :property :node :default :any} type) + {:ok? true :value value} + + :else + {:ok? true :value value}))) + +(defn- normalize-property-values + [property value] + (let [many? (= :many (get-in property [:schema :cardinality])) + values (if many? + (if (and (coll? value) (not (string? value))) value [value]) + [value])] + (loop [remaining values + normalized []] + (if (empty? remaining) + {:ok? true + :value (if many? (vec normalized) (first normalized))} + (let [result (coerce-property-value-basic property (first remaining))] + (if-not (:ok? result) + {:ok? false + :message (:message result)} + (recur (rest remaining) (conj normalized (:value result))))))))) + +(defn- property-public? + [property] + (true? (get-in property [:schema :public?]))) + +(defn- parse-properties-option + [value] + (if-not (seq value) + {:ok? true :value nil} + (let [parsed (parse-edn-option value)] + (cond + (nil? parsed) + (invalid-options-result "properties must be valid EDN map") + + (not (map? parsed)) + (invalid-options-result "properties must be a map") + + (empty? parsed) + (invalid-options-result "properties must be a non-empty map") + + :else + (loop [entries (seq parsed) + acc {}] + (if (empty? entries) + {:ok? true :value acc} + (let [[k v] (first entries) + key* (normalize-property-key k)] + (if-not key* + (invalid-options-result (str "invalid property key: " k)) + (let [property (get db-property/built-in-properties key*)] + (cond + (nil? property) + (invalid-options-result (str "unknown built-in property: " key*)) + + (not (property-public? property)) + (invalid-options-result (str "property is not public: " key*)) + + :else + (let [{:keys [ok? value message]} (normalize-property-values property v)] + (if-not ok? + (invalid-options-result (str "invalid value for " key* ": " message)) + (recur (rest entries) (assoc acc key* value)))))))))))))) + (defn invalid-options? [opts] (let [pos (some-> (:pos opts) string/trim string/lower-case) target-id (:target-id opts) target-uuid (some-> (:target-uuid opts) string/trim) target-page (some-> (:target-page-name opts) string/trim) - target-selectors (filter some? [target-id target-uuid target-page])] + target-selectors (filter some? [target-id target-uuid target-page]) + has-blocks? (or (seq (:blocks opts)) (seq (:blocks-file opts))) + has-tags? (seq (some-> (:tags opts) string/trim)) + has-properties? (seq (some-> (:properties opts) string/trim))] (cond (and (seq pos) (not (contains? add-positions pos))) (str "invalid pos: " (:pos opts)) @@ -120,9 +362,145 @@ (and (= pos "sibling") (or (seq target-page) (empty? target-selectors))) "--pos sibling is only valid for block targets" + (and has-blocks? (or has-tags? has-properties?)) + "tags and properties cannot be combined with --blocks or --blocks-file" + :else nil))) +(defn- pull-entity + [config repo selector lookup] + (transport/invoke config :thread-api/pull false [repo selector lookup])) + +(defn- tag-lookup-ref + [tag] + (cond + (uuid? tag) [:block/uuid tag] + (and (string? tag) (common-util/uuid-string? (string/trim tag))) [:block/uuid (uuid (string/trim tag))] + (string? tag) [:block/name (common-util/page-name-sanity-lc tag)] + :else nil)) + +(defn- resolve-tag-entity + [config repo tag] + (let [lookup (tag-lookup-ref tag)] + (when-not lookup + (throw (ex-info "invalid tag value" {:code :invalid-tag :tag tag}))) + (p/let [entity (pull-entity config repo + [:db/id :block/name :block/title :block/uuid :block/tags + :logseq.property/public? :logseq.property/built-in?] + lookup)] + (cond + (nil? (:db/id entity)) + (throw (ex-info "tag not found" {:code :tag-not-found :tag tag})) + + (false? (:logseq.property/public? entity)) + (throw (ex-info "tag is not public" {:code :tag-not-public :tag tag})) + + :else + entity)))) + +(defn- resolve-tags + [config repo tags] + (if (seq tags) + (p/let [entities (p/all (map #(resolve-tag-entity config repo %) tags))] + (vec entities)) + (p/resolved nil))) + +(defn- resolve-entity-id + [config repo lookup] + (p/let [entity (pull-entity config repo [:db/id] lookup)] + (if-let [id (:db/id entity)] + id + (throw (ex-info "entity not found" {:code :entity-not-found :lookup lookup}))))) + +(defn- resolve-page-id + [config repo value] + (cond + (number? value) (p/resolved value) + (uuid? value) (resolve-entity-id config repo [:block/uuid value]) + (and (string? value) (common-util/uuid-string? (string/trim value))) + (resolve-entity-id config repo [:block/uuid (uuid (string/trim value))]) + (string? value) + (p/let [page (ensure-page! config repo value)] + (or (:db/id page) + (throw (ex-info "page not found" {:code :page-not-found :value value})))) + :else + (p/rejected (ex-info "page must be a name or uuid" {:code :invalid-page :value value})))) + +(defn- resolve-class-id + [config repo value] + (p/let [entity (resolve-tag-entity config repo value)] + (:db/id entity))) + +(defn- resolve-property-id + [config repo value] + (let [key (normalize-property-key value)] + (when-not key + (throw (ex-info "property must be a keyword" {:code :invalid-property :value value}))) + (resolve-entity-id config repo [:db/ident key]))) + +(defn- resolve-node-id + [config repo value] + (cond + (number? value) (p/resolved value) + (uuid? value) (resolve-entity-id config repo [:block/uuid value]) + (and (string? value) (common-util/uuid-string? (string/trim value))) + (resolve-entity-id config repo [:block/uuid (uuid (string/trim value))]) + (string? value) + (resolve-page-id config repo value) + :else + (p/rejected (ex-info "node must be a uuid or page name" {:code :invalid-node :value value})))) + +(defn- resolve-date-page-id + [config repo value] + (when-not (string? value) + (throw (ex-info "date must be a string" {:code :invalid-date :value value}))) + (p/let [journal (pull-entity config repo [:logseq.property.journal/title-format] :logseq.class/Journal) + formatter (or (:logseq.property.journal/title-format journal) "MMM do, yyyy") + formatters (date-time-util/safe-journal-title-formatters formatter) + journal-day (date-time-util/journal-title->int value formatters) + title (or (when journal-day + (date-time-util/int->journal-title journal-day formatter)) + value) + page (ensure-page! config repo title)] + (if-let [id (:db/id page)] + id + (throw (ex-info "journal page not found" {:code :page-not-found :value value}))))) + +(defn- resolve-property-value + [config repo property value] + (let [type (get-in property [:schema :type] :default)] + (case type + :page (resolve-page-id config repo value) + :class (resolve-class-id config repo value) + :property (resolve-property-id config repo value) + :entity (resolve-entity-id config repo (cond + (number? value) value + (uuid? value) [:block/uuid value] + (and (string? value) (common-util/uuid-string? (string/trim value))) + [:block/uuid (uuid (string/trim value))] + :else value)) + :node (resolve-node-id config repo value) + :date (resolve-date-page-id config repo value) + (p/resolved value)))) + +(defn- resolve-properties + [config repo properties] + (if-not (seq properties) + (p/resolved nil) + (p/let [entries (p/all + (map (fn [[k v]] + (let [property (get db-property/built-in-properties k) + many? (= :many (get-in property [:schema :cardinality])) + values (if many? + (if (and (coll? v) (not (string? v))) v [v]) + [v])] + (p/let [resolved (p/all (map #(resolve-property-value config repo property %) values)) + final-value (if many? (vec resolved) (first resolved))] + [k final-value]))) + properties))] + (into {} entries)))) + (defn- resolve-add-target [config {:keys [repo target-id target-uuid target-page-name]}] (cond @@ -185,13 +563,24 @@ :message "repo is required for add"}} (let [blocks-result (read-blocks options args) status-text (some-> (:status options) string/trim) - status (when (seq status-text) (normalize-status status-text))] + status (when (seq status-text) (normalize-status status-text)) + tags-result (parse-tags-option (:tags options)) + properties-result (parse-properties-option (:properties options)) + tags (:value tags-result) + properties (:value properties-result) + ensure-uuids? (or status (seq tags) (seq properties))] (cond (and (seq status-text) (nil? status)) {:ok? false :error {:code :invalid-options :message (str "invalid status: " status-text)}} + (not (:ok? tags-result)) + tags-result + + (not (:ok? properties-result)) + properties-result + :else (if-not (:ok? blocks-result) blocks-result @@ -199,7 +588,7 @@ (if-not (:ok? vector-result) vector-result (let [blocks (cond-> (:value vector-result) - status + ensure-uuids? ensure-block-uuids)] {:ok? true :action {:type :add-block @@ -210,6 +599,8 @@ :target-page-name (some-> (:target-page-name options) string/trim) :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") :status status + :tags tags + :properties properties :blocks blocks}})))))))) (defn build-add-page-action @@ -220,11 +611,23 @@ :message "repo is required for add"}} (let [page (some-> (:page options) string/trim)] (if (seq page) - {:ok? true - :action {:type :add-page - :repo repo - :graph (core/repo->graph repo) - :page page}} + (let [tags-result (parse-tags-option (:tags options)) + properties-result (parse-properties-option (:properties options))] + (cond + (not (:ok? tags-result)) + tags-result + + (not (:ok? properties-result)) + properties-result + + :else + {:ok? true + :action {:type :add-page + :repo repo + :graph (core/repo->graph repo) + :page page + :tags (:value tags-result) + :properties (:value properties-result)}})) {:ok? false :error {:code :missing-page-name :message "page name is required"}})))) @@ -234,35 +637,74 @@ (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) target-id (resolve-add-target cfg action) status (:status action) + tags (resolve-tags cfg (:repo action) (:tags action)) + properties (resolve-properties cfg (:repo action) (:properties action)) pos (:pos action) + keep-uuid? (or status (seq tags) (seq properties)) opts (case pos "last-child" {:sibling? false :bottom? true} "sibling" {:sibling? true} {:sibling? false}) opts (cond-> opts - status + keep-uuid? (assoc :keep-uuid? true)) ops [[:insert-blocks [(:blocks action) target-id (assoc opts :outliner-op :insert-blocks)]]] _ (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) - _ (when status - (let [block-ids (->> (:blocks action) - (map :block/uuid) - (remove nil?) - vec)] - (when (seq block-ids) - (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) - [[:batch-set-property [block-ids :logseq.property/status status {}]]] - {}]))))] + block-ids (->> (:blocks action) + (map :block/uuid) + (remove nil?) + vec) + tag-ids (when (seq tags) + (->> tags (map :db/id) (remove nil?) vec)) + _ (when (and status (seq block-ids)) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:batch-set-property [block-ids :logseq.property/status status {}]]] + {}])) + _ (when (and (seq tag-ids) (seq block-ids)) + (p/all + (map (fn [tag-id] + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:batch-set-property [block-ids :block/tags tag-id {}]]] + {}])) + tag-ids))) + _ (when (and (seq properties) (seq block-ids)) + (p/all + (map (fn [[k v]] + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:batch-set-property [block-ids k v {}]]] + {}])) + properties)))] {:status :ok :data {:result nil}}))) (defn execute-add-page [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - ops [[:create-page [(:page action) {}]]] - result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])] + tags (resolve-tags cfg (:repo action) (:tags action)) + tag-ids (when (seq tags) + (->> tags (map :db/id) (remove nil?) vec)) + properties (resolve-properties cfg (:repo action) (:properties action)) + options (cond-> {} + (seq properties) (assoc :properties properties)) + ops [[:create-page [(:page action) options]]] + result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) + _ (when (seq tag-ids) + (p/let [page-name (common-util/page-name-sanity-lc (:page action)) + page (pull-entity cfg (:repo action) [:db/id :block/uuid] [:block/name page-name]) + page-uuid (:block/uuid page)] + (when-not page-uuid + (throw (ex-info "page not found" {:code :page-not-found :page (:page action)}))) + (p/all + (map (fn [tag-id] + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:batch-set-property [[page-uuid] :block/tags tag-id {}]]] + {}])) + tag-ids))))] {:status :ok :data {:result result}}))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index d067d3ba8e..3b2f94877c 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -423,6 +423,16 @@ (is (= "abc" (get-in result [:options :target-uuid]))) (is (= "first-child" (get-in result [:options :pos]))))) + (testing "add block parses with tags and properties" + (let [result (commands/parse-args ["add" "block" + "--content" "hello" + "--tags" "[\"TagA\" \"TagB\"]" + "--properties" "{:logseq.property/publishing-public? true}"])] + (is (true? (:ok? result))) + (is (= :add-block (:command result))) + (is (= "[\"TagA\" \"TagB\"]" (get-in result [:options :tags]))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) + (testing "add block rejects invalid pos" (let [result (commands/parse-args ["add" "block" "--content" "hello" @@ -430,6 +440,20 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) + (testing "add block rejects tags with blocks payload" + (let [result (commands/parse-args ["add" "block" + "--blocks" "[]" + "--tags" "[\"TagA\"]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "add block rejects properties with blocks-file payload" + (let [result (commands/parse-args ["add" "block" + "--blocks-file" "/tmp/blocks.edn" + "--properties" "{:logseq.property/publishing-public? true}"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "add page requires page name" (let [result (commands/parse-args ["add" "page"])] (is (false? (:ok? result))) @@ -441,6 +465,16 @@ (is (= :add-page (:command result))) (is (= "Home" (get-in result [:options :page]))))) + (testing "add page parses with tags and properties" + (let [result (commands/parse-args ["add" "page" + "--page" "Home" + "--tags" "[\"TagA\"]" + "--properties" "{:logseq.property/publishing-public? true}"])] + (is (true? (:ok? result))) + (is (= :add-page (:command result))) + (is (= "[\"TagA\"]" (get-in result [:options :tags]))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) + (testing "remove requires target" (let [result (commands/parse-args ["remove"])] (is (false? (:ok? result))) @@ -732,6 +766,31 @@ (is (= :show (get-in result [:action :type]))) (is (= [1 2] (get-in result [:action :ids])))))) +(deftest test-build-action-add-validates-properties + (testing "add block rejects unknown property" + (let [parsed (commands/parse-args ["add" "block" + "--content" "hello" + "--properties" "{:not/a 1}"]) + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "add block rejects non-public built-in property" + (let [parsed (commands/parse-args ["add" "block" + "--content" "hello" + "--properties" "{:logseq.property/heading 1}"]) + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "add block rejects invalid checkbox value" + (let [parsed (commands/parse-args ["add" "block" + "--content" "hello" + "--properties" "{:logseq.property/publishing-public? \"nope\"}"]) + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) + (deftest test-build-action-move (testing "move requires source selector" (let [parsed {:ok? true :command :move-block :options {:target-id 2}} diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 5657cd3024..bc42047d4e 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -127,6 +127,106 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-add-tags-and-properties + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-tags")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "tags-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "tags-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + add-page-result (run-cli ["--repo" "tags-graph" + "add" "page" + "--page" "TaggedPage" + "--tags" "[\"Quote\"]" + "--properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + add-page-payload (parse-json-output add-page-result) + add-block-result (run-cli ["--repo" "tags-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "Tagged block" + "--tags" "[\"Quote\"]" + "--properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + query-block-tags-result (run-cli ["--repo" "tags-graph" + "query" + "--query" "[:find ?tag :in $ ?title :where [?b :block/title ?title] [?b :block/tags ?t] [?t :block/title ?tag]]" + "--inputs" "[\"Tagged block\"]"] + data-dir cfg-path) + query-block-tags-payload (parse-json-output query-block-tags-result) + block-tag-names (->> (get-in query-block-tags-payload [:data :result]) + (map first) + set) + query-page-tags-result (run-cli ["--repo" "tags-graph" + "query" + "--query" "[:find ?tag :in $ ?title :where [?p :block/title ?title] [?p :block/tags ?t] [?t :block/title ?tag]]" + "--inputs" "[\"TaggedPage\"]"] + data-dir cfg-path) + query-page-tags-payload (parse-json-output query-page-tags-result) + page-tag-names (->> (get-in query-page-tags-payload [:data :result]) + (map first) + set) + query-page-result (run-cli ["--repo" "tags-graph" + "query" + "--query" "[:find ?value :in $ ?title :where [?p :block/title ?title] [?p :logseq.property/publishing-public? ?value]]" + "--inputs" "[\"TaggedPage\"]"] + data-dir cfg-path) + query-page-payload (parse-json-output query-page-result) + page-value (first (first (get-in query-page-payload [:data :result]))) + query-block-result (run-cli ["--repo" "tags-graph" + "query" + "--query" "[:find ?value :in $ ?title :where [?b :block/title ?title] [?b :logseq.property/deadline ?value]]" + "--inputs" "[\"Tagged block\"]"] + data-dir cfg-path) + query-block-payload (parse-json-output query-block-result) + block-deadline (first (first (get-in query-block-payload [:data :result]))) + stop-result (run-cli ["server" "stop" "--repo" "tags-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (contains? block-tag-names "Quote")) + (is (contains? page-tag-names "Quote")) + (is (true? page-value)) + (is (number? block-deadline)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-tags-rejects-missing-tag + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-missing")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "tags-missing-graph"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "tags-missing-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "Block with missing tag" + "--tags" "[\"MissingTag\"]"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + list-tag-result (run-cli ["--repo" "tags-missing-graph" "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + tag-names (->> (get-in list-tag-payload [:data :items]) + (map #(or (:block/title %) (:block/name %))) + set) + stop-result (run-cli ["server" "stop" "--repo" "tags-missing-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 1 (:exit-code add-block-result))) + (is (= "error" (:status add-block-payload))) + (is (not (contains? tag-names "MissingTag"))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-query (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query") From 0309553f972203f15bf64d23a642d26e8a331108 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 27 Jan 2026 19:59:40 +0800 Subject: [PATCH 045/375] 018-logseq-cli-add-tags-builtin-properties.md (2) --- ...-logseq-cli-add-tags-builtin-properties.md | 3 + .../frontend/worker/db_worker_node_lock.cljs | 14 +- src/main/logseq/cli/command/add.cljs | 221 +++++++++++++----- src/main/logseq/cli/server.cljs | 16 +- src/test/logseq/cli/commands_test.cljs | 142 ++++++----- src/test/logseq/cli/integration_test.cljs | 203 ++++++++++++---- 6 files changed, 421 insertions(+), 178 deletions(-) diff --git a/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md index 44d617761f..f9158c0b4b 100644 --- a/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md +++ b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md @@ -38,6 +38,7 @@ Add block supports setting tags on all inserted blocks. Add page supports setting tags on the created page. Add block supports setting built-in properties with correct type coercion. Add page supports setting built-in properties with correct type coercion. +Add block and add page accept tag and property references by id, :block/title, or :db/ident for --tags and --properties. CLI rejects non built-in properties for these new options. CLI rejects built-in properties that are not public and provides a clear error message. CLI rejects adding non public tags to blocks. @@ -110,6 +111,8 @@ Add new options to content-add-spec in src/main/logseq/cli/command/add.cljs for Add new options to add-page-spec in src/main/logseq/cli/command/add.cljs for tags and properties. Use a single EDN map option like --properties for multiple properties and a repeated option like --property for key value pairs if needed. Use a single EDN vector option like --tags and a repeated option like --tag for convenience. +For --tags, allow each entry to be either a tag id, :db/ident, or the tag page's :block/title. +For --properties, allow each property key to be a property id, :db/ident, or the property's :block/title. Update command summary and help output in src/main/logseq/cli/command/core.cljs if needed to reflect new options. ### 2. Tag resolution helpers diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index d272bb4b08..41cbe87711 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -26,13 +26,17 @@ [data-dir repo] (node-path/join (repo-dir data-dir repo) "db-worker.lock")) -(defn- pid-alive? +(defn- pid-status [pid] (when (number? pid) (try (.kill js/process pid 0) - true - (catch :default _ false)))) + :alive + (catch :default e + (case (.-code e) + "ESRCH" :not-found + "EPERM" :no-permission + :error))))) (defn read-lock [path] @@ -53,9 +57,9 @@ (let [data-dir (resolve-data-dir data-dir) path (lock-path data-dir repo) existing (read-lock path)] - (when (and existing (pid-alive? (:pid existing))) + (when (and existing (contains? #{:alive :no-permission} (pid-status (:pid existing)))) (throw (ex-info "graph already locked" {:code :repo-locked :lock existing}))) - (when existing + (when (and existing (= :not-found (pid-status (:pid existing)))) (remove-lock! path)) (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) (let [fd (fs/openSync path "wx") diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 46d6438308..98d4b45333 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -18,8 +18,8 @@ {:content {:desc "Block content for add"} :blocks {:desc "EDN vector of blocks for add"} :blocks-file {:desc "EDN file of blocks for add"} - :tags {:desc "EDN vector of tags"} - :properties {:desc "EDN map of built-in properties"} + :tags {:desc "EDN vector of tags (id, :db/ident, or :block/title)"} + :properties {:desc "EDN map of built-in properties (id, :db/ident, or :block/title)"} :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} @@ -29,8 +29,8 @@ (def ^:private add-page-spec {:page {:desc "Page name"} - :tags {:desc "EDN vector of tags"} - :properties {:desc "EDN map of built-in properties"}}) + :tags {:desc "EDN vector of tags (id, :db/ident, or :block/title)"} + :properties {:desc "EDN map of built-in properties (id, :db/ident, or :block/title)"}}) (def entries [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) @@ -120,19 +120,20 @@ (when (seq value) (common-util/safe-read-string {:log-error? false} value))) -(defn- keyword->name - [value] - (subs (str value) 1)) - (defn- normalize-tag-value [value] (cond (uuid? value) value + (number? value) value (and (string? value) (common-util/uuid-string? (string/trim value))) (uuid (string/trim value)) - (keyword? value) (keyword->name value) + (keyword? value) value (string? value) (let [text (-> value string/trim (string/replace #"^#+" ""))] - (when (seq text) text)) + (cond + (string/blank? text) nil + (common-util/valid-edn-keyword? text) + (common-util/safe-read-string {:log-error? false} text) + :else text)) :else nil)) (defn- parse-tags-option @@ -153,7 +154,7 @@ :else (let [tags (mapv normalize-tag-value parsed)] (if (some nil? tags) - (invalid-options-result "tags must be strings, keywords, or uuids") + (invalid-options-result "tags must be strings, keywords, uuids, or ids") {:ok? true :value tags})))))) (defn- normalize-property-key @@ -169,6 +170,39 @@ :else (keyword text))) :else nil)) +(def ^:private built-in-properties-by-title + (into {} + (keep (fn [[ident {:keys [title]}]] + (when (string? title) + [(common-util/page-name-sanity-lc title) ident]))) + db-property/built-in-properties)) + +(defn- property-title->ident + [value] + (when (string? value) + (let [text (string/trim value)] + (when (seq text) + (get built-in-properties-by-title (common-util/page-name-sanity-lc text)))))) + +(defn- normalize-property-key-input + [value] + (cond + (keyword? value) {:type :ident :value value} + (number? value) {:type :id :value value} + (string? value) + (let [text (string/trim value)] + (cond + (string/blank? text) nil + (common-util/valid-edn-keyword? text) + (let [parsed (common-util/safe-read-string {:log-error? false} text)] + (when (keyword? parsed) + {:type :ident :value parsed})) + :else + (if-let [ident (property-title->ident text)] + {:type :ident :value ident} + {:type :ident :value (keyword text)}))) + :else nil)) + (defn- parse-boolean-value [value] (cond @@ -320,27 +354,32 @@ (invalid-options-result "properties must be a non-empty map") :else - (loop [entries (seq parsed) + (loop [prop-entries (seq parsed) acc {}] - (if (empty? entries) + (if (empty? prop-entries) {:ok? true :value acc} - (let [[k v] (first entries) - key* (normalize-property-key k)] - (if-not key* + (let [[k v] (first prop-entries) + key-result (normalize-property-key-input k)] + (if-not key-result (invalid-options-result (str "invalid property key: " k)) - (let [property (get db-property/built-in-properties key*)] - (cond - (nil? property) - (invalid-options-result (str "unknown built-in property: " key*)) + (let [{:keys [type value]} key-result + key-ident value] + (if (= type :id) + (recur (rest prop-entries) (assoc acc key-ident v)) + (let [property (get db-property/built-in-properties key-ident)] + (cond + (nil? property) + (invalid-options-result (str "unknown built-in property: " key-ident)) - (not (property-public? property)) - (invalid-options-result (str "property is not public: " key*)) + (not (property-public? property)) + (invalid-options-result (str "property is not public: " key-ident)) - :else - (let [{:keys [ok? value message]} (normalize-property-values property v)] - (if-not ok? - (invalid-options-result (str "invalid value for " key* ": " message)) - (recur (rest entries) (assoc acc key* value)))))))))))))) + :else + (let [{:keys [ok? value message]} (normalize-property-values property v) + normalized-value value] + (if-not ok? + (invalid-options-result (str "invalid value for " key-ident ": " message)) + (recur (rest prop-entries) (assoc acc key-ident normalized-value)))))))))))))))) (defn invalid-options? [opts] @@ -375,8 +414,10 @@ (defn- tag-lookup-ref [tag] (cond + (number? tag) tag (uuid? tag) [:block/uuid tag] (and (string? tag) (common-util/uuid-string? (string/trim tag))) [:block/uuid (uuid (string/trim tag))] + (keyword? tag) [:db/ident tag] (string? tag) [:block/name (common-util/page-name-sanity-lc tag)] :else nil)) @@ -475,11 +516,11 @@ :class (resolve-class-id config repo value) :property (resolve-property-id config repo value) :entity (resolve-entity-id config repo (cond - (number? value) value - (uuid? value) [:block/uuid value] - (and (string? value) (common-util/uuid-string? (string/trim value))) - [:block/uuid (uuid (string/trim value))] - :else value)) + (number? value) value + (uuid? value) [:block/uuid value] + (and (string? value) (common-util/uuid-string? (string/trim value))) + [:block/uuid (uuid (string/trim value))] + :else value)) :node (resolve-node-id config repo value) :date (resolve-date-page-id config repo value) (p/resolved value)))) @@ -488,18 +529,70 @@ [config repo properties] (if-not (seq properties) (p/resolved nil) - (p/let [entries (p/all - (map (fn [[k v]] - (let [property (get db-property/built-in-properties k) - many? (= :many (get-in property [:schema :cardinality])) - values (if many? - (if (and (coll? v) (not (string? v))) v [v]) - [v])] - (p/let [resolved (p/all (map #(resolve-property-value config repo property %) values)) - final-value (if many? (vec resolved) (first resolved))] - [k final-value]))) - properties))] - (into {} entries)))) + (p/let [resolved-entries (p/all + (map (fn [[k v]] + (p/let [{:keys [ident property]} + (cond + (keyword? k) + (let [property (get db-property/built-in-properties k)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property k}))) + (p/resolved {:ident k :property property})) + + (number? k) + (p/let [entity (pull-entity config repo [:db/ident] k) + ident (:db/ident entity) + property (get db-property/built-in-properties ident)] + (cond + (nil? ident) + (throw (ex-info "property not found" + {:code :property-not-found :property k})) + + (nil? property) + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property ident})) + + (not (property-public? property)) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident})) + + :else + {:ident ident :property property})) + + (string? k) + (let [ident (or (property-title->ident k) + (normalize-property-key k)) + property (get db-property/built-in-properties ident)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident}))) + (p/resolved {:ident ident :property property})) + + :else + (p/rejected (ex-info "invalid property key" + {:code :invalid-property :property k}))) + {:keys [ok? value message]} (normalize-property-values property v)] + (when-not ok? + (throw (ex-info "invalid property value" + {:code :invalid-property-value + :property ident + :message message}))) + (let [many? (= :many (get-in property [:schema :cardinality])) + values (if many? + (if (and (coll? value) (not (string? value))) value [value]) + [value])] + (p/let [resolved (p/all (map #(resolve-property-value config repo property %) values)) + final-value (if many? (vec resolved) (first resolved))] + [ident final-value])))) + properties))] + (into {} resolved-entries)))) (defn- resolve-add-target [config {:keys [repo target-id target-uuid target-page-name]}] @@ -582,26 +675,26 @@ properties-result :else - (if-not (:ok? blocks-result) - blocks-result - (let [vector-result (ensure-blocks (:value blocks-result))] - (if-not (:ok? vector-result) - vector-result - (let [blocks (cond-> (:value vector-result) - ensure-uuids? - ensure-block-uuids)] - {:ok? true - :action {:type :add-block - :repo repo - :graph (core/repo->graph repo) - :target-id (:target-id options) - :target-uuid (some-> (:target-uuid options) string/trim) - :target-page-name (some-> (:target-page-name options) string/trim) - :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") - :status status - :tags tags - :properties properties - :blocks blocks}})))))))) + (if-not (:ok? blocks-result) + blocks-result + (let [vector-result (ensure-blocks (:value blocks-result))] + (if-not (:ok? vector-result) + vector-result + (let [blocks (cond-> (:value vector-result) + ensure-uuids? + ensure-block-uuids)] + {:ok? true + :action {:type :add-block + :repo repo + :graph (core/repo->graph repo) + :target-id (:target-id options) + :target-uuid (some-> (:target-uuid options) string/trim) + :target-page-name (some-> (:target-page-name options) string/trim) + :pos (or (some-> (:pos options) string/trim string/lower-case) "last-child") + :status status + :tags tags + :properties properties + :blocks blocks}})))))))) (defn build-add-page-action [options repo] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index dfadd11647..ac01e79d3c 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -29,13 +29,17 @@ [data-dir repo] (node-path/join (repo-dir data-dir repo) "db-worker.lock")) -(defn- pid-alive? +(defn- pid-status [pid] (when (number? pid) (try (.kill js/process pid 0) - true - (catch :default _ false)))) + :alive + (catch :default e + (case (.-code e) + "ESRCH" :not-found + "EPERM" :no-permission + :error))))) (defn- read-lock [path] @@ -117,7 +121,7 @@ (nil? lock) (p/resolved nil) - (not (pid-alive? (:pid lock))) + (= :not-found (pid-status (:pid lock))) (do (remove-lock! path) (p/resolved nil)) @@ -219,13 +223,13 @@ {:ok? true :data {:repo repo}}) (p/catch (fn [_] - (when (and (pid-alive? (:pid lock)) + (when (and (= :alive (pid-status (:pid lock))) (not= (:pid lock) (.-pid js/process))) (try (.kill js/process (:pid lock) "SIGTERM") (catch :default e (log/warn :cli-server-stop-sigterm-failed e)))) - (when-not (pid-alive? (:pid lock)) + (when (= :not-found (pid-status (:pid lock))) (remove-lock! path)) (if (fs/existsSync path) {:ok? false diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 3b2f94877c..2adad4054b 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.commands-test (:require [cljs.test :refer [async deftest is testing]] [clojure.string :as string] + [logseq.cli.command.add :as add-command] [logseq.cli.command.show :as show-command] [logseq.cli.commands :as commands] [logseq.cli.server :as cli-server] @@ -402,6 +403,63 @@ (is (= :invalid-options (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-add-remove + (testing "remove requires target" + (let [result (commands/parse-args ["remove"])] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "remove parses with id" + (let [result (commands/parse-args ["remove" "--id" "10"])] + (is (true? (:ok? result))) + (is (= :remove (:command result))) + (is (= 10 (get-in result [:options :id]))))) + + (testing "remove parses with uuid" + (let [result (commands/parse-args ["remove" "--uuid" "abc"])] + (is (true? (:ok? result))) + (is (= :remove (:command result))) + (is (= "abc" (get-in result [:options :uuid]))))) + + (testing "remove parses with page" + (let [result (commands/parse-args ["remove" "--page" "Home"])] + (is (true? (:ok? result))) + (is (= :remove (:command result))) + (is (= "Home" (get-in result [:options :page]))))) + + (testing "remove rejects multiple selectors" + (let [result (commands/parse-args ["remove" "--id" "1" "--page" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "remove rejects empty id vector" + (let [result (commands/parse-args ["remove" "--id" "[]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "remove rejects invalid id vector" + (let [result (commands/parse-args ["remove" "--id" "[1 \"no\"]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "move requires source selector" + (let [result (commands/parse-args ["move" "--target-id" "10"])] + (is (false? (:ok? result))) + (is (= :missing-source (get-in result [:error :code]))))) + + (testing "move requires target selector" + (let [result (commands/parse-args ["move" "--id" "1"])] + (is (false? (:ok? result))) + (is (= :missing-target (get-in result [:error :code]))))) + + (testing "move parses with source and target" + (let [result (commands/parse-args ["move" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] + (is (true? (:ok? result))) + (is (= :move-block (:command result))) + (is (= "abc" (get-in result [:options :uuid]))) + (is (= "def" (get-in result [:options :target-uuid]))) + (is (= "last-child" (get-in result [:options :pos])))))) + +(deftest test-verb-subcommand-parse-add (testing "add block requires content source" (let [result (commands/parse-args ["add" "block"])] (is (false? (:ok? result))) @@ -473,63 +531,7 @@ (is (true? (:ok? result))) (is (= :add-page (:command result))) (is (= "[\"TagA\"]" (get-in result [:options :tags]))) - (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) - - (testing "remove requires target" - (let [result (commands/parse-args ["remove"])] - (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code]))))) - - (testing "remove parses with id" - (let [result (commands/parse-args ["remove" "--id" "10"])] - (is (true? (:ok? result))) - (is (= :remove (:command result))) - (is (= 10 (get-in result [:options :id]))))) - - (testing "remove parses with uuid" - (let [result (commands/parse-args ["remove" "--uuid" "abc"])] - (is (true? (:ok? result))) - (is (= :remove (:command result))) - (is (= "abc" (get-in result [:options :uuid]))))) - - (testing "remove parses with page" - (let [result (commands/parse-args ["remove" "--page" "Home"])] - (is (true? (:ok? result))) - (is (= :remove (:command result))) - (is (= "Home" (get-in result [:options :page]))))) - - (testing "remove rejects multiple selectors" - (let [result (commands/parse-args ["remove" "--id" "1" "--page" "Home"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) - - (testing "remove rejects empty id vector" - (let [result (commands/parse-args ["remove" "--id" "[]"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) - - (testing "remove rejects invalid id vector" - (let [result (commands/parse-args ["remove" "--id" "[1 \"no\"]"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) - - (testing "move requires source selector" - (let [result (commands/parse-args ["move" "--target-id" "10"])] - (is (false? (:ok? result))) - (is (= :missing-source (get-in result [:error :code]))))) - - (testing "move requires target selector" - (let [result (commands/parse-args ["move" "--id" "1"])] - (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code]))))) - - (testing "move parses with source and target" - (let [result (commands/parse-args ["move" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] - (is (true? (:ok? result))) - (is (= :move-block (:command result))) - (is (= "abc" (get-in result [:options :uuid]))) - (is (= "def" (get-in result [:options :target-uuid]))) - (is (= "last-child" (get-in result [:options :pos])))))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties])))))) (deftest test-verb-subcommand-parse-move-target-page (testing "move parses with target page" @@ -775,6 +777,15 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) + (testing "add block accepts property title key" + (let [parsed (commands/parse-args ["add" "block" + "--content" "hello" + "--properties" "{\"Publishing Public?\" true}"]) + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :logseq.property/publishing-public? + (-> result :action :properties keys first))))) + (testing "add block rejects non-public built-in property" (let [parsed (commands/parse-args ["add" "block" "--content" "hello" @@ -791,6 +802,23 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-build-action-add-accepts-tag-ids + (testing "add block accepts numeric tag ids" + (let [parsed (commands/parse-args ["add" "block" + "--content" "hello" + "--tags" "[42]"]) + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= [42] (get-in result [:action :tags])))))) + +(deftest test-tag-lookup-ref-accepts-id + (let [tag-lookup-ref #'add-command/tag-lookup-ref] + (is (= 42 (tag-lookup-ref 42))))) + +(deftest test-normalize-property-key-input-accepts-id + (let [normalize-property-key-input #'add-command/normalize-property-key-input] + (is (= {:type :id :value 42} (normalize-property-key-input 42))))) + (deftest test-build-action-move (testing "move requires source selector" (let [parsed {:ok? true :command :move-block :options {:target-id 2}} diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index bc42047d4e..d3b4b626cc 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -6,6 +6,7 @@ [clojure.string :as string] [frontend.test.node-helper :as node-helper] [logseq.cli.main :as cli-main] + [logseq.db.frontend.property :as db-property] [promesa.core :as p])) (defn- run-cli @@ -42,6 +43,14 @@ [node] (or (:block/children node) (:children node))) +(defn- item-id + [item] + (or (:db/id item) (:id item))) + +(defn- item-title + [item] + (or (:block/title item) (:block/name item) (:title item) (:name item))) + (defn- find-block-by-title [node title] (when node @@ -49,6 +58,55 @@ node (some #(find-block-by-title % title) (node-children node))))) +(defn- setup-tags-graph + [data-dir] + (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "tags-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "tags-graph" "add" "page" "--page" "Home"] data-dir cfg-path)] + {:cfg-path cfg-path :repo "tags-graph"})) + +(defn- stop-repo! + [data-dir cfg-path repo] + (p/let [result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path)] + (parse-json-output result))) + +(defn- run-query + [data-dir cfg-path repo query inputs] + (p/let [result (run-cli ["--repo" repo "query" "--query" query "--inputs" inputs] + data-dir cfg-path)] + (parse-json-output result))) + +(defn- query-tags + [data-dir cfg-path repo title] + (p/let [payload (run-query data-dir cfg-path repo + "[:find ?tag :in $ ?title :where [?b :block/title ?title] [?b :block/tags ?t] [?t :block/title ?tag]]" + (pr-str [title]))] + (->> (get-in payload [:data :result]) + (map first) + set))) + +(defn- query-property + [data-dir cfg-path repo title property] + (p/let [payload (run-query data-dir cfg-path repo + (str "[:find ?value :in $ ?title :where [?e :block/title ?title] [?e " + property + " ?value]]") + (pr-str [title]))] + (first (first (get-in payload [:data :result]))))) + +(defn- list-items + [data-dir cfg-path repo list-type] + (p/let [result (run-cli ["--repo" repo "list" list-type] data-dir cfg-path)] + (parse-json-output result))) + +(defn- find-item-id + [items title] + (->> items + (some (fn [item] + (when (= title (item-title item)) item))) + item-id)) + (deftest test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -127,13 +185,10 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-tags-and-properties +(deftest test-cli-add-tags-and-properties-by-name (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags")] - (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "tags-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "tags-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) add-page-result (run-cli ["--repo" "tags-graph" "add" "page" "--page" "TaggedPage" @@ -149,49 +204,105 @@ "--properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) - _ (p/delay 100) - query-block-tags-result (run-cli ["--repo" "tags-graph" - "query" - "--query" "[:find ?tag :in $ ?title :where [?b :block/title ?title] [?b :block/tags ?t] [?t :block/title ?tag]]" - "--inputs" "[\"Tagged block\"]"] - data-dir cfg-path) - query-block-tags-payload (parse-json-output query-block-tags-result) - block-tag-names (->> (get-in query-block-tags-payload [:data :result]) - (map first) - set) - query-page-tags-result (run-cli ["--repo" "tags-graph" - "query" - "--query" "[:find ?tag :in $ ?title :where [?p :block/title ?title] [?p :block/tags ?t] [?t :block/title ?tag]]" - "--inputs" "[\"TaggedPage\"]"] + add-block-ident-result (run-cli ["--repo" "tags-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "Tagged block ident" + "--tags" "[:logseq.class/Quote-block]"] data-dir cfg-path) - query-page-tags-payload (parse-json-output query-page-tags-result) - page-tag-names (->> (get-in query-page-tags-payload [:data :result]) - (map first) - set) - query-page-result (run-cli ["--repo" "tags-graph" - "query" - "--query" "[:find ?value :in $ ?title :where [?p :block/title ?title] [?p :logseq.property/publishing-public? ?value]]" - "--inputs" "[\"TaggedPage\"]"] - data-dir cfg-path) - query-page-payload (parse-json-output query-page-result) - page-value (first (first (get-in query-page-payload [:data :result]))) - query-block-result (run-cli ["--repo" "tags-graph" - "query" - "--query" "[:find ?value :in $ ?title :where [?b :block/title ?title] [?b :logseq.property/deadline ?value]]" - "--inputs" "[\"Tagged block\"]"] + add-block-ident-payload (parse-json-output add-block-ident-result) + deadline-prop-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) + publishing-prop-title (get-in db-property/built-in-properties [:logseq.property/publishing-public? :title]) + add-page-title-result (run-cli ["--repo" "tags-graph" + "add" "page" + "--page" "TaggedPageTitle" + "--properties" (str "{\"" publishing-prop-title "\" true}")] + data-dir cfg-path) + add-page-title-payload (parse-json-output add-page-title-result) + add-block-title-result (run-cli ["--repo" "tags-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "Tagged block title" + "--properties" (str "{\"" deadline-prop-title "\" \"2026-01-25T12:00:00Z\"}")] + data-dir cfg-path) + add-block-title-payload (parse-json-output add-block-title-result) + _ (p/delay 100) + block-tag-names (query-tags data-dir cfg-path repo "Tagged block") + block-ident-tag-names (query-tags data-dir cfg-path repo "Tagged block ident") + page-tag-names (query-tags data-dir cfg-path repo "TaggedPage") + page-value (query-property data-dir cfg-path repo "TaggedPage" ":logseq.property/publishing-public?") + page-title-value (query-property data-dir cfg-path repo "TaggedPageTitle" ":logseq.property/publishing-public?") + block-deadline (query-property data-dir cfg-path repo "Tagged block" ":logseq.property/deadline") + block-deadline-title (query-property data-dir cfg-path repo "Tagged block title" ":logseq.property/deadline") + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (= 0 (:exit-code add-block-ident-result))) + (is (= "ok" (:status add-block-ident-payload))) + (is (string? deadline-prop-title)) + (is (string? publishing-prop-title)) + (is (= 0 (:exit-code add-page-title-result))) + (is (= "ok" (:status add-page-title-payload))) + (is (= 0 (:exit-code add-block-title-result))) + (is (= "ok" (:status add-block-title-payload))) + (is (contains? block-tag-names "Quote")) + (is (contains? block-ident-tag-names "Quote")) + (is (contains? page-tag-names "Quote")) + (is (true? page-value)) + (is (true? page-title-value)) + (is (number? block-deadline)) + (is (number? block-deadline-title)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-tags-and-properties-by-id + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-id")] + (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) + list-tag-payload (list-items data-dir cfg-path repo "tag") + quote-tag-id (find-item-id (get-in list-tag-payload [:data :items]) "Quote") + list-property-payload (list-items data-dir cfg-path repo "property") + deadline-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) + publishing-title (get-in db-property/built-in-properties [:logseq.property/publishing-public? :title]) + deadline-id (find-item-id (get-in list-property-payload [:data :items]) deadline-title) + publishing-id (find-item-id (get-in list-property-payload [:data :items]) publishing-title) + add-page-id-result (run-cli ["--repo" repo + "add" "page" + "--page" "TaggedPageId" + "--tags" (pr-str [quote-tag-id]) + "--properties" (pr-str {publishing-id true})] data-dir cfg-path) - query-block-payload (parse-json-output query-block-result) - block-deadline (first (first (get-in query-block-payload [:data :result]))) - stop-result (run-cli ["server" "stop" "--repo" "tags-graph"] data-dir cfg-path) - stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code add-page-result))) - (is (= "ok" (:status add-page-payload))) - (is (= 0 (:exit-code add-block-result))) - (is (= "ok" (:status add-block-payload))) - (is (contains? block-tag-names "Quote")) - (is (contains? page-tag-names "Quote")) - (is (true? page-value)) - (is (number? block-deadline)) + add-page-id-payload (parse-json-output add-page-id-result) + add-block-id-result (run-cli ["--repo" repo + "add" "block" + "--target-page-name" "Home" + "--content" "Tagged block id" + "--tags" (pr-str [quote-tag-id]) + "--properties" (pr-str {deadline-id "2026-01-25T12:00:00Z"})] + data-dir cfg-path) + add-block-id-payload (parse-json-output add-block-id-result) + _ (p/delay 100) + page-id-value (query-property data-dir cfg-path repo "TaggedPageId" ":logseq.property/publishing-public?") + block-deadline-id (query-property data-dir cfg-path repo "Tagged block id" ":logseq.property/deadline") + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= "ok" (:status list-tag-payload))) + (is (number? quote-tag-id)) + (is (= "ok" (:status list-property-payload))) + (is (number? deadline-id)) + (is (number? publishing-id)) + (is (= 0 (:exit-code add-page-id-result)) + (pr-str (:error add-page-id-payload))) + (is (= "ok" (:status add-page-id-payload))) + (is (= 0 (:exit-code add-block-id-result)) + (pr-str (:error add-block-id-payload))) + (is (= "ok" (:status add-block-id-payload))) + (is (true? page-id-value)) + (is (number? block-deadline-id)) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From 3f68b543e3bee46776049b4502a7a78a65e6d394 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 28 Jan 2026 14:31:28 +0800 Subject: [PATCH 046/375] 019-logseq-cli-data-dir-permissions.md --- .../019-logseq-cli-data-dir-permissions.md | 82 +++++++++ src/main/frontend/worker/db_worker_node.cljs | 155 ++++++++++-------- src/main/logseq/cli/data_dir.cljs | 42 +++++ src/main/logseq/cli/format.cljs | 1 + src/main/logseq/cli/main.cljs | 83 ++++++---- src/main/logseq/cli/server.cljs | 23 +++ .../frontend/worker/db_worker_node_test.cljs | 15 ++ src/test/logseq/cli/data_dir_test.cljs | 41 +++++ src/test/logseq/cli/integration_test.cljs | 40 +++++ 9 files changed, 379 insertions(+), 103 deletions(-) create mode 100644 docs/agent-guide/019-logseq-cli-data-dir-permissions.md create mode 100644 src/main/logseq/cli/data_dir.cljs create mode 100644 src/test/logseq/cli/data_dir_test.cljs diff --git a/docs/agent-guide/019-logseq-cli-data-dir-permissions.md b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md new file mode 100644 index 0000000000..51dd430f41 --- /dev/null +++ b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md @@ -0,0 +1,82 @@ +# Logseq CLI Data-dir Permission Checks Plan + +Goal: Make logseq-cli validate read/write access for `data-dir` before it tries to start or communicate with db-worker-node, and surface a clear error when permissions are missing. + +Architecture: The CLI resolves `data-dir` in `logseq.cli.config` and uses it via `logseq.cli.server` to spawn and manage `db-worker-node`. `db-worker-node` also depends on `data-dir` for logs, locks, and SQLite storage via `frontend.worker.platform.node`. + +Tech Stack: ClojureScript, Node.js fs APIs, promesa, logseq-cli, db-worker-node. + +## Problem statement + +`data-dir` is used for locks, logs, and the local SQLite DB. Today, permission issues show up as late runtime exceptions (e.g., during lock creation or log file writes) with unclear error output. The CLI should proactively check that `data-dir` is readable and writable and return a clear error before attempting db-worker-node actions. + +## Current behavior summary + +- `logseq.cli.config/resolve-config` defaults `:data-dir` to `~/.logseq/cli-graphs`. +- `logseq.cli.server` resolves and uses `data-dir` for locks and server discovery. +- `frontend.worker.db-worker-node` writes logs and lock files under `data-dir` and delegates storage to `frontend.worker.platform.node`, which creates directories as needed. +- No explicit read/write permission checks exist; errors bubble up from fs operations. + +## Requirements + +- CLI validates that `data-dir` is a directory with read and write permission. +- If `data-dir` does not exist, CLI attempts to create it (recursive). If creation or access fails, CLI returns an error. +- CLI surfaces a clear error code and message that includes the failing path. +- The check must run before any db-worker-node lifecycle or graph access that relies on `data-dir`. + +## Non-goals + +- Do not change db-worker-node storage layout or lock format. +- Do not add new CLI options for data-dir. +- Do not change API server behavior. + +## Design decisions + +- Treat `data-dir` as required to be read/write for all local-graph CLI commands. +- Convert permission failures into a consistent CLI error code (e.g., `:data-dir-permission`) and message. +- Reuse the same permission check in db-worker-node entrypoint to guard direct invocation. + +## Implementation plan + +### 1) Add a data-dir permission helper + +- Create a helper namespace (e.g., `src/main/logseq/cli/data_dir.cljs`) that: + - Expands `~` and normalizes the path. + - If missing, attempts `fs.mkdirSync` with `{:recursive true}`. + - Verifies the path is a directory (`fs.statSync`). + - Verifies read/write access with `fs.accessSync` (R_OK | W_OK). + - Throws `ex-info` with `{:code :data-dir-permission :path :cause }` on failure. + +### 2) Wire the check into CLI flow + +- In `src/main/logseq/cli/main.cljs`, after `config/resolve-config`, call the permission helper before `commands/build-action`/`commands/execute`. +- If the CLI supports API-token-only commands that do not touch local graphs, gate the check to only run for actions that require local graph access or server management. +- Map thrown permission errors into CLI error output with a clear message (e.g., "data-dir is not readable/writable: "). + +### 3) Add a safety check in db-worker-node + +- In `src/main/frontend/worker/db_worker_node.cljs`, run the same permission helper (or a small local equivalent) before `install-file-logger!` and before `platform-node/node-platform`. +- When this check fails, print a concise error to stderr and exit with code 1 to avoid partial startup. + +### 4) Update CLI error formatting + +- In `src/main/logseq/cli/format.cljs`, add an error hint for `:data-dir-permission` (e.g., "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR"). +- Ensure error output contains the path and permission type (read/write). + +### 5) Tests + +- Add unit tests in `src/test/logseq/cli` for the permission helper: + - Non-existent path that can be created succeeds. + - Path that is a file (not directory) fails. + - Read-only directory fails (use chmod to remove write permission in tmp dir). +- Add an integration test that runs CLI with `--data-dir` pointing to a non-writable directory and asserts the CLI returns error code `:data-dir-permission`. +- Add a graph-create case where `graph-dir` cannot be created (no mkdir permission) and assert a clear error is returned. +- Add a db-worker-node test (if there is a suitable harness) or extend existing CLI integration tests to assert db-worker-node start fails fast with the new error. + +## Open questions + +Resolved: +- Always check `data-dir` permissions, even when an API-server token is provided. +- Only create `data-dir` when a command needs it (local graph or server operations), not eagerly for all commands. + +--- diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 36775f6121..9b92a70b4a 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -8,6 +8,7 @@ [frontend.worker.platform.node :as platform-node] [frontend.worker.state :as worker-state] [lambdaisland.glogi :as log] + [logseq.cli.data-dir :as data-dir] [logseq.db :as ldb] [promesa.core :as p])) @@ -300,65 +301,68 @@ port 0] (if-not (seq repo) (p/rejected (ex-info "repo is required" {:code :missing-repo})) - (do - (install-file-logger! {:data-dir data-dir - :repo repo - :log-level (keyword (or log-level "info"))}) - (reset! *ready? false) - (set-main-thread-stub!) - (-> (p/let [platform (platform-node/node-platform {:data-dir data-dir - :event-fn handle-event!}) - proxy (db-core/init-core! platform) - _ ( (p/let [platform (platform-node/node-platform {:data-dir data-dir + :event-fn handle-event!}) + proxy (db-core/init-core! platform) + _ ( (stop!) - (p/finally (fn [] - (log/info :db-worker-node-stopped nil) - (.exit js/process 0)))))] - (.on js/process "SIGINT" shutdown) - (.on js/process "SIGTERM" shutdown))))) + (-> (p/let [{:keys [stop!] :as daemon} + (start-daemon! {:data-dir data-dir + :repo repo + :rtc-ws-url rtc-ws-url + :log-level (:log-level opts)})] + (log/info :db-worker-node-ready {:host (:host daemon) :port (:port daemon)}) + (let [shutdown (fn [] + (-> (stop!) + (p/finally (fn [] + (log/info :db-worker-node-stopped nil) + (.exit js/process 0)))))] + (.on js/process "SIGINT" shutdown) + (.on js/process "SIGTERM" shutdown))) + (p/catch (fn [error] + (let [data (ex-data error) + message (or (.-message error) (str error))] + (when (= :data-dir-permission (:code data)) + (.error js/console message) + (.exit js/process 1)) + (throw error))))))) diff --git a/src/main/logseq/cli/data_dir.cljs b/src/main/logseq/cli/data_dir.cljs new file mode 100644 index 0000000000..1c9b5d71e1 --- /dev/null +++ b/src/main/logseq/cli/data_dir.cljs @@ -0,0 +1,42 @@ +(ns logseq.cli.data-dir + "Data-dir validation and normalization for the CLI and db-worker-node." + (:require ["fs" :as fs] + ["os" :as os] + ["path" :as node-path] + [clojure.string :as string])) + +(def ^:private default-data-dir "~/.logseq/cli-graphs") + +(defn- expand-home + [path] + (if (and (seq path) (string/starts-with? path "~")) + (node-path/join (.homedir os) (subs path 1)) + path)) + +(defn normalize-data-dir + [path] + (node-path/resolve (expand-home (or path default-data-dir)))) + +(defn ensure-data-dir! + [path] + (let [path (normalize-data-dir path)] + (try + (when-not (fs/existsSync path) + (fs/mkdirSync path #js {:recursive true})) + (let [stat (fs/statSync path)] + (when-not (.isDirectory stat) + (throw (ex-info (str "data-dir is not a directory: " path) + {:code :data-dir-permission + :path path + :cause "ENOTDIR"})))) + (let [constants (.-constants fs) + mode (bit-or (.-R_OK constants) (.-W_OK constants))] + (fs/accessSync path mode)) + path + (catch :default e + (if (= :data-dir-permission (:code (ex-data e))) + (throw e) + (throw (ex-info (str "data-dir is not readable/writable: " path) + {:code :data-dir-permission + :path path + :cause (.-code e)}))))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 4806bb3974..f44f82d110 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -88,6 +88,7 @@ :missing-content "Use --content or pass content as args" :missing-query "Use --query " :unknown-query "Use `logseq query list` to see available queries" + :data-dir-permission "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR" nil)) (defn- format-error diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 903ecc4362..a0987d83d1 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -4,6 +4,7 @@ (:require [clojure.string :as string] [logseq.cli.commands :as commands] [logseq.cli.config :as config] + [logseq.cli.data-dir :as data-dir] [logseq.cli.format :as format] [logseq.cli.version :as version] [promesa.core :as p])) @@ -39,39 +40,59 @@ :output (version/format-version)}) :else - (let [cfg (config/resolve-config (:options parsed)) - action-result (commands/build-action parsed cfg)] - (if-not (:ok? action-result) - (p/resolved {:exit-code 1 - :output (format/format-result {:status :error - :error (:error action-result) - :command (:command parsed) - :context (select-keys (:options parsed) - [:repo :graph :page :block])} - cfg)}) - (-> (commands/execute (:action action-result) cfg) - (p/then (fn [result] - (let [opts (cond-> cfg - (:output-format result) - (assoc :output-format (:output-format result)))] - {:exit-code 0 - :output (format/format-result result opts)}))) - (p/catch (fn [error] - (let [data (ex-data error) - message (cond - (and (= :http-error (:code data)) (seq (:body data))) - (str "http request failed (" (:status data) "): " (:body data)) + (let [cfg (config/resolve-config (:options parsed))] + (try + (let [cfg (assoc cfg :data-dir (data-dir/ensure-data-dir! (:data-dir cfg))) + action-result (commands/build-action parsed cfg)] + (if-not (:ok? action-result) + (p/resolved {:exit-code 1 + :output (format/format-result {:status :error + :error (:error action-result) + :command (:command parsed) + :context (select-keys (:options parsed) + [:repo :graph :page :block])} + cfg)}) + (-> (commands/execute (:action action-result) cfg) + (p/then (fn [result] + (let [opts (cond-> cfg + (:output-format result) + (assoc :output-format (:output-format result)))] + {:exit-code 0 + :output (format/format-result result opts)}))) + (p/catch (fn [error] + (let [data (ex-data error) + message (cond + (and (= :http-error (:code data)) (seq (:body data))) + (str "http request failed (" (:status data) "): " (:body data)) - (some? (:message data)) - (:message data) + (some? (:message data)) + (:message data) - :else - (or (.-message error) (str error)))] - {:exit-code 1 - :output (format/format-result {:status :error - :error {:code :exception - :message message}} - cfg)})))))))))) + :else + (or (.-message error) (str error)))] + (if (= :data-dir-permission (:code data)) + {:exit-code 1 + :output (format/format-result {:status :error + :error {:code :data-dir-permission + :message message + :path (:path data)}} + cfg)} + {:exit-code 1 + :output (format/format-result {:status :error + :error {:code :exception + :message message}} + cfg)}))))))) + (catch :default error + (let [data (ex-data error) + message (or (.-message error) (str error))] + (if (= :data-dir-permission (:code data)) + (p/resolved {:exit-code 1 + :output (format/format-result {:status :error + :error {:code :data-dir-permission + :message message + :path (:path data)}} + cfg)}) + (throw error)))))))))) (defn main [& args] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index ac01e79d3c..da44445acf 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -25,6 +25,28 @@ [data-dir repo] (node-path/join data-dir (worker-util/encode-graph-dir-name repo))) +(defn- ensure-repo-dir! + [data-dir repo] + (let [path (repo-dir data-dir repo)] + (try + (when-not (fs/existsSync path) + (fs/mkdirSync path #js {:recursive true})) + (let [stat (fs/statSync path)] + (when-not (.isDirectory stat) + (throw (ex-info (str "graph-dir is not a directory: " path) + {:code :data-dir-permission + :path path + :cause "ENOTDIR"})))) + (let [constants (.-constants fs) + mode (bit-or (.-R_OK constants) (.-W_OK constants))] + (fs/accessSync path mode)) + path + (catch :default e + (throw (ex-info (str "graph-dir is not readable/writable: " path) + {:code :data-dir-permission + :path path + :cause (.-code e)})))))) + (defn lock-path [data-dir repo] (node-path/join (repo-dir data-dir repo) "db-worker.lock")) @@ -180,6 +202,7 @@ [config repo] (let [data-dir (resolve-data-dir config) path (lock-path data-dir repo)] + (ensure-repo-dir! data-dir repo) (p/let [existing (read-lock path) _ (cleanup-stale-lock! path existing) _ (when (not (fs/existsSync path)) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 4b61784e98..2cbfd92982 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -95,6 +95,21 @@ date-str (yyyymmdd (js/Date.))] (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) +(deftest db-worker-node-data-dir-permission-error + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly") + repo (str "logseq_db_perm_" (subs (str (random-uuid)) 0 8))] + (fs/chmodSync data-dir 365) + (-> (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + (p/then (fn [_] + (is false "expected data-dir permission error"))) + (p/catch (fn [e] + (let [data (ex-data e)] + (is (= :data-dir-permission (:code data))) + (is (= (node-path/resolve data-dir) (:path data)))))) + (p/finally (fn [] (done))))))) + (deftest db-worker-node-creates-log-file (async done (let [daemon (atom nil) diff --git a/src/test/logseq/cli/data_dir_test.cljs b/src/test/logseq/cli/data_dir_test.cljs new file mode 100644 index 0000000000..2720459357 --- /dev/null +++ b/src/test/logseq/cli/data_dir_test.cljs @@ -0,0 +1,41 @@ +(ns logseq.cli.data-dir-test + (:require ["fs" :as fs] + ["path" :as node-path] + [cljs.test :refer [deftest is testing]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.data-dir :as data-dir])) + +(deftest ensure-data-dir-creates-missing-dir + (testing "creates missing directories and returns normalized path" + (let [base (node-helper/create-tmp-dir "data-dir") + target (node-path/join base "nested" "dir")] + (is (not (fs/existsSync target))) + (let [resolved (data-dir/ensure-data-dir! target)] + (is (fs/existsSync target)) + (is (.isDirectory (fs/statSync target))) + (is (= (node-path/resolve target) resolved)))))) + +(deftest ensure-data-dir-rejects-file-path + (testing "rejects paths that are files" + (let [base (node-helper/create-tmp-dir "data-dir-file") + target (node-path/join base "file.txt")] + (fs/writeFileSync target "x") + (try + (data-dir/ensure-data-dir! target) + (is false "expected data-dir permission error") + (catch :default e + (let [data (ex-data e)] + (is (= :data-dir-permission (:code data))) + (is (= (node-path/resolve target) (:path data))))))))) + +(deftest ensure-data-dir-rejects-read-only-dir + (testing "rejects directories without write permission" + (let [target (node-helper/create-tmp-dir "data-dir-readonly")] + (fs/chmodSync target 365) + (try + (data-dir/ensure-data-dir! target) + (is false "expected data-dir permission error") + (catch :default e + (let [data (ex-data e)] + (is (= :data-dir-permission (:code data))) + (is (= (node-path/resolve target) (:path data))))))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index d3b4b626cc..316da209e4 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -4,7 +4,9 @@ [cljs.reader :as reader] [cljs.test :refer [deftest is async]] [clojure.string :as string] + [frontend.worker-common.util :as worker-util] [frontend.test.node-helper :as node-helper] + [logseq.cli.command.core :as command-core] [logseq.cli.main :as cli-main] [logseq.db.frontend.property :as db-property] [promesa.core :as p])) @@ -121,6 +123,44 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-data-dir-permission-error + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly")] + (fs/chmodSync data-dir 365) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + result (run-cli ["graph" "list"] data-dir cfg-path) + payload (parse-json-output result)] + (is (= 1 (:exit-code result))) + (is (= "error" (:status payload))) + (is (= "data-dir-permission" (get-in payload [:error :code]))) + (is (string/includes? (get-in payload [:error :message]) data-dir)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-graph-create-readonly-graph-dir + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-graph-readonly") + repo "readonly-graph" + repo-id (command-core/resolve-repo repo) + repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo-id))] + (fs/mkdirSync repo-dir #js {:recursive true}) + (fs/chmodSync repo-dir 365) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + result (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + payload (parse-json-output result)] + (is (= 1 (:exit-code result))) + (is (= "error" (:status payload))) + (is (= "data-dir-permission" (get-in payload [:error :code]))) + (is (string/includes? (get-in payload [:error :message]) repo-dir)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-graph-create-and-info (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From e6f20fe622ba31ff9116bcd927b16fd54a0f09a3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 28 Jan 2026 17:34:28 +0800 Subject: [PATCH 047/375] 020-logseq-cli-default-paths-move.md --- .../agent-guide/002-logseq-cli-subcommands.md | 2 +- .../003-db-worker-node-cli-orchestration.md | 4 +- .../012-logseq-cli-graph-storage.md | 8 +- .../019-logseq-cli-data-dir-permissions.md | 2 +- .../020-logseq-cli-default-paths-move.md | 77 +++++++++++++++++++ .../task--db-worker-nodejs-compatible.md | 2 +- docs/cli/logseq-cli.md | 6 +- src/main/frontend/worker/db_worker_node.cljs | 2 +- .../frontend/worker/db_worker_node_lock.cljs | 2 +- src/main/frontend/worker/platform/node.cljs | 2 +- src/main/logseq/cli/command/add.cljs | 2 +- src/main/logseq/cli/command/core.cljs | 8 +- src/main/logseq/cli/command/list.cljs | 2 +- src/main/logseq/cli/command/move.cljs | 2 +- src/main/logseq/cli/command/show.cljs | 2 +- src/main/logseq/cli/config.cljs | 4 +- src/main/logseq/cli/data_dir.cljs | 2 +- src/main/logseq/cli/server.cljs | 2 +- src/test/logseq/cli/config_test.cljs | 7 ++ src/test/logseq/cli/data_dir_test.cljs | 7 ++ 20 files changed, 120 insertions(+), 25 deletions(-) create mode 100644 docs/agent-guide/020-logseq-cli-default-paths-move.md diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index eb9b1fbbe8..fdc829537e 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -56,7 +56,7 @@ Global options apply to all subcommands and are parsed before subcommand options | --- | --- | --- | | --help | Show help | Available at top level and per subcommand. | | --version | Show version | Prints build time and revision. | -| --config PATH | Config file path | Defaults to ~/.logseq/cli.edn. | +| --config PATH | Config file path | Defaults to ~/logseq/cli.edn. | | --repo REPO | Graph name | Used as current repo. | | --timeout-ms MS | Request timeout | Integer milliseconds. | | --output FORMAT | Output format | One of human, json, edn. | diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index e699f5bc84..eb88386fb9 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -44,7 +44,7 @@ Key changes: - Reject `thread-api/create-or-open-db`, `thread-api/unsafe-unlink-db`, etc. when repo differs. - Return 409/400 with `:repo-mismatch` error shape. - **Lock file**: - - Location: inside repo dir (e.g. `~/.logseq/cli-graphs//db-worker.lock`). + - Location: inside repo dir (e.g. `~/logseq/cli-graphs//db-worker.lock`). - Content: JSON `{repo, pid, host, port, startedAt}`. - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”. - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM. @@ -108,7 +108,7 @@ Implementation notes: - Answer: No --repo needed, using 'out-of-band access to data-dir' way 2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). - lockfile name:`db-worker.lock`, - - Location: inside repo dir (e.g. `~/.logseq/cli-graphs//db-worker.lock`). + - Location: inside repo dir (e.g. `~/logseq/cli-graphs//db-worker.lock`). - only consider linux/macos for now 3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle. 4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI? diff --git a/docs/agent-guide/012-logseq-cli-graph-storage.md b/docs/agent-guide/012-logseq-cli-graph-storage.md index d210dfc21f..c044f8bc84 100644 --- a/docs/agent-guide/012-logseq-cli-graph-storage.md +++ b/docs/agent-guide/012-logseq-cli-graph-storage.md @@ -3,12 +3,12 @@ ## Context logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logseq/db-worker/` and use per-graph directories named like `.logseq-pool-/` (with partial encoding). This plan captures the non-functional updates requested: -1) Rename `~/.logseq/db-worker/` to `~/.logseq/cli-graphs/` for CLI-managed graphs. +1) Rename `~/.logseq/db-worker/` to `~/logseq/cli-graphs/` for CLI-managed graphs. 2) Rename per-graph directory format from `.logseq-pool-/` to `/`. 3) Ensure graph names that are not valid directory names are encoded, and decoding is symmetric when reading/listing. ## Goals -- Move CLI graph storage to `~/.logseq/cli-graphs` by default. +- Move CLI graph storage to `~/logseq/cli-graphs` by default. - Use a clean per-graph directory name equal to the (encoded) graph name, without `.` prefix or `logseq-pool-` prefix. - Provide a reversible encode/decode for graph names so list/read operations reconstruct the original graph name. - CLI commands and outputs should hide the internal `logseq_db_` prefix; user-facing graph names strip that db prefix. @@ -29,7 +29,7 @@ logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logse ## Proposed Approach ### 1) New default data dir -- Change default data dir for CLI and db-worker-node from `~/.logseq/db-worker` to `~/.logseq/cli-graphs`. +- Change default data dir for CLI and db-worker-node from `~/.logseq/db-worker` to `~/logseq/cli-graphs`. - Update help text and any user-facing docs mentioning the old default. ### 2) New per-graph directory naming @@ -51,7 +51,7 @@ logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logse - Files: `src/main/frontend/worker_common/util.cljc`, potentially `deps/cli/src/logseq/cli/common/graph.cljs`. 2) Update data dir defaults - - Change defaults to `~/.logseq/cli-graphs` in: + - Change defaults to `~/logseq/cli-graphs` in: - `src/main/logseq/cli/config.cljs` - `src/main/logseq/cli/server.cljs` - `src/main/frontend/worker/db_worker_node_lock.cljs` diff --git a/docs/agent-guide/019-logseq-cli-data-dir-permissions.md b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md index 51dd430f41..9819e53edf 100644 --- a/docs/agent-guide/019-logseq-cli-data-dir-permissions.md +++ b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md @@ -12,7 +12,7 @@ Tech Stack: ClojureScript, Node.js fs APIs, promesa, logseq-cli, db-worker-node. ## Current behavior summary -- `logseq.cli.config/resolve-config` defaults `:data-dir` to `~/.logseq/cli-graphs`. +- `logseq.cli.config/resolve-config` defaults `:data-dir` to `~/logseq/cli-graphs`. - `logseq.cli.server` resolves and uses `data-dir` for locks and server discovery. - `frontend.worker.db-worker-node` writes logs and lock files under `data-dir` and delegates storage to `frontend.worker.platform.node`, which creates directories as needed. - No explicit read/write permission checks exist; errors bubble up from fs operations. diff --git a/docs/agent-guide/020-logseq-cli-default-paths-move.md b/docs/agent-guide/020-logseq-cli-default-paths-move.md new file mode 100644 index 0000000000..fea9f1bcad --- /dev/null +++ b/docs/agent-guide/020-logseq-cli-default-paths-move.md @@ -0,0 +1,77 @@ +# Logseq CLI Default Paths Move Plan + +Goal: Move the default `--data-dir` location to `~/logseq/cli-graphs` and the default `cli.edn` location to `~/logseq/cli.edn`, keeping logseq-cli and db-worker-node consistent. + +Architecture: logseq-cli resolves defaults in `logseq.cli.config` and `logseq.cli.data-dir`, then hands `data-dir` into `logseq.cli.server` which spawns and manages db-worker-node. db-worker-node itself also resolves `data-dir` for logs, locks, and SQLite storage via `frontend.worker.platform.node` and `frontend.worker.db-worker-node-lock`. + +Tech Stack: ClojureScript, Node.js fs/path, logseq-cli, db-worker-node. + +## Problem statement + +Defaults currently live under `~/.logseq/`, but CLI data is not the same as desktop app data and should live under `~/logseq/` for better discoverability and separation. We need to update the defaults in both logseq-cli and db-worker-node, and update docs/help text to match. + +## Current behavior summary + +- logseq-cli uses `~/logseq/cli-graphs` as the default data dir. +- db-worker-node help text and internal resolution also default to `~/logseq/cli-graphs`. +- logseq-cli defaults config path to `~/logseq/cli.edn`. +- Docs reference `~/logseq/cli.edn` and `~/logseq/cli-graphs`. + +## Requirements + +- Default `data-dir` becomes `~/logseq/cli-graphs` everywhere it is derived. +- Default config path becomes `~/logseq/cli.edn`. +- `--data-dir` and `--config` flags continue to override defaults. +- `LOGSEQ_CLI_DATA_DIR` and `LOGSEQ_CLI_CONFIG` (if present) continue to override defaults. +- Help text and docs must match the new defaults. + +## Non-goals + +- Do not migrate existing data automatically. +- Do not change CLI flags, env var names, or db-worker-node storage layout. +- Do not change runtime behavior beyond the default locations. + +## Design decisions + +- Keep default paths defined in a single place per subsystem (CLI vs db-worker-node), but ensure they resolve to the same new location. +- Do not auto-detect the old location as a fallback to avoid surprises; users can pass `--data-dir` / `--config` explicitly if needed. +- Document the change and provide a brief migration note in CLI docs. + +## Implementation plan + +### 1) Update default `data-dir` constants and resolution + +- `src/main/logseq/cli/data_dir.cljs` + - Change `default-data-dir` from `~/logseq/cli-graphs` to `~/logseq/cli-graphs`. +- `src/main/logseq/cli/server.cljs` + - Update `resolve-data-dir` default to `~/logseq/cli-graphs` (keeps server defaults aligned when config is absent). +- `src/main/frontend/worker/db_worker_node_lock.cljs` + - Update `resolve-data-dir` default to `~/logseq/cli-graphs`. +- `src/main/frontend/worker/platform/node.cljs` + - Update `node-platform` default for `data-dir` to `~/logseq/cli-graphs`. +- `src/main/frontend/worker/db_worker_node.cljs` + - Update the `--data-dir` help text default to `~/logseq/cli-graphs`. + +### 2) Update default config path for CLI + +- `src/main/logseq/cli/config.cljs` + - Change `config-path` default from `~/logseq/cli.edn` to `~/logseq/cli.edn`. + - Update any inline default map (`resolve-config` default options) to match if present. + +### 3) Update docs and internal references + +- `docs/cli/logseq-cli.md` + - Replace references to `~/logseq/cli.edn` and `~/logseq/cli-graphs` with the new paths. + - Add a short migration note: existing data/config can be used by passing `--data-dir` / `--config`. +- `docs/agent-guide/*.md` + - Update any references to the old defaults (notably `docs/agent-guide/002-logseq-cli-subcommands.md`, `docs/agent-guide/003-db-worker-node-cli-orchestration.md`, `docs/agent-guide/012-logseq-cli-graph-storage.md`, `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`, and `docs/agent-guide/task--db-worker-nodejs-compatible.md`). + +### 4) Tests + +- Unit tests likely unaffected, but adjust any tests or snapshots that assert default path strings (search for `~/logseq/cli-graphs` or `~/logseq/cli.edn` in tests). +- If tests assert CLI help output or default config path, update expected strings accordingly. + +## Notes + +- Do not add a one-time warning for the old `~/logseq/cli-graphs` location. If a config is needed, prefer `cli.edn` under the selected `data-dir`. +- Do not add any fallback or compatibility for `~/logseq/cli.edn`. The old location is ignored. diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index 499fc04c0f..45595e5b4a 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -129,7 +129,7 @@ The db-worker should be runnable as a standalone process for Node.js environment - Provide a CLI entry (example: `bin/logseq-db-worker` or `./dist/db-worker-node.js`). - CLI flags (suggested): - Binds to localhost on a random port and records it in the repo lock file. - - `--data-dir` (path for sqlite files, required or default to `~/.logseq/cli-graphs`) + - `--data-dir` (path for sqlite files, required or default to `~/logseq/cli-graphs`) - `--repo` (optional: auto-open a repo on boot) - `--rtc-ws-url` (optional) - `--log-level` (default `info`) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 2ea1e7c08e..32c5cc613e 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -30,7 +30,11 @@ logseq graph list ## Configuration -Optional configuration file: `~/.logseq/cli.edn` +Optional configuration file: `~/logseq/cli.edn` + +Default data dir: `~/logseq/cli-graphs`. + +Migration note: If you previously used `~/.logseq/cli-graphs` or `~/.logseq/cli.edn`, pass `--data-dir` or `--config` to continue using those locations. Supported keys include: - `:repo` diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 9b92a70b4a..de5d237cdf 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -226,7 +226,7 @@ (defn- show-help! [] (println "db-worker-node options:") - (println " --data-dir (default ~/.logseq/cli-graphs)") + (println " --data-dir (default ~/logseq/cli-graphs)") (println " --repo (required)") (println " --rtc-ws-url (optional)") (println " --log-level (default info)") diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index 41cbe87711..5203422167 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -16,7 +16,7 @@ (defn resolve-data-dir [data-dir] - (expand-home (or data-dir "~/.logseq/cli-graphs"))) + (expand-home (or data-dir "~/logseq/cli-graphs"))) (defn repo-dir [data-dir repo] diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index 8296b42d31..d793593eea 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -186,7 +186,7 @@ (defn node-platform [{:keys [data-dir event-fn]}] - (let [data-dir (expand-home (or data-dir "~/.logseq/cli-graphs")) + (let [data-dir (expand-home (or data-dir "~/logseq/cli-graphs")) kv (kv-store data-dir)] (p/do! (ensure-dir! data-dir) diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 98d4b45333..0a031bffdb 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -24,7 +24,7 @@ :coerce :long} :target-uuid {:desc "Target block UUID"} :target-page-name {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling)"} + :pos {:desc "Position (first-child, last-child, sibling). Default: last-child"} :status {:desc "Task status (todo, doing, done, etc.)"}}) (def ^:private add-page-spec diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 5b15728fda..f718de38a0 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -10,12 +10,12 @@ :coerce :boolean} :version {:desc "Show version" :coerce :boolean} - :config {:desc "Path to cli.edn"} + :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)"} :repo {:desc "Graph name"} - :data-dir {:desc "Path to db-worker data dir"} - :timeout-ms {:desc "Request timeout in ms" + :data-dir {:desc "Path to db-worker data dir (default ~/logseq/cli-graphs)"} + :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} - :output {:desc "Output format (human, json, edn)"}}) + :output {:desc "Output format (human, json, edn). Default: human"}}) (defn global-spec [] diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 6053e7e167..d5effa343c 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -14,7 +14,7 @@ :offset {:desc "Offset results" :coerce :long} :sort {:desc "Sort field"} - :order {:desc "Sort order (asc, desc)"}}) + :order {:desc "Sort order (asc, desc). Default: asc"}}) (def ^:private list-page-spec (merge list-common-spec diff --git a/src/main/logseq/cli/command/move.cljs b/src/main/logseq/cli/command/move.cljs index b4cba44a05..b0fcfc43f1 100644 --- a/src/main/logseq/cli/command/move.cljs +++ b/src/main/logseq/cli/command/move.cljs @@ -15,7 +15,7 @@ :coerce :long} :target-uuid {:desc "Target block UUID"} :target-page {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling)"}}) + :pos {:desc "Position (first-child, last-child, sibling). Default: first-child"}}) (def entries [(core/command-entry ["move"] :move-block "Move block" move-spec)]) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index a996f0ade3..a5927d1f6f 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -12,7 +12,7 @@ {:id {:desc "Block db/id or EDN vector of ids"} :uuid {:desc "Block UUID"} :page {:desc "Page name"} - :level {:desc "Limit tree depth" + :level {:desc "Limit tree depth (default 10)" :coerce :long}}) (def entries diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 2f3ddf2e99..629791c8ff 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -26,7 +26,7 @@ (defn- default-config-path [] - (node-path/join (.homedir os) ".logseq" "cli.edn")) + (node-path/join (.homedir os) "logseq" "cli.edn")) (defn- read-config-file [config-path] @@ -75,7 +75,7 @@ [opts] (let [defaults {:timeout-ms 10000 :output-format nil - :data-dir "~/.logseq/cli-graphs" + :data-dir "~/logseq/cli-graphs" :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) diff --git a/src/main/logseq/cli/data_dir.cljs b/src/main/logseq/cli/data_dir.cljs index 1c9b5d71e1..014a3b9ae9 100644 --- a/src/main/logseq/cli/data_dir.cljs +++ b/src/main/logseq/cli/data_dir.cljs @@ -5,7 +5,7 @@ ["path" :as node-path] [clojure.string :as string])) -(def ^:private default-data-dir "~/.logseq/cli-graphs") +(def ^:private default-data-dir "~/logseq/cli-graphs") (defn- expand-home [path] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index da44445acf..e20e81b0bc 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -19,7 +19,7 @@ (defn resolve-data-dir [config] - (expand-home (or (:data-dir config) "~/.logseq/cli-graphs"))) + (expand-home (or (:data-dir config) "~/logseq/cli-graphs"))) (defn- repo-dir [data-dir repo] diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 251de45e5e..57d08c821c 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -5,6 +5,7 @@ [goog.object :as gobj] [logseq.cli.config :as config] ["fs" :as fs] + ["os" :as os] ["path" :as node-path])) (defn- with-env @@ -77,6 +78,12 @@ :output "json"})] (is (= :edn (:output-format result))))) +(deftest test-default-paths + (let [result (config/resolve-config {}) + expected-config-path (node-path/join (.homedir os) "logseq" "cli.edn")] + (is (= expected-config-path (:config-path result))) + (is (= "~/logseq/cli-graphs" (:data-dir result))))) + (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") diff --git a/src/test/logseq/cli/data_dir_test.cljs b/src/test/logseq/cli/data_dir_test.cljs index 2720459357..f1d97ab444 100644 --- a/src/test/logseq/cli/data_dir_test.cljs +++ b/src/test/logseq/cli/data_dir_test.cljs @@ -1,5 +1,6 @@ (ns logseq.cli.data-dir-test (:require ["fs" :as fs] + ["os" :as os] ["path" :as node-path] [cljs.test :refer [deftest is testing]] [frontend.test.node-helper :as node-helper] @@ -39,3 +40,9 @@ (let [data (ex-data e)] (is (= :data-dir-permission (:code data))) (is (= (node-path/resolve target) (:path data))))))))) + +(deftest normalize-data-dir-default + (testing "defaults to ~/logseq/cli-graphs" + (let [expected (node-path/resolve (node-path/join (.homedir os) "logseq" "cli-graphs")) + resolved (data-dir/normalize-data-dir nil)] + (is (= expected resolved))))) From f17e7856f8640b19dd00632f7150830cd8befce3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 29 Jan 2026 14:09:41 +0800 Subject: [PATCH 048/375] 021-logseq-cli-reference-uuid-rewrite.md --- .../021-logseq-cli-reference-uuid-rewrite.md | 87 ++++++++++++ src/main/logseq/cli/command/add.cljs | 105 +++++++++++++- src/test/logseq/cli/integration_test.cljs | 132 ++++++++++++++++++ 3 files changed, 319 insertions(+), 5 deletions(-) create mode 100644 docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md diff --git a/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md b/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md new file mode 100644 index 0000000000..e5c723a4bc --- /dev/null +++ b/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md @@ -0,0 +1,87 @@ +# Logseq CLI Add Reference UUID Rewrite Plan + +Goal: For logseq-cli add block/page content, replace every `[[]]` with `[[block-uuid]]` before calling db-worker-node thread-api, creating missing pages when needed. + +Architecture: logseq-cli (`src/main/logseq/cli/command/add.cljs`) sends `:thread-api/apply-outliner-ops` calls through `logseq.cli.transport`. db-worker-node executes `outliner-op/apply-ops!` without normalizing refs. Frontend already normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI does not. + +Tech Stack: ClojureScript, logseq-cli, db-worker-node, Datascript. + +## Problem statement + +Today logseq-cli passes block content to db-worker-node as-is. If content includes `[[Page Name]]`, db-worker-node does not automatically resolve or create that page for outliner ops. We need to normalize content refs to uuid references up front so db-worker-node receives canonical refs and missing pages are created deterministically. + +## Current behavior summary + +- `logseq-cli add block` builds blocks and calls `:thread-api/apply-outliner-ops` with `:insert-blocks`. +- `logseq-cli add page` calls `:thread-api/apply-outliner-ops` with `:create-page` only (no content normalization). +- db-worker-node `:thread-api/apply-outliner-ops` applies ops verbatim and does not resolve page refs in block content. +- Frontend editor normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI paths do not use it. + +## Requirements + +- For add block and add page content, replace every `[[]]` with `[[block-uuid]]` before invoking db-worker-node thread-api. +- Page reference: + - If `` is not a UUID, treat it as a page title. + - If the page does not exist, create it first. + - Replace `[[Page Title]]` with `[[page-uuid]]` (case-insensitive). +- Block reference: + - If `` is a UUID, treat it as a block ref. + - The block must exist; otherwise return a CLI error. +- Do not change other syntax (e.g. `((uuid))` block refs, tags, or macros) unless they are inside `[[...]]`. + +## Non-goals + +- Do not alter how `((uuid))` block refs are parsed or stored. +- Do not introduce automatic block creation for missing block UUIDs. +- Do not change CLI command flags or output format. + +## Design decisions + +- Reuse `logseq.db.frontend.content/title-ref->id-ref` so CLI behavior matches frontend normalization rules. +- Extract `[[...]]` refs using the existing page-ref regex from `logseq.common.util.page-ref` to avoid implementing a new parser. +- Resolve refs once per CLI action, cache page-name -> uuid, and then update all affected blocks before the first `:thread-api/apply-outliner-ops` call. +- Handle page creation with the existing `ensure-page!` helper in `src/main/logseq/cli/command/add.cljs` for consistent behavior. + +## Implementation plan + +### 1) Add reference extraction and resolution helpers + +- `src/main/logseq/cli/command/add.cljs` + - Add a helper to extract `[[...]]` tokens from a block title using `logseq.common.util.page-ref/page-ref-re`. + - Add a helper that partitions refs into: + - `uuid-refs`: `[[]]` values + - `page-refs`: `[[]]` values + - Add a resolver that: + - For `page-refs`, calls `ensure-page!` (once per unique title) and returns `{:block/uuid uuid :block/title title}`. + - For `uuid-refs`, pulls the entity by `[:block/uuid uuid]` and errors if missing. + +### 2) Normalize add block content before outliner ops + +- `src/main/logseq/cli/command/add.cljs` + - In `execute-add-block`, before building ops: + - Walk the `:blocks` tree (top-level and any nested `:block/children`) and collect all `[[...]]` refs. + - Resolve refs once per action using the resolver from step 1. + - For each block with `:block/title`, rewrite it with `db-content/title-ref->id-ref` using the resolved refs and `{:replace-tag? false}`. + - Use the rewritten blocks in the `:insert-blocks` op. + +### 3) Normalize add page content (when present) + +- `src/main/logseq/cli/command/add.cljs` + - If add page grows to accept initial blocks/content (or if `:create-page` options start supporting content), reuse the same ref normalization flow from step 2 before invoking `:create-page` or `:insert-blocks`. + - If no content is provided, no change is needed in the current implementation. + +### 4) Tests + +- `test/logseq/cli/integration_test.cljs` + - Add an integration test for `add block` with content `"See [[New Page]]"`: + - Assert the page is created. + - Pull the inserted block and verify its title contains `[[]]` instead of `[[New Page]]`. + - Add an integration test for `add block` with content `"See [[]]"`: + - Assert the UUID is preserved and no new page is created. + - Add an integration test for `add block` with content `"See [[]]"`: + - Assert CLI returns an error with a clear message. + +## Notes + +- If we later decide to normalize tags (`#tag`) or macro-based refs, we can extend the resolver to call `db-content/title-ref->id-ref` with `:replace-tag? true` and add tests accordingly. +- Keep all normalization in CLI to avoid changing db-worker-node semantics for other callers. diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 0a031bffdb..cce1a9eab5 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -8,8 +8,10 @@ [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] + [logseq.common.util.page-ref :as page-ref] [logseq.common.util.date-time :as date-time-util] [logseq.common.uuid :as common-uuid] + [logseq.db.frontend.content :as db-content] [logseq.db.frontend.property :as db-property] [logseq.db.frontend.property.type :as db-property-type] [promesa.core :as p])) @@ -46,14 +48,15 @@ (defn- ensure-page! [config repo page-name] - (p/let [page (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]])] + (let [page-name-lc (common-util/page-name-sanity-lc page-name)] + (p/let [page (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])] (if (:db/id page) page (p/let [_ (transport/invoke config :thread-api/apply-outliner-ops false [repo [[:create-page [page-name {}]]] {}])] (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]]))))) + [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])))))) (def ^:private add-positions #{"first-child" "last-child" "sibling"}) @@ -109,6 +112,91 @@ (assoc block :block/uuid (common-uuid/gen-uuid))))) blocks)) +(defn- extract-page-refs + [title] + (when (string? title) + (->> (re-seq page-ref/page-ref-re title) + (map second) + (remove string/blank?)))) + +(defn- collect-page-refs + [blocks] + (->> blocks + (mapcat (fn walk [block] + (let [refs (extract-page-refs (:block/title block)) + children (:block/children block)] + (if (seq children) + (concat refs (mapcat walk children)) + refs)))) + (remove string/blank?) + vec)) + +(defn- partition-ref-values + [refs] + (reduce + (fn [acc ref-value] + (let [value (string/trim ref-value)] + (cond + (string/blank? value) + acc + + (common-util/uuid-string? value) + (update acc :uuid-refs conj value) + + :else + (update acc :page-refs conj value)))) + {:uuid-refs [] :page-refs []} + refs)) + +(defn- resolve-page-ref-entities + [config repo page-refs] + (if (seq page-refs) + (let [unique (reduce (fn [acc ref-value] + (let [value (string/trim ref-value)] + (if (string/blank? value) + acc + (assoc acc (common-util/page-name-sanity-lc value) value)))) + {} + page-refs)] + (p/let [resolved (p/all + (map (fn [[_ page-name]] + (p/let [page (ensure-page! config repo page-name) + page-uuid (:block/uuid page)] + (when-not page-uuid + (throw (ex-info "page not found" + {:code :page-not-found + :page page-name}))) + {:block/uuid page-uuid + :block/title (or (:block/title page) page-name)})) + unique))] + (vec resolved))) + (p/resolved nil))) + +(defn- ensure-block-refs-exist! + [config repo uuid-refs] + (when (seq uuid-refs) + (p/all + (map (fn [uuid-ref] + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] [:block/uuid (uuid uuid-ref)]])] + (when-not (:db/id entity) + (throw (ex-info (str "block ref not found: " uuid-ref) + {:code :block-ref-not-found + :uuid uuid-ref}))))) + (distinct uuid-refs))))) + +(defn- normalize-block-title-refs + [blocks refs] + (mapv (fn update-block [block] + (let [block' (if (string? (:block/title block)) + (update block :block/title + #(db-content/title-ref->id-ref % refs :replace-tag? false)) + block)] + (if (seq (:block/children block')) + (update block' :block/children #(normalize-block-title-refs % refs)) + block'))) + blocks)) + (defn- invalid-options-result [message] {:ok? false @@ -729,6 +817,13 @@ [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) target-id (resolve-add-target cfg action) + ref-values (collect-page-refs (:blocks action)) + {:keys [uuid-refs page-refs]} (partition-ref-values ref-values) + _ (ensure-block-refs-exist! cfg (:repo action) uuid-refs) + refs (or (resolve-page-ref-entities cfg (:repo action) page-refs) []) + blocks (if (seq refs) + (normalize-block-title-refs (:blocks action) refs) + (:blocks action)) status (:status action) tags (resolve-tags cfg (:repo action) (:tags action)) properties (resolve-properties cfg (:repo action) (:properties action)) @@ -741,11 +836,11 @@ opts (cond-> opts keep-uuid? (assoc :keep-uuid? true)) - ops [[:insert-blocks [(:blocks action) + ops [[:insert-blocks [blocks target-id (assoc opts :outliner-op :insert-blocks)]]] _ (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) - block-ids (->> (:blocks action) + block-ids (->> blocks (map :block/uuid) (remove nil?) vec) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 316da209e4..bd97773b70 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -8,6 +8,7 @@ [frontend.test.node-helper :as node-helper] [logseq.cli.command.core :as command-core] [logseq.cli.main :as cli-main] + [logseq.common.util :as common-util] [logseq.db.frontend.property :as db-property] [promesa.core :as p])) @@ -29,6 +30,16 @@ [result] (js->clj (js/JSON.parse (:output result)) :keywordize-keys true)) +(defn- parse-json-output-safe + [result label] + (try + (parse-json-output result) + (catch :default e + (throw (ex-info (str "json parse failed: " label) + {:label label + :output (:output result)} + e))))) + (defn- parse-edn-output [result] (reader/read-string (:output result))) @@ -220,6 +231,127 @@ (is (contains? (get-in show-payload [:data :root]) :uuid)) (is (= "ok" (:status remove-page-payload))) (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-block-rewrites-page-ref + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "ref-rewrite-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "ref-rewrite-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "ref-rewrite-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "See [[New Page]]"] + data-dir cfg-path) + add-block-payload (parse-json-output-safe add-block-result "add-block") + _ (p/delay 100) + list-page-result (run-cli ["--repo" "ref-rewrite-graph" "list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output-safe list-page-result "list-page") + page-titles (->> (get-in list-page-payload [:data :items]) + (map #(or (:block/title %) (:title %))) + set) + query-payload (run-query data-dir cfg-path "ref-rewrite-graph" + "[:find ?title :in $ ?page-name :where [?p :block/name ?page-name] [?b :block/page ?p] [?b :block/title ?title]]" + (pr-str [(common-util/page-name-sanity-lc "Home")])) + titles (map first (get-in query-payload [:data :result])) + ref-title (some #(when (and (string? %) + (string/includes? % "See [[") + (string/includes? % "]]")) + %) + titles) + ref-value (when ref-title + (second (first (re-seq #"\[\[(.*?)\]\]" ref-title)))) + stop-result (run-cli ["server" "stop" "--repo" "ref-rewrite-graph"] data-dir cfg-path) + stop-payload (parse-json-output-safe stop-result "server-stop")] + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (contains? page-titles "New Page")) + (is (string? ref-value)) + (is (common-util/uuid-string? ref-value)) + (is (string? ref-title)) + (is (not (string/includes? ref-title "[[New Page]]"))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-block-keeps-uuid-ref + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-uuid-ref")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "uuid-ref-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" "uuid-ref-graph" + "add" "block" + "--target-page-name" "Home" + "--content" "Target block"] + data-dir cfg-path) + _ (p/delay 100) + target-query-payload (run-query data-dir cfg-path "uuid-ref-graph" + "[:find ?uuid :in $ ?title :where [?b :block/title ?title] [?b :block/uuid ?uuid]]" + (pr-str ["Target block"])) + target-uuid (first (first (get-in target-query-payload [:data :result]))) + add-block-result (run-cli ["--repo" "uuid-ref-graph" + "add" "block" + "--target-page-name" "Home" + "--content" (str "See [[" target-uuid "]]")] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + list-page-result (run-cli ["--repo" "uuid-ref-graph" "list" "page"] data-dir cfg-path) + list-page-payload (parse-json-output list-page-result) + page-titles (->> (get-in list-page-payload [:data :items]) + (map #(or (:block/title %) (:title %))) + set) + ref-query-payload (run-query data-dir cfg-path "uuid-ref-graph" + "[:find ?title :in $ ?page-name :where [?p :block/name ?page-name] [?b :block/page ?p] [?b :block/title ?title]]" + (pr-str [(common-util/page-name-sanity-lc "Home")])) + titles (map first (get-in ref-query-payload [:data :result])) + ref-title (some #(when (and (string? %) + (string/includes? % (str "[[" target-uuid "]]"))) + %) + titles) + stop-result (run-cli ["server" "stop" "--repo" "uuid-ref-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (string? target-uuid)) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (not (contains? page-titles target-uuid))) + (is (string? ref-title)) + (is (string/includes? ref-title (str "[[" target-uuid "]]"))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-add-block-missing-uuid-ref-errors + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-missing-uuid-ref")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "missing-uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + missing-uuid (str (random-uuid)) + add-block-result (run-cli ["--repo" "missing-uuid-ref-graph" + "add" "block" + "--target-page-name" "Home" + "--content" (str "See [[" missing-uuid "]]")] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + stop-result (run-cli ["server" "stop" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 1 (:exit-code add-block-result))) + (is (= "error" (:status add-block-payload))) + (is (string/includes? (get-in add-block-payload [:error :message]) missing-uuid)) + (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) From 19513ed6bb9e35089749be57e052818447e82c98 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 29 Jan 2026 20:07:25 +0800 Subject: [PATCH 049/375] fix: invalid exec sql in node --- src/main/frontend/worker/platform/node.cljs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index d793593eea..e2a193d8dd 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -78,7 +78,9 @@ (let [sql (gobj/get opts-or-sql "sql") bind (gobj/get opts-or-sql "bind") row-mode (gobj/get opts-or-sql "rowMode") - bind' (if (and bind (object? bind)) + bind' (cond + (array? bind) bind + (and bind (object? bind)) (let [out (js-obj)] (doseq [key (js/Object.keys bind)] (let [value (gobj/get bind key) @@ -88,7 +90,7 @@ :else key)] (gobj/set out normalized value))) out) - bind) + :else bind) ^js stmt (.prepare db sql)] (if (= row-mode "array") (do From 09e25603dce620d27d049b0b818119d4efa15bdd Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 29 Jan 2026 21:48:41 +0800 Subject: [PATCH 050/375] update doc --- docs/cli/logseq-cli.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 32c5cc613e..311d3cea1f 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -5,13 +5,9 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to ## Build the CLI ```bash -LOGSEQ_BUILD_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \ -LOGSEQ_REVISION=$(git rev-parse --short HEAD) \ -clojure -M:cljs compile logseq-cli +clojure -M:cljs compile logseq-cli db-worker-node ``` -If `LOGSEQ_BUILD_TIME` or `LOGSEQ_REVISION` are not provided, the CLI prints defaults in `--version` output. - ## db-worker-node lifecycle `logseq` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. @@ -22,7 +18,12 @@ If `LOGSEQ_BUILD_TIME` or `LOGSEQ_REVISION` are not provided, the CLI prints def node ./dist/logseq.js graph list ``` -If installed globally, run: +You can also use npm link to make ./dist/logseq.js globally available, run: +``` +npm link +``` + +If installed(or linked) globally, run: ```bash logseq graph list From 35ae52b63bc8701505478643dbaa59b3632d0c9f Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 30 Jan 2026 16:09:46 +0800 Subject: [PATCH 051/375] 022-logseq-cli-help-show-output.md --- .../022-logseq-cli-help-show-output.md | 122 ++++++++++++++++++ src/main/logseq/cli/command/add.cljs | 8 +- src/main/logseq/cli/command/core.cljs | 8 +- src/main/logseq/cli/command/show.cljs | 34 +++-- src/test/logseq/cli/commands_test.cljs | 63 ++++++++- src/test/logseq/cli/integration_test.cljs | 87 ++++++------- 6 files changed, 257 insertions(+), 65 deletions(-) create mode 100644 docs/agent-guide/022-logseq-cli-help-show-output.md diff --git a/docs/agent-guide/022-logseq-cli-help-show-output.md b/docs/agent-guide/022-logseq-cli-help-show-output.md new file mode 100644 index 0000000000..00d72911bb --- /dev/null +++ b/docs/agent-guide/022-logseq-cli-help-show-output.md @@ -0,0 +1,122 @@ +# Logseq CLI Help and Show Output Cleanup Implementation Plan + +Goal: Improve logseq-cli help readability and ensure show JSON and EDN outputs use :db/id without :block/uuid. + +Architecture: The CLI help text is assembled in logseq.cli.command.core and babashka.cli spec descriptions, while show output is built in logseq.cli.command.show and formatted in logseq.cli.format before returning JSON or EDN. + +Tech Stack: ClojureScript, logseq-cli, babashka.cli, db-worker-node. + +Related: Relates to 018-logseq-cli-add-tags-builtin-properties.md. + +## Problem statement + +The current help output lists [options] for nearly every command, which clutters the help display and reduces readability. + +The tags and properties option help text does not clearly state that identifiers can be id, :db/ident, or :block/title across all relevant help contexts. + +The show command includes :block/uuid in JSON and EDN output, even though :db/id is already present and preferred for programmatic consumers. + +``` +logseq-cli + | help text built from babashka.cli specs + v +logseq.cli.command.core + | parse args and build command payloads + v +logseq.cli.command.show + | fetch tree data from db-worker-node + v +logseq.cli.format + | render human, json, edn output + v +stdout +``` + +## Testing Plan + +I will add or update unit tests that assert help summaries do not include repeated [options] in the command list, and that tags and properties descriptions include the identifier guidance. + +I will add or update unit tests that verify show JSON and EDN outputs do not include :block/uuid and still include :db/id for root and child nodes. + +I will add or update integration tests for show JSON output to assert :db/id is present and :block/uuid is absent in root, tags, and linked references where applicable. + +NOTE: I will write all tests before I add any implementation behavior. + +## Requirements + +The top level help output and group summaries must avoid repeating [options] for each command listing while still documenting that options exist in the usage line. + +The help description for --tags and --properties must explicitly state that identifiers can be id, :db/ident, or :block/title. + +The show command must omit :block/uuid from JSON and EDN outputs while preserving :db/id for the same entities. + +The show command human output must be unchanged. + +## Non-goals + +Do not change CLI command behavior or supported flags beyond help text updates. + +Do not change db-worker-node behavior or its API surface. + +Do not change the structure of human output for show, list, add, or query commands. + +## Design decisions + +Limit the help output adjustment to formatting in logseq.cli.command.core so command behavior and parsing remain unchanged. + +Apply the identifier clarification to all --tags and --properties options in logseq.cli.command.add and any other specs that expose those flags. + +Strip :block/uuid only for show outputs in JSON and EDN formats by post-processing tree data just before returning payloads. + +## Implementation plan + +1. Follow @test-driven-development for every change in this plan. + +2. Add a unit test in src/test/logseq/cli/commands_test.cljs that asserts command list rows in top level and group help do not contain [options] after the command name. + +3. Add a unit test in src/test/logseq/cli/commands_test.cljs that asserts the --tags and --properties option descriptions include the text supporting id, :db/ident, and :block/title identifiers. + +4. Add a unit test in src/test/logseq/cli/format_test.cljs or src/test/logseq/cli/commands_test.cljs that asserts show JSON and EDN outputs strip :block/uuid while retaining :db/id in root and child nodes. + +5. Update integration tests in src/test/logseq/cli/integration_test.cljs that currently assert :uuid or :block/uuid in show JSON output to instead assert :db/id and absence of :block/uuid. + +6. Adjust logseq.cli.command.core command listing formatting so only the usage line includes [options], and the command listing uses the bare command path without the suffix. + +7. Update the --tags and --properties option descriptions in src/main/logseq/cli/command/add.cljs to include the identifier guidance sentence in a consistent phrasing. + +8. Add a helper in src/main/logseq/cli/command/show.cljs or src/main/logseq/cli/format.cljs that removes :block/uuid keys from show JSON and EDN payloads, and apply it in execute-show when output-format is :json or :edn. + +9. Run the updated unit tests and integration tests from the Testing Plan and confirm all pass. + +## Edge cases + +Command help should still show [options] in the usage line for commands that accept options, but not in the command list table. + +Multi-id show results should strip :block/uuid from each tree entry without changing the error map shape. + +Linked references and tag entities should keep :db/id in the output even when :block/uuid is removed. + +## Testing Details + +I will add tests that verify help summaries and option descriptions at the command summary level and not by matching the raw babashka.cli output. + +I will add tests that parse show JSON and EDN output and assert :block/uuid is missing while :db/id remains on block nodes. + +I will update integration tests that read show JSON output to match the new key expectations without changing the test setup logic. + +## Implementation Details + +- Update logseq.cli.command.core formatting to render command rows without [options]. +- Keep usage lines intact so users still see options availability in usage sections. +- Align help text for --tags and --properties to a single wording that mentions id, :db/ident, and :block/title. +- Add a show-specific sanitization step for json and edn output only. +- Keep the show tree data used for human output unchanged to avoid regressions. +- Ensure strip logic is recursive so :block/uuid is removed from nested children and linked references. +- Prefer clojure.walk/postwalk for key removal to minimize custom traversal code. +- Document the new behavior in tests rather than adding new user-facing docs. + +## Question + +This is resolved. Only add supports --tags and --properties today, so we will update help text there only. + +--- diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index cce1a9eab5..63985c5879 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -20,8 +20,8 @@ {:content {:desc "Block content for add"} :blocks {:desc "EDN vector of blocks for add"} :blocks-file {:desc "EDN file of blocks for add"} - :tags {:desc "EDN vector of tags (id, :db/ident, or :block/title)"} - :properties {:desc "EDN map of built-in properties (id, :db/ident, or :block/title)"} + :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} + :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."} :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} @@ -31,8 +31,8 @@ (def ^:private add-page-spec {:page {:desc "Page name"} - :tags {:desc "EDN vector of tags (id, :db/ident, or :block/title)"} - :properties {:desc "EDN map of built-in properties (id, :db/ident, or :block/title)"}}) + :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} + :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."}}) (def entries [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index f718de38a0..5889881ada 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -47,12 +47,16 @@ (cond-> base has-options? (str " [options]")))) +(defn- command-label + [cmds] + (string/join " " cmds)) + (defn- format-commands [table] (let [rows (->> table (filter (comp seq :cmds)) - (map (fn [{:keys [cmds desc spec]}] - (let [command (command-usage cmds spec)] + (map (fn [{:keys [cmds desc]}] + (let [command (command-label cmds)] {:command command :desc desc})))) width (apply max 0 (map (comp count :command) rows))] diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index a5927d1f6f..7959bafff3 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.command.show "Show-related CLI commands." (:require [clojure.string :as string] + [clojure.walk :as walk] [logseq.cli.command.id :as id-command] [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] @@ -508,6 +509,15 @@ (walk root #{}) #{}))) +(defn- strip-block-uuid + [tree-data] + (walk/postwalk + (fn [entry] + (if (map? entry) + (dissoc entry :block/uuid) + entry)) + tree-data)) + (defn execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -539,12 +549,14 @@ results (vec (remove (fn [{:keys [ok? id]}] (and ok? (contained? id))) results)) + sanitize-tree (fn [tree] + (strip-block-uuid tree)) payload (case format :edn {:status :ok :data (mapv (fn [{:keys [ok? tree id error]}] (if ok? - tree + (sanitize-tree tree) (multi-id-error-entry id error))) results) :output-format :edn} @@ -553,7 +565,7 @@ {:status :ok :data (mapv (fn [{:keys [ok? tree id error]}] (if ok? - tree + (sanitize-tree tree) (multi-id-error-entry id error))) results) :output-format :json} @@ -568,15 +580,17 @@ payload) (p/let [tree-data (build-tree-data cfg action)] (case format - :edn - {:status :ok - :data tree-data - :output-format :edn} + :edn + (let [tree-data (strip-block-uuid tree-data)] + {:status :ok + :data tree-data + :output-format :edn}) - :json - {:status :ok - :data tree-data - :output-format :json} + :json + (let [tree-data (strip-block-uuid tree-data)] + {:status :ok + :data tree-data + :output-format :json}) {:status :ok :data {:message (tree->text-with-linked-refs tree-data)}})))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 2adad4054b..66cfc12bae 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -8,6 +8,25 @@ [logseq.cli.transport :as transport] [promesa.core :as p])) +(defn- command-lines + [summary] + (let [lines (string/split-lines summary) + section (if (some #{"Commands:"} lines) "Commands:" "Subcommands:") + start (inc (.indexOf lines section)) + end (.indexOf lines "Global options:") + entries (subvec (vec lines) start end)] + (->> entries + (filter #(string/starts-with? % " ")) + (remove string/blank?)))) + +(defn- contains-block-uuid? + [value] + (cond + (map? value) (or (contains? value :block/uuid) + (some contains-block-uuid? (vals value))) + (sequential? value) (some contains-block-uuid? value) + :else false)) + (deftest test-help-output (testing "top-level help lists command groups" (let [result (commands/parse-args ["--help"]) @@ -26,6 +45,12 @@ (is (string/includes? summary "graph")) (is (string/includes? summary "server")))) + (testing "top-level help command list omits [options]" + (let [summary (:summary (commands/parse-args ["--help"])) + lines (command-lines summary)] + (is (seq lines)) + (is (every? #(not (string/includes? % "[options]")) lines)))) + (testing "top-level help separates global and command options" (let [summary (:summary (commands/parse-args ["--help"]))] (is (string/includes? summary "Global options:")) @@ -84,7 +109,13 @@ summary (:summary result)] (is (true? (:help? result))) (is (string/includes? summary "query list")) - (is (string/includes? summary "query [options]"))))) + (is (string/includes? summary "query")))) + + (testing "group help command list omits [options]" + (let [summary (:summary (commands/parse-args ["list"])) + lines (command-lines summary)] + (is (seq lines)) + (is (every? #(not (string/includes? % "[options]")) lines))))) (deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" @@ -302,6 +333,36 @@ (is (= (str "1 See [[Target [[Inner]]]]") (tree->text tree-data)))))) +(deftest test-help-tags-properties-identifiers + (testing "add help mentions tag and property identifiers" + (let [summary (:summary (commands/parse-args ["add" "block" "--help"]))] + (is (string/includes? summary "Identifiers can be id, :db/ident, or :block/title."))) + (let [summary (:summary (commands/parse-args ["add" "page" "--help"]))] + (is (string/includes? summary "Identifiers can be id, :db/ident, or :block/title."))))) + +(deftest test-show-json-edn-strips-block-uuid + (testing "show json/edn removes :block/uuid recursively while keeping :db/id" + (let [tree-data {:root {:db/id 1 + :block/uuid "root-uuid" + :block/title "Root" + :block/children [{:db/id 2 + :block/uuid "child-uuid" + :block/title "Child" + :block/tags [{:db/id 3 + :block/uuid "tag-uuid" + :block/title "Tag"}]}]} + :linked-references {:count 1 + :blocks [{:db/id 4 + :block/uuid "ref-uuid" + :block/page {:db/id 5 + :block/uuid "page-uuid" + :block/title "Page"}}]} + :uuid->label {"root-uuid" "Root"}} + stripped (#'show-command/strip-block-uuid tree-data)] + (is (not (contains-block-uuid? stripped))) + (is (= 1 (get-in stripped [:root :db/id]))) + (is (= 2 (get-in stripped [:root :block/children 0 :db/id])))))) + (deftest test-tree->text-uuid-ref-recursion-limit (testing "show tree text limits uuid ref replacement depth" (let [tree->text #'show-command/tree->text diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index bd97773b70..2993f6d998 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -48,10 +48,6 @@ [node] (or (:block/title node) (:block/content node) (:title node) (:content node))) -(defn- node-uuid - [node] - (or (:block/uuid node) (:uuid node))) - (defn- node-children [node] (or (:block/children node) (:children node))) @@ -108,6 +104,20 @@ (pr-str [title]))] (first (first (get-in payload [:data :result]))))) +(defn- query-block-id + [data-dir cfg-path repo title] + (p/let [payload (run-query data-dir cfg-path repo + "[:find ?id . :in $ ?title :where [?b :block/title ?title] [?b :db/id ?id]]" + (pr-str [title]))] + (get-in payload [:data :result]))) + +(defn- query-block-uuid-by-id + [data-dir cfg-path repo id] + (p/let [payload (run-query data-dir cfg-path repo + "[:find ?uuid . :in $ ?id :where [?b :db/id ?id] [?b :block/uuid ?uuid]]" + (pr-str [id]))] + (get-in payload [:data :result]))) + (defn- list-items [data-dir cfg-path repo list-type] (p/let [result (run-cli ["--repo" repo "list" list-type] data-dir cfg-path)] @@ -228,7 +238,8 @@ (is (= "ok" (:status list-property-payload))) (is (vector? (get-in list-property-payload [:data :items]))) (is (= "ok" (:status show-payload))) - (is (contains? (get-in show-payload [:data :root]) :uuid)) + (is (contains? (get-in show-payload [:data :root]) :db/id)) + (is (not (contains? (get-in show-payload [:data :root]) :block/uuid))) (is (= "ok" (:status remove-page-payload))) (is (= "ok" (:status stop-payload))) (done)) @@ -716,16 +727,13 @@ _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) - show-inner (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) - show-inner-payload (parse-json-output show-inner) - inner-node (find-block-by-title (get-in show-inner-payload [:data :root]) "Inner") - inner-uuid (node-uuid inner-node) + inner-id (query-block-id data-dir cfg-path "nested-refs" "Inner") + inner-uuid (query-block-uuid-by-id data-dir cfg-path "nested-refs" inner-id) + middle-content (str "See [[" inner-uuid "]]") _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" - "--content" (str "See [[" inner-uuid "]]")] data-dir cfg-path) - show-middle (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) - show-middle-payload (parse-json-output show-middle) - middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) "See [[Inner]]") - middle-uuid (node-uuid middle-node) + "--content" middle-content] data-dir cfg-path) + middle-id (query-block-id data-dir cfg-path "nested-refs" middle-content) + middle-uuid (query-block-uuid-by-id data-dir cfg-path "nested-refs" middle-id) _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) @@ -751,27 +759,21 @@ _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - target-show-before (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) - target-before-payload (parse-json-output target-show-before) - target-uuid (or (get-in target-before-payload [:data :root :block/uuid]) - (get-in target-before-payload [:data :root :uuid])) - target-title (or (get-in target-before-payload [:data :root :block/title]) - (get-in target-before-payload [:data :root :block/name]) - "TargetPage") + target-id (query-block-id data-dir cfg-path "linked-refs-graph" "TargetPage") + target-uuid (query-block-uuid-by-id data-dir cfg-path "linked-refs-graph" target-id) + target-title "TargetPage" ref-content (str "See [[" target-uuid "]]") ref-title (str "See [[" target-title "]]") _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) source-payload (parse-json-output source-show) ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-title) - ref-uuid (or (:block/uuid ref-node) (:uuid ref-node)) + ref-id (:db/id ref-node) target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) target-payload (parse-json-output target-show) linked-refs (get-in target-payload [:data :linked-references]) linked-blocks (:blocks linked-refs) - linked-uuids (set (map (fn [block] - (or (:block/uuid block) (:uuid block))) - linked-blocks)) + linked-ids (set (map :db/id linked-blocks)) linked-page-titles (set (keep (fn [block] (or (get-in block [:block/page :block/title]) (get-in block [:block/page :block/name]) @@ -782,8 +784,8 @@ stop-payload (parse-json-output stop-result)] (is (some? target-uuid)) (is (= "ok" (:status target-payload))) - (is (some? ref-uuid)) - (is (contains? linked-uuids ref-uuid)) + (is (some? ref-id)) + (is (contains? linked-ids ref-id)) (is (contains? linked-page-titles "SourcePage")) (is (= "ok" (:status stop-payload))) (done)) @@ -800,10 +802,8 @@ _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) - source-show (run-cli ["--repo" "move-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) - source-payload (parse-json-output source-show) - parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") - parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + parent-id (query-block-id data-dir cfg-path "move-graph" "Parent Block") + parent-uuid (query-block-uuid-by-id data-dir cfg-path "move-graph" parent-id) _ (run-cli ["--repo" "move-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--target-page-name" "TargetPage"] data-dir cfg-path) move-payload (parse-json-output move-result) @@ -831,10 +831,8 @@ _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) - parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) - parent-payload (parse-json-output parent-show) - parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") - parent-uuid (or (:block/uuid parent-node) (:uuid parent-node)) + parent-id (query-block-id data-dir cfg-path "add-pos-graph" "Parent") + parent-uuid (query-block-uuid-by-id data-dir cfg-path "add-pos-graph" parent-id) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "first-child" "--content" "First"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) final-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) @@ -941,11 +939,11 @@ (is (some? page-id)) (is (some? page-uuid)) (is (= "ok" (:status show-by-id-payload))) - (is (= (str page-uuid) (str (or (get-in show-by-id-payload [:data :root :uuid]) - (get-in show-by-id-payload [:data :root :block/uuid]))))) + (is (= page-id (get-in show-by-id-payload [:data :root :db/id]))) + (is (not (contains? (get-in show-by-id-payload [:data :root]) :block/uuid))) (is (= "ok" (:status show-by-uuid-payload))) - (is (= (str page-uuid) (str (or (get-in show-by-uuid-payload [:data :root :uuid]) - (get-in show-by-uuid-payload [:data :root :block/uuid]))))) + (is (= page-id (get-in show-by-uuid-payload [:data :root :db/id]))) + (is (not (contains? (get-in show-by-uuid-payload [:data :root]) :block/uuid))) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] @@ -1043,14 +1041,7 @@ data-dir cfg-path) parent-payload (parse-json-output parent-query) parent-id (get-in parent-payload [:data :result]) - show-parent (run-cli ["--repo" "show-multi-id-contained-graph" - "show" - "--page-name" "ParentPage" - "--format" "json"] - data-dir cfg-path) - show-parent-payload (parse-json-output show-parent) - parent-node (find-block-by-title (get-in show-parent-payload [:data :root]) "Parent Block") - parent-uuid (node-uuid parent-node) + parent-uuid (query-block-uuid-by-id data-dir cfg-path "show-multi-id-contained-graph" parent-id) _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] @@ -1174,7 +1165,7 @@ (is (pos? (:count linked))) (is (seq (:blocks linked))) (is (some? ref-block)) - (is (some? (or (:block/uuid ref-block) (:uuid ref-block)))) + (is (some? (:db/id ref-block))) (is (some? (or (get-in ref-block [:page :title]) (get-in ref-block [:page :name])))) (is (= "ok" (:status stop-payload))) From 8a67ef5552f944e2c5342eacc7fdf01cb9b17243 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 30 Jan 2026 18:06:05 +0800 Subject: [PATCH 052/375] 023-logseq-cli-help-show-styling.md --- deps/cli/src/logseq/cli.cljs | 55 +- .../023-logseq-cli-help-show-styling.md | 108 ++++ package.json | 3 +- src/main/frontend/worker/db_worker_node.cljs | 11 +- src/main/logseq/cli/command/core.cljs | 36 +- src/main/logseq/cli/command/show.cljs | 33 +- src/main/logseq/cli/format.cljs | 6 +- src/main/logseq/cli/main.cljs | 2 - src/main/logseq/cli/style.cljs | 78 +++ .../frontend/worker/db_worker_node_test.cljs | 549 +++++++++--------- src/test/logseq/cli/commands_test.cljs | 283 ++++++--- src/test/logseq/cli/format_test.cljs | 60 +- src/test/logseq/cli/main_test.cljs | 11 + src/test/logseq/cli/style_test.cljs | 23 + 14 files changed, 868 insertions(+), 390 deletions(-) create mode 100644 docs/agent-guide/023-logseq-cli-help-show-styling.md create mode 100644 src/main/logseq/cli/style.cljs create mode 100644 src/test/logseq/cli/style_test.cljs diff --git a/deps/cli/src/logseq/cli.cljs b/deps/cli/src/logseq/cli.cljs index e9dc1af7b0..f825c71595 100644 --- a/deps/cli/src/logseq/cli.cljs +++ b/deps/cli/src/logseq/cli.cljs @@ -6,17 +6,35 @@ [clojure.string :as string] [logseq.cli.common.graph :as cli-common-graph] [logseq.cli.spec :as cli-spec] + [logseq.cli.style :as style] [logseq.cli.text-util :as cli-text-util] [nbb.error] [promesa.core :as p])) +(defn- escape-regex + [value] + (string/replace value #"[\\.^$|?*+()\\[\\]{}]" "\\\\$&")) + +(defn- bold-command-names + [value commands] + (reduce (fn [acc command] + (let [pattern (re-pattern (str "(?m)^(\\s*)" (escape-regex command) "(\\s+)"))] + (string/replace acc pattern (fn [[_ prefix spacing]] + (str prefix (style/bold command) spacing))))) + value + commands)) + (defn- format-commands [{:keys [table]}] - (let [table (mapv (fn [{:keys [cmds desc spec]}] - (cond-> [(str (string/join " " cmds) - (when spec " [options]"))] - desc (conj desc))) - (filter (comp seq :cmds) table))] - (cli/format-table {:rows table}))) + (let [entries (->> table + (filter (comp seq :cmds))) + rows (mapv (fn [{:keys [cmds desc spec]}] + (cond-> [(str (string/join " " cmds) + (when spec " [options]"))] + desc (conj desc))) + entries) + commands (map (comp #(string/join " " %) :cmds) entries)] + (-> (cli/format-table {:rows rows}) + (bold-command-names commands)))) (def ^:private default-spec {:version {:coerce :boolean @@ -25,9 +43,10 @@ (declare table) (defn- print-general-help [_m] - (println (str "Usage: logseq [command] [options]\n\nOptions:\n" - (cli/format-opts {:spec default-spec}))) - (println (str "\nCommands:\n" (format-commands {:table table})))) + (println (str "Usage: logseq [command] [options]\n\n" + (style/bold "Options") ":\n" + (style/bold-options (cli/format-opts {:spec default-spec})))) + (println (str "\n" (style/bold "Commands") ":\n" (format-commands {:table table})))) (defn- default-command [{{:keys [version]} :opts :as m}] @@ -45,10 +64,11 @@ (str " " (string/join " " (map #(str "[" (name %) "]") (:args->opts cmd-map))))) (when (:spec cmd-map) - (str " [options]\n\nOptions:\n" - (cli/format-opts {:spec (:spec cmd-map)}))) + (str " [options]\n\n" (style/bold "Options") ":\n" + (style/bold-options (cli/format-opts {:spec (:spec cmd-map)})))) (when (:description cmd-map) - (str "\n\nDescription:\n" (cli-text-util/wrap-text (:description cmd-map) 80)))))) + (str "\n\n" (style/bold "Description") ":\n" + (cli-text-util/wrap-text (:description cmd-map) 80)))))) (defn- help-command [{{:keys [command help]} :opts}] (if-let [cmd-map (and command (some #(when (= command (first (:cmds %))) %) table))] @@ -56,7 +76,7 @@ ;; handle help --help (if-let [cmd-map (and help (some #(when (= "help" (first (:cmds %))) %) table))] (print-command-help "help" cmd-map) - (println "Command" (pr-str command) "does not exist")))) + (println (style/bold "Command") (pr-str command) "does not exist")))) (defn- lazy-load-fn "Lazy load fn to speed up start time. After nbb requires ~30 namespaces, start time gets close to 1s. @@ -146,9 +166,12 @@ (if (and (= :org.babashka/cli type') (= :require cause)) (do - (println "Error: Command missing required" - (if (get-in data [:spec option]) "option" "argument") - (pr-str (name option))) + (println (style/bold-keywords + (str "Error: Command missing required " + (if (get-in data [:spec option]) "option" "argument") + " " + (style/bold (pr-str (name option)))) + ["command" "option" "argument"])) (when-let [cmd-m (some #(when (= {:spec (:spec %) :require (:require %)} (select-keys data [:spec :require])) %) table)] diff --git a/docs/agent-guide/023-logseq-cli-help-show-styling.md b/docs/agent-guide/023-logseq-cli-help-show-styling.md new file mode 100644 index 0000000000..557697196a --- /dev/null +++ b/docs/agent-guide/023-logseq-cli-help-show-styling.md @@ -0,0 +1,108 @@ +# Logseq CLI Help And Show Styling Implementation Plan + +Goal: Add picocolors-based styling for help output and show human output in logseq-cli and db-worker-node. + +Architecture: Introduce a small shared styling helper that wraps picocolors and is used by CLI help renderers and show tree formatting. +Help output headings and error strings will apply bold to keywords while show output will color status labels and bold tag suffixes without changing data payloads. + +Tech Stack: ClojureScript, Node.js, picocolors, babashka.cli. + +Related: Relates to docs/agent-guide/022-logseq-cli-help-show-output.md. + +## Problem statement + +The current help output in logseq-cli and db-worker-node is plain text and does not emphasize command names or option names, which makes scanning harder. +The show command human output also does not visually differentiate status values or tags, which makes scanning large trees harder. +The tree glyphs in show output currently have the same visual weight as content, which makes the structure harder to scan. + +## Testing Plan + +I will add unit tests for help summary formatting that assert bold styling is applied to command names, option names, and error messages for missing options. +I will add unit tests for show tree text rendering that verify status labels are colorized and tag suffixes are bolded while preserving the existing tree glyph alignment when ANSI codes are stripped. +I will add a unit test that verifies tree glyphs are rendered in a lighter style without altering alignment when ANSI is stripped. +I will add a unit test for db-worker-node help output that asserts bold styling on command names and option names and that the help text still omits auth-token. +I will add unit tests for the styling helper to ensure it can be disabled for tests by stripping ANSI when comparing output. +NOTE: I will write all tests before I add any implementation behavior. + +## Scope and constraints + +This plan targets logseq-cli in src/main/logseq/cli and db-worker-node help output in src/main/frontend/worker/db_worker_node.cljs. +This plan must use picocolors for color and bold styling and should not change JSON or EDN output formats. +This plan should not introduce new CLI options unless required to gate coloring for tests. +Styling must only be applied when color is supported, and dumb terminals must receive plaintext output. +The `logseq -h` help output should omit the commands list section. + +## Files and ownership + +| Area | Path | Notes | +| --- | --- | --- | +| npm dependency | package.json | Add picocolors dependency used by ClojureScript Node targets. | +| npm lockfile | yarn.lock | Update to include picocolors. | +| CLI help summary | src/main/logseq/cli/command/core.cljs | Apply bold styling to command names and option names in help summaries and error text. | +| CLI show output | src/main/logseq/cli/command/show.cljs | Apply status color and tag bold styling in tree labels. | +| CLI show output | src/main/logseq/cli/command/show.cljs | Apply lighter styling to tree glyphs in human output. | +| CLI formatting helpers | src/main/logseq/cli/format.cljs | Avoid impacting non-human output, and ensure show uses styled message for human output only. | +| CLI legacy help | deps/cli/src/logseq/cli.cljs | Apply bold styling to command names and option names in help output for legacy cli entrypoint. | +| db-worker-node help | src/main/frontend/worker/db_worker_node.cljs | Apply bold styling to command names and option names in help output lines. | +| CLI tests | src/test/logseq/cli/commands_test.cljs | Update help summary and show tree tests to tolerate ANSI and assert styling intent. | +| CLI format tests | src/test/logseq/cli/format_test.cljs | Add or update tests to ensure human show output includes styled text but JSON and EDN do not. | +| db-worker-node tests | src/test/frontend/worker/db_worker_node_test.cljs | Extend help output test to validate bold styling. | + +## Implementation plan + +1. Add picocolors to package.json dependencies and update yarn.lock with the new dependency using the existing package manager. +2. Create a small styling helper in a new namespace such as src/main/logseq/cli/style.cljs that wraps picocolors functions for bold and color and exposes a no-color flag for tests. +3. Add a companion helper in a shared location for db-worker-node, or reuse the same namespace if it is available in that build target, to avoid duplicated color logic. +4. In src/main/logseq/cli/style.cljs, add a color support check that disables styling when color is not supported or TERM is dumb. +5. In src/main/logseq/cli/command/core.cljs, wrap help summary command names and option names with the new bold helper. +6. In src/main/logseq/cli/command/core.cljs, update the invalid options error formatting so missing required option names are bolded in the error message. +7. In deps/cli/src/logseq/cli.cljs, apply the same bold styling to command names and option names in the help output for the legacy cli entrypoint. +8. In src/main/frontend/worker/db_worker_node.cljs, update show-help! output to bold command names and option names in the help text. +9. In src/main/logseq/cli/command/show.cljs, add a status style function that maps known status labels to distinct colors, and bolds the status text, using picocolors. +10. In src/main/logseq/cli/command/show.cljs, update the tag suffix rendering to wrap each #tag with bold styling and ensure tags remain separated by spaces. +11. In src/main/logseq/cli/command/show.cljs, style the tree glyphs with a dim or gray color using picocolors while leaving ids and labels unstyled. +12. In src/main/logseq/cli/command/show.cljs, ensure status formatting and glyph styling are only applied to the human output path and do not alter the underlying data used for JSON or EDN outputs. +13. Update src/test/logseq/cli/commands_test.cljs to compare help summaries using an ANSI-stripping helper so assertions remain stable, and to assert bold styling for command and option names. +14. Update src/test/logseq/cli/commands_test.cljs show tree text tests to assert that the status prefix and tag suffix are styled when ANSI is preserved, and to verify tree alignment and glyph lightening using stripped output. +15. Add or update tests in src/test/logseq/cli/format_test.cljs to verify that human show output includes styled prefixes while JSON and EDN outputs remain unchanged. +16. Update src/test/frontend/worker/db_worker_node_test.cljs to assert that the help output bolds command and option names and still omits auth-token. +17. Run bb dev:lint-and-test to ensure all lint and unit tests pass. + +## Edge cases + +The status value may be a keyword with namespaces such as :logseq.property.status/todo and should still map to the same color for TODO. +The status label may be missing or blank, and the show output should remain unchanged in that case. +Tag labels may include uppercase or punctuation and should still render as bolded tags without losing the leading #. +Help output should still be readable when ANSI colors are not supported, and tests should be resilient by stripping ANSI sequences. +Tree glyph styling should not break alignment when ANSI codes are stripped. +Styling should be fully disabled when color is not supported or TERM is dumb. + +## Open questions + +Should picocolors styling be applied only when stdout is a TTY, or should it always render for human output regardless of terminal support. +Which specific status to color mapping is preferred for the full set of Logseq statuses such as NOW, LATER, WAITING, CANCELLED, and TODO variants. + +## Testing Details + +The tests will verify visible behavior by asserting that help output includes bolded command names and option names and that show output includes styled status and tags when rendered to human text. +The tests will also assert that JSON and EDN outputs remain unchanged and that ANSI codes do not break alignment by validating stripped output. +The tests will continue to avoid asserting internal data structures and instead focus on rendered output behavior. + +## Implementation Details + +- Use a small helper that can apply bold and color via picocolors and also expose a strip-ansi helper for tests. +- Keep styling limited to human output paths and avoid touching transport or data payloads. +- Centralize the status to color mapping in one function to keep future changes easy. +- Apply bold to command names and option names in help output and error strings. +- Preserve existing spacing and alignment by applying styling after label construction rather than before width calculations. +- Apply a lighter style to tree glyphs only, not to ids or labels. +- Gate styling behind color support checks so dumb terminals get plaintext output. +- Ensure any new helper is available to both the CLI and db-worker-node build targets. +- Update tests to use ANSI stripping for alignment assertions and explicit style presence for keyword checks. +- Avoid adding new configuration flags unless tests cannot reliably assert output without them. + +## Question + +Styling is limited to help info and show human output for now. + +--- diff --git a/package.json b/package.json index 4293f8118a..7c23865116 100644 --- a/package.json +++ b/package.json @@ -158,8 +158,8 @@ "@tabler/icons-react": "^2.47.0", "@tabler/icons-webfont": "^2.47.0", "@tippyjs/react": "4.2.5", - "bignumber.js": "^9.0.2", "better-sqlite3": "12.6.0", + "bignumber.js": "^9.0.2", "chokidar": "3.5.1", "chrono-node": "2.2.4", "codemirror": "5.65.18", @@ -185,6 +185,7 @@ "path-complete-extname": "1.0.0", "pdfjs-dist": "4.2.67", "photoswipe": "^5.3.7", + "picocolors": "^1.1.1", "pixi-graph-fork": "0.2.0", "pixi.js": "6.2.0", "posthog-js": "1.10.2", diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index de5d237cdf..68dbc89c93 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -8,6 +8,7 @@ [frontend.worker.platform.node :as platform-node] [frontend.worker.state :as worker-state] [lambdaisland.glogi :as log] + [logseq.cli.style :as style] [logseq.cli.data-dir :as data-dir] [logseq.db :as ldb] [promesa.core :as p])) @@ -225,11 +226,11 @@ (defn- show-help! [] - (println "db-worker-node options:") - (println " --data-dir (default ~/logseq/cli-graphs)") - (println " --repo (required)") - (println " --rtc-ws-url (optional)") - (println " --log-level (default info)") + (println (str (style/bold "db-worker-node") " " (style/bold "options") ":")) + (println (str " " (style/bold "--data-dir") " (default ~/logseq/cli-graphs)")) + (println (str " " (style/bold "--repo") " (required)")) + (println (str " " (style/bold "--rtc-ws-url") " (optional)")) + (println (str " " (style/bold "--log-level") " (default info)")) (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)")) (defn- pad2 diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 5889881ada..6f050506c2 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -2,6 +2,7 @@ "Shared CLI parsing utilities." (:require [babashka.cli :as cli] [clojure.string :as string] + [logseq.cli.style :as style] [logseq.common.config :as common-config])) (def ^:private global-spec* @@ -58,28 +59,33 @@ (map (fn [{:keys [cmds desc]}] (let [command (command-label cmds)] {:command command + :command-styled (style/bold command) :desc desc})))) width (apply max 0 (map (comp count :command) rows))] (->> rows - (map (fn [{:keys [command desc]}] + (map (fn [{:keys [command command-styled desc]}] (let [padding (apply str (repeat (- width (count command)) " "))] - (cond-> (str " " command padding) + (cond-> (str " " command-styled padding) (seq desc) (str " " desc))))) (string/join "\n")))) +(defn- format-opts + [spec] + (style/bold-options (cli/format-opts {:spec spec}))) + (defn group-summary [group table] (let [group-table (filter #(= group (first (:cmds %))) table)] (string/join "\n" [(str "Usage: logseq " group " [options]") "" - "Subcommands:" + (str (style/bold "Subcommands") ":") (format-commands group-table) "" - "Global options:" - (cli/format-opts {:spec global-spec*}) + (str "Global " (style/bold "options") ":") + (format-opts global-spec*) "" - "Command options:" + (str "Command " (style/bold "options") ":") (str " See `logseq " group " --help`")]))) (defn top-level-summary @@ -94,13 +100,13 @@ (string/join "\n" ["Usage: logseq [options]" "" - "Commands:" + (str (style/bold "Commands") ":") (string/join "\n\n" (map render-group groups)) "" - "Global options:" - (cli/format-opts {:spec global-spec*}) + (str "Global " (style/bold "options") ":") + (format-opts global-spec*) "" - "Command options:" + (str "Command " (style/bold "options") ":") " See `logseq --help`"]))) (defn command-summary @@ -109,11 +115,11 @@ (string/join "\n" [(str "Usage: logseq " (command-usage cmds spec)) "" - "Global options:" - (cli/format-opts {:spec global-spec*}) + (str "Global " (style/bold "options") ":") + (format-opts global-spec*) "" - "Command options:" - (cli/format-opts {:spec command-spec})]))) + (str "Command " (style/bold "options") ":") + (format-opts command-spec)]))) (defn normalize-opts [opts] @@ -139,7 +145,7 @@ [summary message] {:ok? false :error {:code :invalid-options - :message message} + :message (style/bold-options message)} :summary summary}) (defn unknown-command-result diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 7959bafff3..29f56ef0b3 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -5,6 +5,7 @@ [logseq.cli.command.id :as id-command] [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] + [logseq.cli.style :as style] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [promesa.core :as p])) @@ -97,7 +98,7 @@ (map tag-label) (remove string/blank?))] (when (seq labels) - (string/join " " (map #(str "#" %) labels))))) + (string/join " " (map #(style/bold (str "#" %)) labels))))) (defn- status-from-ident [ident] @@ -106,6 +107,24 @@ status (or (last parts) name*)] (string/upper-case status))) +(def ^:private status-color-map + {"TODO" style/yellow + "DOING" style/blue + "NOW" style/cyan + "LATER" style/magenta + "WAITING" style/magenta + "DONE" style/green + "CANCELED" style/red + "CANCELLED" style/red}) + +(defn- style-status + [status] + (when (seq status) + (let [label (str status) + lookup (string/upper-case label) + color-fn (get status-color-map lookup identity)] + (style/bold (color-fn label))))) + (defn- status-label [node] (let [status (:logseq.property/status node)] @@ -123,10 +142,11 @@ (let [title (:block/title node) content (:block/content node) status (status-label node) + status* (style-status status) uuid->label (:uuid->label node) text (or title content) base (cond - (and text (seq status)) (str status " " text) + (and text (seq status)) (str status* " " text) text text (:block/name node) (:block/name node) (:block/uuid node) (some-> (:block/uuid node) str)) @@ -397,6 +417,8 @@ id-padding (apply str (repeat (inc id-width) " ")) split-lines (fn [value] (string/split (or value "") #"\n")) + style-glyph (fn [value] + (style/dim value)) lines (atom []) walk (fn walk [node prefix] (let [children (:block/children node) @@ -408,10 +430,13 @@ rows (split-lines (label child)) first-row (first rows) rest-rows (rest rows) - line (str (pad-id child) " " prefix branch first-row)] + line (str (pad-id child) " " + (style-glyph prefix) + (style-glyph branch) + first-row)] (swap! lines conj line) (doseq [row rest-rows] - (swap! lines conj (str id-padding next-prefix row))) + (swap! lines conj (str id-padding (style-glyph next-prefix) row))) (walk child next-prefix)))))] (let [rows (split-lines (label root)) first-row (first rows) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index f44f82d110..65125bdf09 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -3,6 +3,7 @@ (:require [clojure.string :as string] [clojure.walk :as walk] [logseq.cli.command.core :as command-core] + [logseq.cli.style :as style] [logseq.common.util :as common-util])) (defn- normalize-json @@ -94,8 +95,9 @@ (defn- format-error [error] (let [{:keys [code message]} error - hint (error-hint error)] - (cond-> (str "Error (" (name (or code :error)) "): " message) + hint (error-hint error) + message* (style/bold-keywords message ["option" "command" "argument"])] + (cond-> (str "Error (" (name (or code :error)) "): " message*) hint (str "\nHint: " hint)))) (defn- maybe-ident-header diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index a0987d83d1..c6a998935a 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -13,8 +13,6 @@ [summary] (string/join "\n" ["logseq [options]" - "" - "Commands: list page, list tag, list property, add block, add page, move, remove, query, query list, show, graph list, graph create, graph switch, graph remove, graph validate, graph info, graph export, graph import, server list, server status, server start, server stop, server restart" "" "Options:" summary])) diff --git a/src/main/logseq/cli/style.cljs b/src/main/logseq/cli/style.cljs new file mode 100644 index 0000000000..239880285b --- /dev/null +++ b/src/main/logseq/cli/style.cljs @@ -0,0 +1,78 @@ +(ns logseq.cli.style + "CLI styling helpers based on picocolors." + (:require ["picocolors" :as pc] + [clojure.string :as string])) + +(def ansi-pattern + #"\u001b\[[0-9;]*m") + +(def ^:private option-pattern + #"--[A-Za-z0-9][A-Za-z0-9-]*") + +(def ^:dynamic *color-enabled?* + nil) + +(defn- term-dumb? + [] + (= "dumb" (some-> js/process .-env (aget "TERM")))) + +(defn- color-supported? + [] + (and (some-> js/process .-stdout .-isTTY) + (.-isColorSupported pc) + (not (term-dumb?)))) + +(defn- color-enabled? + [] + (if (some? *color-enabled?*) + (boolean *color-enabled?*) + (color-supported?))) + +(defn- ->text + [value] + (if (nil? value) + "" + (str value))) + +(defn strip-ansi + [value] + (string/replace (or value "") ansi-pattern "")) + +(defn- colors + [] + (.createColors pc (color-enabled?))) + +(defn- apply-style + [style-key value] + (let [text (->text value)] + (if (seq text) + (let [palette (colors) + style-fn (aget palette style-key)] + (if (fn? style-fn) + (style-fn text) + text)) + text))) + +(defn bold [value] (apply-style "bold" value)) +(defn dim [value] (apply-style "dim" value)) +(defn red [value] (apply-style "red" value)) +(defn green [value] (apply-style "green" value)) +(defn yellow [value] (apply-style "yellow" value)) +(defn blue [value] (apply-style "blue" value)) +(defn magenta [value] (apply-style "magenta" value)) +(defn cyan [value] (apply-style "cyan" value)) + +(defn bold-keywords + [value keywords] + (reduce (fn [acc word] + (let [pattern (js/RegExp. (str "\\b" word "\\b") "gi")] + (string/replace acc pattern (fn [match] + (bold match))))) + (->text value) + keywords)) + +(defn bold-options + [value] + (let [text (->text value)] + (string/replace text option-pattern (fn [match] + (bold match))))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 2cbfd92982..1dd3197349 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -1,15 +1,16 @@ (ns frontend.worker.db-worker-node-test - (:require ["http" :as http] + (:require ["fs" :as fs] + ["http" :as http] + ["path" :as node-path] [cljs.test :refer [async deftest is]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker-common.util :as worker-util] [frontend.worker.db-worker-node :as db-worker-node] [goog.object :as gobj] + [logseq.cli.style :as style] [logseq.db :as ldb] - [promesa.core :as p] - ["fs" :as fs] - ["path" :as node-path])) + [promesa.core :as p])) (defn- http-request [opts body] @@ -28,6 +29,17 @@ (.on req "error" reject) (finish!))))) +(defn- escape-regex + [value] + (let [pattern (js/RegExp. "[.*+?^${}()|[\\]\\\\]" "g")] + (string/replace value pattern "\\\\$&"))) + +(defn- contains-bold? + [value token] + (let [token (escape-regex token) + pattern (re-pattern (str "\\u001b\\[[0-9;]*m" token "\\u001b\\[[0-9;]*m"))] + (boolean (re-find pattern value)))) + (defn- http-get [host port path] (http-request {:hostname host @@ -97,60 +109,60 @@ (deftest db-worker-node-data-dir-permission-error (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly") - repo (str "logseq_db_perm_" (subs (str (random-uuid)) 0 8))] - (fs/chmodSync data-dir 365) - (-> (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - (p/then (fn [_] - (is false "expected data-dir permission error"))) - (p/catch (fn [e] - (let [data (ex-data e)] - (is (= :data-dir-permission (:code data))) - (is (= (node-path/resolve data-dir) (:path data)))))) - (p/finally (fn [] (done))))))) + (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly") + repo (str "logseq_db_perm_" (subs (str (random-uuid)) 0 8))] + (fs/chmodSync data-dir 365) + (-> (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + (p/then (fn [_] + (is false "expected data-dir permission error"))) + (p/catch (fn [e] + (let [data (ex-data e)] + (is (= :data-dir-permission (:code data))) + (is (= (node-path/resolve data-dir) (:path data)))))) + (p/finally (fn [] (done))))))) (deftest db-worker-node-creates-log-file (async done - (let [daemon (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-log") - repo (str "logseq_db_log_" (subs (str (random-uuid)) 0 8)) - log-file (log-path data-dir repo)] - (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - _ (reset! daemon {:stop! stop!}) - _ (p/delay 50)] - (is (fs/existsSync log-file))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (if-let [stop! (:stop! @daemon)] - (-> (stop!) (p/finally (fn [] (done)))) - (done)))))))) + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-log") + repo (str "logseq_db_log_" (subs (str (random-uuid)) 0 8)) + log-file (log-path data-dir repo)] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (p/delay 50)] + (is (fs/existsSync log-file))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) (deftest db-worker-node-log-file-has-entries (async done - (let [daemon (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-log-entries") - repo (str "logseq_db_log_entries_" (subs (str (random-uuid)) 0 8)) - log-file (log-path data-dir repo)] - (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - _ (reset! daemon {:stop! stop!}) - _ (invoke host port "thread-api/create-or-open-db" [repo {}]) - _ (p/delay 50) - contents (when (fs/existsSync log-file) - (.toString (fs/readFileSync log-file) "utf8"))] - (is (fs/existsSync log-file)) - (is (pos? (count contents)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (if-let [stop! (:stop! @daemon)] - (-> (stop!) (p/finally (fn [] (done)))) - (done)))))))) + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-log-entries") + repo (str "logseq_db_log_entries_" (subs (str (random-uuid)) 0 8)) + log-file (log-path data-dir repo)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + _ (p/delay 50) + contents (when (fs/existsSync log-file) + (.toString (fs/readFileSync log-file) "utf8"))] + (is (fs/existsSync log-file)) + (is (pos? (count contents)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) (deftest db-worker-node-log-retention (let [enforce-log-retention! #'db-worker-node/enforce-log-retention! @@ -201,8 +213,15 @@ (deftest db-worker-node-help-omits-auth-token (let [show-help! #'db-worker-node/show-help! - output (with-out-str (show-help!))] - (is (not (string/includes? output "--auth-token"))))) + output (binding [style/*color-enabled?* true] + (with-out-str (show-help!)))] + (is (not (string/includes? (style/strip-ansi output) "--auth-token"))) + (is (re-find #"\u001b\[[0-9;]*moptions\u001b\[[0-9;]*m:" output)) + (is (contains-bold? output "db-worker-node")) + (is (contains-bold? output "--data-dir")) + (is (contains-bold? output "--repo")) + (is (contains-bold? output "--rtc-ws-url")) + (is (contains-bold? output "--log-level")))) (deftest db-worker-node-repo-error-handles-keyword-methods (let [repo-error #'db-worker-node/repo-error @@ -222,233 +241,233 @@ (deftest db-worker-node-daemon-smoke-test (async done - (let [daemon (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-daemon") - repo (str "logseq_db_smoke_" (subs (str (random-uuid)) 0 8)) - now (js/Date.now) - page-uuid (random-uuid) - block-uuid (random-uuid)] - (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! - {:data-dir data-dir - :repo repo}) - health (http-get host port "/healthz") - ready (http-get host port "/readyz") - _ (do - (reset! daemon {:host host :port port :stop! stop!}) - (println "[db-worker-node-test] daemon started" {:host host :port port}) - (println "[db-worker-node-test] /healthz" health) - (is (= 200 (:status health))) - (println "[db-worker-node-test] /readyz" ready) - (is (= 200 (:status ready))) - (println "[db-worker-node-test] repo" repo)) - _ (invoke host port "thread-api/create-or-open-db" [repo {}]) - dbs (invoke host port "thread-api/list-db" []) - _ (do - (println "[db-worker-node-test] list-db" dbs) - (is (some #(= repo (:name %)) dbs))) - lock-file (lock-path data-dir repo) - _ (is (fs/existsSync lock-file)) - lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) - _ (is (= repo (gobj/get lock-contents "repo"))) - _ (is (= host (gobj/get lock-contents "host"))) - _ (invoke host port "thread-api/transact" - [repo - [{:block/uuid page-uuid - :block/title "Smoke Page" - :block/name "smoke-page" - :block/tags #{:logseq.class/Page} - :block/created-at now - :block/updated-at now} - {:block/uuid block-uuid - :block/title "Smoke Test" - :block/page [:block/uuid page-uuid] - :block/parent [:block/uuid page-uuid] - :block/order "a0" - :block/created-at now - :block/updated-at now}] - {} - nil]) - result (invoke host port "thread-api/q" + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-daemon") + repo (str "logseq_db_smoke_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid) + block-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! + {:data-dir data-dir + :repo repo}) + health (http-get host port "/healthz") + ready (http-get host port "/readyz") + _ (do + (reset! daemon {:host host :port port :stop! stop!}) + (println "[db-worker-node-test] daemon started" {:host host :port port}) + (println "[db-worker-node-test] /healthz" health) + (is (= 200 (:status health))) + (println "[db-worker-node-test] /readyz" ready) + (is (= 200 (:status ready))) + (println "[db-worker-node-test] repo" repo)) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + dbs (invoke host port "thread-api/list-db" []) + _ (do + (println "[db-worker-node-test] list-db" dbs) + (is (some #(= repo (:name %)) dbs))) + lock-file (lock-path data-dir repo) + _ (is (fs/existsSync lock-file)) + lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + _ (is (= repo (gobj/get lock-contents "repo"))) + _ (is (= host (gobj/get lock-contents "host"))) + _ (invoke host port "thread-api/transact" [repo - ['[:find ?e - :in $ ?uuid - :where [?e :block/uuid ?uuid]] - block-uuid]])] - (println "[db-worker-node-test] q result" result) - (is (seq result))) - (p/catch (fn [e] - (println "[db-worker-node-test] e:" e) - (is false (str e)))) - (p/finally (fn [] - (if-let [stop! (:stop! @daemon)] - (-> (stop!) - (p/finally (fn [] - (is (not (fs/existsSync (lock-path data-dir repo)))) - (done)))) - (done)))))))) + [{:block/uuid page-uuid + :block/title "Smoke Page" + :block/name "smoke-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now} + {:block/uuid block-uuid + :block/title "Smoke Test" + :block/page [:block/uuid page-uuid] + :block/parent [:block/uuid page-uuid] + :block/order "a0" + :block/created-at now + :block/updated-at now}] + {} + nil]) + result (invoke host port "thread-api/q" + [repo + ['[:find ?e + :in $ ?uuid + :where [?e :block/uuid ?uuid]] + block-uuid]])] + (println "[db-worker-node-test] q result" result) + (is (seq result))) + (p/catch (fn [e] + (println "[db-worker-node-test] e:" e) + (is false (str e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) + (p/finally (fn [] + (is (not (fs/existsSync (lock-path data-dir repo)))) + (done)))) + (done)))))))) (deftest db-worker-node-import-edn (async done - (let [daemon-a (atom nil) - daemon-b (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-import-edn") - repo-a (str "logseq_db_import_edn_a_" (subs (str (random-uuid)) 0 8)) - repo-b (str "logseq_db_import_edn_b_" (subs (str (random-uuid)) 0 8)) - now (js/Date.now) - page-uuid (random-uuid)] - (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-a}) - _ (reset! daemon-a {:stop! stop!}) - _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) - _ (invoke host port "thread-api/transact" - [repo-a - [{:block/uuid page-uuid - :block/title "Import Page" - :block/name "import-page" - :block/tags #{:logseq.class/Page} - :block/created-at now - :block/updated-at now}] - {} - nil]) - export-edn (invoke host port "thread-api/export-edn" [repo-a {:export-type :graph}])] - (is (map? export-edn)) - (p/let [_ ((:stop! @daemon-a)) - {:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-b}) - _ (reset! daemon-b {:stop! stop!}) - _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) - _ (invoke host port "thread-api/import-edn" [repo-b export-edn]) - result (invoke host port "thread-api/q" - [repo-b - ['[:find ?e - :in $ ?title - :where [?e :block/title ?title]] - "Import Page"]])] - (is (seq result)))) - (p/catch (fn [e] - (println "[db-worker-node-test] import-edn error:" e) - (is false (str e)))) - (p/finally (fn [] - (let [stop-a (:stop! @daemon-a) - stop-b (:stop! @daemon-b)] - (cond - (and stop-a stop-b) - (-> (stop-a) - (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) + (let [daemon-a (atom nil) + daemon-b (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-import-edn") + repo-a (str "logseq_db_import_edn_a_" (subs (str (random-uuid)) 0 8)) + repo-b (str "logseq_db_import_edn_b_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-a}) + _ (reset! daemon-a {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) + _ (invoke host port "thread-api/transact" + [repo-a + [{:block/uuid page-uuid + :block/title "Import Page" + :block/name "import-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now}] + {} + nil]) + export-edn (invoke host port "thread-api/export-edn" [repo-a {:export-type :graph}])] + (is (map? export-edn)) + (p/let [_ ((:stop! @daemon-a)) + {:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-b}) + _ (reset! daemon-b {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) + _ (invoke host port "thread-api/import-edn" [repo-b export-edn]) + result (invoke host port "thread-api/q" + [repo-b + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + "Import Page"]])] + (is (seq result)))) + (p/catch (fn [e] + (println "[db-worker-node-test] import-edn error:" e) + (is false (str e)))) + (p/finally (fn [] + (let [stop-a (:stop! @daemon-a) + stop-b (:stop! @daemon-b)] + (cond + (and stop-a stop-b) + (-> (stop-a) + (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) - stop-a - (-> (stop-a) (p/finally (fn [] (done)))) + stop-a + (-> (stop-a) (p/finally (fn [] (done)))) - stop-b - (-> (stop-b) (p/finally (fn [] (done)))) + stop-b + (-> (stop-b) (p/finally (fn [] (done)))) - :else - (done))))))))) + :else + (done))))))))) (deftest db-worker-node-import-db-base64 (async done - (let [daemon-a (atom nil) - daemon-b (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-import-sqlite") - repo-a (str "logseq_db_import_sqlite_a_" (subs (str (random-uuid)) 0 8)) - repo-b (str "logseq_db_import_sqlite_b_" (subs (str (random-uuid)) 0 8)) - now (js/Date.now) - page-uuid (random-uuid)] - (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-a}) - _ (reset! daemon-a {:stop! stop!}) - _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) - _ (invoke host port "thread-api/transact" - [repo-a - [{:block/uuid page-uuid - :block/title "SQLite Import Page" - :block/name "sqlite-import-page" - :block/tags #{:logseq.class/Page} - :block/created-at now - :block/updated-at now}] - {} - nil]) - export-base64 (invoke host port "thread-api/export-db-base64" [repo-a])] - (is (string? export-base64)) - (is (pos? (count export-base64))) - (p/let [_ ((:stop! @daemon-a)) - {:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-b}) - _ (reset! daemon-b {:stop! stop!}) - _ (invoke host port "thread-api/import-db-base64" [repo-b export-base64]) - _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) - result (invoke host port "thread-api/q" - [repo-b - ['[:find ?e - :in $ ?title - :where [?e :block/title ?title]] - "SQLite Import Page"]])] - (is (seq result)))) - (p/catch (fn [e] - (println "[db-worker-node-test] import-sqlite error:" e) - (is false (str e)))) - (p/finally (fn [] - (let [stop-a (:stop! @daemon-a) - stop-b (:stop! @daemon-b)] - (cond - (and stop-a stop-b) - (-> (stop-a) - (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) + (let [daemon-a (atom nil) + daemon-b (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-import-sqlite") + repo-a (str "logseq_db_import_sqlite_a_" (subs (str (random-uuid)) 0 8)) + repo-b (str "logseq_db_import_sqlite_b_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-a}) + _ (reset! daemon-a {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) + _ (invoke host port "thread-api/transact" + [repo-a + [{:block/uuid page-uuid + :block/title "SQLite Import Page" + :block/name "sqlite-import-page" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now}] + {} + nil]) + export-base64 (invoke host port "thread-api/export-db-base64" [repo-a])] + (is (string? export-base64)) + (is (pos? (count export-base64))) + (p/let [_ ((:stop! @daemon-a)) + {:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo-b}) + _ (reset! daemon-b {:stop! stop!}) + _ (invoke host port "thread-api/import-db-base64" [repo-b export-base64]) + _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) + result (invoke host port "thread-api/q" + [repo-b + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + "SQLite Import Page"]])] + (is (seq result)))) + (p/catch (fn [e] + (println "[db-worker-node-test] import-sqlite error:" e) + (is false (str e)))) + (p/finally (fn [] + (let [stop-a (:stop! @daemon-a) + stop-b (:stop! @daemon-b)] + (cond + (and stop-a stop-b) + (-> (stop-a) + (p/finally (fn [] (-> (stop-b) (p/finally (fn [] (done))))))) - stop-a - (-> (stop-a) (p/finally (fn [] (done)))) + stop-a + (-> (stop-a) (p/finally (fn [] (done)))) - stop-b - (-> (stop-b) (p/finally (fn [] (done)))) + stop-b + (-> (stop-b) (p/finally (fn [] (done)))) - :else - (done))))))))) + :else + (done))))))))) (deftest db-worker-node-repo-mismatch-test (async done - (let [daemon (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-repo-mismatch") - repo (str "logseq_db_mismatch_" (subs (str (random-uuid)) 0 8)) - other-repo (str repo "_other")] - (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - _ (reset! daemon {:host host :port port :stop! stop!}) - {:keys [status body]} (invoke-raw host port "thread-api/create-or-open-db" [other-repo {}]) - parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] - (is (= 409 status)) - (is (= false (:ok parsed))) - (is (= "repo-mismatch" (get-in parsed [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (if-let [stop! (:stop! @daemon)] - (-> (stop!) (p/finally (fn [] (done)))) - (done)))))))) + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-repo-mismatch") + repo (str "logseq_db_mismatch_" (subs (str (random-uuid)) 0 8)) + other-repo (str repo "_other")] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + {:keys [status body]} (invoke-raw host port "thread-api/create-or-open-db" [other-repo {}]) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (is (= 409 status)) + (is (= false (:ok parsed))) + (is (= "repo-mismatch" (get-in parsed [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) (deftest db-worker-node-lock-prevents-multiple-daemons (async done - (let [daemon (atom nil) - data-dir (node-helper/create-tmp-dir "db-worker-lock") - repo (str "logseq_db_lock_" (subs (str (random-uuid)) 0 8))] - (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - _ (reset! daemon {:stop! stop!})] - (-> (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) - (p/then (fn [_] - (is false "expected lock error"))) - (p/catch (fn [e] - (is (= :repo-locked (-> (ex-data e) :code))))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (if-let [stop! (:stop! @daemon)] - (-> (stop!) (p/finally (fn [] (done)))) - (done)))))))) + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-lock") + repo (str "logseq_db_lock_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!})] + (-> (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + (p/then (fn [_] + (is false "expected lock error"))) + (p/catch (fn [e] + (is (= :repo-locked (-> (ex-data e) :code))))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 66cfc12bae..a537401968 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -5,12 +5,32 @@ [logseq.cli.command.show :as show-command] [logseq.cli.commands :as commands] [logseq.cli.server :as cli-server] + [logseq.cli.style :as style] [logseq.cli.transport :as transport] [promesa.core :as p])) +(defn- strip-ansi + [value] + (style/strip-ansi value)) + +(defn- contains-ansi? + [value] + (boolean (re-find style/ansi-pattern value))) + +(defn- escape-regex + [value] + (let [pattern (js/RegExp. "[.*+?^${}()|[\\]\\\\]" "g")] + (string/replace value pattern "\\\\$&"))) + +(defn- contains-bold? + [value token] + (let [token (escape-regex token) + pattern (re-pattern (str "\\u001b\\[[0-9;]*m" token "\\u001b\\[[0-9;]*m"))] + (boolean (re-find pattern value)))) + (defn- command-lines [summary] - (let [lines (string/split-lines summary) + (let [lines (string/split-lines (strip-ansi summary)) section (if (some #{"Commands:"} lines) "Commands:" "Subcommands:") start (inc (.indexOf lines section)) end (.indexOf lines "Global options:") @@ -29,98 +49,157 @@ (deftest test-help-output (testing "top-level help lists command groups" - (let [result (commands/parse-args ["--help"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (not (string/includes? summary "--auth-token"))) - (is (not (string/includes? summary "--retries"))) - (is (string/includes? summary "Graph Inspect and Edit")) - (is (string/includes? summary "Graph Management")) - (is (string/includes? summary "list")) - (is (string/includes? summary "add")) - (is (string/includes? summary "remove")) - (is (string/includes? summary "move")) - (is (string/includes? summary "query")) - (is (string/includes? summary "show")) - (is (string/includes? summary "graph")) - (is (string/includes? summary "server")))) + (is (not (string/includes? plain-summary "--auth-token"))) + (is (not (string/includes? plain-summary "--retries"))) + (is (string/includes? plain-summary "Graph Inspect and Edit")) + (is (string/includes? plain-summary "Graph Management")) + (is (string/includes? plain-summary "list")) + (is (string/includes? plain-summary "add")) + (is (string/includes? plain-summary "remove")) + (is (string/includes? plain-summary "move")) + (is (string/includes? plain-summary "query")) + (is (string/includes? plain-summary "show")) + (is (string/includes? plain-summary "graph")) + (is (string/includes? plain-summary "server")) + (is (contains-bold? summary "list page")) + (is (contains-bold? summary "list tag")) + (is (contains-bold? summary "list property")) + (is (contains-bold? summary "add block")) + (is (contains-bold? summary "add page")) + (is (contains-bold? summary "remove")) + (is (contains-bold? summary "move")) + (is (contains-bold? summary "query")) + (is (contains-bold? summary "query list")) + (is (contains-bold? summary "show")) + (is (contains-bold? summary "graph list")) + (is (contains-bold? summary "graph create")) + (is (contains-bold? summary "server list")) + (is (contains-bold? summary "server start")) + (is (contains-bold? summary "--help")) + (is (contains-bold? summary "--repo")) + (is (re-find #"\u001b\[[0-9;]*mCommands\u001b\[[0-9;]*m:" summary)) + (is (re-find #"\u001b\[[0-9;]*moptions\u001b\[[0-9;]*m:" summary)))) (testing "top-level help command list omits [options]" - (let [summary (:summary (commands/parse-args ["--help"])) + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["--help"]))) lines (command-lines summary)] (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines)))) (testing "top-level help separates global and command options" - (let [summary (:summary (commands/parse-args ["--help"]))] - (is (string/includes? summary "Global options:")) - (is (string/includes? summary "Command options:"))))) + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["--help"]))) + plain-summary (strip-ansi summary)] + (is (string/includes? plain-summary "Global options:")) + (is (string/includes? plain-summary "Command options:"))))) (deftest test-parse-args-help (testing "graph group shows subcommands" - (let [result (commands/parse-args ["graph"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["graph"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "graph list")) - (is (string/includes? summary "graph create")) - (is (string/includes? summary "graph export")) - (is (string/includes? summary "graph import")))) + (is (string/includes? plain-summary "graph list")) + (is (string/includes? plain-summary "graph create")) + (is (string/includes? plain-summary "graph export")) + (is (string/includes? plain-summary "graph import")) + (is (contains-bold? summary "graph list")) + (is (contains-bold? summary "graph create")) + (is (contains-bold? summary "graph export")) + (is (contains-bold? summary "graph import")))) (testing "list group shows subcommands" - (let [result (commands/parse-args ["list"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["list"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "list page")) - (is (string/includes? summary "list tag")) - (is (string/includes? summary "list property")) - (is (string/includes? summary "Global options:")) - (is (string/includes? summary "Command options:")))) + (is (string/includes? plain-summary "list page")) + (is (string/includes? plain-summary "list tag")) + (is (string/includes? plain-summary "list property")) + (is (contains-bold? summary "list page")) + (is (contains-bold? summary "list tag")) + (is (contains-bold? summary "list property")) + (is (string/includes? plain-summary "Global options:")) + (is (string/includes? plain-summary "Command options:")))) (testing "add group shows subcommands" - (let [result (commands/parse-args ["add"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["add"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "add block")) - (is (string/includes? summary "add page")))) + (is (string/includes? plain-summary "add block")) + (is (string/includes? plain-summary "add page")) + (is (contains-bold? summary "add block")) + (is (contains-bold? summary "add page")))) (testing "remove command shows help" - (let [result (commands/parse-args ["remove" "--help"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["remove" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "Usage: logseq remove")) - (is (string/includes? summary "Command options:")))) + (is (string/includes? plain-summary "Usage: logseq remove")) + (is (string/includes? plain-summary "Command options:")) + (is (contains-bold? summary "--id")) + (is (contains-bold? summary "--uuid")) + (is (contains-bold? summary "--page")))) (testing "move command shows help" - (let [result (commands/parse-args ["move" "--help"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["move" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "Usage: logseq move")) - (is (string/includes? summary "Command options:")))) + (is (string/includes? plain-summary "Usage: logseq move")) + (is (string/includes? plain-summary "Command options:")) + (is (contains-bold? summary "--id")) + (is (contains-bold? summary "--uuid")) + (is (contains-bold? summary "--target-id")) + (is (contains-bold? summary "--target-uuid")))) (testing "server group shows subcommands" - (let [result (commands/parse-args ["server"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["server"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "server list")) - (is (string/includes? summary "server start")))) + (is (string/includes? plain-summary "server list")) + (is (string/includes? plain-summary "server start")) + (is (contains-bold? summary "server list")) + (is (contains-bold? summary "server start")))) (testing "query group shows subcommands" - (let [result (commands/parse-args ["query"]) - summary (:summary result)] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["query"])) + summary (:summary result) + plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? summary "query list")) - (is (string/includes? summary "query")))) + (is (string/includes? plain-summary "query list")) + (is (string/includes? plain-summary "query")) + (is (contains-bold? summary "query list")) + (is (contains-bold? summary "query")))) (testing "group help command list omits [options]" - (let [summary (:summary (commands/parse-args ["list"])) + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["list"]))) lines (command-lines summary)] (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) (deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" - (let [result (commands/parse-args ["graph"]) - summary (:summary result) + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["graph"])) + summary (strip-ansi (:summary result)) subcommand-lines (let [lines (string/split-lines summary) start (inc (.indexOf lines "Subcommands:"))] (->> lines @@ -134,8 +213,9 @@ (is (apply = desc-starts)))) (testing "list group aligns subcommand columns" - (let [result (commands/parse-args ["list"]) - summary (:summary result) + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["list"])) + summary (strip-ansi (:summary result)) subcommand-lines (let [lines (string/split-lines summary) start (inc (.indexOf lines "Subcommands:"))] (->> lines @@ -208,12 +288,18 @@ :block/children [{:db/id 3 :block/title "Grandchild A1"}]} {:db/id 4 - :block/title "Child B"}]}}] + :block/title "Child B"}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (contains-ansi? output)) + (is (string/includes? output (style/dim "├── "))) + (is (string/includes? output (style/dim "└── "))) + (is (string/includes? output (style/dim "│ "))) (is (= (str "1 Root\n" "2 ├── Child A\n" "3 │ └── Grandchild A1\n" "4 └── Child B") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-aligns-mixed-id-widths (testing "show tree text aligns glyph column with mixed-width ids" @@ -225,12 +311,14 @@ :block/children [{:db/id 3 :block/title "Grand"}]} {:db/id 1000 - :block/title "Child B"}]}}] + :block/title "Child B"}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] (is (= (str "7 Root\n" "88 ├── Child A\n" "3 │ └── Grand\n" "1000 └── Child B") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-multiline (testing "show tree text renders multiline blocks under glyph column" @@ -244,14 +332,16 @@ {:db/id 174 :block/title "block-line1\nblock-line2"} {:db/id 175 - :block/title "cccc"}]}}] + :block/title "cccc"}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] (is (= (str "168 Jan 18th, 2026\n" "169 ├── b1\n" "173 ├── aaaxx\n" "174 ├── block-line1\n" " │ block-line2\n" "175 └── cccc") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-prefixes-status (testing "show tree text prefixes status before block titles" @@ -263,10 +353,14 @@ :block/children [{:db/id 2 :block/title "Child" :logseq.property/status {:db/ident :logseq.property/status.canceled - :block/title "CANCELED"}}]}}] + :block/title "CANCELED"}}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (string/includes? output (style/bold "TODO"))) + (is (string/includes? output (style/bold "CANCELED"))) (is (= (str "1 TODO Root\n" "2 └── CANCELED Child") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-status-multiline-alignment (testing "show tree text keeps multiline alignment when status prefix is present" @@ -276,11 +370,13 @@ :block/children [{:db/id 22 :block/title "line1\nline2" :logseq.property/status {:db/ident :logseq.property/status.todo - :block/title "TODO"}}]}}] + :block/title "TODO"}}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] (is (= (str "1 Root\n" "22 └── TODO line1\n" " line2") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-linked-references-tree (testing "show tree text renders linked references as trees with db/id in first column" @@ -297,7 +393,10 @@ {:db/id 11 :block/title "Ref B" :block/page {:db/id 101 - :block/title "Page B"}}]}}] + :block/title "Page B"}}]}} + output (binding [style/*color-enabled?* true] + (tree->text-with-linked-refs tree-data))] + (is (re-find #"\u001b\[[0-9;]*mTODO" output)) (is (= (str "1 Root\n" "\n" "Linked References (2)\n" @@ -306,7 +405,7 @@ "\n" "101 Page B\n" "11 └── Ref B") - (tree->text-with-linked-refs tree-data)))))) + (strip-ansi output)))))) (deftest test-tree->text-appends-tags (testing "show tree text appends block tags to content" @@ -316,10 +415,30 @@ :block/children [{:db/id 2 :block/title "Child" :block/tags [{:block/title "RTC"} - {:block/name "task"}]}]}}] + {:block/name "task"}]}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (string/includes? output (style/bold "#RTC"))) + (is (string/includes? output (style/bold "#task"))) (is (= (str "1 Root\n" "2 └── Child #RTC #task") - (tree->text tree-data)))))) + (strip-ansi output)))))) + +(deftest test-tree->text-status-colors + (testing "show tree text uses green for DONE status" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child" + :logseq.property/status {:db/ident :logseq.property/status.done + :block/title "DONE"}}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (string/includes? output (style/green "DONE"))) + (is (= (str "1 Root\n" + "2 └── DONE Child") + (strip-ansi output)))))) (deftest test-tree->text-replaces-uuid-refs (testing "show tree text replaces inline [[uuid]] with referenced block content recursively" @@ -329,16 +448,22 @@ tree-data {:root {:db/id 1 :block/title (str "See [[" uuid "]]")} :uuid->label {(string/lower-case uuid) (str "Target [[" nested "]]") - (string/lower-case nested) "Inner"}}] + (string/lower-case nested) "Inner"}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] (is (= (str "1 See [[Target [[Inner]]]]") - (tree->text tree-data)))))) + (strip-ansi output)))))) (deftest test-help-tags-properties-identifiers (testing "add help mentions tag and property identifiers" - (let [summary (:summary (commands/parse-args ["add" "block" "--help"]))] - (is (string/includes? summary "Identifiers can be id, :db/ident, or :block/title."))) - (let [summary (:summary (commands/parse-args ["add" "page" "--help"]))] - (is (string/includes? summary "Identifiers can be id, :db/ident, or :block/title."))))) + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["add" "block" "--help"])))] + (is (string/includes? (strip-ansi summary) + "Identifiers can be id, :db/ident, or :block/title."))) + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["add" "page" "--help"])))] + (is (string/includes? (strip-ansi summary) + "Identifiers can be id, :db/ident, or :block/title."))))) (deftest test-show-json-edn-strips-block-uuid (testing "show json/edn removes :block/uuid recursively while keeping :db/id" @@ -623,7 +748,7 @@ (testing "show rejects invalid id edn" (let [result (commands/parse-args ["show" "--id" "[1"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code])))))) + (is (= :invalid-options (get-in result [:error :code]))))) (testing "show rejects legacy page-name option" (let [result (commands/parse-args ["show" "--page-name" "Home"])] @@ -633,7 +758,7 @@ (testing "show rejects format option" (let [result (commands/parse-args ["show" "--format" "json" "--page" "Home"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-query (testing "query shows group help" diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 76f29de945..4532743395 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -1,6 +1,9 @@ (ns logseq.cli.format-test (:require [cljs.test :refer [deftest is testing]] - [logseq.cli.format :as format])) + [clojure.string :as string] + [logseq.cli.command.show :as show-command] + [logseq.cli.format :as format] + [logseq.cli.style :as style])) (deftest test-format-success (testing "json output via output-format" @@ -167,6 +170,61 @@ {:output-format nil})] (is (= "Line 1\nLine 2" result))))) +(deftest test-human-output-show-styled-prefixes + (testing "show preserves styled status and tags in human output" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child" + :logseq.property/status {:db/ident :logseq.property/status.todo + :block/title "TODO"} + :block/tags [{:block/title "TagA"}]}]}} + styled (binding [style/*color-enabled?* true] + (tree->text tree-data)) + result (format/format-result {:status :ok + :command :show + :data {:message styled}} + {:output-format nil})] + (is (string/includes? result (style/bold "TODO"))) + (is (string/includes? result (style/bold "#TagA"))) + (is (= (str "1 Root\n" + "2 └── TODO Child #TagA") + (style/strip-ansi result)))))) + +(deftest test-human-output-show-preserves-styling + (testing "show returns styled text without stripping ANSI" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child"}]}} + styled (binding [style/*color-enabled?* true] + (tree->text tree-data)) + result (format/format-result {:status :ok + :command :show + :data {:message styled}} + {:output-format nil})] + (is (= styled result)) + (is (re-find #"\u001b\[[0-9;]*m" result))))) + +(deftest test-show-json-edn-output-ignores-styled-message + (testing "show json/edn outputs serialize data without ANSI styling" + (let [tree-data {:root {:db/id 1 + :block/title "Root"}} + json-result (format/format-result {:status :ok + :command :show + :data tree-data} + {:output-format :json}) + edn-result (format/format-result {:status :ok + :command :show + :data tree-data} + {:output-format :edn})] + (is (string/includes? json-result "\"root\"")) + (is (string/includes? edn-result ":root")) + (is (not (re-find #"\u001b\[[0-9;]*m" json-result))) + (is (not (re-find #"\u001b\[[0-9;]*m" edn-result)))))) + (deftest test-human-output-query (testing "query renders raw result" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/main_test.cljs b/src/test/logseq/cli/main_test.cljs index 3fb098bf88..d211a998aa 100644 --- a/src/test/logseq/cli/main_test.cljs +++ b/src/test/logseq/cli/main_test.cljs @@ -14,3 +14,14 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done)))))) + +(deftest test-help-output-omits-command-list + (async done + (-> (p/let [result (cli-main/run! ["--help"] {:exit? false}) + output (:output result)] + (is (= 0 (:exit-code result))) + (is (not (string/includes? output "Commands: list page")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))) + (p/finally done)))) diff --git a/src/test/logseq/cli/style_test.cljs b/src/test/logseq/cli/style_test.cljs new file mode 100644 index 0000000000..808f6559fa --- /dev/null +++ b/src/test/logseq/cli/style_test.cljs @@ -0,0 +1,23 @@ +(ns logseq.cli.style-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.style :as style])) + +(deftest test-strip-ansi + (testing "strip-ansi removes ANSI sequences" + (is (= "Hello" + (style/strip-ansi "\u001b[1mHello\u001b[22m"))) + (is (= "Hello" + (style/strip-ansi "\u001b[31mHello\u001b[39m"))))) + +(deftest test-style-disabled + (testing "style helpers return plain text when color is disabled" + (binding [style/*color-enabled?* false] + (is (= "Hi" (style/bold "Hi"))) + (is (= "Hi" (style/dim "Hi"))) + (is (= "Hi" (style/green "Hi")))))) + +(deftest test-style-enabled + (testing "style helpers include ANSI when color is enabled" + (binding [style/*color-enabled?* true] + (is (not= "Hi" (style/bold "Hi"))) + (is (re-find #"\u001b\[[0-9;]*m" (style/bold "Hi")))))) From 598aad055fb57ad4c315a68f3d5b299dda8ccc2b Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 31 Jan 2026 15:58:35 +0800 Subject: [PATCH 053/375] 024-logseq-cli-show-updates.md --- .../024-logseq-cli-show-updates.md | 99 ++++++++++++ src/main/logseq/cli/command/show.cljs | 46 ++++-- src/test/logseq/cli/commands_test.cljs | 146 +++++++++++++++++- 3 files changed, 274 insertions(+), 17 deletions(-) create mode 100644 docs/agent-guide/024-logseq-cli-show-updates.md diff --git a/docs/agent-guide/024-logseq-cli-show-updates.md b/docs/agent-guide/024-logseq-cli-show-updates.md new file mode 100644 index 0000000000..4bc6d1d19e --- /dev/null +++ b/docs/agent-guide/024-logseq-cli-show-updates.md @@ -0,0 +1,99 @@ +# Logseq CLI Show Output & Linked References Options Plan + +Goal: Update the `logseq show` command to (1) render the ID column in human output with lighter styling (matching tree glyphs), (2) include `:db/ident` on block entities in JSON/EDN output when present, and (3) add an option to toggle Linked References (default enabled). +Architecture: Keep the existing logseq-cli → db-worker-node HTTP transport and thread-api calls; adjust show command selectors and output formatting only. +Tech Stack: ClojureScript, babashka.cli, db-worker-node transport, Logseq pull selectors, CLI styling helpers. +Related: Builds on `logseq.cli.command.show` tree rendering and linked references fetch; see `docs/agent-guide/010-logseq-cli-show-linked-references.md` and `docs/agent-guide/023-logseq-cli-help-show-styling.md`. + +## Problem Statement + +The `show` command currently renders the ID column with the same styling as regular text, does not expose `:db/ident` in JSON/EDN output for blocks, and always fetches/prints Linked References. We need to make the ID column visually lighter, surface `:db/ident` when it exists, and add a switch to disable linked references (defaulting to true). + +## Non-Goals + +- Changing db-worker-node APIs or transport protocols. +- Altering the show command’s existing tree structure or block ordering. +- Changing JSON/EDN output structure beyond adding `:db/ident` when present. + +## Current Behavior (Key Points) + +- Human output is rendered by `tree->text` in `src/main/logseq/cli/command/show.cljs`, with tree glyphs styled via `style/dim` but IDs unstyled. +- JSON/EDN output pulls specific selectors via `tree-block-selector` / `linked-ref-selector` and strips only `:block/uuid`. +- Linked References are always fetched and rendered via `fetch-linked-references` + `tree->text-with-linked-refs`. + +## Proposed Changes + +1) **Lighter ID column in human output** + - Style the padded ID column with the same dim styling used for tree glyphs. + - Apply the change in `tree->text` so both root and child rows render IDs dimmed. + +2) **Include `:db/ident` in JSON/EDN for blocks (when present)** + - Add `:db/ident` to block pull selectors: + - `tree-block-selector` (children) + - `linked-ref-selector` (linked references) + - root entity pulls in `fetch-tree` for `:id`, `:uuid`, and `:page` + - No post-processing is required because absent attributes will not appear in the pulled maps. + +3) **Optional Linked References (default true)** + - Add a boolean option `--linked-references` to `show-spec`, defaulting to true. + - Pass the option through `build-action` and into `build-tree-data`. + - When disabled: + - Skip `fetch-linked-references` entirely. + - Omit the `Linked References` section from human output. + - Omit `:linked-references` from JSON/EDN output. + +## Implementation Plan + +1) **Add show option wiring** + - Update `show-spec` in `src/main/logseq/cli/command/show.cljs` with a new boolean option (choose name + description, note default true). + - Extend `build-action` to include the option in `:action` (e.g., `:linked-references?`). + - Update `invalid-options?` if needed for any new validation. + +2) **Gate linked references fetching/rendering** + - Update `build-tree-data` to honor the new option: + - If enabled, keep current logic. + - If disabled, skip `fetch-linked-references`, avoid UUID label resolution based on linked refs, and avoid `tree->text-with-linked-refs`. + - Update `execute-show` to select `tree->text` vs `tree->text-with-linked-refs` based on the option. + - Ensure JSON/EDN output omits `:linked-references` when disabled. + +3) **Add `:db/ident` to selectors** + - Update `tree-block-selector`, `linked-ref-selector`, and root entity pull vectors in `fetch-tree` to include `:db/ident`. + - Ensure this does not add nil values (pull should omit missing attributes). + +4) **Dim ID column in human output** + - In `tree->text`, apply `style/dim` to the padded ID string (e.g., `pad-id` output) for both root and child rows. + - Keep existing branch/prefix dim styling unchanged. + +5) **Help output updates** + - Ensure `logseq show --help` lists the new linked references option (covered automatically by `show-spec`). + +## Testing Plan + +- Update or add unit tests in: + - `src/test/logseq/cli/format_test.cljs` to verify: + - ANSI dim styling is applied to the ID column in human output when color is enabled. + - No regressions in strip-ansi output. + - `src/test/logseq/cli/commands_test.cljs` to verify: + - `logseq show --help` includes the new linked references option. +- Add show-specific tests for JSON/EDN output: + - Ensure `:db/ident` appears when present (use a stubbed tree map or test fixtures if available). + - Ensure `:linked-references` is omitted when the option disables them. + +## Edge Cases + +- Blocks with no `:db/ident` should not gain a nil or empty key in output. +- When linked refs are disabled, UUID label resolution should not depend on linked refs (ensure `collect-uuid-refs` handles empty refs). +- Multi-id output should still honor the linked references option per target and not render linked refs when disabled. + +## Files to Touch + +- `src/main/logseq/cli/command/show.cljs` (options, selectors, tree output, linked refs gating) +- `src/test/logseq/cli/format_test.cljs` (ID dim styling test) +- `src/test/logseq/cli/commands_test.cljs` (help output) +- Possibly `src/test/logseq/cli/command_show_test.cljs` or similar if a dedicated show test file exists + +## Open Questions + +- None. + +--- diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 29f56ef0b3..5ba7dc9d78 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -14,6 +14,8 @@ {:id {:desc "Block db/id or EDN vector of ids"} :uuid {:desc "Block UUID"} :page {:desc "Page name"} + :linked-references {:desc "Include linked references (default true)" + :coerce :boolean} :level {:desc "Limit tree depth (default 10)" :coerce :long}}) @@ -38,6 +40,7 @@ (def ^:private tree-block-selector [:db/id + :db/ident :block/uuid :block/title :block/content @@ -48,6 +51,7 @@ (def ^:private linked-ref-selector [:db/id + :db/ident :block/uuid :block/title :block/content @@ -342,7 +346,7 @@ (cond (some? id) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title + [repo [:db/id :db/ident :block/name :block/uuid :block/title {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] id])] @@ -360,7 +364,7 @@ (if-not (common-util/uuid-string? uuid-str) (p/rejected (ex-info "block must be a uuid" {:code :invalid-block})) (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title + [repo [:db/id :db/ident :block/name :block/uuid :block/title {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] @@ -368,7 +372,7 @@ entity (if (:db/id entity) entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/name :block/uuid :block/title + [repo [:db/id :db/ident :block/name :block/uuid :block/title {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] @@ -385,7 +389,7 @@ (seq page) (p/let [page-entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/title + [repo [:db/id :db/ident :block/uuid :block/title {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/name page]])] @@ -414,7 +418,7 @@ (let [id-str (str (node-id node)) padding (max 0 (- id-width (count id-str)))] (str id-str (apply str (repeat padding " "))))) - id-padding (apply str (repeat (inc id-width) " ")) + id-padding (style/dim (apply str (repeat (inc id-width) " "))) split-lines (fn [value] (string/split (or value "") #"\n")) style-glyph (fn [value] @@ -430,7 +434,7 @@ rows (split-lines (label child)) first-row (first rows) rest-rows (rest rows) - line (str (pad-id child) " " + line (str (style/dim (pad-id child)) " " (style-glyph prefix) (style-glyph branch) first-row)] @@ -441,7 +445,7 @@ (let [rows (split-lines (label root)) first-row (first rows) rest-rows (rest rows)] - (swap! lines conj (str (pad-id root) " " first-row)) + (swap! lines conj (str (style/dim (pad-id root)) " " first-row)) (doseq [row rest-rows] (swap! lines conj (str id-padding row)))) (walk root "") @@ -483,6 +487,9 @@ :id (when (and (seq ids) (not multi-id?)) (first ids)) :ids ids :multi-id? multi-id? + :linked-references? (if (contains? options :linked-references) + (:linked-references options) + true) :uuid (:uuid options) :page (:page options) :level (:level options)}}))))) @@ -491,14 +498,16 @@ [config action] (p/let [tree-data (fetch-tree config action) root-id (get-in tree-data [:root :db/id]) - linked-refs (if root-id - (fetch-linked-references config (:repo action) root-id) - {:count 0 :blocks []}) - uuid-refs (collect-uuid-refs tree-data linked-refs) + linked-enabled? (not= false (:linked-references? action)) + linked-refs (when (and linked-enabled? root-id) + (fetch-linked-references config (:repo action) root-id)) + linked-refs* (if linked-enabled? + (or linked-refs {:count 0 :blocks []}) + {:count 0 :blocks []}) + uuid-refs (collect-uuid-refs tree-data linked-refs*) uuid->label (fetch-uuid-labels config (:repo action) uuid-refs) - tree-data (assoc tree-data - :linked-references linked-refs - :uuid->label uuid->label) + tree-data (cond-> (assoc tree-data :uuid->label uuid->label) + linked-enabled? (assoc :linked-references linked-refs*)) tree-data (resolve-uuid-refs-in-tree-data tree-data uuid->label)] tree-data)) @@ -576,6 +585,9 @@ results)) sanitize-tree (fn [tree] (strip-block-uuid tree)) + render-tree (if (false? (:linked-references? action)) + tree->text + tree->text-with-linked-refs) payload (case format :edn {:status :ok @@ -599,7 +611,7 @@ :data {:message (string/join multi-id-delimiter (map (fn [{:keys [ok? tree id error]}] (if ok? - (tree->text-with-linked-refs tree) + (render-tree tree) (multi-id-error-message id error))) results))}})] payload) @@ -618,4 +630,6 @@ :output-format :json}) {:status :ok - :data {:message (tree->text-with-linked-refs tree-data)}})))))) + :data {:message (if (false? (:linked-references? action)) + (tree->text tree-data) + (tree->text-with-linked-refs tree-data))}})))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index a537401968..1588219715 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -301,6 +301,22 @@ "4 └── Child B") (strip-ansi output)))))) +(deftest test-tree->text-dims-id-column + (testing "show tree text dims the id column" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child"}]}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (contains-ansi? output)) + (is (string/includes? output (style/dim "1"))) + (is (string/includes? output (style/dim "2"))) + (is (= (str "1 Root\n" + "2 └── Child") + (strip-ansi output)))))) + (deftest test-tree->text-aligns-mixed-id-widths (testing "show tree text aligns glyph column with mixed-width ids" (let [tree->text #'show-command/tree->text @@ -488,6 +504,129 @@ (is (= 1 (get-in stripped [:root :db/id]))) (is (= 2 (get-in stripped [:root :block/children 0 :db/id])))))) +(deftest test-fetch-tree-includes-db-ident + (async done + (let [fetch-tree #'show-command/fetch-tree + selectors (atom []) + orig-invoke transport/invoke] + (set! transport/invoke (fn [_ method _ args] + (when (= method :thread-api/pull) + (swap! selectors conj (second args))) + (case method + :thread-api/pull (p/resolved {:db/id 1 + :block/page {:db/id 2}}) + :thread-api/q (p/resolved []) + (p/resolved nil)))) + (-> (p/let [_ (fetch-tree {} {:repo "demo" :id 1})] + (is (some #(some #{:db/ident} %) @selectors))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-fetch-blocks-for-page-includes-db-ident + (async done + (let [fetch-blocks-for-page #'show-command/fetch-blocks-for-page + selectors (atom []) + orig-invoke transport/invoke] + (set! transport/invoke (fn [_ method _ args] + (when (= method :thread-api/q) + (let [[_ [query _]] args + pull-form (second query) + selector (nth pull-form 2)] + (swap! selectors conj selector))) + (p/resolved []))) + (-> (p/let [_ (fetch-blocks-for-page {} "demo" 1)] + (is (some #(some #{:db/ident} %) @selectors))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-fetch-linked-references-includes-db-ident + (async done + (let [fetch-linked-references #'show-command/fetch-linked-references + selectors (atom []) + orig-invoke transport/invoke] + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + :thread-api/pull (let [[_ selector _] args] + (swap! selectors conj selector) + (p/resolved {:db/id 10})) + (p/resolved nil)))) + (-> (p/let [_ (fetch-linked-references {} "demo" 1)] + (is (some #(and (some #{:db/ident} %) + (some (fn [entry] + (and (map? entry) + (contains? entry :block/page))) + %)) + @selectors))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-build-tree-data-linked-references-disabled + (async done + (let [build-tree-data #'show-command/build-tree-data + linked-called? (atom false) + orig-fetch-tree show-command/fetch-tree + orig-fetch-linked show-command/fetch-linked-references + orig-collect-uuid-refs show-command/collect-uuid-refs + orig-fetch-uuid-labels show-command/fetch-uuid-labels] + (set! show-command/fetch-tree (fn [_ _] + (p/resolved {:root {:db/id 1}}))) + (set! show-command/fetch-linked-references (fn [& _] + (reset! linked-called? true) + (p/resolved {:count 1 :blocks []}))) + (set! show-command/collect-uuid-refs (fn [_ _] [])) + (set! show-command/fetch-uuid-labels (fn [& _] (p/resolved {}))) + (-> (p/let [result (build-tree-data {} {:repo "demo" + :linked-references? false})] + (is (false? @linked-called?)) + (is (not (contains? result :linked-references)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! show-command/fetch-tree orig-fetch-tree) + (set! show-command/fetch-linked-references orig-fetch-linked) + (set! show-command/collect-uuid-refs orig-collect-uuid-refs) + (set! show-command/fetch-uuid-labels orig-fetch-uuid-labels) + (done))))))) + +(deftest test-build-tree-data-linked-references-enabled + (async done + (let [build-tree-data #'show-command/build-tree-data + linked-called? (atom false) + linked {:count 1 :blocks []} + orig-fetch-tree show-command/fetch-tree + orig-fetch-linked show-command/fetch-linked-references + orig-collect-uuid-refs show-command/collect-uuid-refs + orig-fetch-uuid-labels show-command/fetch-uuid-labels] + (set! show-command/fetch-tree (fn [_ _] + (p/resolved {:root {:db/id 1}}))) + (set! show-command/fetch-linked-references (fn [& _] + (reset! linked-called? true) + (p/resolved linked))) + (set! show-command/collect-uuid-refs (fn [_ _] [])) + (set! show-command/fetch-uuid-labels (fn [& _] (p/resolved {}))) + (-> (p/let [result (build-tree-data {} {:repo "demo" + :linked-references? true})] + (is (true? @linked-called?)) + (is (= linked (:linked-references result)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! show-command/fetch-tree orig-fetch-tree) + (set! show-command/fetch-linked-references orig-fetch-linked) + (set! show-command/collect-uuid-refs orig-collect-uuid-refs) + (set! show-command/fetch-uuid-labels orig-fetch-uuid-labels) + (done))))))) + (deftest test-tree->text-uuid-ref-recursion-limit (testing "show tree text limits uuid ref replacement depth" (let [tree->text #'show-command/tree->text @@ -758,7 +897,12 @@ (testing "show rejects format option" (let [result (commands/parse-args ["show" "--format" "json" "--page" "Home"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code])))))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "show help lists linked references option" + (let [summary (:summary (binding [style/*color-enabled?* true] + (commands/parse-args ["show" "--help"])))] + (is (string/includes? (strip-ansi summary) "--linked-references"))))) (deftest test-verb-subcommand-parse-query (testing "query shows group help" From 95353158a879659611cb3dc54b3bafb848f2d647 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 31 Jan 2026 16:01:16 +0800 Subject: [PATCH 054/375] update doc --- docs/agent-guide/001-logseq-cli.md | 4 ++-- docs/agent-guide/002-logseq-cli-subcommands.md | 4 ++-- docs/agent-guide/004-logseq-cli-verb-subcommands.md | 8 ++++---- .../005-logseq-cli-output-and-db-worker-node-log.md | 6 +++--- .../007-logseq-cli-thread-api-and-command-split.md | 2 +- docs/agent-guide/010-logseq-cli-show-linked-references.md | 2 +- docs/agent-guide/011-logseq-cli-search-optimization.md | 6 +++--- .../017-logseq-cli-db-worker-node-housekeeping-2.md | 2 +- 8 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md index ae40f8e606..0be698f64c 100644 --- a/docs/agent-guide/001-logseq-cli.md +++ b/docs/agent-guide/001-logseq-cli.md @@ -113,7 +113,7 @@ Graph removal is attempted while a graph is open. Run a single unit test in red phase. ```bash -bb dev:test -v logseq.cli.config-test/test-config-precedence +bb dev:test -v 'logseq.cli.config-test/test-config-precedence' ``` Expected output includes a failing assertion and ends with a non-zero exit code. @@ -121,7 +121,7 @@ Expected output includes a failing assertion and ends with a non-zero exit code. Run the full unit test suite in green phase. ```bash -bb dev:test -v logseq.cli.* +bb dev:test -v 'logseq.cli.commands-test' ``` Expected output includes 0 failures and 0 errors. diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index fdc829537e..63f31ef6b9 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -94,7 +94,7 @@ Each subcommand uses a nested path and its own options. 14. Update src/main/logseq/cli/config.cljs to add a unified output format option and ensure json and edn are both supported. 15. Update src/main/logseq/cli/format.cljs so that all commands emit consistent human, json, or edn output using a single option path. 16. Update docs/cli/logseq-cli.md to document subcommands, shared output flags, and per-subcommand help examples. -17. Run the unit test suite with bb dev:test -v logseq.cli.* and confirm 0 failures and 0 errors. +17. Run the unit test suite with bb dev:test -v 'logseq.cli.commands-test' and confirm 0 failures and 0 errors. 18. Run lint and tests with bb dev:lint-and-test and confirm a zero exit code. 19. Refactor for naming clarity, shared helpers, and reduced duplication while keeping tests green. @@ -117,7 +117,7 @@ Windows quoting should be covered for block add subcommand with multi-word conte Run a single failing unit test in red phase. ```bash -bb dev:test -v logseq.cli.commands-test/test-help-output +bb dev:test -v 'logseq.cli.commands-test/test-help-output' ``` Expected output includes a failing assertion about subcommand help text and ends with a non-zero exit code. diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md index e2eb899a03..960268f193 100644 --- a/docs/agent-guide/004-logseq-cli-verb-subcommands.md +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -134,7 +134,7 @@ Show has no subcommands and returns the block tree for a page or block. 3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, and list property. 4. Add failing unit tests that assert search defaults to all types and respects --type and positional text. 5. Add failing unit tests that assert show accepts --page-name, --uuid, or --id and rejects missing targets. -6. Run bb dev:test -v logseq.cli.commands-test/test-parse-args and confirm failures are about the new verbs and options. +6. Run bb dev:test -v 'logseq.cli.commands-test/test-parse-args' and confirm failures are about the new verbs and options. 7. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. 8. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last. 9. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface. @@ -144,7 +144,7 @@ Show has no subcommands and returns the block tree for a page or block. 13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --order. 14. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. 15. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. -16. Run bb dev:test -v logseq.cli.integration-test/test-cli-list-and-search and confirm failures before implementation. +16. Run bb dev:test -v 'logseq.cli.integration-test/test-cli-list-and-search' and confirm failures before implementation. 17. Implement behavior for list, add, remove, search, and show until all tests pass. 18. Update docs/cli/logseq-cli.md with new verb-first commands and examples. 19. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors. @@ -167,7 +167,7 @@ Show should return a deterministic order based on :block/order. Run a single unit test in red phase. ```bash -bb dev:test -v logseq.cli.commands-test/test-parse-args +bb dev:test -v 'logseq.cli.commands-test/test-parse-args' ``` Expected output includes failing assertions about the new verb-first commands and ends with a non-zero exit code. @@ -175,7 +175,7 @@ Expected output includes failing assertions about the new verb-first commands an Run the integration tests in red phase. ```bash -bb dev:test -v logseq.cli.integration-test/test-cli-list-add-search-show-remove +bb dev:test -v 'logseq.cli.integration-test/test-cli-list-add-search-show-remove' ``` Expected output includes failing assertions about list and search output and ends with a non-zero exit code. diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md index 3d4774ac6b..c0d041d37c 100644 --- a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -90,7 +90,7 @@ ASCII diagram: 23. Update help text in src/main/frontend/worker/db_worker_node.cljs to document the log file location and log-level flag behavior. 24. Update docs/cli/logseq-cli.md with the new human output expectations and any new formatting options. 25. Run unit tests in the red phase to confirm failures, then implement minimal changes to make them pass. -26. Run bb dev:test -v logseq.cli.* and bb dev:test -v frontend.worker.db-worker-node-test in the green phase. +26. Run bb dev:test -v 'logseq.cli.commands-test' and bb dev:test -v 'frontend.worker.db-worker-node-test' in the green phase. 27. Run bb dev:lint-and-test after all changes to validate lint and unit tests. ## Edge cases @@ -107,7 +107,7 @@ The human formatter should avoid printing large nested maps by default for searc Run a focused unit test during the red phase. ```bash -bb dev:test -v logseq.cli.format-test/test-human-output-list +bb dev:test -v 'logseq.cli.format-test/test-human-output-list' ``` Expected output includes a failing assertion and exits with a non-zero status code. @@ -115,7 +115,7 @@ Expected output includes a failing assertion and exits with a non-zero status co Run the db-worker-node log integration test in the green phase. ```bash -bb dev:test -v frontend.worker.db-worker-node-test/test-log-file-created +bb dev:test -v 'frontend.worker.db-worker-node-test/test-log-file-created' ``` Expected output includes 0 failures and 0 errors. diff --git a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md index 978b62ae0b..e3b67af23e 100644 --- a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md +++ b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md @@ -50,7 +50,7 @@ I will add unit tests to ensure all CLI thread-api method invocations use keywor 16. Update tests in `src/test/logseq/cli/commands_test.cljs` to import any moved namespaces or use the facade namespace, and ensure all help summary snapshots still pass. -17. Run unit tests for CLI and db-worker-node with `bb dev:test -v logseq.cli.commands-test`, `bb dev:test -v logseq.cli.transport-test`, and `bb dev:test -v frontend.worker.db-worker-node-test` and fix failures. +17. Run unit tests for CLI and db-worker-node with `bb dev:test -v 'logseq.cli.commands-test'`, `bb dev:test -v 'logseq.cli.transport-test'`, and `bb dev:test -v 'frontend.worker.db-worker-node-test'` and fix failures. 18. Run the full lint and unit test suite with `bb dev:lint-and-test` after all changes are complete. diff --git a/docs/agent-guide/010-logseq-cli-show-linked-references.md b/docs/agent-guide/010-logseq-cli-show-linked-references.md index 1b3b788d83..931563c135 100644 --- a/docs/agent-guide/010-logseq-cli-show-linked-references.md +++ b/docs/agent-guide/010-logseq-cli-show-linked-references.md @@ -17,7 +17,7 @@ I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/com I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text appends :block/tags in `#Tag` format to the rendered block content. I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the status prefix is present. I will add an integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page and a referencing block, runs show with --format json, and asserts that linked references are present and include the referencing block uuid and page title. -I will run the new unit tests with bb dev:test -v logseq.cli.commands-test and the new integration test namespace with bb dev:test -v logseq.cli.integration-test to confirm failures, then again to confirm passing. +I will run the new unit tests with bb dev:test -v 'logseq.cli.commands-test' and the new integration test namespace with bb dev:test -v 'logseq.cli.integration-test' to confirm failures, then again to confirm passing. I will run bb dev:lint-and-test after implementation to ensure no regressions. NOTE: I will write *all* tests before I add any implementation behavior. diff --git a/docs/agent-guide/011-logseq-cli-search-optimization.md b/docs/agent-guide/011-logseq-cli-search-optimization.md index 921b9812a3..81b44f36d4 100644 --- a/docs/agent-guide/011-logseq-cli-search-optimization.md +++ b/docs/agent-guide/011-logseq-cli-search-optimization.md @@ -58,11 +58,11 @@ This plan does not alter vector search or inference-worker behavior. 1. Read @test-driven-development and follow TDD for every behavior change in this plan. 2. Update CLI parsing tests in `src/test/logseq/cli/commands_test.cljs` to use positional search text and verify default types are all when --type is omitted. -3. Run `bb dev:test -v logseq.cli.commands-test` and confirm the new tests fail for the expected reasons. +3. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm the new tests fail for the expected reasons. 4. Update integration tests in `src/test/logseq/cli/integration_test.cljs` to call search without --text and to assert results still return ok with data. -5. Run `bb dev:test -v logseq.cli.integration-test` and confirm the new tests fail for the expected reasons. +5. Run `bb dev:test -v 'logseq.cli.integration-test'` and confirm the new tests fail for the expected reasons. 6. Update formatting expectations in `src/test/logseq/cli/format_test.cljs` if missing-search-text hints change. -7. Run `bb dev:test -v logseq.cli.format-test` and confirm the new tests fail for the expected reasons. +7. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm the new tests fail for the expected reasons. 8. Remove the :text, :include-content, and :limit options from search spec in `src/main/logseq/cli/command/search.cljs` and update help text accordingly. 9. Update `src/main/logseq/cli/commands.cljs` to require positional search text and to stop referencing :text in missing-search-text logic. 10. Update `src/main/logseq/cli/command/search.cljs` build-action to use the first positional argument as text and ignore any additional positional args for search text. diff --git a/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md index c77feba71b..0763fe5b8b 100644 --- a/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md +++ b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md @@ -24,7 +24,7 @@ I will rerun the CLI tests after each behavioral change to confirm they pass. Command to run tests is shown below. ```bash -bb dev:test -v logseq.cli.commands-test +bb dev:test -v 'logseq.cli.commands-test' ``` Expected test output is described below. From c5847ebb2b078af9d9f3e06c259e6fd6df9a7e4a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 31 Jan 2026 16:19:43 +0800 Subject: [PATCH 055/375] fix test --- src/test/logseq/cli/commands_test.cljs | 167 +++++++++++-------------- 1 file changed, 72 insertions(+), 95 deletions(-) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 1588219715..b9c0e4f709 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -270,7 +270,8 @@ (let [result (commands/parse-args ["--graph" "demo" "graph" "list"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))) - (is (= "unknown option: --graph" (get-in result [:error :message])))))) + (is (= "unknown option: --graph" + (strip-ansi (get-in result [:error :message]))))))) (deftest test-parse-args-global-options (testing "global output option is accepted" @@ -372,8 +373,12 @@ :block/title "CANCELED"}}]}} output (binding [style/*color-enabled?* true] (tree->text tree-data))] - (is (string/includes? output (style/bold "TODO"))) - (is (string/includes? output (style/bold "CANCELED"))) + (is (string/includes? output + (binding [style/*color-enabled?* true] + (style/bold (style/yellow "TODO"))))) + (is (string/includes? output + (binding [style/*color-enabled?* true] + (style/bold (style/red "CANCELED"))))) (is (= (str "1 TODO Root\n" "2 └── CANCELED Child") (strip-ansi output)))))) @@ -504,60 +509,30 @@ (is (= 1 (get-in stripped [:root :db/id]))) (is (= 2 (get-in stripped [:root :block/children 0 :db/id])))))) -(deftest test-fetch-tree-includes-db-ident +(deftest test-show-selectors-include-db-ident (async done - (let [fetch-tree #'show-command/fetch-tree - selectors (atom []) - orig-invoke transport/invoke] - (set! transport/invoke (fn [_ method _ args] - (when (= method :thread-api/pull) - (swap! selectors conj (second args))) - (case method - :thread-api/pull (p/resolved {:db/id 1 - :block/page {:db/id 2}}) - :thread-api/q (p/resolved []) - (p/resolved nil)))) - (-> (p/let [_ (fetch-tree {} {:repo "demo" :id 1})] - (is (some #(some #{:db/ident} %) @selectors))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! transport/invoke orig-invoke) - (done))))))) - -(deftest test-fetch-blocks-for-page-includes-db-ident - (async done - (let [fetch-blocks-for-page #'show-command/fetch-blocks-for-page - selectors (atom []) - orig-invoke transport/invoke] - (set! transport/invoke (fn [_ method _ args] - (when (= method :thread-api/q) - (let [[_ [query _]] args - pull-form (second query) - selector (nth pull-form 2)] - (swap! selectors conj selector))) - (p/resolved []))) - (-> (p/let [_ (fetch-blocks-for-page {} "demo" 1)] - (is (some #(some #{:db/ident} %) @selectors))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! transport/invoke orig-invoke) - (done))))))) - -(deftest test-fetch-linked-references-includes-db-ident - (async done - (let [fetch-linked-references #'show-command/fetch-linked-references - selectors (atom []) + (let [selectors (atom []) + orig-ensure-server! cli-server/ensure-server! orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [config _] config)) (set! transport/invoke (fn [_ method _ args] (case method - :thread-api/get-block-refs (p/resolved [{:db/id 10}]) :thread-api/pull (let [[_ selector _] args] (swap! selectors conj selector) - (p/resolved {:db/id 10})) + (p/resolved {:db/id 1 + :block/page {:db/id 2}})) + :thread-api/q (let [[_ [query _]] args + pull-form (second query) + selector (nth pull-form 2)] + (swap! selectors conj selector) + (p/resolved [])) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) (p/resolved nil)))) - (-> (p/let [_ (fetch-linked-references {} "demo" 1)] + (-> (p/let [_ (show-command/execute-show {:type :show + :repo "demo" + :id 1} + {:output-format :json})] + (is (some #(some #{:db/ident} %) @selectors)) (is (some #(and (some #{:db/ident} %) (some (fn [entry] (and (map? entry) @@ -567,64 +542,66 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke) (done))))))) -(deftest test-build-tree-data-linked-references-disabled +(deftest test-show-linked-references-disabled (async done - (let [build-tree-data #'show-command/build-tree-data - linked-called? (atom false) - orig-fetch-tree show-command/fetch-tree - orig-fetch-linked show-command/fetch-linked-references - orig-collect-uuid-refs show-command/collect-uuid-refs - orig-fetch-uuid-labels show-command/fetch-uuid-labels] - (set! show-command/fetch-tree (fn [_ _] - (p/resolved {:root {:db/id 1}}))) - (set! show-command/fetch-linked-references (fn [& _] - (reset! linked-called? true) - (p/resolved {:count 1 :blocks []}))) - (set! show-command/collect-uuid-refs (fn [_ _] [])) - (set! show-command/fetch-uuid-labels (fn [& _] (p/resolved {}))) - (-> (p/let [result (build-tree-data {} {:repo "demo" - :linked-references? false})] - (is (false? @linked-called?)) - (is (not (contains? result :linked-references)))) + (let [method-calls (atom []) + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [config _] config)) + (set! transport/invoke (fn [_ method _ _] + (swap! method-calls conj method) + (case method + :thread-api/pull (p/resolved {:db/id 1 + :block/page {:db/id 2}}) + :thread-api/q (p/resolved []) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + (p/resolved nil)))) + (-> (p/let [result (show-command/execute-show {:type :show + :repo "demo" + :id 1 + :linked-references? false} + {:output-format :json})] + (is (= :ok (:status result))) + (is (not (contains? (:data result) :linked-references))) + (is (not (some #{:thread-api/get-block-refs} @method-calls)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] - (set! show-command/fetch-tree orig-fetch-tree) - (set! show-command/fetch-linked-references orig-fetch-linked) - (set! show-command/collect-uuid-refs orig-collect-uuid-refs) - (set! show-command/fetch-uuid-labels orig-fetch-uuid-labels) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) (done))))))) -(deftest test-build-tree-data-linked-references-enabled +(deftest test-show-linked-references-enabled (async done - (let [build-tree-data #'show-command/build-tree-data - linked-called? (atom false) - linked {:count 1 :blocks []} - orig-fetch-tree show-command/fetch-tree - orig-fetch-linked show-command/fetch-linked-references - orig-collect-uuid-refs show-command/collect-uuid-refs - orig-fetch-uuid-labels show-command/fetch-uuid-labels] - (set! show-command/fetch-tree (fn [_ _] - (p/resolved {:root {:db/id 1}}))) - (set! show-command/fetch-linked-references (fn [& _] - (reset! linked-called? true) - (p/resolved linked))) - (set! show-command/collect-uuid-refs (fn [_ _] [])) - (set! show-command/fetch-uuid-labels (fn [& _] (p/resolved {}))) - (-> (p/let [result (build-tree-data {} {:repo "demo" - :linked-references? true})] - (is (true? @linked-called?)) - (is (= linked (:linked-references result)))) + (let [method-calls (atom []) + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [config _] config)) + (set! transport/invoke (fn [_ method _ _] + (swap! method-calls conj method) + (case method + :thread-api/pull (p/resolved {:db/id 1 + :block/page {:db/id 2}}) + :thread-api/q (p/resolved []) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + (p/resolved nil)))) + (-> (p/let [result (show-command/execute-show {:type :show + :repo "demo" + :id 1 + :linked-references? true} + {:output-format :json})] + (is (= :ok (:status result))) + (is (contains? (:data result) :linked-references)) + (is (some #{:thread-api/get-block-refs} @method-calls))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] - (set! show-command/fetch-tree orig-fetch-tree) - (set! show-command/fetch-linked-references orig-fetch-linked) - (set! show-command/collect-uuid-refs orig-collect-uuid-refs) - (set! show-command/fetch-uuid-labels orig-fetch-uuid-labels) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) (done))))))) (deftest test-tree->text-uuid-ref-recursion-limit From 2076f8f78864f2130b6d2194a6d3d93c5ca29711 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 31 Jan 2026 20:33:58 +0800 Subject: [PATCH 056/375] 025-logseq-cli-builtin-status-priority-queries.md --- ...seq-cli-builtin-status-priority-queries.md | 94 +++++++++++++++++++ docs/cli/logseq-cli.md | 3 + src/main/logseq/cli/command/query.cljs | 29 ++++-- src/test/logseq/cli/command/query_test.cljs | 43 ++++++++- src/test/logseq/cli/integration_test.cljs | 53 +++++++++++ 5 files changed, 212 insertions(+), 10 deletions(-) create mode 100644 docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md diff --git a/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md b/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md new file mode 100644 index 0000000000..b185015d8b --- /dev/null +++ b/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md @@ -0,0 +1,94 @@ +# Logseq CLI Built-in Status/Priority Queries Plan + +Goal: Add built-in `query` names `list-status` and `list-priority` that return the available Status/Priority options from a graph. +Architecture: Keep the existing logseq-cli → db-worker-node transport and thread-api usage; implement `list-status`/`list-priority` via `:thread-api/q` rather than adding a dedicated thread-api. +Tech Stack: ClojureScript, logseq-cli, db-worker-node, Datascript, logseq.db.frontend.property. +Related: `src/main/logseq/cli/command/query.cljs`, `src/main/frontend/worker/db_core.cljs`, `deps/db/src/logseq/db/frontend/property.cljs`. + +## Problem Statement + +Logseq CLI’s `query` built-ins are limited to Datascript queries. Users cannot easily discover valid Status or Priority options for a graph (e.g., TODO/DOING/DONE, Urgent/High/Low) without knowing the property internals or running custom queries. We need simple built-ins that list the closed values configured for Status and Priority. + +## Non-Goals + +- Changing property schemas or defaults. +- Replacing the existing Datascript query flow. +- Adding new UI commands outside `query` or changing CLI output formats. + +## Current Behavior (Key Points) + +- Built-in queries are defined in `built-in-query-specs` in `src/main/logseq/cli/command/query.cljs` and executed via `:thread-api/q`. +- Status and Priority options are stored as closed values on `:logseq.property/status` and `:logseq.property/priority` (see `logseq.db.frontend.property/get-closed-property-values`). +- There is no CLI helper to list those closed values. + +## Proposed Changes + +1) **Use `:thread-api/q` for closed values** + - Implement `list-status`/`list-priority` by issuing a Datascript query via `:thread-api/q`. + - Return a vector of maps with `:db/ident` and `:db/id`. + - Use `:find` with an ellipsis form to return a vector, e.g. `:find [(pull ?e [:db/ident :db/id]) ...]` (instead of `:find (pull ?e [:db/ident :db/id])`). + +2) **Add built-in query names `list-status` and `list-priority`** + - Add entries to `built-in-query-specs` that specify a non-Datascript execution path (e.g., `:method`/`:handler` metadata). + - Wire `build-action` to create a dedicated action type when these names are used. + - Update `execute-query` to call the new thread API and return results as `{:result ["TODO" ...]}` so existing output formatting works unchanged. + +3) **Expose built-ins in `query list`** + - Ensure `query list` includes the new entries (with `doc` and `inputs: []`). + - Keep existing “custom overrides built-in” semantics. + +## Implementation Plan + +1) **db-worker-node API** + - No new thread-api should be added; use `:thread-api/q` only. + +2) **CLI built-in wiring** + - Extend `built-in-query-specs` in `src/main/logseq/cli/command/query.cljs`: + - `list-status` → `:property-ident :logseq.property/status` + - `list-priority` → `:property-ident :logseq.property/priority` + - Include `:doc` and `:inputs []`. + - Update `normalize-query-entry` / `find-query` handling to preserve the extra metadata. + - Update `build-action`: + - When the built-in includes a `:property-ident` (or `:method`), create an action like + `{:type :query-closed-values :repo ... :property-ident ...}`. + - Update `execute-query`: + - Branch on the new action type to call `transport/invoke` with `:thread-api/q` and return `{:result values}` (vector of maps with `:db/ident` and `:db/id`). + +3) **Output and docs** + - No change to formatters; `format-query-results` already prints vectors. + - Update CLI docs if needed (e.g., `docs/cli/logseq-cli.md`) to mention the two built-ins. + +## Testing Plan + +- **Unit tests** (`src/test/logseq/cli/command/query_test.cljs`): + - `list-queries` includes `list-status` and `list-priority` with empty inputs. + - `build-action` for `--name list-status` returns `:query-closed-values` action with property ident. + - Custom query overrides built-in name still works. + +- **db-worker-node tests**: no new thread-api, so no additional db-worker-node test needed. + +- **Integration** (`src/test/logseq/cli/integration_test.cljs`): + - Start a graph, run `logseq query --name list-status` / `list-priority`, assert status `ok` and a non-empty vector. + - If stable defaults are known, assert a known value is present; otherwise, seed closed values in the test graph. + +## Edge Cases + +- Property has no closed values → return an empty vector (not an error). +- Closed values may be stored in either `:block/title` or `:logseq.property/value`; if `:db/ident` is absent, the map should still include the key with a nil value. +- Ensure ordering follows `:block/order` from `get-closed-property-values`. + +## Files to Touch + +- `src/main/frontend/worker/db_core.cljs` (no new thread-api) +- `src/main/logseq/cli/command/query.cljs` (built-in specs, build-action branching, execute-query) +- `src/test/logseq/cli/command/query_test.cljs` (unit coverage) +- `src/test/frontend/worker/db_worker_node_test.cljs` or a db-core test (not required for new thread-api) +- `src/test/logseq/cli/integration_test.cljs` (CLI end-to-end) +- `docs/cli/logseq-cli.md` (optional docs update) + +## Open Questions + +- Use `:thread-api/q` to return structured values: `{:db/ident .., :db/id ..}`. +- Expose `list-status`/`list-priority` via `query --name` only (no dedicated subcommands). + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 311d3cea1f..379aa9a3a7 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -78,6 +78,8 @@ Inspect and edit commands: - `remove --id |--uuid |--page ` - remove blocks (by db/id or UUID) or pages - `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) - `query --query [--inputs ]` - run a Datascript query against the graph +- `query --name [--inputs ]` - run a named query (built-in or from `cli.edn`) +- `query list` - list available named queries - `show --page [--level ]` - show page tree - `show --uuid [--level ]` - show block tree - `show --id [--level ]` - show block tree by db/id @@ -113,6 +115,7 @@ Output formats: - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. +- Built-in named queries currently include `block-search`, `task-search`, `recent-updated`, `list-status`, and `list-priority`. Use `query list` to see the full set for your config. - Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. - `show` human output prints the `:db/id` as the first column followed by a tree: diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index ebf61d89eb..87b86dcf9b 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -69,7 +69,23 @@ [(missing? $ ?e :logseq.property/built-in?)] [(* ?recent-days 86400000) ?recent-days-ms] [(- ?now-ms ?recent-days-ms) ?days-ago] - [(>= ?updated-at ?days-ago)]]}}) + [(>= ?updated-at ?days-ago)]]} + + "list-status" + {:doc "List closed values for the Status property." + :inputs [] + :query '[:find [(pull ?value [:db/id :db/ident :block/order]) ...] + :where + [?property :db/ident :logseq.property/status] + [?value :block/closed-value-property ?property]]} + + "list-priority" + {:doc "List closed values for the Priority property." + :inputs [] + :query '[:find [(pull ?value [:db/id :db/ident :block/order]) ...] + :where + [?property :db/ident :logseq.property/priority] + [?value :block/closed-value-property ?property]]}}) (defn- parse-edn [label value] @@ -264,11 +280,12 @@ (defn execute-query [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - args (into [(:query action)] (:inputs action)) - results (transport/invoke cfg :thread-api/q false [(:repo action) args])] - {:status :ok - :data {:result results}}))) + (case (:type action) + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + args (into [(:query action)] (:inputs action)) + results (transport/invoke cfg :thread-api/q false [(:repo action) args])] + {:status :ok + :data {:result results}})))) (defn execute-query-list [_action config] diff --git a/src/test/logseq/cli/command/query_test.cljs b/src/test/logseq/cli/command/query_test.cljs index a6becea914..1c28aa6944 100644 --- a/src/test/logseq/cli/command/query_test.cljs +++ b/src/test/logseq/cli/command/query_test.cljs @@ -79,7 +79,7 @@ "logseq_db_demo" config)] (is (true? (:ok? result))) - (is (= ["doing" nil nil] (get-in result [:action :inputs]))))) + (is (= [:logseq.property/status.doing nil nil] (get-in result [:action :inputs]))))) (testing "missing required inputs returns invalid-options" (let [config {:custom-queries {"task-search" @@ -109,7 +109,7 @@ "logseq_db_demo" config)] (is (true? (:ok? result))) - (is (= ["doing" "fallback-title" 7] (get-in result [:action :inputs]))))) + (is (= [:logseq.property/status.doing "fallback-title" 7] (get-in result [:action :inputs]))))) (testing "built-in task-search uses defaults for optional inputs" (let [result (query-command/build-action {:name "task-search" @@ -118,18 +118,53 @@ {})] (is (true? (:ok? result))) (let [inputs (get-in result [:action :inputs])] - (is (= ["doing" "" 0] (subvec inputs 0 3))) + (is (= [:logseq.property/status.doing "" 0] (subvec inputs 0 3))) (is (number? (nth inputs 3))))))) +(deftest test-build-action-closed-value-queries + (testing "list-status builds standard query action" + (let [result (query-command/build-action {:name "list-status"} + "logseq_db_demo" + {})] + (is (true? (:ok? result))) + (is (= :query (get-in result [:action :type]))) + (is (= "logseq_db_demo" (get-in result [:action :repo]))) + (is (= '[:find [(pull ?value [:db/id :db/ident :block/order]) ...] + :where + [?property :db/ident :logseq.property/status] + [?value :block/closed-value-property ?property]] + (get-in result [:action :query]))))) + + (testing "list-priority builds standard query action" + (let [result (query-command/build-action {:name "list-priority"} + "logseq_db_demo" + {})] + (is (true? (:ok? result))) + (is (= :query (get-in result [:action :type]))) + (is (= :logseq.property/priority (get-in result [:action :query 3 2]))) + (is (= '[:find [(pull ?value [:db/id :db/ident :block/order]) ...] + :where + [?property :db/ident :logseq.property/priority] + [?value :block/closed-value-property ?property]] + (get-in result [:action :query])))))) + (deftest test-query-list-merges-built-in-and-custom (testing "built-in and custom queries are both listed" (let [queries (query-command/list-queries {:custom-queries {"custom-q" {:query '[:find ?e]}}}) names (set (map :name queries))] (is (contains? names "block-search")) (is (contains? names "task-search")) + (is (contains? names "list-status")) + (is (contains? names "list-priority")) (is (contains? names "custom-q")))) (testing "custom query overrides built-in name" (let [queries (query-command/list-queries {:custom-queries {"block-search" {:query '[:find ?e]}}}) block-search (first (filter #(= "block-search" (:name %)) queries))] - (is (= :custom (:source block-search)))))) + (is (= :custom (:source block-search))))) + + (testing "list-status and list-priority have empty inputs" + (let [queries (query-command/list-queries {}) + by-name (into {} (map (fn [entry] [(:name entry) entry]) queries))] + (is (= [] (get-in by-name ["list-status" :inputs]))) + (is (= [] (get-in by-name ["list-priority" :inputs])))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 2993f6d998..23131a9fad 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -621,6 +621,59 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-query-list-status-priority + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-status-query")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--repo" "status-query-graph"] data-dir cfg-path) + create-payload (parse-json-output create-result) + _ (p/delay 100) + list-result (run-cli ["query" "list"] data-dir cfg-path) + list-payload (parse-json-output list-result) + names (set (map :name (get-in list-payload [:data :queries]))) + status-result (run-cli ["--repo" "status-query-graph" + "query" + "--name" "list-status"] + data-dir cfg-path) + status-payload (parse-json-output status-result) + status-values (get-in status-payload [:data :result]) + priority-result (run-cli ["--repo" "status-query-graph" + "query" + "--name" "list-priority"] + data-dir cfg-path) + priority-payload (parse-json-output priority-result) + priority-values (get-in priority-payload [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" "status-query-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status create-payload))) + (is (= "ok" (:status list-payload))) + (is (contains? names "list-status")) + (is (contains? names "list-priority")) + (is (= 0 (:exit-code status-result))) + (is (= "ok" (:status status-payload))) + (is (vector? status-values)) + (when (seq status-values) + (let [row (first status-values) + value (if (vector? row) (first row) row)] + (is (map? value)) + (is (contains? value :ident)) + (is (contains? value :id)))) + (is (= 0 (:exit-code priority-result))) + (is (= "ok" (:status priority-payload))) + (is (vector? priority-values)) + (when (seq priority-values) + (let [row (first priority-values) + value (if (vector? row) (first row) row)] + (is (map? value)) + (is (contains? value :ident)) + (is (contains? value :id)))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-query-recent-updated (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-recent-updated")] From dac05879767b5edbb558dde10f11a0a52fae15ee Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 31 Jan 2026 22:09:01 +0800 Subject: [PATCH 057/375] 026-logseq-cli-query-output.md (1) --- .../026-logseq-cli-query-output.md | 74 +++++++++++ src/main/logseq/cli/command/id.cljs | 2 +- src/main/logseq/cli/command/show.cljs | 42 ++++++- src/main/logseq/cli/commands.cljs | 89 ++++++++----- src/main/logseq/cli/format.cljs | 14 ++- src/test/logseq/cli/command/show_test.cljs | 33 +++++ src/test/logseq/cli/format_test.cljs | 36 +++++- src/test/logseq/cli/integration_test.cljs | 119 +++++++++++++++--- 8 files changed, 354 insertions(+), 55 deletions(-) create mode 100644 docs/agent-guide/026-logseq-cli-query-output.md create mode 100644 src/test/logseq/cli/command/show_test.cljs diff --git a/docs/agent-guide/026-logseq-cli-query-output.md b/docs/agent-guide/026-logseq-cli-query-output.md new file mode 100644 index 0000000000..3e6f3978bf --- /dev/null +++ b/docs/agent-guide/026-logseq-cli-query-output.md @@ -0,0 +1,74 @@ +# Logseq CLI Query Output Piping Implementation Plan + +Goal: Remove the space-to-comma transformation in CLI query human output while keeping query results usable in shell pipelines like xargs and direct stdin piping with logseq show. + +Architecture: Adjust human formatting for query results to preserve EDN output for general results and emit line-oriented output only for scalar id collections. +Architecture: Extend logseq show -id to read ids from stdin when no id argument is provided and update integration tests to validate both xargs and direct stdin pipelines. + +Tech Stack: ClojureScript, Logseq CLI, db-worker-node, Node-based integration tests. + +Related: Relates to docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md. + +## Testing Plan + +I will add an integration test that ensures a human-output query can be piped into xargs and then into logseq show for multiple ids. +I will add an integration test that ensures a human-output query can be piped directly into logseq show -id via stdin when no id value is passed. +I will update the existing integration test that currently pipes human query output directly into logseq show so it asserts the new line-oriented behavior and stdin ingestion. +I will add a focused unit test for format-query-results that covers scalar collections, non-scalar collections, and nil results. +I will add a unit test for show id parsing that covers missing id arg with stdin provided and missing id arg without stdin. + +NOTE: I will write all tests before I add any implementation behavior. + +## Problem statement + +The CLI currently replaces spaces with commas in format-query-results, which is a lossy transformation that makes output less readable. +Removing that transformation will reintroduce spaces in EDN vectors and lists, which breaks shell pipelines that rely on whitespace splitting. +We need to remove the space-to-comma logic while still ensuring the pipeline commands in the request work reliably. +The show command needs to accept ids from stdin when -id is present but no explicit id argument is provided. + +## Plan + +1. Read the existing formatting logic in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs and document current behavior in a quick note. +2. Read the show command option parsing in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs and /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/id.cljs to understand current id validation. +3. Locate the current integration test that verifies human query output piping in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs and map the required changes. +4. Define the new output rule for format-query-results as a comment in the plan: if the result is a sequential collection of scalar ids, output one id per line, otherwise output the EDN string unchanged. +5. Define the new show -id stdin rule as a comment in the plan: when -id is present but no id value is provided, read stdin, trim it, and treat it as the id or id vector string for parsing. +6. Write the failing unit tests for format-query-results in a new test namespace under /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs that covers nil, scalar-id sequences, and nested maps. +7. Write failing unit tests for show id parsing in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/show_test.cljs that cover stdin provided, stdin blank, and explicit id values. +8. Run the unit tests to confirm the failure before implementation using bb dev:test with the new namespaces. +9. Write a failing integration test that uses the exact pipeline command from the request by invoking a shell command from the test harness in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs for xargs usage. +10. Write a failing integration test that uses the exact pipeline command from the request for direct stdin piping into logseq show -id with no argument. +11. Run the integration tests to confirm the failure before implementation using bb dev:test with the specific test names. +12. Implement the new format-query-results behavior in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs using a helper predicate for scalar ids and line-join logic. +13. Remove the string/replace space-to-comma logic from format-query-results in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs. +14. Implement stdin ingestion for show -id in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, using a helper to read from stdin only when -id is present and no id argument is provided. +15. Update the existing human-output pipeline integration test to align with the new output behavior and assert both xargs and direct stdin usage. +16. Re-run the unit tests and the integration tests to validate the new behavior. +17. Update deps/cli/README.md examples if they reference comma-transformed output, and add a short note describing the new xargs-friendly and stdin-friendly behavior for id lists. + +## Testing Details + +The unit tests will exercise behavior by calling format-query-results with representative results and asserting the exact emitted string for id lists and non-id data. +The integration tests will execute logseq query with a task-search or custom datalog query, pipe the output through xargs into logseq show, and pipe directly into logseq show -id via stdin with no id argument. +The show stdin behavior test will assert that missing -id input without stdin returns a clear error and that stdin is parsed the same as an id argument. + +## Implementation Details + +- Add a helper predicate for scalar ids that accepts integers and rejects maps, vectors, and strings. +- Detect sequential results that are entirely scalar ids and return a newline-joined string of ids. +- Preserve the existing safe-read-string validation to avoid changing behavior for invalid EDN strings. +- Keep non-scalar results as their original EDN string output. +- Maintain nil handling so that "nil" still prints as "nil". +- Ensure the output is stable for xargs by avoiding embedded spaces in the line-oriented mode. +- Add stdin reading to show -id when the option is present but its value is missing. +- Parse stdin through the same id parsing function used for explicit -id values. +- Keep existing errors when -id is missing and stdin is empty or whitespace. +- Update integration tests to use the exact pipeline command from the requirement. +- Use @test-driven-development for all implementation steps. + +## Question + +The line-oriented output applies only to vectors of numeric ids. +Logseq show -id reads stdin only when -id is present with no value. + +--- diff --git a/src/main/logseq/cli/command/id.cljs b/src/main/logseq/cli/command/id.cljs index 164ad37fec..1d5ed82c00 100644 --- a/src/main/logseq/cli/command/id.cljs +++ b/src/main/logseq/cli/command/id.cljs @@ -39,7 +39,7 @@ (every? valid-id? parsed) {:ok? true :value (vec parsed) :multi? true} :else (invalid "id vector must contain only integers"))) - (re-matches #"-?\\d+" text) + (re-matches #"-?\d+" text) {:ok? true :value [(js/parseInt text 10)] :multi? false} :else diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 5ba7dc9d78..abed3d9227 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.command.show "Show-related CLI commands." - (:require [clojure.string :as string] + (:require ["fs" :as fs] + [clojure.string :as string] [clojure.walk :as walk] [logseq.cli.command.id :as id-command] [logseq.cli.command.core :as core] @@ -24,15 +25,47 @@ (def ^:private multi-id-delimiter "\n================================================================\n") +(defn read-stdin + [] + (.toString (fs/readFileSync 0) "utf8")) + +(defn- normalize-stdin-id + [value] + (let [text (string/trim (or value ""))] + (cond + (string/blank? text) text + (string/starts-with? text "[") text + (re-matches #"-?\d+" text) text + :else + (let [tokens (->> (string/split text #"\s+") + (remove string/blank?))] + (if (and (seq tokens) (every? #(re-matches #"-?\d+" %) tokens)) + (str "[" (string/join " " tokens) "]") + text))))) + +(defn- resolve-stdin-id + [options] + (if (:id-from-stdin? options) + (let [stdin (if (contains? options :stdin) + (:stdin options) + (read-stdin))] + (assoc options :id (normalize-stdin-id stdin))) + options)) + (defn invalid-options? [opts] (let [level (:level opts) - id-result (id-command/parse-id-option (:id opts))] + id-value (:id opts) + id-missing? (and (:id-from-stdin? opts) + (or (nil? id-value) + (and (string? id-value) (string/blank? id-value)))) + id-result (when-not id-missing? + (id-command/parse-id-option id-value))] (cond (and (some? level) (< level 1)) "level must be >= 1" - (and (some? (:id opts)) (not (:ok? id-result))) + (and (some? id-value) (not id-missing?) (not (:ok? id-result))) (:message id-result) :else @@ -469,7 +502,8 @@ {:ok? false :error {:code :missing-repo :message "repo is required for show"}} - (let [id-result (id-command/parse-id-option (:id options)) + (let [options (resolve-stdin-id options) + id-result (id-command/parse-id-option (:id options)) ids (:value id-result) multi-id? (:multi? id-result) targets (filter some? [(:id options) (:uuid options) (:page options)])] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 0bedcd411f..ac38c6063e 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -105,6 +105,28 @@ ;; Global option parsing lives in logseq.cli.command.core. +(defn- index-of + [coll value] + (first (keep-indexed (fn [idx item] + (when (= item value) idx)) + coll))) + +(defn- inject-stdin-id-arg + [args] + (if (and (seq args) (= "show" (first args))) + (if-let [idx (index-of args "--id")] + (let [next-token (nth args (inc idx) nil) + missing-value? (or (nil? next-token) + (string/starts-with? next-token "-"))] + (if missing-value? + {:args (vec (concat (subvec args 0 (inc idx)) + [""] + (subvec args (inc idx)))) + :id-from-stdin? true} + {:args args :id-from-stdin? false})) + {:args args :id-from-stdin? false}) + {:args args :id-from-stdin? false})) + (defn- unknown-command-message [{:keys [dispatch wrong-input]}] (string/join " " (cond-> (vec dispatch) @@ -223,44 +245,53 @@ [raw-args] (let [summary (command-core/top-level-summary table) legacy-graph-opt? (command-core/legacy-graph-opt? raw-args) - {:keys [opts args]} (command-core/parse-leading-global-opts raw-args)] - (if legacy-graph-opt? + {:keys [opts args]} (command-core/parse-leading-global-opts raw-args) + {:keys [args id-from-stdin?]} (inject-stdin-id-arg (vec args))] + (cond + legacy-graph-opt? (command-core/invalid-options-result summary "unknown option: --graph") - (if (:version opts) + + (:version opts) (command-core/ok-result :version opts [] summary) - (if (empty? args) + + (empty? args) (if (:help opts) (command-core/help-result summary) {:ok? false :error {:code :missing-command :message "missing command"} - :summary summary}) - (if (and (= 1 (count args)) (#{"graph" "server" "list" "add" "query"} (first args))) - (command-core/help-result (command-core/group-summary (first args) table)) - (try - (let [result (cli/dispatch table args {:spec global-spec})] - (if (nil? result) - (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args))) - (finalize-command summary (update result :opts #(merge opts (or % {})))))) - (catch :default e - (let [{:keys [cause] :as data} (ex-data e)] - (cond - (= cause :input-exhausted) - (if (:help opts) - (command-core/help-result summary) - {:ok? false - :error {:code :missing-command - :message "missing command"} - :summary summary}) + :summary summary}) - (= cause :no-match) - (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) + (and (= 1 (count args)) (#{"graph" "server" "list" "add" "query"} (first args))) + (command-core/help-result (command-core/group-summary (first args) table)) - (some? data) - (command-core/cli-error->result summary data) + :else + (try + (let [result (cli/dispatch table args {:spec global-spec})] + (if (nil? result) + (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args))) + (finalize-command summary + (update result :opts #(cond-> (merge opts (or % {})) + id-from-stdin? (assoc :id-from-stdin? true)))))) + (catch :default e + (let [{:keys [cause] :as data} (ex-data e)] + (cond + (= cause :input-exhausted) + (if (:help opts) + (command-core/help-result summary) + {:ok? false + :error {:code :missing-command + :message "missing command"} + :summary summary}) - :else - (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args))))))))))))) + (= cause :no-match) + (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) + + (some? data) + (command-core/cli-error->result summary data) + + :else + (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args)))))))))) ;; Repo/graph helpers live in logseq.cli.command.core. @@ -296,8 +327,6 @@ ;; Show helpers live in logseq.cli.command.show. - - ;; Show helpers live in logseq.cli.command.show. ;; Repo normalization lives in logseq.cli.command.core. diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 65125bdf09..f4c8daf9ce 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -182,13 +182,25 @@ (:pid server)]) (or servers [])))) +(defn- scalar-id? + [value] + (and (number? value) (integer? value))) + +(defn- scalar-id-seq? + [value] + (and (sequential? value) + (seq value) + (every? scalar-id? value))) + (defn- format-query-results [result] (let [edn-str (pr-str result) parsed (common-util/safe-read-string {:log-error? false} edn-str) valid? (or (some? parsed) (= "nil" (string/trim edn-str)))] (if valid? - (string/replace edn-str " " ",") + (if (scalar-id-seq? result) + (string/join "\n" (map str result)) + edn-str) edn-str))) (defn- format-query-list diff --git a/src/test/logseq/cli/command/show_test.cljs b/src/test/logseq/cli/command/show_test.cljs new file mode 100644 index 0000000000..a36f749769 --- /dev/null +++ b/src/test/logseq/cli/command/show_test.cljs @@ -0,0 +1,33 @@ +(ns logseq.cli.command.show-test + (:require [cljs.test :refer [deftest is testing]] + [clojure.string :as string] + [logseq.cli.command.show :as show-command])) + +(deftest test-build-action-stdin-id + (testing "reads id from stdin when id flag is present without a value" + (let [result (show-command/build-action {:id "" + :id-from-stdin? true + :stdin "42"} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= 42 (get-in result [:action :id]))) + (is (= [42] (get-in result [:action :ids]))) + (is (false? (get-in result [:action :multi-id?]))))) + + (testing "reads multi-id vector from stdin" + (let [result (show-command/build-action {:id "" + :id-from-stdin? true + :stdin "[1 2 3]"} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= [1 2 3] (get-in result [:action :ids]))) + (is (true? (get-in result [:action :multi-id?]))))) + + (testing "blank stdin returns invalid options" + (let [result (show-command/build-action {:id "" + :id-from-stdin? true + :stdin " "} + "logseq_db_demo")] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))) + (is (string/includes? (get-in result [:error :message]) "id"))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 4532743395..c733b050d7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -101,7 +101,7 @@ (testing "remove page renders a succinct success line" (let [result (format/format-result {:status :ok - :command :remove-page + :command :remove :context {:repo "demo-repo" :page "Home"} :data {:result {:ok true}}} @@ -170,6 +170,38 @@ {:output-format nil})] (is (= "Line 1\nLine 2" result))))) +(deftest test-human-output-query-results + (testing "scalar id collections render one id per line" + (let [result (format/format-result {:status :ok + :command :query + :data {:result [1 2 3]}} + {:output-format nil})] + (is (= "1\n2\n3" result)))) + + (testing "non-scalar collections preserve EDN formatting" + (let [value [{:db/id 1 :block/title "Alpha"} + {:db/id 2 :block/title "Beta"}] + result (format/format-result {:status :ok + :command :query + :data {:result value}} + {:output-format nil})] + (is (= (pr-str value) result)))) + + (testing "mixed scalar collections preserve EDN formatting" + (let [value [1 "two" 3] + result (format/format-result {:status :ok + :command :query + :data {:result value}} + {:output-format nil})] + (is (= (pr-str value) result)))) + + (testing "nil results render as nil" + (let [result (format/format-result {:status :ok + :command :query + :data {:result nil}} + {:output-format nil})] + (is (= "nil" result))))) + (deftest test-human-output-show-styled-prefixes (testing "show preserves styled status and tags in human output" (let [tree->text #'show-command/tree->text @@ -231,7 +263,7 @@ :command :query :data {:result [[1] [2] [3]]}} {:output-format nil})] - (is (= "[[1],[2],[3]]" result))))) + (is (= "[[1] [2] [3]]" result))))) (deftest test-human-output-query-list (testing "query list renders a table with count" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 23131a9fad..a804234eb6 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1,11 +1,13 @@ (ns logseq.cli.integration-test - (:require ["fs" :as fs] + (:require ["child_process" :as child-process] + ["fs" :as fs] ["path" :as node-path] [cljs.reader :as reader] [cljs.test :refer [deftest is async]] [clojure.string :as string] [frontend.worker-common.util :as worker-util] [frontend.test.node-helper :as node-helper] + [logseq.cli.command.show :as show-command] [logseq.cli.command.core :as command-core] [logseq.cli.main :as cli-main] [logseq.common.util :as common-util] @@ -44,6 +46,28 @@ [result] (reader/read-string (:output result))) +(defn- shell-escape + [value] + (let [text (str value)] + (str "'" (string/replace text #"'" "'\"'\"'") "'"))) + +(defn- run-shell + [command] + (try + (child-process/execSync command #js {:encoding "utf8" + :shell "/bin/bash"}) + (catch :default e + (let [err ^js e + stdout (some-> (.-stdout err) (.toString "utf8")) + stderr (some-> (.-stderr err) (.toString "utf8"))] + (throw (ex-info (str "shell command failed: " command + "\nstdout: " (or stdout "") + "\nstderr: " (or stderr "")) + {:command command + :stdout stdout + :stderr stderr} + e)))))) + (defn- node-title [node] (or (:block/title node) (:block/content node) (:title node) (:content node))) @@ -1156,30 +1180,91 @@ " :where" " [?e :block/title ?title]" " [(clojure.string/includes? ?title ?q)]]") - query-result (run-cli ["--repo" "query-pipe-graph" + node-bin (shell-escape (.-execPath js/process)) + cli-bin (shell-escape (node-path/resolve "static/logseq-cli.js")) + data-arg (shell-escape data-dir) + cfg-arg (shell-escape cfg-path) + repo-arg (shell-escape "query-pipe-graph") + query-arg (shell-escape query-text) + inputs-arg (shell-escape (pr-str ["Pipe"])) + query-cmd (string/join " " + [node-bin cli-bin + "--data-dir" data-arg + "--config" cfg-arg + "--repo" repo-arg + "--output" "human" + "query" + "--query" query-arg + "--inputs" inputs-arg]) + show-cmd (string/join " " + [node-bin cli-bin + "--data-dir" data-arg + "--config" cfg-arg + "--repo" repo-arg + "--output" "human" + "show" + "--id"]) + pipeline (str query-cmd " | xargs -I{} " show-cmd " {}") + output (run-shell pipeline) + stop-result (run-cli ["server" "stop" "--repo" "query-pipe-graph"] + data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (string/includes? output "Pipe One")) + (is (string/includes? output "Pipe Two")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-query-human-output-pipes-to-show-stdin + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-query-stdin")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" "query-stdin-graph"] data-dir cfg-path) + _ (run-cli ["--repo" "query-stdin-graph" "add" "page" "--page" "PipePage"] + data-dir cfg-path) + _ (run-cli ["--repo" "query-stdin-graph" "add" "block" + "--target-page-name" "PipePage" + "--content" "Pipe One"] + data-dir cfg-path) + _ (run-cli ["--repo" "query-stdin-graph" "add" "block" + "--target-page-name" "PipePage" + "--content" "Pipe Two"] + data-dir cfg-path) + _ (p/delay 100) + query-text (str "[:find [?e ...]" + " :in $ ?q" + " :where" + " [?e :block/title ?title]" + " [(clojure.string/includes? ?title ?q)]]") + query-result (run-cli ["--repo" "query-stdin-graph" "--output" "human" "query" "--query" query-text "--inputs" (pr-str ["Pipe"])] data-dir cfg-path) - ids-edn (string/trim (:output query-result)) - show-json-result (run-cli ["--repo" "query-pipe-graph" "show" - "--id" ids-edn - "--format" "json"] - data-dir cfg-path) - show-json-payload (parse-json-output show-json-result) - show-data (:data show-json-payload) + ids-text (string/trim (:output query-result)) + show-result (with-redefs [show-command/read-stdin (fn [] ids-text)] + (run-cli ["--repo" "query-stdin-graph" + "--output" "json" + "show" + "--id"] + data-dir cfg-path)) + show-payload (parse-json-output show-result) + show-data (:data show-payload) + root (some-> show-data first :root) root-titles (set (map (comp node-title :root) show-data)) - stop-result (run-cli ["server" "stop" "--repo" "query-pipe-graph"] + pipe-one (find-block-by-title root "Pipe One") + pipe-two (find-block-by-title root "Pipe Two") + stop-result (run-cli ["server" "stop" "--repo" "query-stdin-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code query-result))) - (is (seq ids-edn)) - (is (= 0 (:exit-code show-json-result))) - (is (= "ok" (:status show-json-payload))) - (is (vector? show-data)) - (is (contains? root-titles "Pipe One")) - (is (contains? root-titles "Pipe Two")) + (is (= "ok" (:status show-payload))) + (is (contains? root-titles "PipePage")) + (is (some? pipe-one)) + (is (some? pipe-two)) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From e9b68d5280765a87789d0d64989b33a281e343da Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 1 Feb 2026 08:54:42 +0800 Subject: [PATCH 058/375] 026-logseq-cli-query-output.md (2) --- .../026-logseq-cli-query-output.md | 58 +++++++++---------- src/main/logseq/cli/command/show.cljs | 23 ++++++-- src/main/logseq/cli/format.cljs | 14 +---- src/test/logseq/cli/command/show_test.cljs | 18 +++++- src/test/logseq/cli/format_test.cljs | 4 +- src/test/logseq/cli/integration_test.cljs | 23 ++++++++ 6 files changed, 87 insertions(+), 53 deletions(-) diff --git a/docs/agent-guide/026-logseq-cli-query-output.md b/docs/agent-guide/026-logseq-cli-query-output.md index 3e6f3978bf..7c36123334 100644 --- a/docs/agent-guide/026-logseq-cli-query-output.md +++ b/docs/agent-guide/026-logseq-cli-query-output.md @@ -1,9 +1,9 @@ # Logseq CLI Query Output Piping Implementation Plan -Goal: Remove the space-to-comma transformation in CLI query human output while keeping query results usable in shell pipelines like xargs and direct stdin piping with logseq show. +Goal: Remove the space-to-comma transformation in CLI query human output while preserving EDN output and ensuring `logseq show --id` accepts ids both via argument and stdin pipelines. -Architecture: Adjust human formatting for query results to preserve EDN output for general results and emit line-oriented output only for scalar id collections. -Architecture: Extend logseq show -id to read ids from stdin when no id argument is provided and update integration tests to validate both xargs and direct stdin pipelines. +Architecture: Keep human formatting for query results as EDN output (no line-oriented transformation) and update tests to validate pipeline usage with intact EDN. +Architecture: Extend logseq show --id to read ids from stdin when present, whether or not an id argument is provided, and update integration tests to validate both xargs and direct stdin pipelines. Tech Stack: ClojureScript, Logseq CLI, db-worker-node, Node-based integration tests. @@ -11,64 +11,60 @@ Related: Relates to docs/agent-guide/025-logseq-cli-builtin-status-priority-quer ## Testing Plan -I will add an integration test that ensures a human-output query can be piped into xargs and then into logseq show for multiple ids. -I will add an integration test that ensures a human-output query can be piped directly into logseq show -id via stdin when no id value is passed. -I will update the existing integration test that currently pipes human query output directly into logseq show so it asserts the new line-oriented behavior and stdin ingestion. -I will add a focused unit test for format-query-results that covers scalar collections, non-scalar collections, and nil results. -I will add a unit test for show id parsing that covers missing id arg with stdin provided and missing id arg without stdin. +I will add an integration test that ensures a human-output query can be piped into xargs and then into logseq show --id for multiple ids, with the query output preserved as EDN. +I will add an integration test that ensures a human-output query can be piped directly into logseq show --id via stdin, with or without an explicit id argument. +I will update the existing integration test that currently pipes human query output directly into logseq show so it asserts EDN preservation and stdin ingestion. +I will add a focused unit test for format-query-results that covers EDN output for scalar collections, non-scalar collections, and nil results. +I will add a unit test for show id parsing that covers stdin provided, stdin blank, and explicit id values. NOTE: I will write all tests before I add any implementation behavior. ## Problem statement The CLI currently replaces spaces with commas in format-query-results, which is a lossy transformation that makes output less readable. -Removing that transformation will reintroduce spaces in EDN vectors and lists, which breaks shell pipelines that rely on whitespace splitting. -We need to remove the space-to-comma logic while still ensuring the pipeline commands in the request work reliably. -The show command needs to accept ids from stdin when -id is present but no explicit id argument is provided. +We need to remove the space-to-comma logic while preserving EDN output (including spaces) for query results. +Pipelines should remain usable by allowing logseq show --id to accept ids from stdin and from explicit arguments interchangeably. ## Plan 1. Read the existing formatting logic in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs and document current behavior in a quick note. 2. Read the show command option parsing in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs and /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/id.cljs to understand current id validation. 3. Locate the current integration test that verifies human query output piping in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs and map the required changes. -4. Define the new output rule for format-query-results as a comment in the plan: if the result is a sequential collection of scalar ids, output one id per line, otherwise output the EDN string unchanged. -5. Define the new show -id stdin rule as a comment in the plan: when -id is present but no id value is provided, read stdin, trim it, and treat it as the id or id vector string for parsing. -6. Write the failing unit tests for format-query-results in a new test namespace under /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs that covers nil, scalar-id sequences, and nested maps. -7. Write failing unit tests for show id parsing in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/show_test.cljs that cover stdin provided, stdin blank, and explicit id values. +4. Define the new output rule for format-query-results as a comment in the plan: always return EDN output unchanged (including spaces), and remove space-to-comma logic. +5. Define the new show --id stdin rule as a comment in the plan: when stdin is non-empty, parse it as the id or id vector string and use it regardless of whether an id argument is provided. +6. Write the failing unit tests for format-query-results in a new test namespace under /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs that covers nil, scalar-id sequences, and nested maps, all preserved as EDN strings. +7. Write failing unit tests for show id parsing in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/show_test.cljs that cover stdin provided (overriding or complementing args), stdin blank, and explicit id values. 8. Run the unit tests to confirm the failure before implementation using bb dev:test with the new namespaces. 9. Write a failing integration test that uses the exact pipeline command from the request by invoking a shell command from the test harness in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs for xargs usage. -10. Write a failing integration test that uses the exact pipeline command from the request for direct stdin piping into logseq show -id with no argument. +10. Write a failing integration test that uses the exact pipeline command from the request for direct stdin piping into logseq show --id with or without an argument. 11. Run the integration tests to confirm the failure before implementation using bb dev:test with the specific test names. -12. Implement the new format-query-results behavior in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs using a helper predicate for scalar ids and line-join logic. -13. Remove the string/replace space-to-comma logic from format-query-results in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs. -14. Implement stdin ingestion for show -id in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, using a helper to read from stdin only when -id is present and no id argument is provided. -15. Update the existing human-output pipeline integration test to align with the new output behavior and assert both xargs and direct stdin usage. +12. Implement the new format-query-results behavior in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs by removing the space-to-comma logic and preserving EDN output. +13. Implement stdin ingestion for show --id in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, using a helper to read from stdin when non-empty regardless of whether an id argument is provided. +14. Update the existing human-output pipeline integration test to align with the EDN-preserving output and assert both xargs and direct stdin usage. 16. Re-run the unit tests and the integration tests to validate the new behavior. 17. Update deps/cli/README.md examples if they reference comma-transformed output, and add a short note describing the new xargs-friendly and stdin-friendly behavior for id lists. ## Testing Details -The unit tests will exercise behavior by calling format-query-results with representative results and asserting the exact emitted string for id lists and non-id data. -The integration tests will execute logseq query with a task-search or custom datalog query, pipe the output through xargs into logseq show, and pipe directly into logseq show -id via stdin with no id argument. -The show stdin behavior test will assert that missing -id input without stdin returns a clear error and that stdin is parsed the same as an id argument. +The unit tests will exercise behavior by calling format-query-results with representative results and asserting the exact emitted EDN string for id lists and non-id data. +The integration tests will execute logseq query with a task-search or custom datalog query, pipe the EDN output through xargs into logseq show --id, and pipe directly into logseq show --id via stdin. +The show stdin behavior test will assert that missing --id input without stdin returns a clear error and that stdin is parsed the same as an explicit id argument. ## Implementation Details -- Add a helper predicate for scalar ids that accepts integers and rejects maps, vectors, and strings. -- Detect sequential results that are entirely scalar ids and return a newline-joined string of ids. - Preserve the existing safe-read-string validation to avoid changing behavior for invalid EDN strings. - Keep non-scalar results as their original EDN string output. - Maintain nil handling so that "nil" still prints as "nil". -- Ensure the output is stable for xargs by avoiding embedded spaces in the line-oriented mode. -- Add stdin reading to show -id when the option is present but its value is missing. -- Parse stdin through the same id parsing function used for explicit -id values. -- Keep existing errors when -id is missing and stdin is empty or whitespace. +- Remove the space-to-comma transformation while keeping EDN output intact, including spaces. +- Add stdin reading to show --id when stdin is non-empty, even if an id argument is provided. +- Parse stdin through the same id parsing function used for explicit --id values. +- Keep existing errors when --id is missing and stdin is empty or whitespace. - Update integration tests to use the exact pipeline command from the requirement. - Use @test-driven-development for all implementation steps. ## Question -The line-oriented output applies only to vectors of numeric ids. -Logseq show -id reads stdin only when -id is present with no value. +Query results remain in EDN form (e.g., `[1 2 3]`) with no line-oriented transformation. +Logseq show --id accepts ids via explicit argument and via stdin pipelines (stdin takes precedence when provided). --- diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index abed3d9227..e21893410b 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -29,6 +29,13 @@ [] (.toString (fs/readFileSync 0) "utf8")) +(defn- stdin-available? + [] + (try + (let [stat (fs/fstatSync 0)] + (or (.isFIFO stat) (.isFile stat))) + (catch :default _ false))) + (defn- normalize-stdin-id [value] (let [text (string/trim (or value ""))] @@ -45,12 +52,16 @@ (defn- resolve-stdin-id [options] - (if (:id-from-stdin? options) - (let [stdin (if (contains? options :stdin) - (:stdin options) - (read-stdin))] - (assoc options :id (normalize-stdin-id stdin))) - options)) + (let [id-present? (or (contains? options :id) (some? (:id options))) + stdin (cond + (contains? options :stdin) (:stdin options) + (:id-from-stdin? options) (read-stdin) + (and id-present? (stdin-available?)) (read-stdin) + :else nil) + normalized (normalize-stdin-id stdin)] + (if (string/blank? normalized) + options + (assoc options :id normalized)))) (defn invalid-options? [opts] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index f4c8daf9ce..1e6ac00f7c 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -182,25 +182,13 @@ (:pid server)]) (or servers [])))) -(defn- scalar-id? - [value] - (and (number? value) (integer? value))) - -(defn- scalar-id-seq? - [value] - (and (sequential? value) - (seq value) - (every? scalar-id? value))) - (defn- format-query-results [result] (let [edn-str (pr-str result) parsed (common-util/safe-read-string {:log-error? false} edn-str) valid? (or (some? parsed) (= "nil" (string/trim edn-str)))] (if valid? - (if (scalar-id-seq? result) - (string/join "\n" (map str result)) - edn-str) + edn-str edn-str))) (defn- format-query-list diff --git a/src/test/logseq/cli/command/show_test.cljs b/src/test/logseq/cli/command/show_test.cljs index a36f749769..bf8c326c6c 100644 --- a/src/test/logseq/cli/command/show_test.cljs +++ b/src/test/logseq/cli/command/show_test.cljs @@ -23,7 +23,23 @@ (is (= [1 2 3] (get-in result [:action :ids]))) (is (true? (get-in result [:action :multi-id?]))))) - (testing "blank stdin returns invalid options" + (testing "stdin overrides explicit id when present" + (let [result (show-command/build-action {:id "99" + :stdin "[1 2]"} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= [1 2] (get-in result [:action :ids]))) + (is (true? (get-in result [:action :multi-id?]))))) + + (testing "blank stdin falls back to explicit id" + (let [result (show-command/build-action {:id "99" + :stdin " "} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= 99 (get-in result [:action :id]))) + (is (= [99] (get-in result [:action :ids]))))) + + (testing "blank stdin returns invalid options when id is missing" (let [result (show-command/build-action {:id "" :id-from-stdin? true :stdin " "} diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index c733b050d7..819047886e 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -171,12 +171,12 @@ (is (= "Line 1\nLine 2" result))))) (deftest test-human-output-query-results - (testing "scalar id collections render one id per line" + (testing "scalar id collections preserve EDN formatting" (let [result (format/format-result {:status :ok :command :query :data {:result [1 2 3]}} {:output-format nil})] - (is (= "1\n2\n3" result)))) + (is (= "[1 2 3]" result)))) (testing "non-scalar collections preserve EDN formatting" (let [value [{:db/id 1 :block/title "Alpha"} diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index a804234eb6..af9adc724b 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1180,6 +1180,20 @@ " :where" " [?e :block/title ?title]" " [(clojure.string/includes? ?title ?q)]]") + query-json-result (run-cli ["--repo" "query-pipe-graph" + "query" + "--query" query-text + "--inputs" (pr-str ["Pipe"])] + data-dir cfg-path) + query-json-payload (parse-json-output query-json-result) + query-ids (get-in query-json-payload [:data :result]) + query-human-result (run-cli ["--repo" "query-pipe-graph" + "--output" "human" + "query" + "--query" query-text + "--inputs" (pr-str ["Pipe"])] + data-dir cfg-path) + query-human-output (string/trim (:output query-human-result)) node-bin (shell-escape (.-execPath js/process)) cli-bin (shell-escape (node-path/resolve "static/logseq-cli.js")) data-arg (shell-escape data-dir) @@ -1209,6 +1223,7 @@ stop-result (run-cli ["server" "stop" "--repo" "query-pipe-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] + (is (= (pr-str query-ids) query-human-output)) (is (string/includes? output "Pipe One")) (is (string/includes? output "Pipe Two")) (is (= "ok" (:status stop-payload))) @@ -1239,6 +1254,13 @@ " :where" " [?e :block/title ?title]" " [(clojure.string/includes? ?title ?q)]]") + query-json-result (run-cli ["--repo" "query-stdin-graph" + "query" + "--query" query-text + "--inputs" (pr-str ["Pipe"])] + data-dir cfg-path) + query-json-payload (parse-json-output query-json-result) + query-ids (get-in query-json-payload [:data :result]) query-result (run-cli ["--repo" "query-stdin-graph" "--output" "human" "query" @@ -1261,6 +1283,7 @@ stop-result (run-cli ["server" "stop" "--repo" "query-stdin-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] + (is (= (pr-str query-ids) ids-text)) (is (= "ok" (:status show-payload))) (is (contains? root-titles "PipePage")) (is (some? pipe-one)) From bd0f4ebe6a35f658a1d450fce61f2a68a05d35e4 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 1 Feb 2026 23:38:53 +0800 Subject: [PATCH 059/375] 027-logseq-cli-update-command.md --- .../027-logseq-cli-update-command.md | 133 +++++++++++ src/main/logseq/cli/command/add.cljs | 119 +++++++++- src/main/logseq/cli/command/core.cljs | 2 +- .../cli/command/{move.cljs => update.cljs} | 125 ++++++++-- src/main/logseq/cli/commands.cljs | 28 +-- src/main/logseq/cli/format.cljs | 20 +- src/test/logseq/cli/commands_test.cljs | 188 +++++++++++---- src/test/logseq/cli/format_test.cljs | 22 +- src/test/logseq/cli/integration_test.cljs | 219 ++++++++++++------ 9 files changed, 672 insertions(+), 184 deletions(-) create mode 100644 docs/agent-guide/027-logseq-cli-update-command.md rename src/main/logseq/cli/command/{move.cljs => update.cljs} (51%) diff --git a/docs/agent-guide/027-logseq-cli-update-command.md b/docs/agent-guide/027-logseq-cli-update-command.md new file mode 100644 index 0000000000..1911937726 --- /dev/null +++ b/docs/agent-guide/027-logseq-cli-update-command.md @@ -0,0 +1,133 @@ +# Logseq CLI Update Command (Move Refactor) + +## Summary +Introduce a new `update` CLI command that subsumes the current `move` command and adds tag/property updates. The new command keeps all existing move capabilities/options, makes move targets optional (no target means no move), and adds update/remove semantics for tags and properties. The plan is grounded in current `logseq-cli` parsing/execution and db-worker-node outliner ops. + +## Goals +- Replace `move` command behavior with `update` while keeping all current move options and semantics. +- Allow `update` to move blocks *and/or* update tags/properties in one command. +- Support new options: + - `--update-tags`, `--update-properties` + - `--remove-tags`, `--remove-properties` +- Reuse tag/property parsing and resolution rules from `add block`: + - Identifiers accept `db/id`, `db/ident`, `block/title`. + - `--update-tags` and `--update-properties` accept the same EDN format as `add block` `--tags`/`--properties`. + - `--remove-tags` and `--remove-properties` accept EDN vectors of tag/property identifiers. + +## Non-goals +- Changing db-worker-node APIs or introducing new outliner ops. +- Expanding `update` beyond blocks (no page-level update scope in this iteration). +- Changing move semantics or target resolution. + +## Background: Current Move Command +- CLI move implementation lives in `src/main/logseq/cli/command/move.cljs`. +- It requires a source (`--id` or `--uuid`) and a target (`--target-id`, `--target-uuid`, or `--target-page`). +- Execution uses `:thread-api/apply-outliner-ops` with `[:move-blocks ...]`. +- Move output formatting in `src/main/logseq/cli/format.cljs` uses `format-move-block`. + +## Proposed Behavior (Update Command) +### Required/Optional Inputs +- **Source is required**: one of `--id` or `--uuid`. +- **Move targets are optional**: + - `--target-page`, `--target-uuid`, `--target-id`, `--pos` are optional. + - If no target is provided, no move occurs and the command only updates tags/properties. + - If a target is provided, move executes with the same semantics as today. +- **Update/remove options are optional**: + - `--update-tags` (EDN vector, same as `add block --tags`) + - `--update-properties` (EDN map, same as `add block --properties`) + - `--remove-tags` (EDN vector of tag identifiers) + - `--remove-properties` (EDN vector of property identifiers) + +### Validation Rules +- Only one source selector is allowed: `--id` or `--uuid`. +- Only one target selector is allowed: `--target-id`, `--target-uuid`, or `--target-page`. +- `--pos sibling` is only valid when the target is a block (not a page), same as `move`. +- At least one of the following must be provided: target (move) or update/remove options. +- `--update-tags` and `--update-properties` accept the same EDN grammar and validations as in `add`. +- `--remove-tags` and `--remove-properties` must be non-empty vectors (EDN), with identifiers validated similarly to add. + +### Execution Semantics +- Resolve source block to `db/id` using existing move helpers. +- If move target provided, resolve target entity using existing move helpers and compute `pos` opts. +- Tag/property updates use current outliner ops (no new db-worker changes): + - **Add/update tags**: `[:batch-set-property [block-ids :block/tags tag-id {}]]` for each tag. + - **Add/update properties**: `[:batch-set-property [block-ids property-id value {}]]`. + - **Remove tags**: `[:batch-delete-property-value [block-ids :block/tags tag-id]]`. + - **Remove properties**: `[:batch-remove-property [block-ids property-id]]`, with property resolution aligned to `add` (built-in/public property rules). +- Combine operations into a single `apply-outliner-ops` call when possible to keep updates atomic and reduce roundtrips. + +## Design and Implementation Plan +### 1) Create `update` command module +- New file: `src/main/logseq/cli/command/update.cljs`. +- Start from `move.cljs` and expand the spec to include update/remove tag/property options. +- Extract shared helpers from `move.cljs` (e.g., `resolve-source`, `resolve-target`, `pos->opts`, `invalid-options?`) into a small shared namespace or move into `update.cljs`. + +### 2) Reuse tag/property parsing and resolution from `add` +- Refactor `src/main/logseq/cli/command/add.cljs` to expose reusable helpers for: + - `parse-tags-option`, `parse-properties-option` (for update) + - `resolve-tags`, `resolve-properties` (for update) + - Tag/property identifier normalization functions if needed for remove vectors +- Keep the parsing behavior exactly consistent with `add block`. +- Add new helper in `add` (or a shared namespace) to parse **remove vectors**: + - `parse-tags-vector-option` (vector of tag identifiers) + - `parse-properties-vector-option` (vector of property identifiers) + +### 3) Update top-level command registry +- Add `update` to `src/main/logseq/cli/commands.cljs` table and summary groups. +- Remove `move` from the command registry and help output (no alias/compat). + +### 4) Update parse/validation logic +- In `finalize-command` (`src/main/logseq/cli/commands.cljs`): + - Add `update`-specific validation for sources, targets, and update/remove options. + - Reuse `update-command/invalid-options?`. + - Allow missing target when update/remove options are present. + - Keep error messaging aligned with existing CLI patterns. + +### 5) Implement update action building +- `build-action` returns a combined action with: + - `:type :update-block` + - `:id`/`:uuid` for source, optional target selectors, `:pos` default `first-child` when move is requested + - `:update-tags`, `:update-properties`, `:remove-tags`, `:remove-properties` +- `execute-update`: + - Resolve source; resolve target only when move is requested. + - Build a vector of outliner ops, in order: move (if present), then remove tags/properties, then update/add tags/properties (order can be adjusted if needed for predictable results). + - Use `:thread-api/apply-outliner-ops` once with all ops. + +### 6) Update output formatting +- `src/main/logseq/cli/format.cljs`: + - Add `format-update-block` to describe move + updates in a concise line. + - Update dispatcher to handle `:update-block`. + +### 7) Tests +- Unit tests in `src/test/logseq/cli/commands_test.cljs`: + - Parsing: `update` accepts `--id/--uuid`, optional target, and new options. + - Validation: missing source, invalid target selector combinations, invalid pos, invalid EDN options. + - Ensure that `update` without target but with update/remove options is accepted. +- Format tests in `src/test/logseq/cli/format_test.cljs` for `:update-block`. +- Integration tests in `src/test/logseq/cli/integration_test.cljs`: + - Move-only update should behave the same as current `move`. + - Update tags/properties on an existing block. + - Remove tags/properties and validate via `show` or query. + +## Open Questions +- If only `--pos` is provided without any target selector, return the error: `--pos is only valid when a target is provided`. + +## Risks / Edge Cases +- If tag/property parsing rules diverge from `add`, user experience becomes inconsistent. Refactor shared parsing to avoid drift. +- Combining move and property updates in one `apply-outliner-ops` call needs to preserve correct operation order; keep move first unless property updates depend on position. +- Ensure non-page validation for source and block target remains intact when refactoring. + +## Implementation Checklist (Concrete File Touches) +- Add: `src/main/logseq/cli/command/update.cljs`. +- Update: `src/main/logseq/cli/commands.cljs` (table, validation, action dispatch). +- Update: `src/main/logseq/cli/command/core.cljs` (top-level summary group list to include update). +- Update: `src/main/logseq/cli/format.cljs` (formatting for update). +- Update: `src/test/logseq/cli/commands_test.cljs`. +- Update: `src/test/logseq/cli/format_test.cljs`. +- Update: `src/test/logseq/cli/integration_test.cljs`. + +## Verification +- Run unit tests: `bb dev:test -v logseq.cli.commands-test`. +- Run format tests: `bb dev:test -v logseq.cli.format-test`. +- Run CLI integration tests (move/update subset): `bb dev:test -v logseq.cli.integration-test`. +- Optional full suite: `bb dev:lint-and-test`. diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 63985c5879..6e89244c2f 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -51,12 +51,12 @@ (let [page-name-lc (common-util/page-name-sanity-lc page-name)] (p/let [page (transport/invoke config :thread-api/pull false [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])] - (if (:db/id page) - page - (p/let [_ (transport/invoke config :thread-api/apply-outliner-ops false - [repo [[:create-page [page-name {}]]] {}])] - (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])))))) + (if (:db/id page) + page + (p/let [_ (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:create-page [page-name {}]]] {}])] + (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])))))) (def ^:private add-positions #{"first-child" "last-child" "sibling"}) @@ -224,7 +224,7 @@ :else text)) :else nil)) -(defn- parse-tags-option +(defn parse-tags-option [value] (if-not (seq value) {:ok? true :value nil} @@ -245,6 +245,10 @@ (invalid-options-result "tags must be strings, keywords, uuids, or ids") {:ok? true :value tags})))))) +(defn parse-tags-vector-option + [value] + (parse-tags-option value)) + (defn- normalize-property-key [value] (cond @@ -272,7 +276,7 @@ (when (seq text) (get built-in-properties-by-title (common-util/page-name-sanity-lc text)))))) -(defn- normalize-property-key-input +(defn normalize-property-key-input [value] (cond (keyword? value) {:type :ident :value value} @@ -426,7 +430,7 @@ [property] (true? (get-in property [:schema :public?]))) -(defn- parse-properties-option +(defn parse-properties-option [value] (if-not (seq value) {:ok? true :value nil} @@ -469,6 +473,44 @@ (invalid-options-result (str "invalid value for " key-ident ": " message)) (recur (rest prop-entries) (assoc acc key-ident normalized-value)))))))))))))))) +(defn parse-properties-vector-option + [value] + (if-not (seq value) + {:ok? true :value nil} + (let [parsed (parse-edn-option value)] + (cond + (nil? parsed) + (invalid-options-result "properties must be valid EDN vector") + + (not (vector? parsed)) + (invalid-options-result "properties must be a vector") + + (empty? parsed) + (invalid-options-result "properties must be a non-empty vector") + + :else + (loop [prop-entries (seq parsed) + acc []] + (if (empty? prop-entries) + {:ok? true :value acc} + (let [entry (first prop-entries) + key-result (normalize-property-key-input entry)] + (if-not key-result + (invalid-options-result (str "invalid property key: " entry)) + (let [{:keys [type value]} key-result] + (if (= type :id) + (recur (rest prop-entries) (conj acc value)) + (let [property (get db-property/built-in-properties value)] + (cond + (nil? property) + (invalid-options-result (str "unknown built-in property: " value)) + + (not (property-public? property)) + (invalid-options-result (str "property is not public: " value)) + + :else + (recur (rest prop-entries) (conj acc value)))))))))))))) + (defn invalid-options? [opts] (let [pos (some-> (:pos opts) string/trim string/lower-case) @@ -528,7 +570,7 @@ :else entity)))) -(defn- resolve-tags +(defn resolve-tags [config repo tags] (if (seq tags) (p/let [entities (p/all (map #(resolve-tag-entity config repo %) tags))] @@ -613,7 +655,7 @@ :date (resolve-date-page-id config repo value) (p/resolved value)))) -(defn- resolve-properties +(defn resolve-properties [config repo properties] (if-not (seq properties) (p/resolved nil) @@ -682,6 +724,61 @@ properties))] (into {} resolved-entries)))) +(defn resolve-property-identifiers + [config repo properties] + (if-not (seq properties) + (p/resolved nil) + (p/let [resolved-entries (p/all + (map (fn [k] + (cond + (keyword? k) + (let [property (get db-property/built-in-properties k)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property k}))) + (p/resolved k)) + + (number? k) + (p/let [entity (pull-entity config repo [:db/ident] k) + ident (:db/ident entity) + property (get db-property/built-in-properties ident)] + (cond + (nil? ident) + (throw (ex-info "property not found" + {:code :property-not-found :property k})) + + (nil? property) + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property ident})) + + (not (property-public? property)) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident})) + + :else + ident)) + + (string? k) + (let [ident (or (property-title->ident k) + (normalize-property-key k)) + property (get db-property/built-in-properties ident)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident}))) + (p/resolved ident)) + + :else + (p/rejected (ex-info "invalid property key" + {:code :invalid-property :property k})))) + properties))] + (vec resolved-entries)))) + (defn- resolve-add-target [config {:keys [repo target-id target-uuid target-page-name]}] (cond diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 6f050506c2..0ec5a91039 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -91,7 +91,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "move" "query" "show"}} + :commands #{"list" "add" "remove" "update" "query" "show"}} {:title "Graph Management" :commands #{"graph" "server"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/move.cljs b/src/main/logseq/cli/command/update.cljs similarity index 51% rename from src/main/logseq/cli/command/move.cljs rename to src/main/logseq/cli/command/update.cljs index b0fcfc43f1..4f6fa24d0c 100644 --- a/src/main/logseq/cli/command/move.cljs +++ b/src/main/logseq/cli/command/update.cljs @@ -1,13 +1,14 @@ -(ns logseq.cli.command.move - "Move-related CLI commands." +(ns logseq.cli.command.update + "Update-related CLI commands." (:require [clojure.string :as string] + [logseq.cli.command.add :as add-command] [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [promesa.core :as p])) -(def ^:private move-spec +(def ^:private update-spec {:id {:desc "Source block db/id" :coerce :long} :uuid {:desc "Source block UUID"} @@ -15,12 +16,16 @@ :coerce :long} :target-uuid {:desc "Target block UUID"} :target-page {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling). Default: first-child"}}) + :pos {:desc "Position (first-child, last-child, sibling). Default: first-child"} + :update-tags {:desc "Tags to add/update (EDN vector)"} + :update-properties {:desc "Properties to add/update (EDN map)"} + :remove-tags {:desc "Tags to remove (EDN vector)"} + :remove-properties {:desc "Properties to remove (EDN vector)"}}) (def entries - [(core/command-entry ["move"] :move-block "Move block" move-spec)]) + [(core/command-entry ["update"] :update-block "Update block" update-spec)]) -(def ^:private move-positions +(def ^:private update-positions #{"first-child" "last-child" "sibling"}) (defn invalid-options? @@ -29,9 +34,14 @@ source-selectors (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) target-selectors (filter some? [(:target-id opts) (:target-uuid opts) - (some-> (:target-page opts) string/trim)])] + (some-> (:target-page opts) string/trim)]) + has-update-tags? (seq (some-> (:update-tags opts) string/trim)) + has-update-properties? (seq (some-> (:update-properties opts) string/trim)) + has-remove-tags? (seq (some-> (:remove-tags opts) string/trim)) + has-remove-properties? (seq (some-> (:remove-properties opts) string/trim)) + has-updates? (or has-update-tags? has-update-properties? has-remove-tags? has-remove-properties?)] (cond - (and (seq pos) (not (contains? move-positions pos))) + (and (seq pos) (not (contains? update-positions pos))) (str "invalid pos: " (:pos opts)) (> (count source-selectors) 1) @@ -43,6 +53,12 @@ (and (= pos "sibling") (seq (some-> (:target-page opts) string/trim))) "--pos sibling is only valid for block targets" + (and (seq pos) (empty? target-selectors)) + "--pos is only valid when a target is provided" + + (and (empty? target-selectors) (not has-updates?)) + "target or update/remove options are required" + :else nil))) @@ -104,12 +120,13 @@ (throw (ex-info "target block not found" {:code :target-not-found}))))) (seq target-page) - (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid :block/name :block/title] - [:block/name target-page]])] + (let [page-name (common-util/page-name-sanity-lc target-page)] + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid :block/name :block/title] + [:block/name page-name]])] (if (:db/id entity) entity - (throw (ex-info "page not found" {:code :page-not-found})))) + (throw (ex-info "page not found" {:code :page-not-found}))))) :else (p/rejected (ex-info "target is required" {:code :missing-target})))) @@ -127,13 +144,23 @@ (if-not (seq repo) {:ok? false :error {:code :missing-repo - :message "repo is required for move"}} + :message "repo is required for update"}} (let [id (:id options) uuid (some-> (:uuid options) string/trim) target-id (:target-id options) target-uuid (some-> (:target-uuid options) string/trim) page-name (some-> (:target-page options) string/trim) pos (some-> (:pos options) string/trim string/lower-case) + update-tags-result (add-command/parse-tags-option (:update-tags options)) + update-properties-result (add-command/parse-properties-option (:update-properties options)) + remove-tags-result (add-command/parse-tags-vector-option (:remove-tags options)) + remove-properties-result (add-command/parse-properties-vector-option (:remove-properties options)) + update-tags (:value update-tags-result) + update-properties (:value update-properties-result) + remove-tags (:value remove-tags-result) + remove-properties (:value remove-properties-result) + has-target? (or (some? target-id) (seq target-uuid) (seq page-name)) + has-updates? (or (seq update-tags) (seq update-properties) (seq remove-tags) (seq remove-properties)) source-label (cond (seq uuid) uuid (some? id) (str id) @@ -149,14 +176,26 @@ :error {:code :missing-source :message "source block is required"}} - (not (or (some? target-id) (seq target-uuid) (seq page-name))) + (and (not has-target?) (not has-updates?)) {:ok? false - :error {:code :missing-target - :message "target is required"}} + :error {:code :invalid-options + :message "target or update/remove options are required"}} + + (not (:ok? update-tags-result)) + update-tags-result + + (not (:ok? update-properties-result)) + update-properties-result + + (not (:ok? remove-tags-result)) + remove-tags-result + + (not (:ok? remove-properties-result)) + remove-properties-result :else {:ok? true - :action {:type :move-block + :action {:type :update-block :repo repo :graph (core/repo->graph repo) :id id @@ -164,18 +203,56 @@ :target-id target-id :target-uuid target-uuid :target-page page-name - :pos (or pos "first-child") + :pos (when has-target? (or pos "first-child")) + :update-tags update-tags + :update-properties update-properties + :remove-tags remove-tags + :remove-properties remove-properties :source source-label :target target-label}})))) -(defn execute-move +(defn execute-update [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) source (resolve-source cfg (:repo action) action) - target (resolve-target cfg (:repo action) action) - opts (pos->opts (:pos action)) - ops [[:move-blocks [[(:db/id source)] (:db/id target) opts]]] - result (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) ops {}])] + target (when (or (:target-id action) + (:target-uuid action) + (seq (:target-page action))) + (resolve-target cfg (:repo action) action)) + opts (when target (pos->opts (:pos action))) + update-tags (add-command/resolve-tags cfg (:repo action) (:update-tags action)) + remove-tags (add-command/resolve-tags cfg (:repo action) (:remove-tags action)) + update-properties (add-command/resolve-properties cfg (:repo action) (:update-properties action)) + remove-properties (add-command/resolve-property-identifiers cfg (:repo action) + (:remove-properties action)) + block-id (:db/id source) + block-ids [block-id] + update-tag-ids (when (seq update-tags) + (->> update-tags (map :db/id) (remove nil?) vec)) + remove-tag-ids (when (seq remove-tags) + (->> remove-tags (map :db/id) (remove nil?) vec)) + ops (cond-> [] + target (conj [:move-blocks [[(:db/id source)] (:db/id target) opts]])) + ops (cond-> ops + (seq remove-tag-ids) + (into (map (fn [tag-id] + [:batch-delete-property-value [block-ids :block/tags tag-id]]) + remove-tag-ids)) + (seq remove-properties) + (into (map (fn [property-id] + [:batch-remove-property [block-ids property-id]]) + remove-properties)) + (seq update-tag-ids) + (into (map (fn [tag-id] + [:batch-set-property [block-ids :block/tags tag-id {}]]) + update-tag-ids)) + (seq update-properties) + (into (map (fn [[k v]] + [:batch-set-property [block-ids k v {}]]) + update-properties))) + result (if (seq ops) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) ops {}]) + (p/resolved nil))] {:status :ok :data {:result result}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index ac38c6063e..8191e2cf52 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -6,11 +6,11 @@ [logseq.cli.command.core :as command-core] [logseq.cli.command.graph :as graph-command] [logseq.cli.command.list :as list-command] - [logseq.cli.command.move :as move-command] [logseq.cli.command.query :as query-command] [logseq.cli.command.remove :as remove-command] [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] + [logseq.cli.command.update :as update-command] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -98,8 +98,8 @@ server-command/entries list-command/entries add-command/entries - move-command/entries remove-command/entries + update-command/entries query-command/entries show-command/entries))) @@ -147,10 +147,7 @@ remove-targets (filter some? [(:id opts) (some-> (:uuid opts) string/trim) (some-> (:page opts) string/trim)]) - move-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]) - move-targets (filter some? [(:target-id opts) - (some-> (:target-uuid opts) string/trim) - (some-> (:target-page opts) string/trim)])] + update-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)])] (cond (:help opts) (command-core/help-result cmd-summary) @@ -177,15 +174,12 @@ (and (= command :remove) (> (count remove-targets) 1)) (command-core/invalid-options-result summary "only one of --id, --uuid, or --page is allowed") - (and (= command :move-block) (move-command/invalid-options? opts)) - (command-core/invalid-options-result summary (move-command/invalid-options? opts)) + (and (= command :update-block) (update-command/invalid-options? opts)) + (command-core/invalid-options-result summary (update-command/invalid-options? opts)) - (and (= command :move-block) (empty? move-sources)) + (and (= command :update-block) (empty? update-sources)) (missing-source-result summary) - (and (= command :move-block) (empty? move-targets)) - (missing-target-result summary) - (and (= command :show) (empty? show-targets)) (missing-target-result summary) @@ -366,8 +360,8 @@ :add-page (add-command/build-add-page-action options repo) - :move-block - (move-command/build-action options repo) + :update-block + (update-command/build-action options repo) :remove (remove-command/build-action options repo) @@ -411,7 +405,7 @@ :list-property (list-command/execute-list-property action config) :add-block (add-command/execute-add-block action config) :add-page (add-command/execute-add-page action config) - :move-block (move-command/execute-move action config) + :update-block (update-command/execute-update action config) :remove (remove-command/execute-remove action config) :query (query-command/execute-query action config) :query-list (query-command/execute-query-list action config) @@ -426,4 +420,6 @@ :message "unknown action"}}))] (assoc result :command (or (:command action) (:type action)) - :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks :source :target]))))) + :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks + :source :target :update-tags :update-properties + :remove-tags :remove-properties]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 1e6ac00f7c..c03b804204 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -246,17 +246,25 @@ (some? id) (str "Removed block: " id " (repo: " repo ")") :else (str "Removed item (repo: " repo ")"))) -(defn- format-move-block - [{:keys [repo source target]}] - (str "Moved block: " source " -> " target " (repo: " repo ")")) +(defn- format-update-block + [{:keys [repo source target update-tags update-properties remove-tags remove-properties]}] + (let [change-parts (cond-> [] + (seq update-tags) (conj (str "tags:+" (count update-tags))) + (seq update-properties) (conj (str "properties:+" (count update-properties))) + (seq remove-tags) (conj (str "remove-tags:+" (count remove-tags))) + (seq remove-properties) (conj (str "remove-properties:+" (count remove-properties)))) + changes (when (seq change-parts) + (str ", " (string/join ", " change-parts))) + move-fragment (when (seq target) + (str " -> " target))] + (str "Updated block: " source (or move-fragment "") " (repo: " repo (or changes "") ")"))) (defn- format-graph-export [{:keys [export-type output]}] (str "Exported " export-type " to " output)) (defn- format-graph-import - [{:keys [import-type input] :as xxx}] - (prn :xxx xxx) + [{:keys [import-type input]}] (str "Imported " import-type " from " input)) (defn- format-graph-action @@ -288,7 +296,7 @@ :add-block (format-add-block context) :add-page (format-add-page context) :remove (format-remove context) - :move-block (format-move-block context) + :update-block (format-update-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) :query (format-query-results (:result data)) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index b9c0e4f709..53a44ac0e3 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -61,7 +61,7 @@ (is (string/includes? plain-summary "list")) (is (string/includes? plain-summary "add")) (is (string/includes? plain-summary "remove")) - (is (string/includes? plain-summary "move")) + (is (string/includes? plain-summary "update")) (is (string/includes? plain-summary "query")) (is (string/includes? plain-summary "show")) (is (string/includes? plain-summary "graph")) @@ -72,7 +72,7 @@ (is (contains-bold? summary "add block")) (is (contains-bold? summary "add page")) (is (contains-bold? summary "remove")) - (is (contains-bold? summary "move")) + (is (contains-bold? summary "update")) (is (contains-bold? summary "query")) (is (contains-bold? summary "query list")) (is (contains-bold? summary "show")) @@ -153,18 +153,22 @@ (is (contains-bold? summary "--uuid")) (is (contains-bold? summary "--page")))) - (testing "move command shows help" + (testing "update command shows help" (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["move" "--help"])) + (commands/parse-args ["update" "--help"])) summary (:summary result) plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? plain-summary "Usage: logseq move")) + (is (string/includes? plain-summary "Usage: logseq update")) (is (string/includes? plain-summary "Command options:")) (is (contains-bold? summary "--id")) (is (contains-bold? summary "--uuid")) (is (contains-bold? summary "--target-id")) - (is (contains-bold? summary "--target-uuid")))) + (is (contains-bold? summary "--target-uuid")) + (is (contains-bold? summary "--update-tags")) + (is (contains-bold? summary "--update-properties")) + (is (contains-bold? summary "--remove-tags")) + (is (contains-bold? summary "--remove-properties")))) (testing "server group shows subcommands" (let [result (binding [style/*color-enabled?* true] @@ -743,23 +747,39 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "move requires source selector" - (let [result (commands/parse-args ["move" "--target-id" "10"])] + (testing "update requires source selector" + (let [result (commands/parse-args ["update" "--target-id" "10"])] (is (false? (:ok? result))) (is (= :missing-source (get-in result [:error :code]))))) - (testing "move requires target selector" - (let [result (commands/parse-args ["move" "--id" "1"])] + (testing "update requires target or update/remove options" + (let [result (commands/parse-args ["update" "--id" "1"])] (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code]))))) - (testing "move parses with source and target" - (let [result (commands/parse-args ["move" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] + (testing "update parses with source and target" + (let [result (commands/parse-args ["update" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] (is (true? (:ok? result))) - (is (= :move-block (:command result))) + (is (= :update-block (:command result))) (is (= "abc" (get-in result [:options :uuid]))) (is (= "def" (get-in result [:options :target-uuid]))) - (is (= "last-child" (get-in result [:options :pos])))))) + (is (= "last-child" (get-in result [:options :pos]))))) + + (testing "update parses with tags and properties" + (let [result (commands/parse-args ["update" "--id" "1" + "--update-tags" "[\"TagA\"]" + "--update-properties" "{:logseq.property/publishing-public? true}"])] + (is (true? (:ok? result))) + (is (= :update-block (:command result))) + (is (= "[\"TagA\"]" (get-in result [:options :update-tags]))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :update-properties]))))) + + (testing "update allows update without move target" + (let [result (commands/parse-args ["update" "--uuid" "abc" + "--update-tags" "[\"TagA\"]"])] + (is (true? (:ok? result))) + (is (= :update-block (:command result))) + (is (= "abc" (get-in result [:options :uuid])))))) (deftest test-verb-subcommand-parse-add (testing "add block requires content source" @@ -835,11 +855,11 @@ (is (= "[\"TagA\"]" (get-in result [:options :tags]))) (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties])))))) -(deftest test-verb-subcommand-parse-move-target-page - (testing "move parses with target page" - (let [result (commands/parse-args ["move" "--id" "1" "--target-page" "Home"])] +(deftest test-verb-subcommand-parse-update-target-page + (testing "update parses with target page" + (let [result (commands/parse-args ["update" "--id" "1" "--target-page" "Home"])] (is (true? (:ok? result))) - (is (= :move-block (:command result))) + (is (= :update-block (:command result))) (is (= 1 (get-in result [:options :id]))) (is (= "Home" (get-in result [:options :target-page])))))) @@ -953,7 +973,7 @@ (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] ["remove" "--wat"] - ["move" "--wat"] + ["update" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] (is (false? (:ok? result))) @@ -1126,45 +1146,119 @@ (let [normalize-property-key-input #'add-command/normalize-property-key-input] (is (= {:type :id :value 42} (normalize-property-key-input 42))))) -(deftest test-build-action-move - (testing "move requires source selector" - (let [parsed {:ok? true :command :move-block :options {:target-id 2}} +(deftest test-build-action-update + (testing "update requires source selector" + (let [parsed {:ok? true :command :update-block :options {:target-id 2}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-source (get-in result [:error :code]))))) - (testing "move requires target selector" - (let [parsed {:ok? true :command :move-block :options {:id 1}} + (testing "update requires target or update/remove options" + (let [parsed {:ok? true :command :update-block :options {:id 1}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code])))))) - -(deftest test-move-parse-validation - (testing "move rejects multiple source selectors" - (let [result (commands/parse-args ["move" "--id" "1" "--uuid" "abc" "--target-id" "2"])] - (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "move rejects multiple target selectors" - (let [result (commands/parse-args ["move" "--id" "1" "--target-id" "2" "--target-uuid" "def"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (testing "update accepts update tags without target" + (let [parsed {:ok? true + :command :update-block + :options {:id 1 :update-tags "[\"TagA\"]"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :update-block (get-in result [:action :type]))) + (is (= ["TagA"] (get-in result [:action :update-tags]))))) - (testing "move rejects invalid position" - (let [result (commands/parse-args ["move" "--id" "1" "--target-id" "2" "--pos" "middle"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) - - (testing "move rejects sibling pos for page target" - (let [result (commands/parse-args ["move" "--id" "1" "--target-page" "Home" "--pos" "sibling"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) - - (testing "move rejects legacy target-page-name option" - (let [result (commands/parse-args ["move" "--id" "1" "--target-page-name" "Home"])] + (testing "update rejects invalid update tags" + (let [parsed {:ok? true + :command :update-block + :options {:id 1 :update-tags "{:tag \"no\"}"}} + result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-update-parse-validation + (testing "update rejects multiple source selectors" + (let [result (commands/parse-args ["update" "--id" "1" "--uuid" "abc" "--target-id" "2"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update rejects multiple target selectors" + (let [result (commands/parse-args ["update" "--id" "1" "--target-id" "2" "--target-uuid" "def"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update rejects invalid position" + (let [result (commands/parse-args ["update" "--id" "1" "--target-id" "2" "--pos" "middle"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update rejects sibling pos for page target" + (let [result (commands/parse-args ["update" "--id" "1" "--target-page" "Home" "--pos" "sibling"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update rejects legacy target-page-name option" + (let [result (commands/parse-args ["update" "--id" "1" "--target-page-name" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update rejects pos without target" + (let [result (commands/parse-args ["update" "--id" "1" "--pos" "last-child" "--update-tags" "[\"TagA\"]"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) + +(deftest test-execute-update-builds-batch-ops + (async done + (let [ops* (atom nil) + calls* (atom []) + action {:type :update-block + :repo "demo" + :id 1 + :target-id 2 + :pos "last-child" + :update-tags [:tag/new] + :remove-tags [:tag/old] + :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} + :remove-properties [:logseq.property/publishing-public?]}] + (with-redefs [cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond + (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil))) + add-command/resolve-properties (fn [_ _ properties] (p/resolved properties)) + add-command/resolve-property-identifiers (fn [_ _ properties] (p/resolved properties)) + transport/invoke (fn [_ method _ args] + (swap! calls* conj {:method method :args args}) + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 1) + {:db/id 1 + :block/name nil + :block/uuid (uuid "00000000-0000-0000-0000-000000000001")} + (= lookup 2) + {:db/id 2 + :block/name nil + :block/uuid (uuid "00000000-0000-0000-0000-000000000002")} + :else {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))] + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] + [:batch-delete-property-value [[1] :block/tags 202]] + [:batch-remove-property [[1] :logseq.property/publishing-public?]] + [:batch-set-property [[1] :block/tags 101 {}]] + [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + @ops*)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e " calls: " @calls*)) + (done)))))))) + (deftest test-execute-requires-existing-graph (async done (with-redefs [cli-server/list-graphs (fn [_] []) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 819047886e..1f2634b9d7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -108,15 +108,29 @@ {:output-format nil})] (is (= "Removed page: Home (repo: demo-repo)" result)))) - (testing "move block renders a succinct success line" + (testing "update block renders a succinct success line" (let [result (format/format-result {:status :ok - :command :move-block + :command :update-block :context {:repo "demo-repo" :source "source-uuid" - :target "target-uuid"} + :target "target-uuid" + :update-tags ["TagA"] + :update-properties {:logseq.property/publishing-public? true} + :remove-tags ["TagB"] + :remove-properties [:logseq.property/deadline]} :data {:result {:ok true}}} {:output-format nil})] - (is (= "Moved block: source-uuid -> target-uuid (repo: demo-repo)" result))))) + (is (= "Updated block: source-uuid -> target-uuid (repo: demo-repo, tags:+1, properties:+1, remove-tags:+1, remove-properties:+1)" result)))) + + (testing "update without move target renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :update-block + :context {:repo "demo-repo" + :source "source-uuid" + :update-tags ["TagA"]} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Updated block: source-uuid (repo: demo-repo, tags:+1)" result))))) (deftest test-human-output-graph-import-export (testing "graph export renders a succinct success line" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index af9adc724b..6131d0e96d 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -16,11 +16,19 @@ (defn- run-cli [args data-dir cfg-path] - (let [args-with-output (if (some #{"--output"} args) - args - (concat args ["--output" "json"])) + (let [args (vec args) + output-idx (.indexOf args "--output") + [args output-args] (if (and (>= output-idx 0) + (< (inc output-idx) (count args))) + [(vec (concat (subvec args 0 output-idx) + (subvec args (+ output-idx 2)))) + ["--output" (nth args (inc output-idx))]] + [args []]) + output-args (if (seq output-args) + output-args + ["--output" "json"]) global-opts ["--data-dir" data-dir "--config" cfg-path] - final-args (vec (concat global-opts args-with-output))] + final-args (vec (concat global-opts output-args args))] (-> (cli-main/run! final-args {:exit? false}) (p/then (fn [result] (let [res (if (map? result) @@ -30,7 +38,12 @@ (defn- parse-json-output [result] - (js->clj (js/JSON.parse (:output result)) :keywordize-keys true)) + (try + (js->clj (js/JSON.parse (:output result)) :keywordize-keys true) + (catch :default e + (throw (ex-info "json parse failed" + {:output (:output result)} + e))))) (defn- parse-json-output-safe [result label] @@ -76,6 +89,10 @@ [node] (or (:block/children node) (:children node))) +(defn- node-id + [node] + (or (:db/id node) (:id node))) + (defn- item-id [item] (or (:db/id item) (:id item))) @@ -112,35 +129,32 @@ (defn- query-tags [data-dir cfg-path repo title] - (p/let [payload (run-query data-dir cfg-path repo - "[:find ?tag :in $ ?title :where [?b :block/title ?title] [?b :block/tags ?t] [?t :block/title ?tag]]" - (pr-str [title]))] - (->> (get-in payload [:data :result]) - (map first) - set))) + (let [name (common-util/page-name-sanity-lc title)] + (p/let [payload (run-query data-dir cfg-path repo + "[:find ?tag :in $ ?title ?name :where (or [?b :block/title ?title] [?b :block/content ?title] [?b :block/name ?name]) [?b :block/tags ?t] (or [?t :block/title ?tag] [?t :block/name ?tag])]" + (pr-str [title name]))] + (->> (get-in payload [:data :result]) + (map first) + set)))) (defn- query-property [data-dir cfg-path repo title property] - (p/let [payload (run-query data-dir cfg-path repo - (str "[:find ?value :in $ ?title :where [?e :block/title ?title] [?e " - property - " ?value]]") - (pr-str [title]))] - (first (first (get-in payload [:data :result]))))) + (let [name (common-util/page-name-sanity-lc title)] + (p/let [payload (run-query data-dir cfg-path repo + (str "[:find ?value :in $ ?title ?name :where (or [?e :block/title ?title] [?e :block/content ?title] [?e :block/name ?name]) [?e " + property + " ?value]]") + (pr-str [title name]))] + (first (first (get-in payload [:data :result])))))) -(defn- query-block-id +(defn- query-block-uuid-by-title [data-dir cfg-path repo title] - (p/let [payload (run-query data-dir cfg-path repo - "[:find ?id . :in $ ?title :where [?b :block/title ?title] [?b :db/id ?id]]" - (pr-str [title]))] - (get-in payload [:data :result]))) - -(defn- query-block-uuid-by-id - [data-dir cfg-path repo id] - (p/let [payload (run-query data-dir cfg-path repo - "[:find ?uuid . :in $ ?id :where [?b :db/id ?id] [?b :block/uuid ?uuid]]" - (pr-str [id]))] - (get-in payload [:data :result]))) + (let [name (common-util/page-name-sanity-lc title)] + (p/let [_ (p/delay 300) + payload (run-query data-dir cfg-path repo + "[:find ?uuid . :in $ ?title ?name :where (or [?b :block/title ?title] [?b :block/content ?title] [?b :block/name ?name]) [?b :block/uuid ?uuid]]" + (pr-str [title name]))] + (get-in payload [:data :result])))) (defn- list-items [data-dir cfg-path repo list-type] @@ -246,9 +260,9 @@ add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - show-result (run-cli ["--repo" "content-graph" "show" "--page-name" "TestPage" "--format" "json"] data-dir cfg-path) + show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage" ] data-dir cfg-path) show-payload (parse-json-output show-result) - remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--page" "TestPage"] data-dir cfg-path) + remove-page-result (run-cli ["--repo" "content-graph" "remove" "--page" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -262,7 +276,8 @@ (is (= "ok" (:status list-property-payload))) (is (vector? (get-in list-property-payload [:data :items]))) (is (= "ok" (:status show-payload))) - (is (contains? (get-in show-payload [:data :root]) :db/id)) + (is (some? (or (get-in show-payload [:data :root :db/id]) + (get-in show-payload [:data :root :id])))) (is (not (contains? (get-in show-payload [:data :root]) :block/uuid))) (is (= "ok" (:status remove-page-payload))) (is (= "ok" (:status stop-payload))) @@ -509,7 +524,49 @@ (pr-str (:error add-block-id-payload))) (is (= "ok" (:status add-block-id-payload))) (is (true? page-id-value)) - (is (number? block-deadline-id)) + (is (number? block-deadline-id)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-update-tags-and-properties + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-update-tags")] + (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) + tag-a-name "Quote" + tag-b-name "Math" + add-block-result (run-cli ["--repo" repo + "add" "block" + "--target-page-name" "Home" + "--content" "Update block" + "--tags" "[:logseq.class/Quote-block]" + "--properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + show-home (run-cli ["--repo" repo "show" "--page" "Home"] data-dir cfg-path) + show-home-payload (parse-json-output show-home) + block-node (find-block-by-title (get-in show-home-payload [:data :root]) "Update block") + block-id (node-id block-node) + update-result (run-cli ["--repo" repo + "update" + "--id" (str block-id) + "--update-tags" "[:logseq.class/Math-block]" + "--remove-tags" "[:logseq.class/Quote-block]" + "--update-properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}" + "--remove-properties" "[:logseq.property/publishing-public?]"] + data-dir cfg-path) + update-payload (parse-json-output update-result) + _ (p/delay 300) + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (string? tag-a-name)) + (is (string? tag-b-name)) + (is (some? block-id)) + (is (= "ok" (:status update-payload))) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] @@ -719,8 +776,8 @@ page-id (or (:db/id page-item) (:id page-item)) show-result (run-cli ["--repo" "recent-updated-graph" "show" - "--page-name" "RecentPage" - "--format" "json"] + "--page" "RecentPage" + ] data-dir cfg-path) show-payload (parse-json-output show-result) show-root (get-in show-payload [:data :root]) @@ -804,16 +861,20 @@ _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) - inner-id (query-block-id data-dir cfg-path "nested-refs" "Inner") - inner-uuid (query-block-uuid-by-id data-dir cfg-path "nested-refs" inner-id) + show-nested (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) + show-nested-payload (parse-json-output show-nested) + _inner-node (find-block-by-title (get-in show-nested-payload [:data :root]) "Inner") + inner-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" "Inner") middle-content (str "See [[" inner-uuid "]]") _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" middle-content] data-dir cfg-path) - middle-id (query-block-id data-dir cfg-path "nested-refs" middle-content) - middle-uuid (query-block-uuid-by-id data-dir cfg-path "nested-refs" middle-id) + show-middle (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) + show-middle-payload (parse-json-output show-middle) + _middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) middle-content) + middle-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" middle-content) _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) - show-outer (run-cli ["--repo" "nested-refs" "show" "--page-name" "NestedPage" "--format" "json"] data-dir cfg-path) + show-outer (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage" ] data-dir cfg-path) show-outer-payload (parse-json-output show-outer) outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) @@ -836,21 +897,23 @@ _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - target-id (query-block-id data-dir cfg-path "linked-refs-graph" "TargetPage") - target-uuid (query-block-uuid-by-id data-dir cfg-path "linked-refs-graph" target-id) + target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) + _target-show-payload (parse-json-output target-show) + target-uuid (query-block-uuid-by-title data-dir cfg-path "linked-refs-graph" "TargetPage") target-title "TargetPage" ref-content (str "See [[" target-uuid "]]") ref-title (str "See [[" target-title "]]") _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) - source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "SourcePage" "--format" "json"] data-dir cfg-path) + _ (p/delay 100) + source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "SourcePage" ] data-dir cfg-path) source-payload (parse-json-output source-show) ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-title) - ref-id (:db/id ref-node) - target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + ref-id (or (:db/id ref-node) (:id ref-node)) + target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage" ] data-dir cfg-path) target-payload (parse-json-output target-show) linked-refs (get-in target-payload [:data :linked-references]) linked-blocks (:blocks linked-refs) - linked-ids (set (map :db/id linked-blocks)) + linked-ids (set (map #(or (:db/id %) (:id %)) linked-blocks)) linked-page-titles (set (keep (fn [block] (or (get-in block [:block/page :block/title]) (get-in block [:block/page :block/name]) @@ -870,7 +933,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-move-block +(deftest test-cli-update-block-move (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -879,19 +942,22 @@ _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) - parent-id (query-block-id data-dir cfg-path "move-graph" "Parent Block") - parent-uuid (query-block-uuid-by-id data-dir cfg-path "move-graph" parent-id) - _ (run-cli ["--repo" "move-graph" "add" "block" "--target-uuid" (str parent-uuid) "--content" "Child Block"] data-dir cfg-path) - move-result (run-cli ["--repo" "move-graph" "move" "--uuid" (str parent-uuid) "--target-page-name" "TargetPage"] data-dir cfg-path) - move-payload (parse-json-output move-result) - target-show (run-cli ["--repo" "move-graph" "show" "--page-name" "TargetPage" "--format" "json"] data-dir cfg-path) + _ (p/delay 100) + source-show (run-cli ["--repo" "move-graph" "show" "--page" "SourcePage"] data-dir cfg-path) + source-payload (parse-json-output source-show) + parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") + parent-id (node-id parent-node) + _ (run-cli ["--repo" "move-graph" "add" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) + update-result (run-cli ["--repo" "move-graph" "update" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) + update-payload (parse-json-output update-result) + target-show (run-cli ["--repo" "move-graph" "show" "--page" "TargetPage" ] data-dir cfg-path) target-payload (parse-json-output target-show) moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") child-node (find-block-by-title moved-node "Child Block") stop-result (run-cli ["server" "stop" "--repo" "move-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= "ok" (:status move-payload))) - (is (some? parent-uuid)) + (is (= "ok" (:status update-payload))) + (is (some? parent-id)) (is (some? moved-node)) (is (some? child-node)) (is (= "ok" (:status stop-payload))) @@ -908,17 +974,20 @@ _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) - parent-id (query-block-id data-dir cfg-path "add-pos-graph" "Parent") - parent-uuid (query-block-uuid-by-id data-dir cfg-path "add-pos-graph" parent-id) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "first-child" "--content" "First"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-uuid" (str parent-uuid) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) - final-show (run-cli ["--repo" "add-pos-graph" "show" "--page-name" "PosPage" "--format" "json"] data-dir cfg-path) + _ (p/delay 100) + parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) + parent-payload (parse-json-output parent-show) + parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") + parent-id (node-id parent-node) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) + final-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage" ] data-dir cfg-path) final-payload (parse-json-output final-show) final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") child-titles (map node-title (node-children final-parent)) stop-result (run-cli ["server" "stop" "--repo" "add-pos-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (some? parent-uuid)) + (is (some? parent-id)) (is (= ["First" "Last"] (vec child-titles))) (is (= "ok" (:status stop-payload))) (done)) @@ -1005,9 +1074,9 @@ (get-in list-page-payload [:data :items])) page-id (or (:db/id page-item) (:id page-item)) page-uuid (or (:block/uuid page-item) (:uuid page-item)) - show-by-id-result (run-cli ["show" "--id" (str page-id) "--format" "json"] data-dir cfg-path) + show-by-id-result (run-cli ["show" "--id" (str page-id) ] data-dir cfg-path) show-by-id-payload (parse-json-output show-by-id-result) - show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) "--format" "json"] data-dir cfg-path) + show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) ] data-dir cfg-path) show-by-uuid-payload (parse-json-output show-by-uuid-result) stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -1016,10 +1085,12 @@ (is (some? page-id)) (is (some? page-uuid)) (is (= "ok" (:status show-by-id-payload))) - (is (= page-id (get-in show-by-id-payload [:data :root :db/id]))) + (is (= page-id (or (get-in show-by-id-payload [:data :root :db/id]) + (get-in show-by-id-payload [:data :root :id])))) (is (not (contains? (get-in show-by-id-payload [:data :root]) :block/uuid))) (is (= "ok" (:status show-by-uuid-payload))) - (is (= page-id (get-in show-by-uuid-payload [:data :root :db/id]))) + (is (= page-id (or (get-in show-by-uuid-payload [:data :root :db/id]) + (get-in show-by-uuid-payload [:data :root :id])))) (is (not (contains? (get-in show-by-uuid-payload [:data :root]) :block/uuid))) (is (= "ok" (:status stop-payload))) (done)) @@ -1060,7 +1131,7 @@ ids-edn (str "[" block-one-id " " block-two-id "]") show-text-result (run-cli ["--repo" "show-multi-id-graph" "show" "--id" ids-edn - "--format" "text" + "--output" "human"] data-dir cfg-path) output (:output show-text-result) @@ -1069,7 +1140,7 @@ idx-delim (string/index-of output "================================================================") show-json-result (run-cli ["--repo" "show-multi-id-graph" "show" "--id" ids-edn - "--format" "json"] + ] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) @@ -1118,16 +1189,15 @@ data-dir cfg-path) parent-payload (parse-json-output parent-query) parent-id (get-in parent-payload [:data :result]) - parent-uuid (query-block-uuid-by-id data-dir cfg-path "show-multi-id-contained-graph" parent-id) _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" - "--target-uuid" (str parent-uuid) + "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) _ (p/delay 100) show-children (run-cli ["--repo" "show-multi-id-contained-graph" "show" - "--page-name" "ParentPage" - "--format" "json"] + "--page" "ParentPage" + ] data-dir cfg-path) show-children-payload (parse-json-output show-children) child-node (find-block-by-title (get-in show-children-payload [:data :root]) "Child Block") @@ -1135,7 +1205,7 @@ ids-edn (str "[" parent-id " " child-id "]") show-json-result (run-cli ["--repo" "show-multi-id-contained-graph" "show" "--id" ids-edn - "--format" "json"] + ] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) @@ -1145,7 +1215,6 @@ stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code parent-query))) (is (some? parent-id)) - (is (some? parent-uuid)) (is (some? child-id)) (is (= 0 (:exit-code show-json-result))) (is (= "ok" (:status show-json-payload))) @@ -1313,7 +1382,7 @@ blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--blocks" blocks-edn] data-dir cfg-path) - show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page-name" "TargetPage" "--format" "json"] + show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage" ] data-dir cfg-path) show-payload (parse-json-output show-result) linked (get-in show-payload [:data :linked-references]) @@ -1326,7 +1395,7 @@ (is (pos? (:count linked))) (is (seq (:blocks linked))) (is (some? ref-block)) - (is (some? (:db/id ref-block))) + (is (some? (or (:db/id ref-block) (:id ref-block)))) (is (some? (or (get-in ref-block [:page :title]) (get-in ref-block [:page :name])))) (is (= "ok" (:status stop-payload))) From 451329a4119edf9b4b9e0f9ba1667615b1d85023 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 1 Feb 2026 23:47:18 +0800 Subject: [PATCH 060/375] fix: graph export/import output string --- src/main/logseq/cli/commands.cljs | 3 ++- src/test/logseq/cli/commands_test.cljs | 8 ++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 8191e2cf52..9f76fe29d0 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -422,4 +422,5 @@ :command (or (:command action) (:type action)) :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks :source :target :update-tags :update-properties - :remove-tags :remove-properties]))))) + :remove-tags :remove-properties + :export-type :output :import-type :input]))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 53a44ac0e3..d4543c1a4e 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1329,6 +1329,10 @@ {})] (is (= :ok (:status edn-result))) (is (= :ok (:status sqlite-result))) + (is (= "edn" (get-in edn-result [:context :export-type]))) + (is (= "/tmp/export.edn" (get-in edn-result [:context :output]))) + (is (= "sqlite" (get-in sqlite-result [:context :export-type]))) + (is (= "/tmp/export.sqlite" (get-in sqlite-result [:context :output]))) (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] [:thread-api/export-db-base64 true ["logseq_db_demo"]]] @invoke-calls)) @@ -1393,6 +1397,10 @@ {})] (is (= :ok (:status edn-result))) (is (= :ok (:status sqlite-result))) + (is (= "edn" (get-in edn-result [:context :import-type]))) + (is (= "/tmp/import.edn" (get-in edn-result [:context :input]))) + (is (= "sqlite" (get-in sqlite-result [:context :import-type]))) + (is (= "/tmp/import.sqlite" (get-in sqlite-result [:context :input]))) (is (= [[:edn "/tmp/import.edn"] [:sqlite "/tmp/import.sqlite"]] @read-calls)) From 8746526e97ebeec372fa786b79b5b6f318447077 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 2 Feb 2026 21:45:05 +0800 Subject: [PATCH 061/375] 028-logseq-cli-verbose-debug.md --- .../028-logseq-cli-verbose-debug.md | 74 +++++++++++++++++++ docs/cli/logseq-cli.md | 4 + src/main/logseq/cli/command/core.cljs | 4 +- src/main/logseq/cli/log.cljs | 53 +++++++++++++ src/main/logseq/cli/main.cljs | 14 ++++ src/main/logseq/cli/server.cljs | 21 +++++- src/main/logseq/cli/transport.cljs | 26 ++++++- src/test/logseq/cli/integration_test.cljs | 33 +++++++++ src/test/logseq/cli/log_test.cljs | 48 ++++++++++++ 9 files changed, 271 insertions(+), 6 deletions(-) create mode 100644 docs/agent-guide/028-logseq-cli-verbose-debug.md create mode 100644 src/main/logseq/cli/log.cljs create mode 100644 src/test/logseq/cli/log_test.cljs diff --git a/docs/agent-guide/028-logseq-cli-verbose-debug.md b/docs/agent-guide/028-logseq-cli-verbose-debug.md new file mode 100644 index 0000000000..ff4b1b867f --- /dev/null +++ b/docs/agent-guide/028-logseq-cli-verbose-debug.md @@ -0,0 +1,74 @@ +# Logseq CLI Verbose Debug Logging Implementation Plan + +Goal: Add a global --verbose flag that enables structured debug logging for CLI options and all db-worker-node API calls without polluting normal command output. + +Architecture: The CLI will install a stderr log handler and enable debug level logging when --verbose is set. +Architecture: The CLI transport and db-worker-node lifecycle utilities will emit structured debug logs for each request and response, using a shared truncation helper to cap large payloads. +Architecture: The db-worker-node process log level remains unchanged, and --verbose only controls logseq-cli logging. + +Tech Stack: ClojureScript, lambdaisland.glogi, Node.js, db-worker-node HTTP API. + +Related: Relates to docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md. + +## Problem statement + +The CLI currently has no verbose debug mode for troubleshooting, so engineers cannot easily see which options were parsed or what db-worker-node API calls were made. +This makes it hard to diagnose failures, especially when db-worker-node responses are large and need safe, truncated logging. +The goal is to add a global --verbose option that emits debug logs for command options and db-worker-node API calls without breaking existing output contracts. + +## Testing Plan + +I will add a unit test that verifies the log truncation helper returns a preview with a stable maximum length and a flag indicating truncation. +I will add a unit test that verifies the log truncation helper handles strings, collections, and nil without throwing. +I will add a unit test that verifies the CLI debug logger emits records only when verbose is enabled by capturing glogi records via a temporary handler. +I will add an integration test that runs the CLI with --verbose and asserts that stderr contains a db-worker-node invoke debug line while stdout remains valid JSON for a simple command. +NOTE: I will write all tests before I add any implementation behavior. + +## Logging coverage + +| Area | File path | Log events | Notes | +| --- | --- | --- | --- | +| CLI option intake | src/main/logseq/cli/main.cljs | Parsed options and resolved config values | Use truncation for long option values like content or blocks. | +| db-worker-node invoke | src/main/logseq/cli/transport.cljs | Request and response debug logs with timing | Truncate response preview and include size metadata. | +| db-worker-node health checks | src/main/logseq/cli/server.cljs | /healthz, /readyz, /v1/shutdown debug logs | Keep quiet unless verbose is true. | +| db-worker-node server logs | src/main/frontend/worker/db_worker_node.cljs | Keep existing logging behavior unchanged. | --verbose does not affect db-worker-node. | + +## Implementation Plan + +1. Read @prompts/review.md and note any CLI and db-worker-node review checklist items that affect logging or output stability. +2. Add :verbose to the CLI global option spec in src/main/logseq/cli/command/core.cljs with a description and boolean coercion so it appears in help output. +3. Add a new logging helper namespace in src/main/logseq/cli/log.cljs that sets a stderr handler, toggles log levels, and exposes a truncate-preview helper that caps output length and records the original size. +4. Add unit tests for the new logging helper in src/test/logseq/cli/log_test.cljs to cover truncation behavior and the verbose gating behavior using a temporary glogi handler. +5. Initialize CLI logging in src/main/logseq/cli/main.cljs after resolving config so --verbose turns on debug level and installs the handler once per run. +6. Emit a debug log in src/main/logseq/cli/main.cljs that includes the command, args, and full options map from the parsed command, using the truncation helper for large values. +7. Add request and response debug logs in src/main/logseq/cli/transport.cljs around invoke, including method, directPass, args preview, response preview, response size, and elapsed time. +8. Add request and response debug logs in src/main/logseq/cli/server.cljs for db-worker-node HTTP calls, and ensure logs are emitted only when --verbose is set. +9. Confirm no changes are made to src/main/frontend/worker/db_worker_node.cljs behavior or log levels for this feature. +10. Add an integration test in src/test/logseq/cli/integration_test.cljs to ensure stdout remains valid JSON when verbose logs are emitted to stderr. +11. Update docs/cli/logseq-cli.md to document the new --verbose flag, that it only affects logseq-cli, and that debug logs go to stderr with large payloads truncated. + +## Edge Cases + +Large query or export results can exceed the preview limit, so logs must include a length field and a truncated preview instead of full payloads. +CLI commands that output JSON or EDN must keep stdout clean, so debug logs must go to stderr only. +Options that contain large text content or EDN blocks must be truncated in logs to avoid massive log lines. +db-worker-node can be started independently, so its debug logs should still be gated by its log-level flag even when the CLI is not involved. + +## Testing Details + +Tests will validate that debug logging is gated by --verbose, that truncation is applied consistently, and that stdout output remains parseable while stderr contains debug logs for a simple db-worker-node invocation. + +## Implementation Details + +- Use lambdaisland.glogi for CLI logging so log-level control is consistent with db-worker-node. +- Install a stderr log handler in the CLI to avoid polluting stdout output formats. +- Truncate previews by character count and include metadata such as original length and truncation flag. +- Log db-worker-node invoke timings so slow calls are visible in verbose mode. +- Keep db-worker-node API behavior and log levels unchanged. +- Ensure all verbose logs are emitted by logseq-cli only. + +## Question + +None. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 379aa9a3a7..a6bc11dc73 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -45,6 +45,10 @@ Supported keys include: CLI flags take precedence over environment variables, which take precedence over the config file. +Verbose logging: +- `--verbose` enables structured debug logs to stderr for CLI option parsing and db-worker-node API calls. +- stdout remains reserved for command output; large payloads are truncated in debug previews. + ## Commands Graph commands: diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 0ec5a91039..279ad1f097 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -16,7 +16,9 @@ :data-dir {:desc "Path to db-worker data dir (default ~/logseq/cli-graphs)"} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} - :output {:desc "Output format (human, json, edn). Default: human"}}) + :output {:desc "Output format (human, json, edn). Default: human"} + :verbose {:desc "Enable verbose debug logging to stderr" + :coerce :boolean}}) (defn global-spec [] diff --git a/src/main/logseq/cli/log.cljs b/src/main/logseq/cli/log.cljs new file mode 100644 index 0000000000..b14fba8357 --- /dev/null +++ b/src/main/logseq/cli/log.cljs @@ -0,0 +1,53 @@ +(ns logseq.cli.log + "CLI logging helpers for verbose debug output." + (:require [lambdaisland.glogi :as log])) + +(def ^:private default-preview-limit 400) + +(defonce ^:private handler-installed? (atom false)) + +(defn truncate-preview + "Returns a preview map with `:preview`, `:length`, and `:truncated?` for `value`. + + Example: + + ```clojure + (truncate-preview {:a 1} 10) + ```" + ([value] + (truncate-preview value default-preview-limit)) + ([value max-len] + (let [text (if (string? value) value (pr-str value)) + length (count text) + limit (max 0 (or max-len 0)) + truncated? (> length limit) + preview (if truncated? + (subs text 0 limit) + text)] + {:preview preview + :length length + :truncated? truncated?}))) + +(defn- format-record + [{:keys [level logger-name message exception time]}] + (cond-> {:time time + :level level + :logger logger-name + :message message} + exception (assoc :exception (str exception)))) + +(defn- stderr-handler + [record] + (.write (.-stderr js/process) + (str (pr-str (format-record record)) "\n"))) + +(defn install-stderr-handler! + [] + (when-not @handler-installed? + (log/add-handler stderr-handler) + (reset! handler-installed? true))) + +(defn set-verbose! + [verbose?] + (install-stderr-handler!) + (log/set-levels {:glogi/root (if verbose? :debug :info)})) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index c6a998935a..36c2feb87a 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -6,7 +6,9 @@ [logseq.cli.config :as config] [logseq.cli.data-dir :as data-dir] [logseq.cli.format :as format] + [logseq.cli.log :as cli-log] [logseq.cli.version :as version] + [lambdaisland.glogi :as log] [promesa.core :as p])) (defn- usage @@ -39,6 +41,18 @@ :else (let [cfg (config/resolve-config (:options parsed))] + (cli-log/set-verbose! (:verbose cfg)) + (log/debug :event :cli/parsed-options + :command (:command parsed) + :args (cli-log/truncate-preview (:args parsed)) + :options (into {} + (map (fn [[k v]] + [k (cli-log/truncate-preview v)]) + (:options parsed))) + :config (into {} + (map (fn [[k v]] + [k (cli-log/truncate-preview v)]) + (dissoc cfg :auth-token)))) (try (let [cfg (assoc cfg :data-dir (data-dir/ensure-data-dir! (:data-dir cfg))) action-result (commands/build-action parsed cfg)] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index e20e81b0bc..7d17521220 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -7,6 +7,7 @@ ["path" :as node-path] [clojure.string :as string] [logseq.cli.command.core :as command-core] + [logseq.cli.log :as cli-log] [frontend.worker-common.util :as worker-util] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -83,6 +84,7 @@ (p/create (fn [resolve reject] (let [timeout-ms (or timeout-ms 5000) + start-ms (js/Date.now) req (.request http #js {:method method @@ -94,15 +96,28 @@ (let [chunks (array)] (.on res "data" (fn [chunk] (.push chunks chunk))) (.on res "end" (fn [] - (let [buf (js/Buffer.concat chunks)] - (resolve {:status (.-statusCode res) - :body (.toString buf "utf8")})))) + (let [buf (js/Buffer.concat chunks) + response {:status (.-statusCode res) + :body (.toString buf "utf8")}] + (log/debug :event :cli.server/http-response + :method method + :path path + :status (:status response) + :elapsed-ms (- (js/Date.now) start-ms) + :body (cli-log/truncate-preview (:body response))) + (resolve response)))) (.on res "error" reject)))) timeout-id (js/setTimeout (fn [] (.destroy req) (reject (ex-info "request timeout" {:code :timeout}))) timeout-ms)] + (log/debug :event :cli.server/http-request + :method method + :host host + :port port + :path path + :body (cli-log/truncate-preview body)) (.on req "error" (fn [err] (js/clearTimeout timeout-id) (reject err))) diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index 519b4937e0..1972e63387 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -3,6 +3,8 @@ (:require [cljs.reader :as reader] [clojure.string :as string] [logseq.db :as ldb] + [logseq.cli.log :as cli-log] + [lambdaisland.glogi :as log] [promesa.core :as p] ["fs" :as fs] ["http" :as http] @@ -79,6 +81,8 @@ (string? method) method (nil? method) nil :else (str method)) + start-ms (js/Date.now) + args-preview (cli-log/truncate-preview args) payload (if direct-pass? {:method method* :directPass true @@ -87,6 +91,11 @@ :directPass false :argsTransit (ldb/write-transit-str args)}) body (js/JSON.stringify (clj->js payload))] + (log/debug :event :cli.transport/invoke + :method method* + :direct-pass? direct-pass? + :args args-preview + :url url) (p/let [{:keys [body]} (request {:method "POST" :url url :headers (base-headers) @@ -94,8 +103,21 @@ :timeout-ms timeout-ms}) {:keys [result resultTransit]} (js->clj (js/JSON.parse body) :keywordize-keys true)] (if direct-pass? - result - (ldb/read-transit-str resultTransit))))) + (let [response-preview (cli-log/truncate-preview result)] + (log/debug :event :cli.transport/response + :method method* + :direct-pass? direct-pass? + :elapsed-ms (- (js/Date.now) start-ms) + :response response-preview) + result) + (let [decoded (ldb/read-transit-str resultTransit) + response-preview (cli-log/truncate-preview decoded)] + (log/debug :event :cli.transport/response + :method method* + :direct-pass? direct-pass? + :elapsed-ms (- (js/Date.now) start-ms) + :response response-preview) + decoded))))) (defn write-output [{:keys [format path data]}] diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 6131d0e96d..1d8537856b 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -81,6 +81,18 @@ :stderr stderr} e)))))) +(defn- capture-stderr! + [] + (let [stderr (.-stderr js/process) + original-write (.-write stderr) + buffer (atom "")] + (set! (.-write stderr) + (fn [chunk] + (swap! buffer str chunk) + true)) + {:buffer buffer + :restore! (fn [] (set! (.-write stderr) original-write))})) + (defn- node-title [node] (or (:block/title node) (:block/content node) (:title node) (:content node))) @@ -531,6 +543,27 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-cli-verbose-logs-to-stderr + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-verbose") + repo "verbose-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + {:keys [buffer restore!]} (capture-stderr!) + result (-> (run-cli ["--verbose" "--repo" repo "graph" "info"] data-dir cfg-path) + (p/finally (fn [] (restore!)))) + payload (parse-json-output-safe result "verbose graph info") + stderr-text @buffer + _ (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path)] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (string/includes? stderr-text ":cli.transport/invoke")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-cli-update-tags-and-properties (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-update-tags")] diff --git a/src/test/logseq/cli/log_test.cljs b/src/test/logseq/cli/log_test.cljs new file mode 100644 index 0000000000..edfa0d7161 --- /dev/null +++ b/src/test/logseq/cli/log_test.cljs @@ -0,0 +1,48 @@ +(ns logseq.cli.log-test + (:require [cljs.test :refer [deftest is testing]] + [lambdaisland.glogi :as log] + [logseq.cli.log :as cli-log])) + +(deftest test-truncate-preview + (testing "truncates long strings with length metadata" + (let [value (apply str (repeat 50 "a")) + result (cli-log/truncate-preview value 10)] + (is (= 50 (:length result))) + (is (= 10 (count (:preview result)))) + (is (true? (:truncated? result))))) + + (testing "does not truncate short strings" + (let [value "short" + result (cli-log/truncate-preview value 10)] + (is (= 5 (:length result))) + (is (= "short" (:preview result))) + (is (false? (:truncated? result))))) + + (testing "handles collections" + (let [value [1 2 3] + result (cli-log/truncate-preview value 100)] + (is (= (count "[1 2 3]") (:length result))) + (is (= "[1 2 3]" (:preview result))) + (is (false? (:truncated? result))))) + + (testing "handles nil" + (let [result (cli-log/truncate-preview nil 10)] + (is (= 3 (:length result))) + (is (= "nil" (:preview result))) + (is (false? (:truncated? result)))))) + +(deftest test-debug-logging-gated-by-verbose + (let [records (atom []) + handler (fn [record] + (swap! records conj record))] + (log/add-handler handler) + (cli-log/set-verbose! false) + (log/debug :event :cli/verbose-test) + (is (empty? @records)) + + (cli-log/set-verbose! true) + (log/debug :event :cli/verbose-test) + (is (= 1 (count @records))) + + (log/remove-handler handler) + (cli-log/set-verbose! false))) From 15fb705315c70fd11474a2f7181c46ec19cc5202 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 4 Feb 2026 23:02:23 +0800 Subject: [PATCH 062/375] 029-logseq-cli-show-properties.md (1) --- .../029-logseq-cli-show-properties.md | 150 +++++++ src/main/logseq/cli/command/show.cljs | 411 ++++++++++++++++-- src/test/logseq/cli/commands_test.cljs | 172 ++++++-- src/test/logseq/cli/integration_test.cljs | 177 +++++--- 4 files changed, 764 insertions(+), 146 deletions(-) create mode 100644 docs/agent-guide/029-logseq-cli-show-properties.md diff --git a/docs/agent-guide/029-logseq-cli-show-properties.md b/docs/agent-guide/029-logseq-cli-show-properties.md new file mode 100644 index 0000000000..fee92b0d46 --- /dev/null +++ b/docs/agent-guide/029-logseq-cli-show-properties.md @@ -0,0 +1,150 @@ +# Logseq CLI Show Properties Implementation Plan + +Goal: Display block properties in the human-readable output of the logseq-cli show command. + +Architecture: Extend the show command pull selectors to include property data from db-worker-node, then enrich tree->text to append formatted property lines per block. +Architecture: Keep JSON and EDN outputs structurally the same, while only altering the human text renderer to include properties beneath each block label. + +Tech Stack: ClojureScript, Datascript pull selectors, logseq-cli, db-worker-node thread-api. + +Related: Builds on 028-logseq-cli-verbose-debug.md. + +## Problem statement + +The logseq-cli show command currently renders block trees without any property visibility. +Users need to see each block's properties directly under the block content in the human output so that show reflects the same metadata they rely on in the UI. +Property lines belong to the same block tree element, so they must not render with tree glyphs. +The display must show ": " and must handle single-value properties as a single line and multi-value properties as an indented list as shown in the example. + +## Testing Plan + +I will add a unit test that asserts tree->text renders single-value properties in one line after the block content. +I will add a unit test that asserts tree->text renders multi-value properties as a dash list aligned under the property name. +I will add a unit test that asserts user property order follows a stable key order. +I will add a unit test that asserts properties do not break multiline block alignment for both root and child blocks. +I will add a unit test that asserts property values only appear in property-kvs and never in block/children. +I will update selector coverage tests to assert property selectors are included in show pulls. +I will add an integration test that asserts show output includes property lines. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Context and integration points + +The show command data path is: CLI action -> transport/invoke -> db-worker-node -> pull/query -> tree data -> tree->text human rendering. +The pull selectors in logseq.cli.command.show define what properties are available to tree->text. +User properties are stored directly on block entities with keys in the :user.property/* namespace. + +ASCII diagram of the flow: + +CLI show + | + | transport/invoke :thread-api/pull / :thread-api/q + v +DB worker node + | + | returns tree data with block maps + v +logseq.cli.command.show/tree->text + | + | renders block label + property lines + v +stdout + +## Files to touch + +| File path | Purpose | +| --- | --- | +| src/main/logseq/cli/command/show.cljs | Add property selectors, build property lines, and render properties in tree->text. | +| src/test/logseq/cli/commands_test.cljs | Add tree->text unit tests for property rendering and selector coverage. | +| src/test/logseq/cli/integration_test.cljs | Add integration coverage for show property rendering output. | + +## Implementation plan (TDD-ordered) + +1. Read @prompts/review.md to align with review expectations. +2. Add failing unit tests for tree->text: + - single-value property rendering + - multi-value property rendering with dash list formatting + alignment + - user property ordering + - multiline alignment with properties +3. Add failing selector coverage test asserting show pull patterns include property selectors. +4. Add failing integration test asserting show output includes property lines. +5. Run each new test and confirm failures reference missing property behavior (not test errors). +6. Update show pull selectors in src/main/logseq/cli/command/show.cljs to include :user.property/* attributes so db-worker-node returns needed data. +7. Add helpers in src/main/logseq/cli/command/show.cljs to: + - collect and sort user property keys + - normalize user property values into a vector of display strings + - format property lines for single vs multi values +8. Extend tree->text in src/main/logseq/cli/command/show.cljs to append property lines after block label lines, preserving indentation rules and no tree glyphs. +9. Re-run unit tests and selector test; confirm they pass. +10. Re-run integration test; confirm it passes. +11. Run `bb dev:lint-and-test` and confirm all linters and tests pass. + +## Edge cases to handle + +User properties with empty values should be skipped to avoid blank lines in the output. +User properties with values that are sets, vectors, or lists should render each item as a separate dash line. +User properties with values that are entities or maps should render as :block/title, :block/name, or :logseq.property/value when present. +User properties must be detected only via the :user.property/* namespace and no other property sources should be displayed. +Blocks without properties should render exactly as they do today. +Linked references output should include properties for each block without breaking the existing tree formatting. +Property lines should not include tree glyphs or branch markers. + +## Implementation details for formatting + +Use a stable sort order for :user.property/* keys, such as ascending by keyword name. +Do not use :block/properties-text-values for db-graph output. +Format user property values from :user.property/* attributes and coerce values into strings using :block/title, :block/name, or :logseq.property/value. +Render single values as "Property: value" and multi values as "Property: - v1" with subsequent lines aligned under the dash list. +Align property lines with the block content column and preserve tree glyph indentation, but do not render additional tree glyphs for properties. + +## Questions + +Properties should display using their :block/title for the property name, derived from the property entity when available, and never fall back to :db/ident. +Hidden or internal properties must be filtered out to avoid noise in CLI output. +Tags must render as #tags only and must not be listed as properties. +:block/properties-text-values is file-graph only and should be ignored for db-graph output. + +## Testing Details + +The new tests will directly exercise tree->text with synthetic tree data that includes :user.property/* attributes to ensure the human output matches the requested format. +The new tests will assert property values are only present in property-kvs and do not appear in block/children output. +The selector test will validate that the show pull patterns include property attributes so db-worker-node can provide the necessary data. +These tests verify output behavior, ordering, and indentation rather than internal helper logic. +The integration test will validate that show output includes property lines end-to-end. + +## Implementation Details + +- Update tree-block-selector and linked-ref-selector to pull :user.property/* attributes. +- Update the id and uuid fetch pull patterns to include the same property fields. +- Add a property normalization helper that returns ordered [key values] pairs. +- Add a value formatting helper that converts each property value into displayable strings. +- Extend tree->text to append property lines after block content lines for both root and child nodes. +- Keep JSON and EDN outputs intact aside from the additional property keys pulled from the db. +- Property values should only appear in property-kvs; do not surface property values in block/children. +- Add new unit tests for single-value, multi-value, ordering, and multiline alignment cases. +- Add a selector test that asserts property selectors are included. + +## Example output + +Current: +``` +5137 Done Add git sha when graph is created for improved debugging #Issue +5138 ├── Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 +5139 ├── When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha +5140 └── When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha +``` + +Target: +``` +5137 Done Add git sha when graph is created for improved debugging #Issue + Background: Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 + Acceptance Criteria: + - When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha + - When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha +``` + +## Question + +JSON and EDN outputs must include property values alongside the existing data. + +--- diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index e21893410b..085dad98dd 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -88,6 +88,7 @@ :block/uuid :block/title :block/content + :logseq.property/created-from-property {:logseq.property/status [:db/ident :block/name :block/title]} :block/order {:block/parent [:db/id]} @@ -99,6 +100,7 @@ :block/uuid :block/title :block/content + :logseq.property/created-from-property {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]} {:block/page [:db/id :block/name :block/title :block/uuid]} @@ -108,7 +110,10 @@ :block/uuid {:block/page [:db/id :block/name :block/title :block/uuid]}]}]) -(declare tree->text) +(declare tree->text + property-value-block? + attach-user-properties + attach-user-properties-to-entity) (def ^:private uuid-ref-pattern #"\[\[([0-9a-fA-F-]{36})\]\]") (def ^:private uuid-ref-max-depth 10) @@ -148,6 +153,102 @@ (when (seq labels) (string/join " " (map #(style/bold (str "#" %)) labels))))) +(def ^:private user-property-namespace "user.property") + +(defn- user-property-key? + [k] + (and (qualified-keyword? k) + (= user-property-namespace (namespace k)))) + +(defn- nonblank-string + [value] + (when (and (string? value) (not (string/blank? value))) + value)) + +(defn- lookup-ref? + [value] + (and (vector? value) + (= 2 (count value)) + (= :block/uuid (first value)) + (uuid? (second value)))) + +(defn- property-value->string + ([value] (property-value->string value nil)) + ([value labels] + (cond + (string? value) (nonblank-string value) + (number? value) (or (get labels value) (str value)) + (uuid? value) (or (get labels value) (str value)) + (lookup-ref? value) (let [uuid (second value)] + (or (get labels uuid) (str uuid))) + (boolean? value) (str value) + (keyword? value) (str value) + (map? value) (or (nonblank-string (:block/title value)) + (nonblank-string (:block/name value)) + (when-let [id (:db/id value)] + (get labels id)) + (when-let [uuid (:block/uuid value)] + (get labels uuid)) + (when-let [val (:logseq.property/value value)] + (if (string? val) + (nonblank-string val) + (str val)))) + (some? value) (str value) + :else nil))) + +(defn- normalize-property-values + ([value] (normalize-property-values value nil)) + ([value labels] + (let [values (cond + (set? value) (seq value) + (sequential? value) value + (nil? value) nil + :else [value]) + rendered (->> values + (map #(property-value->string % labels)) + (remove string/blank?) + vec)] + (if (set? value) + (vec (sort rendered)) + rendered)))) + +(defn- node-user-property-entries + ([node] (node-user-property-entries node nil)) + ([node labels] + (->> node + (filter (fn [[k _]] (user-property-key? k))) + (map (fn [[k v]] [k (normalize-property-values v labels)])) + (remove (fn [[_ values]] (empty? values))) + vec))) + +(defn- sort-property-entries + [property-entries] + (sort-by (comp name first) property-entries)) + +(defn- property-title-for + [property-titles property-key] + (let [title (get property-titles property-key)] + (nonblank-string title))) + +(defn- format-property-lines + [indent title values] + (when (seq values) + (if (= 1 (count values)) + [(str indent title ": " (first values))] + (let [item-indent (str indent " ")] + (into [(str indent title ":")] + (map #(str item-indent "- " %) values)))))) + +(defn- node-property-lines + [node property-titles property-value-labels indent] + (let [property-entries (->> (node-user-property-entries node property-value-labels) + sort-property-entries)] + (->> property-entries + (mapcat (fn [[property-key values]] + (when-let [title (property-title-for property-titles property-key)] + (format-property-lines indent title values)))) + vec))) + (defn- status-from-ident [ident] (let [name* (name ident) @@ -246,7 +347,10 @@ (transport/invoke config :thread-api/pull false [repo linked-ref-selector id])) ref-ids)) [])] - (let [blocks (vec (remove nil? pulled)) + (let [blocks (vec (remove (fn [block] + (or (nil? block) + (property-value-block? block))) + pulled)) page-lookup-key (fn [value] (cond (map? value) (or (:db/id value) @@ -267,7 +371,8 @@ (keep page-id-from) distinct vec)] - (p/let [pages (if (seq page-ids) + (p/let [blocks (attach-user-properties config repo blocks) + pages (if (seq page-ids) (p/all (map (fn [id] (transport/invoke config :thread-api/pull false [repo [:db/id :block/name :block/title :block/uuid] id])) @@ -295,7 +400,7 @@ :blocks blocks}))))) (defn- linked-refs->text - [blocks uuid->label] + [blocks uuid->label property-titles property-value-labels] (let [page-key (fn [block] (let [page (:block/page block)] (or (:db/id page) @@ -316,7 +421,10 @@ (map (fn [[_ page-blocks]] (let [root (page-node (first page-blocks)) root (assoc root :block/children (vec page-blocks))] - (tree->text {:root root :uuid->label uuid->label}))) + (tree->text {:root root + :uuid->label uuid->label + :property-titles property-titles + :property-value-labels property-value-labels}))) groups)))) (defn- extract-uuid-refs @@ -360,13 +468,201 @@ (into {}))) (p/resolved {}))) +(defn- collect-user-property-keys + [{:keys [root linked-references]}] + (letfn [(collect-node [node] + (let [node-keys (->> (keys node) + (filter user-property-key?))] + (reduce (fn [acc child] + (into acc (collect-node child))) + (set node-keys) + (or (:block/children node) []))))] + (let [root-keys (if root (collect-node root) #{}) + linked-keys (reduce (fn [acc block] + (into acc (collect-node block))) + #{} + (or (:blocks linked-references) []))] + (into root-keys linked-keys)))) + +(defn- property-value-label + [entity] + (when (map? entity) + (or (nonblank-string (:block/title entity)) + (nonblank-string (:block/name entity)) + (when-let [val (:logseq.property/value entity)] + (if (string? val) + (nonblank-string val) + (str val)))))) + +(defn- collect-property-value-refs + [{:keys [root linked-references]}] + (letfn [(collect-value [acc value] + (cond + (lookup-ref? value) + (update acc :uuids conj (second value)) + + (uuid? value) + (update acc :uuids conj value) + + (number? value) + (update acc :ids conj value) + + (map? value) + (let [resolved? (or (nonblank-string (:block/title value)) + (nonblank-string (:block/name value)) + (some? (:logseq.property/value value)))] + (if resolved? + acc + (cond-> acc + (:block/uuid value) (update :uuids conj (:block/uuid value)) + (:db/id value) (update :ids conj (:db/id value))))) + + (set? value) + (reduce collect-value acc value) + + (sequential? value) + (reduce collect-value acc value) + + :else acc)) + (collect-node [acc node] + (let [acc (reduce (fn [acc [k v]] + (if (user-property-key? k) + (collect-value acc v) + acc)) + acc + node)] + (reduce collect-node acc (or (:block/children node) []))))] + (let [init {:ids #{} :uuids #{}} + acc-root (if root (collect-node init root) init)] + (reduce collect-node acc-root (or (:blocks linked-references) []))))) + +(defn- property-visible? + [entity] + (and (map? entity) + (not (true? (:logseq.property/hide? entity))) + (not (false? (:logseq.property/public? entity))))) + +(defn- property-entity-title + [entity] + (or (nonblank-string (:block/title entity)) + (nonblank-string (:block/name entity)))) + +(defn- property-value-block? + [block] + (some? (:logseq.property/created-from-property block))) + +(defn- fetch-property-titles + [config repo property-keys] + (if (seq property-keys) + (let [keys (vec property-keys) + selector [:db/id :db/ident :block/title :block/name + :logseq.property/hide? :logseq.property/public?]] + (p/let [entities (p/all (map (fn [property-key] + (transport/invoke config :thread-api/pull false + [repo selector [:db/ident property-key]])) + keys))] + (->> (map vector keys entities) + (keep (fn [[property-key entity]] + (when (property-visible? entity) + (when-let [title (property-entity-title entity)] + [property-key title])))) + (into {})))) + (p/resolved {}))) + +(defn- fetch-property-value-labels + [config repo {:keys [ids uuids]}] + (if (or (seq ids) (seq uuids)) + (let [selector [:db/id :block/uuid :block/title :block/name :logseq.property/value] + ids* (vec ids) + uuids* (vec uuids)] + (p/let [id-entities (if (seq ids*) + (p/all (map (fn [id] + (transport/invoke config :thread-api/pull false + [repo selector id])) + ids*)) + []) + uuid-entities (if (seq uuids*) + (p/all (map (fn [uuid] + (transport/invoke config :thread-api/pull false + [repo selector [:block/uuid uuid]])) + uuids*)) + [])] + (->> (concat id-entities uuid-entities) + (remove nil?) + (reduce (fn [acc entity] + (if-let [label (property-value-label entity)] + (cond-> acc + (:db/id entity) (assoc (:db/id entity) label) + (:block/uuid entity) (assoc (:block/uuid entity) label)) + acc)) + {})))) + (p/resolved {}))) + +(defn- fetch-user-properties + [config repo block-ids] + (if (seq block-ids) + (let [idents-query '[:find [?a ...] + :where + [?e :db/ident ?a] + [(namespace ?a) ?ns] + [(= "user.property" ?ns)]] + props-query '[:find ?b ?a ?v + :in $ [?b ...] [?a ...] + :where + [?b ?a ?v]] + ids (vec block-ids)] + (p/let [property-idents (transport/invoke config :thread-api/q false [repo [idents-query]])] + (if (seq property-idents) + (p/let [rows (transport/invoke config :thread-api/q false + [repo [props-query ids (vec property-idents)]])] + (reduce (fn [acc [block-id attr value]] + (update acc block-id assoc attr value)) + {} + rows)) + {}))) + (p/resolved {}))) + +(defn- attach-user-properties + [config repo blocks] + (let [block-ids (vec (keep :db/id blocks))] + (p/let [id->props (fetch-user-properties config repo block-ids)] + (mapv (fn [block] + (if-let [props (get id->props (:db/id block))] + (merge block props) + block)) + blocks)))) + +(defn- attach-user-properties-to-entity + [config repo entity] + (if-let [block-id (:db/id entity)] + (p/let [id->props (fetch-user-properties config repo [block-id])] + (if-let [props (get id->props block-id)] + (merge entity props) + entity)) + (p/resolved entity))) + +(defn- attach-property-titles + [config repo tree-data] + (let [property-keys (collect-user-property-keys tree-data) + value-refs (collect-property-value-refs tree-data)] + (p/let [titles (fetch-property-titles config repo property-keys) + value-labels (fetch-property-value-labels config repo value-refs)] + (assoc tree-data + :property-titles titles + :property-value-labels value-labels)))) + (defn- fetch-blocks-for-page [config repo page-id] (let [query [:find (list 'pull '?b tree-block-selector) :in '$ '?page-id :where ['?b :block/page '?page-id]]] - (p/let [rows (transport/invoke config :thread-api/q false [repo [query page-id]])] - (mapv first rows)))) + (p/let [rows (transport/invoke config :thread-api/q false [repo [query page-id]]) + blocks (->> rows + (map first) + (remove property-value-block?) + vec) + blocks (attach-user-properties config repo blocks)] + blocks))) (defn- build-tree [blocks root-id max-depth] @@ -394,6 +690,7 @@ {:logseq.property/status [:db/ident :block/name :block/title]} {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] id])] + (p/let [entity (attach-user-properties-to-entity config repo entity)] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks (:db/id entity) max-depth)] @@ -403,6 +700,7 @@ children (build-tree blocks (:db/id entity) max-depth)] {:root (assoc entity :block/children children)}) (throw (ex-info "block not found" {:code :block-not-found}))))) + ) (seq uuid-str) (if-not (common-util/uuid-string? uuid-str) @@ -421,6 +719,7 @@ {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid uuid-str]]))] + (p/let [entity (attach-user-properties-to-entity config repo entity)] (if-let [page-id (get-in entity [:block/page :db/id])] (p/let [blocks (fetch-blocks-for-page config repo page-id) children (build-tree blocks (:db/id entity) max-depth)] @@ -430,6 +729,7 @@ children (build-tree blocks (:db/id entity) max-depth)] {:root (assoc entity :block/children children)}) (throw (ex-info "block not found" {:code :block-not-found})))))) + ) (seq page) (p/let [page-entity (transport/invoke config :thread-api/pull false @@ -437,17 +737,18 @@ {:logseq.property/status [:db/ident :block/name :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/name page]])] - (if-let [page-id (:db/id page-entity)] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks page-id max-depth)] - {:root (assoc page-entity :block/children children)}) - (throw (ex-info "page not found" {:code :page-not-found})))) + (p/let [page-entity (attach-user-properties-to-entity config repo page-entity)] + (if-let [page-id (:db/id page-entity)] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks page-id max-depth)] + {:root (assoc page-entity :block/children children)}) + (throw (ex-info "page not found" {:code :page-not-found}))))) :else (p/rejected (ex-info "block or page required" {:code :missing-target}))))) (defn tree->text - [{:keys [root uuid->label]}] + [{:keys [root uuid->label property-titles property-value-labels]}] (let [label (fn [node] (or (block-label (assoc node :uuid->label uuid->label)) "-")) node-id (fn [node] @@ -468,6 +769,14 @@ style-glyph (fn [value] (style/dim value)) lines (atom []) + property-indent (fn [prefix] + (let [prefix* (string/replace prefix "│" " ")] + (str id-padding (style-glyph prefix*)))) + append-property-lines (fn [node prefix] + (let [indent (property-indent prefix) + prop-lines (node-property-lines node property-titles property-value-labels indent)] + (doseq [line prop-lines] + (swap! lines conj line)))) walk (fn walk [node prefix] (let [children (:block/children node) total (count children)] @@ -485,6 +794,7 @@ (swap! lines conj line) (doseq [row rest-rows] (swap! lines conj (str id-padding (style-glyph next-prefix) row))) + (append-property-lines child next-prefix) (walk child next-prefix)))))] (let [rows (split-lines (label root)) first-row (first rows) @@ -492,11 +802,12 @@ (swap! lines conj (str (style/dim (pad-id root)) " " first-row)) (doseq [row rest-rows] (swap! lines conj (str id-padding row)))) + (append-property-lines root "") (walk root "") (string/join "\n" @lines))) (defn- tree->text-with-linked-refs - [{:keys [linked-references uuid->label] :as tree-data}] + [{:keys [linked-references uuid->label property-titles property-value-labels] :as tree-data}] (let [tree-text (tree->text tree-data) refs (:blocks linked-references) count (:count linked-references)] @@ -504,7 +815,7 @@ (str tree-text "\n\n" "Linked References (" count ")\n" - (linked-refs->text refs uuid->label)) + (linked-refs->text refs uuid->label property-titles property-value-labels)) tree-text))) (defn build-action @@ -597,6 +908,17 @@ entry)) tree-data)) +(defn- render-tree-text + [tree-data action] + (if (false? (:linked-references? action)) + (tree->text tree-data) + (tree->text-with-linked-refs tree-data))) + +(defn- render-tree-text-with-properties + [config action tree-data] + (p/let [tree-data (attach-property-titles config (:repo action) tree-data)] + (render-tree-text tree-data action))) + (defn execute-show [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -630,36 +952,32 @@ results)) sanitize-tree (fn [tree] (strip-block-uuid tree)) - render-tree (if (false? (:linked-references? action)) - tree->text - tree->text-with-linked-refs) - payload (case format - :edn - {:status :ok - :data (mapv (fn [{:keys [ok? tree id error]}] - (if ok? - (sanitize-tree tree) - (multi-id-error-entry id error))) - results) - :output-format :edn} + render-result (fn [{:keys [ok? tree id error]}] + (if ok? + (render-tree-text-with-properties cfg action tree) + (multi-id-error-message id error)))] + (case format + :edn + {:status :ok + :data (mapv (fn [{:keys [ok? tree id error]}] + (if ok? + (sanitize-tree tree) + (multi-id-error-entry id error))) + results) + :output-format :edn} - :json - {:status :ok - :data (mapv (fn [{:keys [ok? tree id error]}] - (if ok? - (sanitize-tree tree) - (multi-id-error-entry id error))) - results) - :output-format :json} + :json + {:status :ok + :data (mapv (fn [{:keys [ok? tree id error]}] + (if ok? + (sanitize-tree tree) + (multi-id-error-entry id error))) + results) + :output-format :json} - {:status :ok - :data {:message (string/join multi-id-delimiter - (map (fn [{:keys [ok? tree id error]}] - (if ok? - (render-tree tree) - (multi-id-error-message id error))) - results))}})] - payload) + (p/let [messages (p/all (map render-result results))] + {:status :ok + :data {:message (string/join multi-id-delimiter messages)}}))) (p/let [tree-data (build-tree-data cfg action)] (case format :edn @@ -674,7 +992,6 @@ :data tree-data :output-format :json}) - {:status :ok - :data {:message (if (false? (:linked-references? action)) - (tree->text tree-data) - (tree->text-with-linked-refs tree-data))}})))))) + (p/let [message (render-tree-text-with-properties cfg action tree-data)] + {:status :ok + :data {:message message}}))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index d4543c1a4e..82d9db411c 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -364,6 +364,85 @@ "175 └── cccc") (strip-ansi output)))))) +(deftest test-tree->text-renders-properties-single-value + (testing "show tree text renders user properties below block labels" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :user.property/background "Because"} + :property-titles {:user.property/background "Background"}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (= (str "1 Root\n" + " Background: Because") + (strip-ansi output))) + (is (not (string/includes? output "└── Background")))))) + +(deftest test-tree->text-renders-properties-multi-value + (testing "show tree text renders multi-value properties as a list" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :user.property/criteria ["One" "Two"]} + :property-titles {:user.property/criteria "Criteria"}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (= (str "1 Root\n" + " Criteria:\n" + " - One\n" + " - Two") + (strip-ansi output)))))) + +(deftest test-tree->text-properties-order + (testing "show tree text renders user properties in stable key order" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :user.property/zeta "Last" + :user.property/alpha "First"} + :property-titles {:user.property/zeta "Zeta" + :user.property/alpha "Alpha"}} + output (strip-ansi (tree->text tree-data)) + alpha-idx (.indexOf output "Alpha:") + zeta-idx (.indexOf output "Zeta:")] + (is (<= 0 alpha-idx)) + (is (<= 0 zeta-idx)) + (is (< alpha-idx zeta-idx))))) + +(deftest test-tree->text-properties-multiline-alignment + (testing "show tree text keeps multiline alignment with properties" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root line1\nRoot line2" + :user.property/background "Root prop" + :block/children [{:db/id 22 + :block/title "Child line1\nChild line2" + :user.property/notes "Child prop"}]} + :property-titles {:user.property/background "Background" + :user.property/notes "Notes"}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (= (str "1 Root line1\n" + " Root line2\n" + " Background: Root prop\n" + "22 └── Child line1\n" + " Child line2\n" + " Notes: Child prop") + (strip-ansi output)))))) + +(deftest test-tree->text-properties-dont-render-as-children + (testing "show tree text does not render property values as children" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :user.property/background "Child"} + :property-titles {:user.property/background "Background"}} + output (binding [style/*color-enabled?* true] + (tree->text tree-data))] + (is (string/includes? output "Background: Child")) + (is (not (string/includes? output "└── Child"))) + (is (not (string/includes? output "├── Child")))))) + (deftest test-tree->text-prefixes-status (testing "show tree text prefixes status before block titles" (let [tree->text #'show-command/tree->text @@ -527,8 +606,11 @@ :block/page {:db/id 2}})) :thread-api/q (let [[_ [query _]] args pull-form (second query) - selector (nth pull-form 2)] - (swap! selectors conj selector) + selector (when (and (seq? pull-form) + (= 'pull (first pull-form))) + (nth pull-form 2))] + (when selector + (swap! selectors conj selector)) (p/resolved [])) :thread-api/get-block-refs (p/resolved [{:db/id 10}]) (p/resolved nil)))) @@ -1211,6 +1293,11 @@ (async done (let [ops* (atom nil) calls* (atom []) + orig-ensure-server! cli-server/ensure-server! + orig-resolve-tags add-command/resolve-tags + orig-resolve-properties add-command/resolve-properties + orig-resolve-property-identifiers add-command/resolve-property-identifiers + orig-invoke transport/invoke action {:type :update-block :repo "demo" :id 1 @@ -1220,44 +1307,49 @@ :remove-tags [:tag/old] :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} :remove-properties [:logseq.property/publishing-public?]}] - (with-redefs [cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) - add-command/resolve-tags (fn [_ _ tags] - (p/resolved (cond - (= tags [:tag/new]) [{:db/id 101}] - (= tags [:tag/old]) [{:db/id 202}] - :else nil))) - add-command/resolve-properties (fn [_ _ properties] (p/resolved properties)) - add-command/resolve-property-identifiers (fn [_ _ properties] (p/resolved properties)) - transport/invoke (fn [_ method _ args] - (swap! calls* conj {:method method :args args}) - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup 1) - {:db/id 1 - :block/name nil - :block/uuid (uuid "00000000-0000-0000-0000-000000000001")} - (= lookup 2) - {:db/id 2 - :block/name nil - :block/uuid (uuid "00000000-0000-0000-0000-000000000002")} - :else {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))] - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] - [:batch-delete-property-value [[1] :block/tags 202]] - [:batch-remove-property [[1] :logseq.property/publishing-public?]] - [:batch-set-property [[1] :block/tags 101 {}]] - [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] - @ops*)) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e " calls: " @calls*)) - (done)))))))) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond + (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil)))) + (set! add-command/resolve-properties (fn [_ _ properties] (p/resolved properties))) + (set! add-command/resolve-property-identifiers (fn [_ _ properties] (p/resolved properties))) + (set! transport/invoke (fn [_ method _ args] + (swap! calls* conj {:method method :args args}) + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 1) + {:db/id 1 + :block/name nil + :block/uuid (uuid "00000000-0000-0000-0000-000000000001")} + (= lookup 2) + {:db/id 2 + :block/name nil + :block/uuid (uuid "00000000-0000-0000-0000-000000000002")} + :else {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] + [:batch-delete-property-value [[1] :block/tags 202]] + [:batch-remove-property [[1] :logseq.property/publishing-public?]] + [:batch-set-property [[1] :block/tags 101 {}]] + [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + @ops*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e " calls: " @calls*)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! add-command/resolve-tags orig-resolve-tags) + (set! add-command/resolve-properties orig-resolve-properties) + (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) + (set! transport/invoke orig-invoke) + (done))))))) (deftest test-execute-requires-existing-graph (async done diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 1d8537856b..fb0bda6cd7 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -5,11 +5,14 @@ [cljs.reader :as reader] [cljs.test :refer [deftest is async]] [clojure.string :as string] - [frontend.worker-common.util :as worker-util] [frontend.test.node-helper :as node-helper] - [logseq.cli.command.show :as show-command] + [frontend.worker-common.util :as worker-util] [logseq.cli.command.core :as command-core] + [logseq.cli.command.show :as show-command] + [logseq.cli.config :as cli-config] [logseq.cli.main :as cli-main] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [logseq.db.frontend.property :as db-property] [promesa.core :as p])) @@ -272,7 +275,7 @@ add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage" ] data-dir cfg-path) + show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) show-payload (parse-json-output show-result) remove-page-result (run-cli ["--repo" "content-graph" "remove" "--page" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) @@ -293,8 +296,8 @@ (is (not (contains? (get-in show-payload [:data :root]) :block/uuid))) (is (= "ok" (:status remove-page-payload))) (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] + (done)) + (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -324,7 +327,7 @@ ref-title (some #(when (and (string? %) (string/includes? % "See [[") (string/includes? % "]]")) - %) + %) titles) ref-value (when ref-title (second (first (re-seq #"\[\[(.*?)\]\]" ref-title)))) @@ -378,7 +381,7 @@ titles (map first (get-in ref-query-payload [:data :result])) ref-title (some #(when (and (string? %) (string/includes? % (str "[[" target-uuid "]]"))) - %) + %) titles) stop-result (run-cli ["server" "stop" "--repo" "uuid-ref-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -442,7 +445,7 @@ "add" "block" "--target-page-name" "Home" "--content" "Tagged block ident" - "--tags" "[:logseq.class/Quote-block]"] + "--tags" "[:logseq.class/Quote-block]"] data-dir cfg-path) add-block-ident-payload (parse-json-output add-block-ident-result) deadline-prop-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) @@ -469,27 +472,27 @@ block-deadline (query-property data-dir cfg-path repo "Tagged block" ":logseq.property/deadline") block-deadline-title (query-property data-dir cfg-path repo "Tagged block title" ":logseq.property/deadline") stop-payload (stop-repo! data-dir cfg-path repo)] - (is (= 0 (:exit-code add-page-result))) - (is (= "ok" (:status add-page-payload))) - (is (= 0 (:exit-code add-block-result))) - (is (= "ok" (:status add-block-payload))) - (is (= 0 (:exit-code add-block-ident-result))) - (is (= "ok" (:status add-block-ident-payload))) - (is (string? deadline-prop-title)) - (is (string? publishing-prop-title)) - (is (= 0 (:exit-code add-page-title-result))) - (is (= "ok" (:status add-page-title-payload))) - (is (= 0 (:exit-code add-block-title-result))) - (is (= "ok" (:status add-block-title-payload))) - (is (contains? block-tag-names "Quote")) - (is (contains? block-ident-tag-names "Quote")) - (is (contains? page-tag-names "Quote")) - (is (true? page-value)) - (is (true? page-title-value)) - (is (number? block-deadline)) - (is (number? block-deadline-title)) - (is (= "ok" (:status stop-payload))) - (done)) + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (= 0 (:exit-code add-block-ident-result))) + (is (= "ok" (:status add-block-ident-payload))) + (is (string? deadline-prop-title)) + (is (string? publishing-prop-title)) + (is (= 0 (:exit-code add-page-title-result))) + (is (= "ok" (:status add-page-title-payload))) + (is (= 0 (:exit-code add-block-title-result))) + (is (= "ok" (:status add-block-title-payload))) + (is (contains? block-tag-names "Quote")) + (is (contains? block-ident-tag-names "Quote")) + (is (contains? page-tag-names "Quote")) + (is (true? page-value)) + (is (true? page-title-value)) + (is (number? block-deadline)) + (is (number? block-deadline-title)) + (is (= "ok" (:status stop-payload))) + (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -536,12 +539,66 @@ (pr-str (:error add-block-id-payload))) (is (= "ok" (:status add-block-id-payload))) (is (true? page-id-value)) - (is (number? block-deadline-id)) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (is (number? block-deadline-id)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest test-cli-show-properties-human-output + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-show-properties") + repo "show-properties-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + cfg (cli-config/resolve-config {:data-dir data-dir + :config-path cfg-path + :output-format :json}) + server (cli-server/ensure-server! cfg repo) + property-ident :user.property/acceptance-criteria + import-data {:properties {property-ident {:logseq.property/type :default + :block/title "Acceptance Criteria"}} + :pages-and-blocks [{:page {:block/title "PropsPage"} + :blocks [{:block/title "Property block" + :build/properties {property-ident "First requirement"}}]}]} + wait-for-property (fn wait-for-property [attempt] + (p/let [value (transport/invoke server :thread-api/q false + [repo ['[:find ?v . + :in $ ?title ?prop + :where + [?b :block/title ?title] + [?b ?prop ?v]] + "Property block" + property-ident]])] + (if (or value (>= attempt 20)) + value + (p/let [_ (p/delay 100)] + (wait-for-property (inc attempt)))))) + _ (transport/invoke server :thread-api/apply-outliner-ops false + [repo [[:batch-import-edn [import-data {}]]] {}]) + _ (p/delay 100) + _ (wait-for-property 0) + page-name (common-util/page-name-sanity-lc "PropsPage") + page-entity (transport/invoke server :thread-api/pull false + [repo [:db/id :block/name :block/title] [:block/name page-name]]) + _ (when-not (:db/id page-entity) + (throw (ex-info "page not found in server" {:page page-name}))) + show-config (assoc cfg :output-format :human) + show-result (show-command/execute-show {:type :show + :repo repo + :page page-name} + show-config) + output (get-in show-result [:data :message]) + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= :ok (:status show-result))) + (is (string/includes? output "Acceptance Criteria: First requirement")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-cli-verbose-logs-to-stderr (async done @@ -641,11 +698,17 @@ query-text "[:find ?e :in $ ?title :where [?e :block/title ?title]]"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "query-graph"] data-dir cfg-path) - create-payload (parse-json-output create-result) - _ (run-cli ["--repo" "query-graph" "add" "page" "--page" "QueryPage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "add" "block" "--target-page-name" "QueryPage" "--content" "Query block"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "add" "block" "--target-page-name" "QueryPage" "--content" "Query block"] data-dir cfg-path) + create-result (run-cli ["graph" "create" "--repo" "query-graph"] data-dir cfg-path) + create-payload (parse-json-output create-result) + _ (run-cli ["--repo" "query-graph" "add" "page" "--page" "QueryPage"] data-dir cfg-path) + _ (run-cli ["--repo" "query-graph" "add" "block" + "--target-page-name" "QueryPage" + "--content" "Query block"] + data-dir cfg-path) + _ (run-cli ["--repo" "query-graph" "add" "block" + "--target-page-name" "QueryPage" + "--content" "Query block"] + data-dir cfg-path) _ (p/delay 100) query-result (run-cli ["--repo" "query-graph" "query" @@ -809,8 +872,7 @@ page-id (or (:db/id page-item) (:id page-item)) show-result (run-cli ["--repo" "recent-updated-graph" "show" - "--page" "RecentPage" - ] + "--page" "RecentPage"] data-dir cfg-path) show-payload (parse-json-output show-result) show-root (get-in show-payload [:data :root]) @@ -907,7 +969,7 @@ middle-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" middle-content) _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) - show-outer (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage" ] data-dir cfg-path) + show-outer (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-outer-payload (parse-json-output show-outer) outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) @@ -938,11 +1000,11 @@ ref-title (str "See [[" target-title "]]") _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) _ (p/delay 100) - source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "SourcePage" ] data-dir cfg-path) + source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "SourcePage"] data-dir cfg-path) source-payload (parse-json-output source-show) ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-title) ref-id (or (:db/id ref-node) (:id ref-node)) - target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage" ] data-dir cfg-path) + target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) target-payload (parse-json-output target-show) linked-refs (get-in target-payload [:data :linked-references]) linked-blocks (:blocks linked-refs) @@ -983,7 +1045,7 @@ _ (run-cli ["--repo" "move-graph" "add" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) update-result (run-cli ["--repo" "move-graph" "update" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) update-payload (parse-json-output update-result) - target-show (run-cli ["--repo" "move-graph" "show" "--page" "TargetPage" ] data-dir cfg-path) + target-show (run-cli ["--repo" "move-graph" "show" "--page" "TargetPage"] data-dir cfg-path) target-payload (parse-json-output target-show) moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") child-node (find-block-by-title moved-node "Child Block") @@ -1014,7 +1076,7 @@ parent-id (node-id parent-node) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) - final-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage" ] data-dir cfg-path) + final-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) final-payload (parse-json-output final-show) final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") child-titles (map node-title (node-children final-parent)) @@ -1107,9 +1169,9 @@ (get-in list-page-payload [:data :items])) page-id (or (:db/id page-item) (:id page-item)) page-uuid (or (:block/uuid page-item) (:uuid page-item)) - show-by-id-result (run-cli ["show" "--id" (str page-id) ] data-dir cfg-path) + show-by-id-result (run-cli ["show" "--id" (str page-id)] data-dir cfg-path) show-by-id-payload (parse-json-output show-by-id-result) - show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid) ] data-dir cfg-path) + show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid)] data-dir cfg-path) show-by-uuid-payload (parse-json-output show-by-uuid-result) stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -1172,8 +1234,7 @@ idx-two (string/index-of output "Multi show two") idx-delim (string/index-of output "================================================================") show-json-result (run-cli ["--repo" "show-multi-id-graph" "show" - "--id" ids-edn - ] + "--id" ids-edn] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) @@ -1198,9 +1259,9 @@ (is (= 2 (count show-data))) (is (contains? root-titles "Multi show one")) (is (contains? root-titles "Multi show two")) - (is (= "ok" (:status stop-payload))) - (done)) - (p/catch (fn [e] + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) @@ -1229,16 +1290,14 @@ _ (p/delay 100) show-children (run-cli ["--repo" "show-multi-id-contained-graph" "show" - "--page" "ParentPage" - ] + "--page" "ParentPage"] data-dir cfg-path) show-children-payload (parse-json-output show-children) child-node (find-block-by-title (get-in show-children-payload [:data :root]) "Child Block") child-id (or (:db/id child-node) (:id child-node)) ids-edn (str "[" parent-id " " child-id "]") show-json-result (run-cli ["--repo" "show-multi-id-contained-graph" "show" - "--id" ids-edn - ] + "--id" ids-edn] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) @@ -1415,7 +1474,7 @@ blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--blocks" blocks-edn] data-dir cfg-path) - show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage" ] + show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) show-payload (parse-json-output show-result) linked (get-in show-payload [:data :linked-references]) From de004d1eaa2ddaa6ba4618fcc7cef7669f60ad77 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 4 Feb 2026 23:26:27 +0800 Subject: [PATCH 063/375] 029-logseq-cli-show-properties.md (2) --- .../029-logseq-cli-show-properties.md | 14 +++++------ src/main/logseq/cli/command/show.cljs | 3 +-- src/test/logseq/cli/commands_test.cljs | 24 ++++++++++++++----- src/test/logseq/cli/integration_test.cljs | 5 ++-- 4 files changed, 29 insertions(+), 17 deletions(-) diff --git a/docs/agent-guide/029-logseq-cli-show-properties.md b/docs/agent-guide/029-logseq-cli-show-properties.md index fee92b0d46..d4da572d5b 100644 --- a/docs/agent-guide/029-logseq-cli-show-properties.md +++ b/docs/agent-guide/029-logseq-cli-show-properties.md @@ -13,7 +13,7 @@ Related: Builds on 028-logseq-cli-verbose-debug.md. The logseq-cli show command currently renders block trees without any property visibility. Users need to see each block's properties directly under the block content in the human output so that show reflects the same metadata they rely on in the UI. -Property lines belong to the same block tree element, so they must not render with tree glyphs. +Property lines belong to the same block tree element, so they must render with the same tree glyphs and indentation rules used for multiline block content (no blank space-only prefix). The display must show ": " and must handle single-value properties as a single line and multi-value properties as an indented list as shown in the example. ## Testing Plan @@ -87,7 +87,7 @@ User properties with values that are entities or maps should render as :block/ti User properties must be detected only via the :user.property/* namespace and no other property sources should be displayed. Blocks without properties should render exactly as they do today. Linked references output should include properties for each block without breaking the existing tree formatting. -Property lines should not include tree glyphs or branch markers. +Property lines should include the same tree glyphs/branch markers used for multiline block content so they align with block text. ## Implementation details for formatting @@ -95,7 +95,7 @@ Use a stable sort order for :user.property/* keys, such as ascending by keyword Do not use :block/properties-text-values for db-graph output. Format user property values from :user.property/* attributes and coerce values into strings using :block/title, :block/name, or :logseq.property/value. Render single values as "Property: value" and multi values as "Property: - v1" with subsequent lines aligned under the dash list. -Align property lines with the block content column and preserve tree glyph indentation, but do not render additional tree glyphs for properties. +Align property lines with the block content column and reuse the same tree glyph indentation rules used for multiline block content. ## Questions @@ -137,10 +137,10 @@ Current: Target: ``` 5137 Done Add git sha when graph is created for improved debugging #Issue - Background: Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 - Acceptance Criteria: - - When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha - - When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha + │ Background: Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 + │ Acceptance Criteria: + │ - When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha + │ - When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha ``` ## Question diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 085dad98dd..95c7d72c5f 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -770,8 +770,7 @@ (style/dim value)) lines (atom []) property-indent (fn [prefix] - (let [prefix* (string/replace prefix "│" " ")] - (str id-padding (style-glyph prefix*)))) + (str id-padding (style-glyph prefix))) append-property-lines (fn [node prefix] (let [indent (property-indent prefix) prop-lines (node-property-lines node property-titles property-value-labels indent)] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 82d9db411c..8d0c23c207 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -369,12 +369,18 @@ (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 1 :block/title "Root" - :user.property/background "Because"} + :block/children [{:db/id 2 + :block/title "Child A" + :user.property/background "Because"} + {:db/id 3 + :block/title "Child B"}]} :property-titles {:user.property/background "Background"}} output (binding [style/*color-enabled?* true] (tree->text tree-data))] (is (= (str "1 Root\n" - " Background: Because") + "2 ├── Child A\n" + " │ Background: Because\n" + "3 └── Child B") (strip-ansi output))) (is (not (string/includes? output "└── Background")))))) @@ -383,14 +389,20 @@ (let [tree->text #'show-command/tree->text tree-data {:root {:db/id 1 :block/title "Root" - :user.property/criteria ["One" "Two"]} + :block/children [{:db/id 2 + :block/title "Child A" + :user.property/criteria ["One" "Two"]} + {:db/id 3 + :block/title "Child B"}]} :property-titles {:user.property/criteria "Criteria"}} output (binding [style/*color-enabled?* true] (tree->text tree-data))] (is (= (str "1 Root\n" - " Criteria:\n" - " - One\n" - " - Two") + "2 ├── Child A\n" + " │ Criteria:\n" + " │ - One\n" + " │ - Two\n" + "3 └── Child B") (strip-ansi output)))))) (deftest test-tree->text-properties-order diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index fb0bda6cd7..05ab081de8 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -562,7 +562,8 @@ :block/title "Acceptance Criteria"}} :pages-and-blocks [{:page {:block/title "PropsPage"} :blocks [{:block/title "Property block" - :build/properties {property-ident "First requirement"}}]}]} + :build/properties {property-ident "First requirement"}} + {:block/title "Sibling block"}]}]} wait-for-property (fn wait-for-property [attempt] (p/let [value (transport/invoke server :thread-api/q false [repo ['[:find ?v . @@ -593,7 +594,7 @@ output (get-in show-result [:data :message]) stop-payload (stop-repo! data-dir cfg-path repo)] (is (= :ok (:status show-result))) - (is (string/includes? output "Acceptance Criteria: First requirement")) + (is (string/includes? output "│ Acceptance Criteria: First requirement")) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From 96dad8194b5d264d8c65bf7c11b989fecc506e18 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 6 Feb 2026 21:36:17 +0800 Subject: [PATCH 064/375] remove redundant ut --- src/test/frontend/worker/platform_test.cljs | 85 --------------------- 1 file changed, 85 deletions(-) delete mode 100644 src/test/frontend/worker/platform_test.cljs diff --git a/src/test/frontend/worker/platform_test.cljs b/src/test/frontend/worker/platform_test.cljs deleted file mode 100644 index f5477421df..0000000000 --- a/src/test/frontend/worker/platform_test.cljs +++ /dev/null @@ -1,85 +0,0 @@ -(ns frontend.worker.platform-test - (:require ["ws" :as ws] - [cljs.test :refer [async deftest is]] - [frontend.common.file.opfs :as opfs] - [frontend.test.node-helper :as node-helper] - [frontend.worker-common.util :as worker-util] - [frontend.worker.platform.browser :as platform-browser] - [frontend.worker.platform.node :as platform-node] - [promesa.core :as p])) - -(defn- wait-for-event - [emitter event] - (p/create - (fn [resolve reject] - (.once emitter event (fn [& args] (resolve args))) - (.once emitter "error" reject)))) - -(defn- fake-websocket - [url] - (this-as this - (set! (.-url this) url) - this)) - -(deftest browser-platform-adapter - (async done - (let [saved-location (.-location js/globalThis) - saved-websocket (.-WebSocket js/globalThis) - kv-state (atom {}) - posted (atom nil)] - (set! (.-location js/globalThis) #js {:href "http://example.test/?publishing=true"}) - (set! (.-WebSocket js/globalThis) fake-websocket) - (with-redefs [opfs/ (p/let [platform (platform-browser/browser-platform) - kv (:kv platform) - storage (:storage platform) - _ (is (fn? (:get kv))) - _ (is (fn? (:set! kv))) - _ (p/let [_ ((:write-text! storage) "foo.txt" "bar") - v ((:read-text! storage) "foo.txt")] - (is (= "read:foo.txt" v))) - _ ((:post-message! (:broadcast platform)) :event {:ok true}) - ws ((:connect (:websocket platform)) "ws://example.test/socket")] - (is (= [:event {:ok true}] @posted)) - (is (= "ws://example.test/socket" (.-url ws)))) - (p/finally (fn [] - (set! (.-location js/globalThis) saved-location) - (set! (.-WebSocket js/globalThis) saved-websocket))) - (p/then (fn [] (done)))))))) - -(deftest node-platform-adapter - (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-platform") - events (atom []) - server (ws/Server. #js {:port 0})] - (.on server "connection" (fn [socket] (.close socket))) - (-> (p/let [_ (wait-for-event server "listening") - port (.-port (.address server)) - platform (platform-node/node-platform - {:data-dir data-dir - :event-fn (fn [type payload] - (swap! events conj [type payload]))}) - storage (:storage platform) - kv (:kv platform) - ws-connect (:connect (:websocket platform)) - _ (p/let [_ ((:write-text! storage) "foo/bar.txt" "hello") - v ((:read-text! storage) "foo/bar.txt")] - (is (= "hello" v))) - _ (p/let [_ ((:set! kv) "alpha" "beta") - v ((:get kv) "alpha")] - (is (= "beta" v))) - _ ((:post-message! (:broadcast platform)) :event {:value 1}) - _ (is (= [[:event {:value 1}]] @events)) - client (ws-connect (str "ws://127.0.0.1:" port)) - _ (p/let [_ (wait-for-event client "open")] - (.close client))] - true) - (p/finally (fn [] - (.close server))) - (p/then (fn [] (done))))))) From 4581814fdc34aaa256f5ee7465541dcaf76b23ea Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 6 Feb 2026 21:54:16 +0800 Subject: [PATCH 065/375] fix test --- src/test/logseq/cli/format_test.cljs | 10 +++++++--- src/test/logseq/cli/integration_test.cljs | 4 +++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 1f2634b9d7..429fc24a9c 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -231,9 +231,13 @@ result (format/format-result {:status :ok :command :show :data {:message styled}} - {:output-format nil})] - (is (string/includes? result (style/bold "TODO"))) - (is (string/includes? result (style/bold "#TagA"))) + {:output-format nil}) + styled-todo (binding [style/*color-enabled?* true] + (style/bold (style/yellow "TODO"))) + styled-tag (binding [style/*color-enabled?* true] + (style/bold "#TagA"))] + (is (string/includes? result styled-todo)) + (is (string/includes? result styled-tag)) (is (= (str "1 Root\n" "2 └── TODO Child #TagA") (style/strip-ansi result)))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 05ab081de8..a6c0116b3b 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -12,6 +12,7 @@ [logseq.cli.config :as cli-config] [logseq.cli.main :as cli-main] [logseq.cli.server :as cli-server] + [logseq.cli.style :as style] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [logseq.db.frontend.property :as db-property] @@ -592,9 +593,10 @@ :page page-name} show-config) output (get-in show-result [:data :message]) + output* (style/strip-ansi output) stop-payload (stop-repo! data-dir cfg-path repo)] (is (= :ok (:status show-result))) - (is (string/includes? output "│ Acceptance Criteria: First requirement")) + (is (string/includes? output* "│ Acceptance Criteria: First requirement")) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From 4b2d725be49571c34e78696b07d9b5531b0a4ac2 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 6 Feb 2026 22:23:02 +0800 Subject: [PATCH 066/375] 030-logseq-cli-db-graph-default-dir-locking.md (1) --- ...logseq-cli-db-graph-default-dir-locking.md | 191 ++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md diff --git a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md new file mode 100644 index 0000000000..2f09deb083 --- /dev/null +++ b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md @@ -0,0 +1,191 @@ +# Logseq CLI DB Graph Default Dir And Write Exclusion Implementation Plan + +Goal: Move the default CLI data directory to `~/logseq/graphs`, store graph directories without requiring the `logseq_db_` prefix, and enforce single-writer behavior for graph files while one db-worker-node instance owns a graph. + +Architecture: Keep `logseq_db_` as the internal repo identifier used by thread-api calls, but introduce a canonical graph directory key that strips the db prefix for filesystem paths. +Architecture: Centralize graph directory resolution and lock ownership checks so `logseq.cli.server`, `frontend.worker.db-worker-node-lock`, and `frontend.worker.platform.node` enforce the same rules. + +Tech Stack: ClojureScript, Node.js `fs` and `path`, promesa, logseq-cli command pipeline, db-worker-node daemon, existing lock file protocol. + +Related: Builds on `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`. +Related: Supercedes `docs/agent-guide/020-logseq-cli-default-paths-move.md`. +Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. + +## Problem statement + +The current CLI default data directory is `~/logseq/cli-graphs`, while this change requires `~/logseq/graphs`. + +The current filesystem directory naming for a graph is based on repo identifiers that frequently include `logseq_db_`, so users still observe prefixed names in the data directory. + +The current lock model prevents launching a second db-worker-node for one repo, but it does not define an explicit write-lease boundary that all graph file mutations must validate before writing. + +We need one coherent model where graph directories are user-facing names, internal repo identifiers stay db-prefixed for thread-api compatibility, and graph writes are denied for non-owners while a server is running. + +This plan does not include compatibility logic, migration, or special handling for old on-disk graph directories that start with `logseq_db_`. + +This plan treats old on-disk `logseq_db_` prefixed graph directories as ignored entries. + +## Testing Plan + +I will follow `@test-driven-development` and add all failing tests before implementation edits. + +I will add unit tests for default path resolution and help text defaults in CLI and db-worker-node code paths. + +I will add unit tests for graph directory canonicalization that prove `logseq_db_demo` resolves to the same on-disk directory as `demo` in the default data directory. + +I will not add migration tests for old prefixed directories because compatibility and migration are out of scope for this change. + +I will add db-worker-node tests that validate write-lease ownership checks fail for non-owner lock metadata and pass for the active owner. + +I will run targeted test namespaces first for fast red and green loops, then run the full lint and test suite. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Required change | +|---|---|---| +| Default data dir | `~/logseq/cli-graphs` in CLI and db-worker-node defaults. | `~/logseq/graphs` in all default derivation and help output locations. | +| Graph dir naming | Graph directory derivation commonly receives repo values that include `logseq_db_`. | Graph directory derivation must use canonical graph names without requiring `logseq_db_`. | +| Graph type assumption | DB graph identity is inferred from repo prefix in several paths. | In default data dir, treat every graph directory as a db graph and map to internal repo with prefix only at invocation boundaries. | +| Write exclusivity | `db-worker.lock` blocks duplicate daemon start, but write ownership is not verified by all mutation paths. | Introduce write-lease ownership checks for all graph file mutation operations executed by db-worker-node. | + +## Integration sketch + +```text +CLI --repo demo + -> command-core resolves internal repo: logseq_db_demo + -> graph-dir resolver maps repo to graph key: demo + -> fs paths use ~/logseq/graphs/demo + -> thread-api calls still use logseq_db_demo + +db-worker-node owner lease + -> creates db-worker.lock with pid + lock-id + -> mutation path checks lock-id + pid ownership + -> non-owner mutation attempt returns :repo-locked +``` + +## Implementation plan + +### Phase 1: Add failing tests for path defaults and naming. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs` that `resolve-config` defaults `:data-dir` to `~/logseq/graphs`. +2. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/data_dir_test.cljs` that `normalize-data-dir` resolves to `$HOME/logseq/graphs`. +3. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `show-help!` prints `default ~/logseq/graphs` for `--data-dir`. +4. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that global help includes `Path to db-worker data dir (default ~/logseq/graphs)`. +5. Run `bb dev:test -v 'logseq.cli.config-test'` and confirm the new default path assertion fails. +6. Run `bb dev:test -v 'logseq.cli.data-dir-test'` and confirm the new default path assertion fails. + +### Phase 2: Add failing tests for prefix-free graph directory semantics. + +7. Add a new test namespace `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that asserts repo `logseq_db_demo` resolves to graph directory key `demo` under default data dir. +8. Add a second test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that repo `demo` also resolves to graph directory key `demo`. +9. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that verifies no migration helper is invoked for legacy prefixed on-disk graph directories. +10. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that canonical directory resolution only targets `` naming in the default data dir. +11. Add a failing CLI server test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that `lock-path` for repo `logseq_db_demo` points to `/demo/db-worker.lock`. +12. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and confirm failures for unimplemented canonicalization and non-migration logic. +13. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm lock-path behavior fails before implementation. + +### Phase 3: Add failing tests for write-lease ownership. + +14. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that simulates lock metadata mismatch and expects write mutation to fail with `:repo-locked`. +15. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that verifies owner write mutation succeeds when lock metadata matches current owner. +16. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that stale lock ownership is rejected after lock replacement. +17. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm the write-lease tests fail before implementation. + +### Phase 4: Implement default directory switch to `~/logseq/graphs`. + +18. Update default path constants in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs` from `~/logseq/cli-graphs` to `~/logseq/graphs`. +19. Update CLI config defaults in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` so `:data-dir` defaults to `~/logseq/graphs`. +20. Update server fallback in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so `resolve-data-dir` defaults to `~/logseq/graphs`. +21. Update db-worker-node default path in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` so `resolve-data-dir` defaults to `~/logseq/graphs`. +22. Update node platform fallback in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` so `node-platform` defaults to `~/logseq/graphs`. +23. Update CLI help text in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` to show `~/logseq/graphs`. +24. Re-run `bb dev:test -v 'logseq.cli.config-test'` and `bb dev:test -v 'logseq.cli.data-dir-test'` and confirm green results. + +### Phase 5: Implement canonical graph directory resolution without required `logseq_db_` prefix. + +25. Add shared graph directory canonicalization helpers in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` to map internal repo names to canonical graph directory keys. +26. Ensure canonicalization strips `logseq_db_` only when the repo is a db repo and preserves encoded filename safety through existing `encode-graph-dir-name` helpers. +27. Explicitly avoid adding legacy directory migration logic from `logseq_db_` to `` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs`. +28. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` to use canonical graph directory resolution for `repo-dir`, `lock-path`, and graph enumeration. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` to use canonical graph directory keys in `storage-pool-name` and `db-exists?` path resolution for node runtime. +30. Re-run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and `bb dev:test -v 'logseq.cli.server-test'` and confirm canonical naming tests pass without migration behavior. + +### Phase 6: Implement write-lease ownership checks for graph file mutations. + +32. Extend lock payload in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` with a generated `lock-id` and keep ownership fields immutable after acquisition. +33. Add `assert-lock-owner!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` that validates lock existence, pid ownership, and `lock-id` match before mutation. +34. Pass a write-guard callback from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` into `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. +35. Invoke the write-guard callback before every graph file mutation path in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`, including sqlite import, text writes, and recursive delete paths. +36. Return consistent `:repo-locked` errors from ownership failures and ensure CLI formatting remains readable through `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +37. Re-run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm ownership tests pass. + +### Phase 7: Update docs and run full verification. + +38. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` for the new default data dir and for prefix-free on-disk graph directory naming. +39. Add a breaking-change note in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` stating old on-disk prefixed graph directories are not automatically migrated. +40. Re-run `bb dev:test -v 'logseq.cli.commands-test'` and `bb dev:test -v 'logseq.cli.main-test'` to verify help text and command behavior coverage. +41. Run `bb dev:lint-and-test` and confirm the suite passes with no regressions. +42. Review changed code against `@prompts/review.md` to catch Clojure and ClojureScript correctness pitfalls before merge. + +## Edge cases to cover + +| Scenario | Expected behavior | +|---|---| +| Graph name contains `/`, `:`, `%`, or unicode. | Directory naming remains reversible through encode and decode helpers. | +| Legacy on-disk directory `logseq_db_demo` exists. | No compatibility or migration is performed by this change, and discovery commands ignore this directory. | +| Lock file is stale with dead pid. | Startup removes stale lock and acquires a fresh lease. | +| Lock file exists with alive non-owner pid. | Startup and direct mutation fail fast with `:repo-locked`. | +| Default data dir has non-graph directories. | Enumeration ignores directories that are not valid graph directories after canonicalization checks. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.config-test' +bb dev:test -v 'logseq.cli.data-dir-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'frontend.worker.db-worker-node-lock-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +bb dev:test -v 'logseq.cli.commands-test' +bb dev:lint-and-test +``` + +Each command should finish with zero failures and zero errors. + +If a red phase command is run before implementation, it should fail specifically on the newly added assertions in that phase. + +## Testing Details + +The tests validate behavior through public CLI and daemon entrypoints instead of validating only implementation internals. + +The naming tests prove that internal db repo prefixes remain stable while on-disk names are canonicalized for user-facing graph directories. + +The ownership tests verify that only the active lock owner can execute graph file mutation paths in db-worker-node. + +The tests intentionally avoid migration coverage because compatibility with old prefixed on-disk graph directories is out of scope. + +## Implementation Details + +- Keep internal repo values db-prefixed for thread-api compatibility and db metadata reads. +- Use canonical graph directory keys for all on-disk path construction in node runtime. +- Do not add compatibility branches or migration logic for old `logseq_db_` prefixed on-disk graph directories. +- Ignore old `logseq_db_` prefixed on-disk graph directories during graph discovery and server listing. +- Add lock ownership token fields and verify ownership before each graph mutation path. +- Enforce ownership checks for all files under a graph directory, including sqlite files, search and client-op files, debug logs, backups, and text files. +- Keep error code semantics stable by reusing `:repo-locked` for ownership and lock conflicts. +- Keep lock-conflict error semantics unified across CLI and db-worker-node as `:repo-locked`. +- Update all default data-dir strings and help text to `~/logseq/graphs`. +- Keep CLI output graph names user-facing and prefix-free. +- Update docs with a breaking-change note for old prefixed on-disk graph directories. +- Follow `@test-driven-development` and `@prompts/review.md` through implementation and verification. + +## Question + +Resolved decisions: + +1. Ignore old on-disk `logseq_db_` prefixed graph directories. +2. Requirement three covers all files under the graph directory. +3. Lock conflicts and ownership failures use unified error code `:repo-locked`. + +--- From a1ff32f6dd4f3d9f808d50b698794d6ee5bd0560 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Feb 2026 09:05:23 +0800 Subject: [PATCH 067/375] 030-logseq-cli-db-graph-default-dir-locking.md (2) --- docs/cli/logseq-cli.md | 4 +- src/main/frontend/worker/db_core.cljs | 9 +- src/main/frontend/worker/db_worker_node.cljs | 49 +++++++--- .../frontend/worker/db_worker_node_lock.cljs | 80 ++++++++++++++- src/main/frontend/worker/platform/node.cljs | 39 ++++---- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/config.cljs | 2 +- src/main/logseq/cli/data_dir.cljs | 2 +- src/main/logseq/cli/server.cljs | 14 +-- .../worker/db_worker_node_lock_test.cljs | 37 +++++++ .../frontend/worker/db_worker_node_test.cljs | 97 +++++++++++++++++-- src/test/logseq/cli/commands_test.cljs | 4 + src/test/logseq/cli/config_test.cljs | 2 +- src/test/logseq/cli/data_dir_test.cljs | 4 +- src/test/logseq/cli/integration_test.cljs | 28 +++--- src/test/logseq/cli/server_test.cljs | 17 ++-- 16 files changed, 312 insertions(+), 78 deletions(-) create mode 100644 src/test/frontend/worker/db_worker_node_lock_test.cljs diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index a6bc11dc73..c3e3b51039 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -33,7 +33,9 @@ logseq graph list Optional configuration file: `~/logseq/cli.edn` -Default data dir: `~/logseq/cli-graphs`. +Default data dir: `~/logseq/graphs`. + +Graph directories on disk are stored as user-facing graph names (for example, `demo/`), not `logseq_db_` prefixed repo identifiers. Migration note: If you previously used `~/.logseq/cli-graphs` or `~/.logseq/cli.edn`, pass `--data-dir` or `--config` to continue using those locations. diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 911024e033..378b2c205c 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -15,6 +15,7 @@ [frontend.worker-common.util :as worker-util] [frontend.worker.db-listener :as db-listener] [frontend.worker.db-metadata :as worker-db-metadata] + [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.db.fix :as db-fix] [frontend.worker.db.migrate :as db-migrate] [frontend.worker.db.validate :as worker-db-validate] @@ -72,7 +73,7 @@ (defn- storage-pool-name [graph] (if (node-runtime?) - (worker-util/encode-graph-dir-name graph) + (db-lock/repo->graph-dir-key graph) (worker-util/get-pool-name graph))) (defn- get-storage-pool @@ -350,9 +351,9 @@ (p/let [storage (platform/storage (platform/current)) graph-names ((:list-graphs storage))] (p/all (map (fn [graph-name] - (p/let [repo (str sqlite-util/db-version-prefix graph-name) - metadata (worker-db-metadata/ (default ~/logseq/cli-graphs)")) + (println (str " " (style/bold "--data-dir") " (default ~/logseq/graphs)")) (println (str " " (style/bold "--repo") " (required)")) (println (str " " (style/bold "--rtc-ws-url") " (optional)")) (println (str " " (style/bold "--log-level") " (default info)")) @@ -308,9 +327,14 @@ :repo repo :log-level (keyword (or log-level "info"))}) (reset! *ready? false) + (reset! *lock-info nil) (set-main-thread-stub!) - (-> (p/let [platform (platform-node/node-platform {:data-dir data-dir - :event-fn handle-event!}) + (-> (p/let [write-guard-fn (fn [] + (let [{:keys [path lock]} @*lock-info] + (db-lock/assert-lock-owner! path lock))) + platform (platform-node/node-platform {:data-dir data-dir + :event-fn handle-event! + :write-guard-fn write-guard-fn}) proxy (db-core/init-core! platform) _ (graph-dir-key + [repo] + (when (seq repo) + (if (string/starts-with? repo common-config/db-version-prefix) + (subs repo (count common-config/db-version-prefix)) + repo))) + +(defn canonical-graph-dir-key? + [graph-dir-key] + (and (seq graph-dir-key) + (not (string/starts-with? graph-dir-key common-config/db-version-prefix)))) + +(defn decode-canonical-graph-dir-key + [encoded-graph-dir-key] + (let [decoded (worker-util/decode-graph-dir-name encoded-graph-dir-key)] + (when (canonical-graph-dir-key? decoded) + decoded))) (defn repo-dir [data-dir repo] - (node-path/join data-dir (worker-util/encode-graph-dir-name repo))) + (node-path/join data-dir (worker-util/encode-graph-dir-name (repo->graph-dir-key repo)))) (defn lock-path [data-dir repo] @@ -65,6 +84,7 @@ (let [fd (fs/openSync path "wx") lock {:repo repo :pid (.-pid js/process) + :lock-id (str (random-uuid)) :host host :port port :startedAt (.toISOString (js/Date.))}] @@ -82,12 +102,64 @@ (p/create (fn [resolve reject] (try - (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) - (resolve lock) + (let [existing (read-lock path) + lock' (if existing + (-> lock + (assoc :repo (:repo existing)) + (assoc :pid (:pid existing)) + (assoc :lock-id (or (:lock-id existing) (:lock-id lock))) + (assoc :startedAt (:startedAt existing))) + lock)] + (fs/writeFileSync path (js/JSON.stringify (clj->js lock'))) + (resolve lock')) (catch :default e (log/error :db-worker-node-lock-update-failed e) (reject e)))))) +(defn assert-lock-owner! + [path {:keys [repo pid lock-id] :as owner-lock}] + (let [lock (read-lock path)] + (cond + (nil? owner-lock) + (throw (ex-info "lock owner missing" + {:code :repo-locked + :path path})) + + (nil? lock) + (throw (ex-info "graph lock missing" + {:code :repo-locked + :path path})) + + (not= :alive (pid-status (:pid lock))) + (throw (ex-info "graph lock is stale" + {:code :repo-locked + :path path + :lock lock})) + + (not= repo (:repo lock)) + (throw (ex-info "graph lock repo mismatch" + {:code :repo-locked + :path path + :lock lock + :owner owner-lock})) + + (not= pid (:pid lock)) + (throw (ex-info "graph lock pid mismatch" + {:code :repo-locked + :path path + :lock lock + :owner owner-lock})) + + (not= lock-id (:lock-id lock)) + (throw (ex-info "graph lock-id mismatch" + {:code :repo-locked + :path path + :lock lock + :owner owner-lock})) + + :else + lock))) + (defn ensure-lock! [{:keys [data-dir repo host port]}] (let [data-dir (resolve-data-dir data-dir) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index e2a193d8dd..19c7b385b0 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -6,7 +6,7 @@ ["path" :as node-path] ["ws" :as ws] [clojure.string :as string] - [frontend.worker-common.util :as worker-util] + [frontend.worker.db-worker-node-lock :as db-lock] [goog.object :as gobj] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -29,8 +29,8 @@ (string/replace-first path #"^/" "")) (defn- repo-dir - [data-dir pool-name] - (node-path/join data-dir pool-name)) + [data-dir repo] + (db-lock/repo-dir data-dir repo)) (defn- pool-path [^js pool path] @@ -57,7 +57,7 @@ db-dirs (->> entries (filter dir?)) graph-names (map (fn [dirent] - (worker-util/decode-graph-dir-name (.-name dirent))) + (db-lock/decode-canonical-graph-dir-key (.-name dirent))) db-dirs)] (->> graph-names (filter some?) @@ -65,8 +65,7 @@ (defn- db-exists? [data-dir graph] - (p/let [pool-name (worker-util/encode-graph-dir-name graph) - db-path (node-path/join (repo-dir data-dir pool-name) "db.sqlite")] + (p/let [db-path (node-path/join (repo-dir data-dir graph) "db.sqlite")] (-> (fs/stat db-path) (p/then (fn [_] true)) (p/catch (fn [_] false))))) @@ -136,26 +135,32 @@ (fs/readFile (pool-path pool path))) (defn- import-db - [pool path data] + [write-guard-fn pool path data] (let [full-path (pool-path pool path) dir (node-path/dirname full-path)] - (p/let [_ (ensure-dir! dir)] + (p/let [_ (when write-guard-fn + (write-guard-fn)) + _ (ensure-dir! dir)] (fs/writeFile full-path (->buffer data))))) (defn- remove-vfs! - [^js pool] + [write-guard-fn ^js pool] (when pool - (fs/rm (.-repoDir pool) #js {:recursive true :force true}))) + (p/let [_ (when write-guard-fn + (write-guard-fn))] + (fs/rm (.-repoDir pool) #js {:recursive true :force true})))) (defn- read-text! [data-dir path] (fs/readFile (path-under-data-dir data-dir path) "utf8")) (defn- write-text! - [data-dir path text] + [write-guard-fn data-dir path text] (let [full-path (path-under-data-dir data-dir path) dir (node-path/dirname full-path)] - (p/let [_ (ensure-dir! dir)] + (p/let [_ (when write-guard-fn + (write-guard-fn)) + _ (ensure-dir! dir)] (fs/writeFile full-path text "utf8")))) (defn- websocket-connect @@ -187,8 +192,8 @@ (fs/writeFile kv-path payload "utf8")))})) (defn node-platform - [{:keys [data-dir event-fn]}] - (let [data-dir (expand-home (or data-dir "~/logseq/cli-graphs")) + [{:keys [data-dir event-fn write-guard-fn]}] + (let [data-dir (expand-home (or data-dir "~/logseq/graphs")) kv (kv-store data-dir)] (p/do! (ensure-dir! data-dir) @@ -203,10 +208,10 @@ :resolve-db-path (fn [_repo pool path] (pool-path pool path)) :export-file export-file - :import-db import-db - :remove-vfs! remove-vfs! + :import-db (fn [pool path data] (import-db write-guard-fn pool path data)) + :remove-vfs! (fn [pool] (remove-vfs! write-guard-fn pool)) :read-text! (fn [path] (read-text! data-dir path)) - :write-text! (fn [path text] (write-text! data-dir path text))} + :write-text! (fn [path text] (write-text! write-guard-fn data-dir path text))} :kv {:get (:get kv) :set! (:set! kv)} :broadcast {:post-message! (fn [type payload] diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 279ad1f097..fc8f6908e0 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -13,7 +13,7 @@ :coerce :boolean} :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)"} :repo {:desc "Graph name"} - :data-dir {:desc "Path to db-worker data dir (default ~/logseq/cli-graphs)"} + :data-dir {:desc "Path to db-worker data dir (default ~/logseq/graphs)"} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} :output {:desc "Output format (human, json, edn). Default: human"} diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 629791c8ff..11dc5c987f 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -75,7 +75,7 @@ [opts] (let [defaults {:timeout-ms 10000 :output-format nil - :data-dir "~/logseq/cli-graphs" + :data-dir "~/logseq/graphs" :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) diff --git a/src/main/logseq/cli/data_dir.cljs b/src/main/logseq/cli/data_dir.cljs index 014a3b9ae9..ccaaf0f385 100644 --- a/src/main/logseq/cli/data_dir.cljs +++ b/src/main/logseq/cli/data_dir.cljs @@ -5,7 +5,7 @@ ["path" :as node-path] [clojure.string :as string])) -(def ^:private default-data-dir "~/logseq/cli-graphs") +(def ^:private default-data-dir "~/logseq/graphs") (defn- expand-home [path] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 7d17521220..1bda79284b 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -6,9 +6,8 @@ ["os" :as os] ["path" :as node-path] [clojure.string :as string] - [logseq.cli.command.core :as command-core] [logseq.cli.log :as cli-log] - [frontend.worker-common.util :as worker-util] + [frontend.worker.db-worker-node-lock :as db-lock] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -20,11 +19,11 @@ (defn resolve-data-dir [config] - (expand-home (or (:data-dir config) "~/logseq/cli-graphs"))) + (expand-home (or (:data-dir config) "~/logseq/graphs"))) (defn- repo-dir [data-dir repo] - (node-path/join data-dir (worker-util/encode-graph-dir-name repo))) + (db-lock/repo-dir data-dir repo)) (defn- ensure-repo-dir! [data-dir repo] @@ -316,7 +315,9 @@ (for [^js entry entries :when (.isDirectory entry) :let [name (.-name entry) - lock (read-lock (node-path/join data-dir name "db-worker.lock"))] + graph-key (db-lock/decode-canonical-graph-dir-key name) + lock (when graph-key + (read-lock (node-path/join data-dir name "db-worker.lock")))] :when lock] (p/let [ready (ready? lock)] {:repo (:repo lock) @@ -333,7 +334,6 @@ (->> entries (filter #(.isDirectory ^js %)) (map (fn [^js dirent] - (worker-util/decode-graph-dir-name (.-name dirent)))) + (db-lock/decode-canonical-graph-dir-key (.-name dirent)))) (filter some?) - (map command-core/repo->graph) (vec)))) diff --git a/src/test/frontend/worker/db_worker_node_lock_test.cljs b/src/test/frontend/worker/db_worker_node_lock_test.cljs new file mode 100644 index 0000000000..e024d15c98 --- /dev/null +++ b/src/test/frontend/worker/db_worker_node_lock_test.cljs @@ -0,0 +1,37 @@ +(ns frontend.worker.db-worker-node-lock-test + (:require ["fs" :as fs] + ["os" :as os] + ["path" :as node-path] + [cljs.test :refer [deftest is testing]] + [frontend.test.node-helper :as node-helper] + [frontend.worker.db-worker-node-lock :as db-lock])) + +(deftest repo-dir-canonicalizes-db-prefixed-repo + (testing "db-prefixed repo name resolves to prefix-free graph directory key" + (let [data-dir "/tmp/logseq-db-worker-node-lock" + expected (node-path/join data-dir "demo")] + (is (= expected (db-lock/repo-dir data-dir "logseq_db_demo")))))) + +(deftest repo-dir-canonicalizes-prefix-free-repo + (testing "prefix-free repo name resolves to same graph directory key" + (let [data-dir "/tmp/logseq-db-worker-node-lock" + expected (node-path/join data-dir "demo")] + (is (= expected (db-lock/repo-dir data-dir "demo")))))) + +(deftest repo-dir-does-not-migrate-legacy-prefixed-dir + (testing "canonical resolution does not rename legacy prefixed directories" + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock") + legacy-dir (node-path/join data-dir "logseq_db_demo") + canonical-dir (node-path/join data-dir "demo")] + (fs/mkdirSync legacy-dir #js {:recursive true}) + (is (= canonical-dir (db-lock/repo-dir data-dir "logseq_db_demo"))) + (is (fs/existsSync legacy-dir)) + (is (not (fs/existsSync canonical-dir)))))) + +(deftest lock-path-default-data-dir-uses-canonical-graph-dir + (testing "default data-dir lock path is built with canonical directory naming" + (let [default-data-dir (db-lock/resolve-data-dir nil) + expected-data-dir (node-path/join (.homedir os) "logseq" "graphs") + expected-lock-path (node-path/join expected-data-dir "demo" "db-worker.lock")] + (is (= expected-data-dir default-data-dir)) + (is (= expected-lock-path (db-lock/lock-path default-data-dir "logseq_db_demo")))))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 1dd3197349..de349058b4 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -5,7 +5,7 @@ [cljs.test :refer [async deftest is]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] - [frontend.worker-common.util :as worker-util] + [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.db-worker-node :as db-worker-node] [goog.object :as gobj] [logseq.cli.style :as style] @@ -86,8 +86,7 @@ (defn- lock-path [data-dir repo] - (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo))] - (node-path/join repo-dir "db-worker.lock"))) + (db-lock/lock-path data-dir repo)) (defn- pad2 [value] @@ -103,7 +102,7 @@ (defn- log-path [data-dir repo] - (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo)) + (let [repo-dir (db-lock/repo-dir data-dir repo) date-str (yyyymmdd (js/Date.))] (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) @@ -168,7 +167,7 @@ (let [enforce-log-retention! #'db-worker-node/enforce-log-retention! data-dir (node-helper/create-tmp-dir "db-worker-log-retention") repo (str "logseq_db_log_retention_" (subs (str (random-uuid)) 0 8)) - repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo)) + repo-dir (db-lock/repo-dir data-dir repo) days ["20240101" "20240102" "20240103" "20240104" "20240105" "20240106" "20240107" "20240108" "20240109"] make-log (fn [day] @@ -214,8 +213,10 @@ (deftest db-worker-node-help-omits-auth-token (let [show-help! #'db-worker-node/show-help! output (binding [style/*color-enabled?* true] - (with-out-str (show-help!)))] + (with-out-str (show-help!))) + plain-output (style/strip-ansi output)] (is (not (string/includes? (style/strip-ansi output) "--auth-token"))) + (is (string/includes? plain-output "(default ~/logseq/graphs)")) (is (re-find #"\u001b\[[0-9;]*moptions\u001b\[[0-9;]*m:" output)) (is (contains-bold? output "db-worker-node")) (is (contains-bold? output "--data-dir")) @@ -471,3 +472,87 @@ (if-let [stop! (:stop! @daemon)] (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) + +(deftest db-worker-node-write-mutation-fails-for-non-owner-pid + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-write-lease-pid") + repo (str "logseq_db_write_lease_pid_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + export-base64 (invoke host port "thread-api/export-db-base64" [repo]) + lock-contents (js->clj (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + :keywordize-keys true) + tampered-lock (assoc lock-contents + :pid (inc (:pid lock-contents)) + :lock-id "non-owner-lock") + _ (fs/writeFileSync lock-file (js/JSON.stringify (clj->js tampered-lock))) + {:keys [status body]} (invoke-raw host port "thread-api/import-db-base64" [repo export-base64]) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (is (= 409 status)) + (is (= false (:ok parsed))) + (is (= "repo-locked" (get-in parsed [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-write-mutation-succeeds-for-active-owner + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-write-lease-owner") + repo (str "logseq_db_write_lease_owner_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + lock-id (gobj/get lock-contents "lock-id") + _ (is (string? lock-id)) + export-base64 (invoke host port "thread-api/export-db-base64" [repo]) + {:keys [status body]} (invoke-raw host port "thread-api/import-db-base64" [repo export-base64]) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (is (= 200 status)) + (is (= true (:ok parsed)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-write-mutation-rejects-stale-lock-after-replacement + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-write-lease-replaced") + repo (str "logseq_db_write_lease_replaced_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + export-base64 (invoke host port "thread-api/export-db-base64" [repo]) + lock-contents (js->clj (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + :keywordize-keys true) + replaced-lock (assoc lock-contents :lock-id "replaced-lock-id") + _ (fs/writeFileSync lock-file (js/JSON.stringify (clj->js replaced-lock))) + {:keys [status body]} (invoke-raw host port "thread-api/import-db-base64" [repo export-base64]) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] + (is (= 409 status)) + (is (= false (:ok parsed))) + (is (= "repo-locked" (get-in parsed [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 8d0c23c207..3ca8b1267f 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -66,6 +66,7 @@ (is (string/includes? plain-summary "show")) (is (string/includes? plain-summary "graph")) (is (string/includes? plain-summary "server")) + (is (string/includes? plain-summary "Path to db-worker data dir (default ~/logseq/graphs)")) (is (contains-bold? summary "list page")) (is (contains-bold? summary "list tag")) (is (contains-bold? summary "list property")) @@ -1305,6 +1306,7 @@ (async done (let [ops* (atom nil) calls* (atom []) + orig-list-graphs cli-server/list-graphs orig-ensure-server! cli-server/ensure-server! orig-resolve-tags add-command/resolve-tags orig-resolve-properties add-command/resolve-properties @@ -1319,6 +1321,7 @@ :remove-tags [:tag/old] :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} :remove-properties [:logseq.property/publishing-public?]}] + (set! cli-server/list-graphs (fn [_] ["demo"])) (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) (set! add-command/resolve-tags (fn [_ _ tags] (p/resolved (cond @@ -1356,6 +1359,7 @@ (p/catch (fn [e] (is false (str "unexpected error: " e " calls: " @calls*)))) (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) (set! cli-server/ensure-server! orig-ensure-server!) (set! add-command/resolve-tags orig-resolve-tags) (set! add-command/resolve-properties orig-resolve-properties) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 57d08c821c..a20df6fdb9 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -82,7 +82,7 @@ (let [result (config/resolve-config {}) expected-config-path (node-path/join (.homedir os) "logseq" "cli.edn")] (is (= expected-config-path (:config-path result))) - (is (= "~/logseq/cli-graphs" (:data-dir result))))) + (is (= "~/logseq/graphs" (:data-dir result))))) (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") diff --git a/src/test/logseq/cli/data_dir_test.cljs b/src/test/logseq/cli/data_dir_test.cljs index f1d97ab444..e17b5c3298 100644 --- a/src/test/logseq/cli/data_dir_test.cljs +++ b/src/test/logseq/cli/data_dir_test.cljs @@ -42,7 +42,7 @@ (is (= (node-path/resolve target) (:path data))))))))) (deftest normalize-data-dir-default - (testing "defaults to ~/logseq/cli-graphs" - (let [expected (node-path/resolve (node-path/join (.homedir os) "logseq" "cli-graphs")) + (testing "defaults to ~/logseq/graphs" + (let [expected (node-path/resolve (node-path/join (.homedir os) "logseq" "graphs")) resolved (data-dir/normalize-data-dir nil)] (is (= expected resolved))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index a6c0116b3b..9ef690009f 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -6,7 +6,7 @@ [cljs.test :refer [deftest is async]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] - [frontend.worker-common.util :as worker-util] + [frontend.worker.db-worker-node-lock :as db-lock] [logseq.cli.command.core :as command-core] [logseq.cli.command.show :as show-command] [logseq.cli.config :as cli-config] @@ -220,7 +220,7 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-graph-readonly") repo "readonly-graph" repo-id (command-core/resolve-repo repo) - repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo-id))] + repo-dir (db-lock/repo-dir data-dir repo-id)] (fs/mkdirSync repo-dir #js {:recursive true}) (fs/chmodSync repo-dir 365) (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -550,7 +550,8 @@ (deftest test-cli-show-properties-human-output (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-show-properties") - repo "show-properties-graph"] + repo "show-properties-graph" + repo-id (command-core/resolve-repo repo)] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) @@ -567,29 +568,30 @@ {:block/title "Sibling block"}]}]} wait-for-property (fn wait-for-property [attempt] (p/let [value (transport/invoke server :thread-api/q false - [repo ['[:find ?v . - :in $ ?title ?prop - :where - [?b :block/title ?title] - [?b ?prop ?v]] - "Property block" - property-ident]])] + [repo-id + ['[:find ?v . + :in $ ?title ?prop + :where + [?b :block/title ?title] + [?b ?prop ?v]] + "Property block" + property-ident]])] (if (or value (>= attempt 20)) value (p/let [_ (p/delay 100)] (wait-for-property (inc attempt)))))) _ (transport/invoke server :thread-api/apply-outliner-ops false - [repo [[:batch-import-edn [import-data {}]]] {}]) + [repo-id [[:batch-import-edn [import-data {}]]] {}]) _ (p/delay 100) _ (wait-for-property 0) page-name (common-util/page-name-sanity-lc "PropsPage") page-entity (transport/invoke server :thread-api/pull false - [repo [:db/id :block/name :block/title] [:block/name page-name]]) + [repo-id [:db/id :block/name :block/title] [:block/name page-name]]) _ (when-not (:db/id page-entity) (throw (ex-info "page not found in server" {:page page-name}))) show-config (assoc cfg :output-format :human) show-result (show-command/execute-show {:type :show - :repo repo + :repo repo-id :page page-name} show-config) output (get-in show-result [:data :message]) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index dd0482d5ac..bc0179d0d7 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -3,7 +3,6 @@ [frontend.test.node-helper :as node-helper] [logseq.cli.server :as cli-server] [promesa.core :as p] - [clojure.string :as string] ["fs" :as fs] ["path" :as node-path] ["child_process" :as child-process])) @@ -32,12 +31,19 @@ (.chdir js/process original-cwd) (set! (.-spawn child-process) original-spawn))))) +(deftest lock-path-uses-canonical-graph-dir + (let [data-dir "/tmp/logseq-db-worker" + repo "logseq_db_demo" + expected (node-path/join data-dir "demo" "db-worker.lock")] + (is (= expected (cli-server/lock-path data-dir repo))))) + (deftest ensure-server-repairs-stale-lock (async done (let [data-dir (node-helper/create-tmp-dir "cli-server") repo (str "logseq_db_stale_" (subs (str (random-uuid)) 0 8)) path (cli-server/lock-path data-dir repo) + cleanup-stale-lock! #'cli-server/cleanup-stale-lock! lock {:repo repo :pid (.-pid js/process) :host "127.0.0.1" @@ -45,13 +51,8 @@ :startedAt (.toISOString (js/Date.))}] (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) - (-> (p/let [cfg (cli-server/ensure-server! {:data-dir data-dir} repo) - _ (is (string/starts-with? (:base-url cfg) "http://127.0.0.1:")) - lock-data (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) - :keywordize-keys true) - _ (is (pos-int? (:port lock-data))) - stop-result (cli-server/stop-server! {:data-dir data-dir} repo)] - (is (:ok? stop-result)) + (-> (p/let [_ (cleanup-stale-lock! path lock)] + (is (not (fs/existsSync path))) (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) From a8b118a721c760fbe0dc04baea316c58034b4600 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Feb 2026 09:33:02 +0800 Subject: [PATCH 068/375] refactor(worker): extract graph dir key helper --- src/main/frontend/worker/db_core.cljs | 4 ++-- src/main/frontend/worker/db_worker_node_lock.cljs | 6 ++---- src/main/frontend/worker/graph_dir.cljs | 11 +++++++++++ src/test/frontend/worker/graph_dir_test.cljs | 11 +++++++++++ 4 files changed, 26 insertions(+), 6 deletions(-) create mode 100644 src/main/frontend/worker/graph_dir.cljs create mode 100644 src/test/frontend/worker/graph_dir_test.cljs diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 378b2c205c..3fed17c119 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -11,11 +11,11 @@ [frontend.common.graph-view :as graph-view] [frontend.common.missionary :as c.m] [frontend.common.thread-api :as thread-api :refer [def-thread-api]] + [frontend.worker.graph-dir :as graph-dir] [frontend.worker.platform :as platform] [frontend.worker-common.util :as worker-util] [frontend.worker.db-listener :as db-listener] [frontend.worker.db-metadata :as worker-db-metadata] - [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.db.fix :as db-fix] [frontend.worker.db.migrate :as db-migrate] [frontend.worker.db.validate :as worker-db-validate] @@ -73,7 +73,7 @@ (defn- storage-pool-name [graph] (if (node-runtime?) - (db-lock/repo->graph-dir-key graph) + (graph-dir/repo->graph-dir-key graph) (worker-util/get-pool-name graph))) (defn- get-storage-pool diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index e63ec8b3d6..77044572bf 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -4,6 +4,7 @@ ["os" :as os] ["path" :as node-path] [clojure.string :as string] + [frontend.worker.graph-dir :as graph-dir] [frontend.worker-common.util :as worker-util] [lambdaisland.glogi :as log] [logseq.common.config :as common-config] @@ -21,10 +22,7 @@ (defn repo->graph-dir-key [repo] - (when (seq repo) - (if (string/starts-with? repo common-config/db-version-prefix) - (subs repo (count common-config/db-version-prefix)) - repo))) + (graph-dir/repo->graph-dir-key repo)) (defn canonical-graph-dir-key? [graph-dir-key] diff --git a/src/main/frontend/worker/graph_dir.cljs b/src/main/frontend/worker/graph_dir.cljs new file mode 100644 index 0000000000..d23a22b56f --- /dev/null +++ b/src/main/frontend/worker/graph_dir.cljs @@ -0,0 +1,11 @@ +(ns frontend.worker.graph-dir + "Platform-agnostic graph directory naming helpers." + (:require [clojure.string :as string] + [logseq.common.config :as common-config])) + +(defn repo->graph-dir-key + [repo] + (when (seq repo) + (if (string/starts-with? repo common-config/db-version-prefix) + (subs repo (count common-config/db-version-prefix)) + repo))) diff --git a/src/test/frontend/worker/graph_dir_test.cljs b/src/test/frontend/worker/graph_dir_test.cljs new file mode 100644 index 0000000000..f9a62241c2 --- /dev/null +++ b/src/test/frontend/worker/graph_dir_test.cljs @@ -0,0 +1,11 @@ +(ns frontend.worker.graph-dir-test + (:require [cljs.test :refer [deftest is testing]] + [frontend.worker.graph-dir :as graph-dir])) + +(deftest repo->graph-dir-key-strips-db-prefix + (testing "db-prefixed repo is mapped to prefix-free graph dir key" + (is (= "demo" (graph-dir/repo->graph-dir-key "logseq_db_demo"))))) + +(deftest repo->graph-dir-key-keeps-prefix-free-name + (testing "prefix-free repo remains unchanged" + (is (= "demo" (graph-dir/repo->graph-dir-key "demo"))))) From 4c10100aa6bdb38cd5d205e86fe608b8fd040f3a Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Feb 2026 15:38:27 +0800 Subject: [PATCH 069/375] 031-logseq-cli-doctor-command.md --- .../031-logseq-cli-doctor-command.md | 183 ++++++++++++++++++ docs/cli/logseq-cli.md | 9 + src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/doctor.cljs | 140 ++++++++++++++ src/main/logseq/cli/commands.cljs | 8 +- src/main/logseq/cli/format.cljs | 32 ++- src/main/logseq/cli/server.cljs | 10 +- src/test/logseq/cli/command/doctor_test.cljs | 135 +++++++++++++ src/test/logseq/cli/commands_test.cljs | 15 ++ src/test/logseq/cli/format_test.cljs | 56 +++++- 10 files changed, 581 insertions(+), 9 deletions(-) create mode 100644 docs/agent-guide/031-logseq-cli-doctor-command.md create mode 100644 src/main/logseq/cli/command/doctor.cljs create mode 100644 src/test/logseq/cli/command/doctor_test.cljs diff --git a/docs/agent-guide/031-logseq-cli-doctor-command.md b/docs/agent-guide/031-logseq-cli-doctor-command.md new file mode 100644 index 0000000000..13cc97b068 --- /dev/null +++ b/docs/agent-guide/031-logseq-cli-doctor-command.md @@ -0,0 +1,183 @@ +# Logseq CLI Doctor Command Implementation Plan + +Goal: Add a `doctor` command that verifies logseq-cli runtime availability before normal command execution, including `db-worker-node.js` existence and `data-dir` read and write readiness. + +Architecture: Add a dedicated `logseq.cli.command.doctor` namespace and wire it into the existing `parse-args` -> `build-action` -> `execute` pipeline in `logseq.cli.commands`. +Architecture: Reuse existing helpers in `logseq.cli.data-dir` and `logseq.cli.server` for permission checks and daemon liveness probes, then return one structured diagnostics report. + +Tech Stack: ClojureScript, babashka.cli command table, Node.js `fs` and `path`, Promesa, existing CLI formatter and test harness. + +Related: Builds on `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`. +Related: Relates to `docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md`. +Related: Relates to `docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md`. +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. + +## Problem statement + +The current CLI fails only when a concrete command touches startup paths, so users discover environment problems late. + +We need a fast explicit health command that confirms whether logseq-cli can run reliably in the current machine context. + +The minimum required checks are the presence of `db-worker-node.js` and read and write access for `data-dir`. + +Note: `dist/db-worker-node.js` is a thin entry wrapper that loads `static/db-worker-node.js`. Doctor should validate the actual runtime target in `static/` rather than only the `dist/` wrapper. + +We should also surface practical runtime risks already modeled by current code, especially stale or unready db-worker instances discovered from lock files and health endpoints. + +This plan keeps scope to diagnostics and does not change daemon lifecycle semantics, lock protocol, or graph migration behavior. + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before adding implementation behavior. + +I will add parser and action-dispatch tests for `doctor` in `commands_test` so command discovery and help output are guarded. + +I will add dedicated `doctor` command tests that cover success, missing script file, and `data-dir` permission failure behavior. + +I will add `format` tests to ensure human and machine-readable output for `doctor` are stable and useful. + +I will run focused test namespaces first to validate RED and GREEN transitions, then run the full lint and test suite. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Required change | +|---|---|---| +| Runtime script path | `spawn-server!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` starts `../dist/db-worker-node.js`, which delegates to `../static/db-worker-node.js`, but no explicit diagnostic command validates that runtime target path readiness. | Add `doctor` check that validates the effective script file existence and readability before startup commands fail. | +| Data-dir readiness | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs` enforces directory creation and read or write access in `ensure-data-dir!`. | Reuse `ensure-data-dir!` inside `doctor` and report a dedicated failing check item. | +| Daemon liveness visibility | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` has `list-servers`, `server-status`, `ready?`, and `healthy?`, but no consolidated health summary command. | Add optional runtime checks in `doctor` that flag non-ready running servers discovered from lock files. | +| CLI discoverability | Top-level help and command table in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` do not include diagnostics entrypoint. | Add `doctor` to command entries and help summaries. | + +## Proposed doctor checks + +| Check id | Behavior | Existing helper to reuse | Failure signal | +|---|---|---|---| +| `db-worker-script` | Verify `../static/db-worker-node.js` exists and is readable as a file (and optionally verify `../dist/db-worker-node.js` wrapper exists). | New shared path helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` plus Node `fs` checks in doctor command. | `:doctor-script-missing` or `:doctor-script-unreadable`. | +| `data-dir` | Verify configured or default data dir can be created and is read and write accessible. | `logseq.cli.data-dir/ensure-data-dir!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs`. | Existing `:data-dir-permission` surfaced as doctor failure detail. | +| `running-servers` | Verify currently locked db-worker instances are reachable on readiness endpoint. | `logseq.cli.server/list-servers` status derivation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`. | `:doctor-server-not-ready` for any server reported as `:starting`. | + +## Integration sketch + +```text +logseq doctor + -> parse-args (commands table) + -> build-action {:type :doctor} + -> execute-doctor + 1) check effective db-worker-node.js runtime path (`static/db-worker-node.js`). + 2) check data-dir accessibility. + 3) inspect running server readiness. + -> format result for human/json/edn. +``` + +## Implementation plan + +### Phase 1: RED for command plumbing. + +1. Add failing assertions in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that top-level help includes `doctor` and bold-styled `doctor` command text. +2. Add a failing parse test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `commands/parse-args ["doctor"]` returning `:ok? true` with command `:doctor`. +3. Add a failing build-action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `{:command :doctor}` producing action type `:doctor`. +4. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm failures are specifically on new doctor assertions. + +### Phase 2: RED for doctor behavior. + +5. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` with namespace and fixtures consistent with existing command tests. +6. Add a failing test that marks script check as failed when `static/db-worker-node.js` path does not exist. +7. Add a failing test that marks data-dir check as failed when `ensure-data-dir!` throws `:data-dir-permission`. +8. Add a failing test that returns all checks passed when script and data-dir are both valid and no running server is unready. +9. Add a failing test that reports runtime warning or failure when `list-servers` includes entries with status `:starting`. +10. Run `bb dev:test -v 'logseq.cli.command.doctor-test'` and confirm all new tests fail for expected reasons. + +### Phase 3: GREEN for command integration. + +11. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` with `entries`, `build-action`, and `execute-doctor` returning structured check results. +12. Wire doctor namespace into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` requires and append `doctor-command/entries` into `table`. +13. Add `:doctor` branch in `build-action` inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +14. Add `:doctor` branch in `execute` inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +15. Update top-level command grouping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` to show `doctor` in help output. +16. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm doctor parse and help tests are green. + +### Phase 4: GREEN for doctor checks. + +17. Extract or add a shared db-worker script path helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so spawn and doctor share one source of truth. +18. Implement script existence and readability check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` using Node `fs` metadata checks. +19. Implement data-dir check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` by invoking `logseq.cli.data-dir/ensure-data-dir!`. +20. Implement running-server readiness check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` using `logseq.cli.server/list-servers`. +21. Return deterministic check ordering and include actionable message per check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs`. +22. Re-run `bb dev:test -v 'logseq.cli.command.doctor-test'` and confirm all doctor behavior tests are green. + +### Phase 5: RED and GREEN for formatting and docs. + +23. Add failing output tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human summary rendering of doctor checks. +24. Add failing output tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for json and edn output preserving structured check payload. +25. Implement doctor-specific human formatter in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +26. Ensure `doctor` output includes overall status and per-check status in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +27. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` with `doctor` command description, examples, and expected failure hints. +28. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm new doctor formatting tests are green. + +### Phase 6: Verify RED to GREEN cycle completion, refactor, and full validation. + +29. Run `bb dev:test -v 'logseq.cli.commands-test'` and ensure no regressions in help parsing and action dispatch. +30. Run `bb dev:test -v 'logseq.cli.command.doctor-test'` and ensure all doctor checks are behavior-driven and stable. +31. Run `bb dev:test -v 'logseq.cli.main-test'` to confirm entrypoint behavior remains compatible. +32. Run `bb dev:test -v 'logseq.cli.server-test'` to verify shared script path changes do not break server startup assumptions. +33. Run `bb dev:test -v 'logseq.cli.format-test'` to validate output contracts. +34. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +35. Review changed code against `@prompts/review.md` before merge. + +## Edge cases to cover + +| Scenario | Expected behavior | +|---|---| +| `static/db-worker-node.js` path exists but points to a directory. | `doctor` reports script check failure with explicit path and reason. | +| `data-dir` path points to a file. | `doctor` fails with `:data-dir-permission` detail and does not continue to misleading pass status. | +| `data-dir` is readable but not writable. | `doctor` fails data-dir check and returns actionable permission hint. | +| Running server lock exists but `/readyz` is not healthy. | `doctor` reports runtime check as failed for that repo. | +| No running server exists. | Runtime server check passes with empty server list and does not force daemon startup. | +| `--output json` is used. | Doctor returns stable machine-readable check list for scripts and automation. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.commands-test' +bb dev:test -v 'logseq.cli.command.doctor-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.main-test' +bb dev:lint-and-test +``` + +Each command should finish with zero failures and zero errors in GREEN phase. + +Each RED phase run should fail on newly added doctor assertions and not on unrelated setup errors. + +## Testing Details + +The tests focus on command behavior and diagnostics outcomes through public parser and executor boundaries. + +The tests avoid implementation-detail assertions and instead validate user-observable results for success and failure cases. + +The formatter tests ensure the same doctor payload is usable for both human troubleshooting and automation output modes. + +## Implementation Details + +- Keep `doctor` as a first-class command in the existing CLI command table. +- Reuse `ensure-data-dir!` instead of reimplementing permission checks. +- Reuse server health status discovery through existing `list-servers` behavior. +- Keep check execution deterministic and output stable for CI parsing. +- Keep command scope read-only for diagnostics and avoid auto-remediation side effects. +- Return explicit error codes for script and runtime health failures. +- Preserve current graph and repo naming semantics and lock protocol behavior. +- Add targeted formatter support so human output is concise and actionable. +- Verify all changes via focused tests before full lint and test pass. +- Follow `@test-driven-development` and `@prompts/review.md` throughout implementation. + +## Question + +Resolved: `doctor` will fail fast on the first failed check. + +Resolved: `doctor` will treat `:starting` servers as warnings when script and data-dir checks pass. + +Resolved: `doctor` will support a future `--repo` scoped deep check that verifies per-graph lock path and repo directory access without starting the daemon. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index c3e3b51039..bb10461a59 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -71,6 +71,7 @@ Server commands: - `server start --repo ` - start db-worker-node for a graph - `server stop --repo ` - stop db-worker-node for a graph - `server restart --repo ` - restart db-worker-node for a graph +- `doctor` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages @@ -120,6 +121,12 @@ Output formats: - Global `--output ` applies to all commands - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- `doctor` output includes overall status (`ok`, `warning`, `error`) and per-check rows for `db-worker-script`, `data-dir`, and `running-servers`. For scripting, `--output json|edn` keeps the structured check payload. +- Common doctor failures: + - `doctor-script-missing`: `db-worker-node.js` runtime target is missing (typically `static/db-worker-node.js`; `dist/db-worker-node.js` is only the wrapper entry). + - `doctor-script-unreadable`: script path exists but is not a readable file. + - `data-dir-permission`: configured data dir is not readable or writable. + - `doctor-server-not-ready`: one or more lock-discovered servers are still in `:starting` state (warning). - `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. - Built-in named queries currently include `block-search`, `task-search`, `recent-updated`, `list-status`, and `list-priority`. Use `query list` to see the full set for your config. - Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. @@ -147,4 +154,6 @@ node ./dist/logseq.js move --uuid --target-page TargetPage node ./dist/logseq.js search "hello" node ./dist/logseq.js show --page TestPage --output json node ./dist/logseq.js server list +node ./dist/logseq.js doctor +node ./dist/logseq.js doctor --output json ``` diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index fc8f6908e0..94ab0db180 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -95,7 +95,7 @@ (let [groups [{:title "Graph Inspect and Edit" :commands #{"list" "add" "remove" "update" "query" "show"}} {:title "Graph Management" - :commands #{"graph" "server"}}] + :commands #{"graph" "server" "doctor"}}] render-group (fn [{:keys [title commands]}] (let [entries (filter #(contains? commands (first (:cmds %))) table)] (string/join "\n" [title (format-commands entries)])))] diff --git a/src/main/logseq/cli/command/doctor.cljs b/src/main/logseq/cli/command/doctor.cljs new file mode 100644 index 0000000000..001545c896 --- /dev/null +++ b/src/main/logseq/cli/command/doctor.cljs @@ -0,0 +1,140 @@ +(ns logseq.cli.command.doctor + "Doctor command for CLI runtime diagnostics." + (:require ["fs" :as fs] + [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.data-dir :as data-dir] + [logseq.cli.server :as cli-server] + [promesa.core :as p])) + +(def entries + [(core/command-entry ["doctor"] :doctor "Run runtime diagnostics" {})]) + +(defn build-action + [] + {:ok? true + :action {:type :doctor}}) + +(defn- doctor-error + [checks code message] + {:status :error + :error {:code code + :message message + :checks checks} + :data {:status :error + :checks checks}}) + +(defn- check-db-worker-script + [action] + (let [path (or (:script-path action) + (cli-server/db-worker-runtime-script-path))] + (try + (cond + (not (fs/existsSync path)) + {:ok? false + :check {:id :db-worker-script + :status :error + :code :doctor-script-missing + :path path + :message (str "db-worker script is missing: " path)}} + + :else + (let [stat (fs/statSync path)] + (if-not (.isFile stat) + {:ok? false + :check {:id :db-worker-script + :status :error + :code :doctor-script-unreadable + :path path + :message (str "db-worker script path is not a file: " path)}} + (let [constants (.-constants fs)] + (fs/accessSync path (.-R_OK constants)) + {:ok? true + :check {:id :db-worker-script + :status :ok + :path path + :message (str "Found readable file: " path)}})))) + (catch :default e + {:ok? false + :check {:id :db-worker-script + :status :error + :code :doctor-script-unreadable + :path path + :cause (.-code e) + :message (str "db-worker script is not readable: " path)}})))) + +(defn- check-data-dir + [config] + (try + (let [path (data-dir/ensure-data-dir! (:data-dir config))] + {:ok? true + :check {:id :data-dir + :status :ok + :path path + :message (str "Read/write access confirmed: " path)}}) + (catch :default e + (let [data (ex-data e) + code (or (:code data) :data-dir-permission) + path (or (:path data) (:data-dir config)) + message (or (.-message e) + "data-dir check failed")] + {:ok? false + :check {:id :data-dir + :status :error + :code code + :path path + :cause (:cause data) + :message message}})))) + +(defn- check-running-servers + [config] + (-> (p/let [servers (or (cli-server/list-servers config) []) + starting (vec (filter #(= :starting (:status %)) servers))] + (if (seq starting) + {:ok? true + :warning? true + :check {:id :running-servers + :status :warning + :code :doctor-server-not-ready + :servers starting + :message (str (count starting) + " server" + (when (> (count starting) 1) "s") + " still starting: " + (string/join ", " (map :repo starting)))}} + {:ok? true + :warning? false + :check {:id :running-servers + :status :ok + :servers servers + :message (if (seq servers) + "All running servers are ready" + "No running db-worker servers detected")}})) + (p/catch (fn [e] + {:ok? false + :check {:id :running-servers + :status :error + :code :doctor-server-check-failed + :message (or (.-message e) + "running server check failed")}})))) + +(defn execute-doctor + [action config] + (p/let [script-check (check-db-worker-script action)] + (if-not (:ok? script-check) + (let [check (:check script-check)] + (doctor-error [check] (:code check) (:message check))) + (let [checks [(:check script-check)] + data-dir-check (check-data-dir config)] + (if-not (:ok? data-dir-check) + (let [check (:check data-dir-check) + checks (conj checks check)] + (doctor-error checks (:code check) (:message check))) + (p/let [server-check (check-running-servers config) + checks (conj checks (:check data-dir-check) (:check server-check))] + (if-not (:ok? server-check) + (let [check (:check server-check)] + (doctor-error checks (:code check) (:message check))) + {:status :ok + :data {:status (if (:warning? server-check) :warning :ok) + :checks checks}}))))))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 9f76fe29d0..cde338c4ec 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -4,6 +4,7 @@ [clojure.string :as string] [logseq.cli.command.add :as add-command] [logseq.cli.command.core :as command-core] + [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] [logseq.cli.command.list :as list-command] [logseq.cli.command.query :as query-command] @@ -101,7 +102,8 @@ remove-command/entries update-command/entries query-command/entries - show-command/entries))) + show-command/entries + doctor-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -375,6 +377,9 @@ :show (show-command/build-action options repo) + :doctor + (doctor-command/build-action) + {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) @@ -410,6 +415,7 @@ :query (query-command/execute-query action config) :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) + :doctor (doctor-command/execute-doctor action config) :server-list (server-command/execute-list action config) :server-status (server-command/execute-status action config) :server-start (server-command/execute-start action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index c03b804204..d5d40642aa 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -15,7 +15,7 @@ value)) (defn- ->json - [{:keys [status data error]}] + [{:keys [status data error command]}] (let [obj (js-obj)] (set! (.-status obj) (name status)) (cond @@ -23,7 +23,10 @@ (set! (.-data obj) (clj->js (normalize-json data))) (= status :error) - (set! (.-error obj) (clj->js (normalize-json (update error :code name))))) + (do + (set! (.-error obj) (clj->js (normalize-json (update error :code name)))) + (when (and (= :doctor command) (some? data)) + (set! (.-data obj) (clj->js (normalize-json data)))))) (js/JSON.stringify obj))) (defn- pad-right @@ -277,6 +280,18 @@ "updated")] (str "Graph " verb ": " graph))) +(defn- format-doctor + [status checks] + (let [header (str "Doctor: " (name (or status :unknown))) + check-lines (mapv (fn [{:keys [id status message]}] + (str "[" (name (or status :unknown)) + "] " + (name (or id :unknown)) + (when (seq message) + (str " - " message)))) + (or checks []))] + (string/join "\n" (into [header] check-lines)))) + (defn- ->human [{:keys [status data error command context]} {:keys [now-ms]}] (let [now-ms (or now-ms (js/Date.now))] @@ -302,20 +317,27 @@ :query (format-query-results (:result data)) :query-list (format-query-list (:queries data)) :show (or (:message data) (pr-str data)) + :doctor (format-doctor (:status data) (:checks data)) (if (and (map? data) (contains? data :message)) (:message data) (pr-str data))) :error - (format-error error) + (if (= :doctor command) + (format-doctor (or (get-in data [:status]) :error) + (or (get-in data [:checks]) + (get-in error [:checks]))) + (format-error error)) (pr-str {:status status :data data :error error})))) (defn- ->edn - [{:keys [status data error]}] + [{:keys [status data error command]}] (pr-str (cond-> {:status status} (= status :ok) (assoc :data data) - (= status :error) (assoc :error error)))) + (= status :error) (assoc :error error) + (and (= status :error) (= :doctor command) (some? data)) + (assoc :data data)))) (defn- normalize-graph-result [result] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 1bda79284b..3bea561842 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -51,6 +51,14 @@ [data-dir repo] (node-path/join (repo-dir data-dir repo) "db-worker.lock")) +(defn db-worker-script-path + [] + (node-path/join js/__dirname "../dist/db-worker-node.js")) + +(defn db-worker-runtime-script-path + [] + (node-path/join js/__dirname "../static/db-worker-node.js")) + (defn- pid-status [pid] (when (number? pid) @@ -205,7 +213,7 @@ (defn- spawn-server! [{:keys [repo data-dir]}] - (let [script (node-path/join js/__dirname "../dist/db-worker-node.js") + (let [script (db-worker-script-path) args #js ["--repo" repo "--data-dir" data-dir] child (.spawn child-process script args #js {:detached true :stdio "ignore"})] diff --git a/src/test/logseq/cli/command/doctor_test.cljs b/src/test/logseq/cli/command/doctor_test.cljs new file mode 100644 index 0000000000..b785083125 --- /dev/null +++ b/src/test/logseq/cli/command/doctor_test.cljs @@ -0,0 +1,135 @@ +(ns logseq.cli.command.doctor-test + (:require [cljs.test :refer [async deftest is]] + [clojure.string :as string] + [logseq.cli.commands :as commands] + [logseq.cli.data-dir :as data-dir] + [logseq.cli.server :as cli-server] + [promesa.core :as p])) + +(deftest test-execute-doctor-script-missing + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers + ensure-data-dir-called? (atom false) + list-servers-called? (atom false)] + (set! data-dir/ensure-data-dir! (fn [_] + (reset! ensure-data-dir-called? true) + "/tmp/logseq-doctor")) + (set! cli-server/list-servers (fn [_] + (reset! list-servers-called? true) + (p/resolved []))) + (-> (p/let [result (commands/execute {:type :doctor + :script-path "/tmp/logseq-cli-missing-db-worker-node.js"} + {})] + (is (= :error (:status result))) + (is (= :doctor-script-missing (get-in result [:error :code]))) + (is (= :db-worker-script + (get-in result [:error :checks 0 :id]))) + (is (= :error + (get-in result [:error :checks 0 :status]))) + (is (false? @ensure-data-dir-called?)) + (is (false? @list-servers-called?))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) + +(deftest test-execute-doctor-data-dir-permission + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers + list-servers-called? (atom false)] + (set! data-dir/ensure-data-dir! (fn [_] + (throw (ex-info "data-dir is not readable/writable: /tmp/nope" + {:code :data-dir-permission + :path "/tmp/nope" + :cause "EACCES"})))) + (set! cli-server/list-servers (fn [_] + (reset! list-servers-called? true) + (p/resolved []))) + (-> (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/nope"})] + (is (= :error (:status result))) + (is (= :data-dir-permission (get-in result [:error :code]))) + (is (= [:db-worker-script :data-dir] + (mapv :id (get-in result [:error :checks])))) + (is (= [:ok :error] + (mapv :status (get-in result [:error :checks])))) + (is (false? @list-servers-called?))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) + +(deftest test-execute-doctor-all-checks-pass + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers] + (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) + (set! cli-server/list-servers (fn [_] (p/resolved []))) + (-> (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/logseq-doctor"})] + (is (= :ok (:status result))) + (is (= :ok (get-in result [:data :status]))) + (is (= [:db-worker-script :data-dir :running-servers] + (mapv :id (get-in result [:data :checks])))) + (is (= [:ok :ok :ok] + (mapv :status (get-in result [:data :checks]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) + +(deftest test-execute-doctor-starting-server-warning + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers] + (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) + (set! cli-server/list-servers (fn [_] + (p/resolved [{:repo "logseq_db_demo" + :status :starting + :host "127.0.0.1" + :port 9010}]))) + (-> (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/logseq-doctor"})] + (is (= :ok (:status result))) + (is (= :warning (get-in result [:data :status]))) + (is (= :running-servers + (get-in result [:data :checks 2 :id]))) + (is (= :warning + (get-in result [:data :checks 2 :status]))) + (is (= :doctor-server-not-ready + (get-in result [:data :checks 2 :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) + +(deftest test-execute-doctor-default-script-checks-static-runtime-target + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers] + (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) + (set! cli-server/list-servers (fn [_] (p/resolved []))) + (-> (p/let [result (commands/execute {:type :doctor} + {:data-dir "/tmp/logseq-doctor"}) + checked-path (get-in result [:data :checks 0 :path])] + (is (= :ok (:status result))) + (is (string/ends-with? checked-path "/static/db-worker-node.js"))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 3ca8b1267f..7f19013492 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -64,6 +64,7 @@ (is (string/includes? plain-summary "update")) (is (string/includes? plain-summary "query")) (is (string/includes? plain-summary "show")) + (is (string/includes? plain-summary "doctor")) (is (string/includes? plain-summary "graph")) (is (string/includes? plain-summary "server")) (is (string/includes? plain-summary "Path to db-worker data dir (default ~/logseq/graphs)")) @@ -77,6 +78,7 @@ (is (contains-bold? summary "query")) (is (contains-bold? summary "query list")) (is (contains-bold? summary "show")) + (is (contains-bold? summary "doctor")) (is (contains-bold? summary "graph list")) (is (contains-bold? summary "graph create")) (is (contains-bold? summary "server list")) @@ -284,6 +286,12 @@ (is (true? (:ok? result))) (is (= "json" (get-in result [:options :output])))))) +(deftest test-parse-args-doctor + (testing "doctor command parses" + (let [result (commands/parse-args ["doctor"])] + (is (true? (:ok? result))) + (is (= :doctor (:command result)))))) + (deftest test-tree->text-format (testing "show tree text uses db/id with tree glyphs" (let [tree->text #'show-command/tree->text @@ -1139,6 +1147,13 @@ (is (true? (:ok? result))) (is (= :server-stop (get-in result [:action :type])))))) +(deftest test-build-action-doctor + (testing "doctor builds action" + (let [parsed {:ok? true :command :doctor :options {}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :doctor (get-in result [:action :type])))))) + (deftest test-build-action-inspect-edit (testing "list page requires repo" (let [parsed {:ok? true :command :list-page :options {}} diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 429fc24a9c..9da6987cb3 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -1,5 +1,6 @@ (ns logseq.cli.format-test - (:require [cljs.test :refer [deftest is testing]] + (:require [cljs.reader :as reader] + [cljs.test :refer [deftest is testing]] [clojure.string :as string] [logseq.cli.command.show :as show-command] [logseq.cli.format :as format] @@ -307,3 +308,56 @@ (is (= (str "Error (missing-graph): graph name is required\n" "Hint: Use --repo ") result))))) + +(deftest test-human-output-doctor + (testing "doctor renders concise check summary" + (let [result (format/format-result {:status :ok + :command :doctor + :data {:status :warning + :checks [{:id :db-worker-script + :status :ok + :message "Found readable file: /tmp/db-worker-node.js"} + {:id :data-dir + :status :ok + :message "Read/write access confirmed: /tmp/logseq/graphs"} + {:id :running-servers + :status :warning + :message "1 server is still starting"}]}} + {:output-format nil})] + (is (= (str "Doctor: warning\n" + "[ok] db-worker-script - Found readable file: /tmp/db-worker-node.js\n" + "[ok] data-dir - Read/write access confirmed: /tmp/logseq/graphs\n" + "[warning] running-servers - 1 server is still starting") + result))))) + +(deftest test-doctor-json-edn-output + (testing "doctor json and edn keep structured checks for failed runs" + (let [payload {:checks [{:id :db-worker-script + :status :ok + :message "Found readable file: /tmp/db-worker-node.js"} + {:id :data-dir + :status :error + :code :data-dir-permission + :message "data-dir is not readable/writable: /tmp/logseq"}]} + json-result (format/format-result {:status :error + :command :doctor + :error {:code :data-dir-permission + :message "data-dir check failed"} + :data payload} + {:output-format :json}) + edn-result (format/format-result {:status :error + :command :doctor + :error {:code :data-dir-permission + :message "data-dir check failed"} + :data payload} + {:output-format :edn}) + parsed-json (js->clj (js/JSON.parse json-result) :keywordize-keys true) + parsed-edn (reader/read-string edn-result)] + (is (= "error" (:status parsed-json))) + (is (= "data-dir-permission" (get-in parsed-json [:error :code]))) + (is (= "data-dir" (get-in parsed-json [:data :checks 1 :id]))) + (is (= "error" (get-in parsed-json [:data :checks 1 :status]))) + (is (= :error (:status parsed-edn))) + (is (= :data-dir-permission (get-in parsed-edn [:error :code]))) + (is (= :data-dir (get-in parsed-edn [:data :checks 1 :id]))) + (is (= :error (get-in parsed-edn [:data :checks 1 :status])))))) From 0b377364aefea9dbd0954b20dc404e11403a3220 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 8 Feb 2026 20:52:12 +0800 Subject: [PATCH 070/375] 032-logseq-cli-show-property-key-bold.md --- .../032-logseq-cli-show-property-key-bold.md | 119 ++++++++++++++++++ src/main/logseq/cli/command/show.cljs | 13 +- src/test/logseq/cli/commands_test.cljs | 13 +- src/test/logseq/cli/format_test.cljs | 28 +++++ 4 files changed, 163 insertions(+), 10 deletions(-) create mode 100644 docs/agent-guide/032-logseq-cli-show-property-key-bold.md diff --git a/docs/agent-guide/032-logseq-cli-show-property-key-bold.md b/docs/agent-guide/032-logseq-cli-show-property-key-bold.md new file mode 100644 index 0000000000..703f9a84d5 --- /dev/null +++ b/docs/agent-guide/032-logseq-cli-show-property-key-bold.md @@ -0,0 +1,119 @@ +# Logseq CLI Show Property Key Bold Implementation Plan + +Goal: Make `` render in bold in `show` human output while keeping the existing `property-kvs` layout and values unchanged. + +Architecture: Keep db-worker-node data fetching and property title resolution unchanged, and only add styling at the final text rendering layer in `logseq.cli.command.show`. + +Tech Stack: ClojureScript, logseq-cli show renderer, db-worker-node transport/thread-api, cljs.test. + +Related: Relates to `docs/agent-guide/029-logseq-cli-show-properties.md` and `docs/agent-guide/023-logseq-cli-help-show-styling.md`. + +## Problem statement + +The current `show` human output prints property lines as plain text in the form `: `. + +`property-kvs` are already rendered in the correct position and indentation under each block, but `` does not have visual emphasis. + +The requested behavior is to bold only `` while preserving `:`, value text, multiline/list formatting, and tree alignment. + +Because logseq-cli `show` depends on db-worker-node data, this change must not alter db-worker-node API contracts, pull results, or non-human output formats. + +## Testing Plan + +I will add a unit test that verifies a single-value property line styles only the property key with ANSI bold when color is enabled. + +I will add a unit test that verifies multi-value property blocks style the property key heading in bold while list item rows remain non-bold. + +I will update existing property rendering tests to keep `strip-ansi` expectations unchanged so layout behavior is locked. + +I will add a format-level test to ensure human `show` output preserves the new bold key styling through `format/format-result`. + +I will run one integration test for show properties to confirm db-worker-node end-to-end behavior remains unchanged after ANSI stripping. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +In scope is styling `` in human output generated by `show`. + +Out of scope is changing property query logic, property sorting rules, property value normalization, and db-worker-node request/response schemas. + +Out of scope is any change to JSON or EDN output. + +## Files to touch + +| File path | Purpose | +| --- | --- | +| `src/main/logseq/cli/command/show.cljs` | Apply bold style to `` in property line formatter only. | +| `src/test/logseq/cli/commands_test.cljs` | Add renderer-focused tests for ANSI bold presence and unchanged stripped layout. | +| `src/test/logseq/cli/format_test.cljs` | Verify formatted human output keeps bold property-key styling. | +| `src/test/logseq/cli/integration_test.cljs` | Confirm end-to-end show property output remains stable after strip-ansi. | + +## Implementation plan + +1. Follow `@test-driven-development` and `@prompts/review.md` before code edits. +2. Add failing renderer unit tests in `src/test/logseq/cli/commands_test.cljs` for bold `` in single and multi-value property output. +3. Run targeted tests and confirm they fail for missing styling behavior and not for test setup errors. +4. Update `format-property-lines` in `src/main/logseq/cli/command/show.cljs` to wrap only the property title segment with `style/bold`. +5. Keep `indent`, colon placement, and value rendering unchanged so current tree alignment remains intact. +6. Add or update `src/test/logseq/cli/format_test.cljs` to ensure `format/format-result` does not strip the new bold style in human output. +7. Run targeted unit tests again and confirm all new assertions pass. +8. Run the existing integration test that covers show properties and confirm stripped output still matches the same textual content. +9. Run full lint and unit test suite for regression coverage. + +## Verification commands + +```bash +bb dev:test -v 'logseq.cli.commands-test/test-tree->text-renders-properties-single-value' +bb dev:test -v 'logseq.cli.commands-test/test-tree->text-renders-properties-multi-value' +bb dev:test -v 'logseq.cli.format-test/test-human-output-show' +bb dev:test -v 'logseq.cli.integration-test/test-cli-show-properties-human-output' +bb dev:lint-and-test +``` + +Expected outcome is that new property-key bold tests pass, pre-existing strip-ansi assertions remain unchanged, integration behavior remains stable, and the full lint/test command exits successfully. + +## Edge cases + +When color is disabled, output should remain plain text and still include `: ` with no ANSI codes. + +For multiline block titles, property-key bold styling must not shift spacing or tree glyph alignment. + +For multi-value properties, only the heading key line should be bold and bullet item values should remain unchanged. + +If a property title is missing or blank, existing skip behavior should remain unchanged. + +## Rollout and risk + +Risk is low because the change is limited to a string formatting helper used only by human output. + +The main regression risk is accidental styling leakage into values or alignment shifts caused by ANSI code placement. + +This risk is controlled by preserving existing strip-ansi golden assertions and adding explicit ANSI presence tests. + +## Testing Details + +The tests verify behavior at three levels, which are renderer output, formatter passthrough, and db-worker-node integration stability. + +Renderer tests assert exact ANSI bold placement around property keys and unchanged plain text after stripping ANSI. + +Formatter tests confirm human output still carries ANSI styling while JSON and EDN behavior is unchanged. + +Integration coverage confirms end-to-end property visibility still works with the same stripped output content. + +## Implementation Details +- Reuse `logseq.cli.style/bold` instead of introducing a new styling helper. +- Change only `format-property-lines` and avoid modifying property discovery helpers. +- Keep sorted property order logic exactly as-is. +- Keep `property-value->string` and `normalize-property-values` untouched. +- Preserve the existing list format for multi-value properties. +- Preserve root and child indentation prefixes from `tree->text`. +- Avoid any db-worker-node API or transport changes. +- Ensure ANSI checks in tests use existing helpers like `style/strip-ansi` or regex. +- Keep all JSON and EDN output paths unchanged. + +## Question + +Only the heading key should be considered the full `` target, so `Criteria` should be bold while `- One` and `- Two` should remain non-bold. + +--- diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 95c7d72c5f..a22aebdd2c 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -232,12 +232,13 @@ (defn- format-property-lines [indent title values] - (when (seq values) - (if (= 1 (count values)) - [(str indent title ": " (first values))] - (let [item-indent (str indent " ")] - (into [(str indent title ":")] - (map #(str item-indent "- " %) values)))))) + (let [title* (style/bold title)] + (when (seq values) + (if (= 1 (count values)) + [(str indent title* ": " (first values))] + (let [item-indent (str indent " ")] + (into [(str indent title* ":")] + (map #(str item-indent "- " %) values))))))) (defn- node-property-lines [node property-titles property-value-labels indent] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 7f19013492..4fa96bcbad 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -386,6 +386,7 @@ :property-titles {:user.property/background "Background"}} output (binding [style/*color-enabled?* true] (tree->text tree-data))] + (is (contains-bold? output "Background")) (is (= (str "1 Root\n" "2 ├── Child A\n" " │ Background: Because\n" @@ -406,6 +407,9 @@ :property-titles {:user.property/criteria "Criteria"}} output (binding [style/*color-enabled?* true] (tree->text tree-data))] + (is (contains-bold? output "Criteria")) + (is (not (contains-bold? output "- One"))) + (is (not (contains-bold? output "- Two"))) (is (= (str "1 Root\n" "2 ├── Child A\n" " │ Criteria:\n" @@ -459,10 +463,11 @@ :user.property/background "Child"} :property-titles {:user.property/background "Background"}} output (binding [style/*color-enabled?* true] - (tree->text tree-data))] - (is (string/includes? output "Background: Child")) - (is (not (string/includes? output "└── Child"))) - (is (not (string/includes? output "├── Child")))))) + (tree->text tree-data)) + output* (strip-ansi output)] + (is (string/includes? output* "Background: Child")) + (is (not (string/includes? output* "└── Child"))) + (is (not (string/includes? output* "├── Child")))))) (deftest test-tree->text-prefixes-status (testing "show tree text prefixes status before block titles" diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 9da6987cb3..bfd3aec065 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -243,6 +243,34 @@ "2 └── TODO Child #TagA") (style/strip-ansi result)))))) +(deftest test-human-output-show-styled-property-keys + (testing "show preserves styled property keys in human output" + (let [tree->text #'show-command/tree->text + tree-data {:root {:db/id 1 + :block/title "Root" + :block/children [{:db/id 2 + :block/title "Child" + :user.property/acceptance-criteria ["One" "Two"]} + {:db/id 3 + :block/title "Sibling"}]} + :property-titles {:user.property/acceptance-criteria "Acceptance Criteria"}} + styled (binding [style/*color-enabled?* true] + (tree->text tree-data)) + result (format/format-result {:status :ok + :command :show + :data {:message styled}} + {:output-format nil})] + (is (re-find #"\u001b\[[0-9;]*mAcceptance Criteria\u001b\[[0-9;]*m" result)) + (is (not (re-find #"\u001b\[[0-9;]*m- One\u001b\[[0-9;]*m" result))) + (is (not (re-find #"\u001b\[[0-9;]*m- Two\u001b\[[0-9;]*m" result))) + (is (= (str "1 Root\n" + "2 ├── Child\n" + " │ Acceptance Criteria:\n" + " │ - One\n" + " │ - Two\n" + "3 └── Sibling") + (style/strip-ansi result)))))) + (deftest test-human-output-show-preserves-styling (testing "show returns styled text without stripping ANSI" (let [tree->text #'show-command/tree->text From cafe75509199705e95f6da9f03174926d23de440 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 11 Feb 2026 22:29:24 +0800 Subject: [PATCH 071/375] refactor: desktop app use db-worker-node as well see also: 033-desktop-db-worker-node-backend.md --- .../033-desktop-db-worker-node-backend.md | 222 ++++++++++++++++++ docs/cli/logseq-cli.md | 6 + src/electron/electron/core.cljs | 4 +- src/electron/electron/db.cljs | 2 + src/electron/electron/db_worker.cljs | 184 +++++++++++++++ src/electron/electron/handler.cljs | 74 ++++-- src/electron/electron/window.cljs | 2 + src/main/electron/ipc.cljs | 15 +- src/main/frontend/handler.cljs | 5 +- src/main/frontend/handler/events.cljs | 11 +- .../frontend/modules/shortcut/config.cljs | 2 +- src/main/frontend/persist_db.cljs | 93 ++++++-- src/main/frontend/persist_db/browser.cljs | 5 +- src/main/frontend/persist_db/remote.cljs | 202 ++++++++++++++++ src/main/frontend/worker/db_worker_node.cljs | 29 ++- src/main/logseq/cli/server.cljs | 186 +++------------ src/main/logseq/db_worker/daemon.cljs | 153 ++++++++++++ src/test/electron/db_worker_manager_test.cljs | 141 +++++++++++ src/test/frontend/electron/ipc_test.cljs | 46 ++++ .../modules/shortcut/config_test.cljs | 14 ++ src/test/frontend/persist_db/remote_test.cljs | 100 ++++++++ src/test/frontend/persist_db_test.cljs | 173 ++++++++++++++ .../frontend/worker/db_worker_node_test.cljs | 113 +++++++++ src/test/logseq/cli/server_test.cljs | 75 ++++++ src/test/logseq/db_worker/daemon_test.cljs | 47 ++++ 25 files changed, 1709 insertions(+), 195 deletions(-) create mode 100644 docs/agent-guide/033-desktop-db-worker-node-backend.md create mode 100644 src/electron/electron/db_worker.cljs create mode 100644 src/main/frontend/persist_db/remote.cljs create mode 100644 src/main/logseq/db_worker/daemon.cljs create mode 100644 src/test/electron/db_worker_manager_test.cljs create mode 100644 src/test/frontend/electron/ipc_test.cljs create mode 100644 src/test/frontend/modules/shortcut/config_test.cljs create mode 100644 src/test/frontend/persist_db/remote_test.cljs create mode 100644 src/test/frontend/persist_db_test.cljs create mode 100644 src/test/logseq/db_worker/daemon_test.cljs diff --git a/docs/agent-guide/033-desktop-db-worker-node-backend.md b/docs/agent-guide/033-desktop-db-worker-node-backend.md new file mode 100644 index 0000000000..c6261231df --- /dev/null +++ b/docs/agent-guide/033-desktop-db-worker-node-backend.md @@ -0,0 +1,222 @@ +# Desktop Db Worker Node Backend Implementation Plan + +Goal: Switch the Electron desktop app graph database backend from OPFS plus periodic disk export to db-worker-node with direct disk SQLite access, so desktop app and logseq-cli can safely use the same data-dir at the same time. + +Architecture: Reuse existing `logseq.cli.server` daemon orchestration and lock semantics, and add an Electron main process graph-scoped daemon manager. + +Architecture: Replace the Electron renderer `PersistentDB` implementation from `frontend.persist-db.browser` to an HTTP plus SSE remote client that talks to `/v1/invoke` and `/v1/events` on db-worker-node. + +Tech Stack: ClojureScript, Electron main plus renderer, Node child_process, db-worker-node HTTP plus SSE API, Electron IPC with transit-json payloads, SQLite files under data-dir, lock files. + +Related: Builds on `docs/agent-guide/003-db-worker-node-cli-orchestration.md`. + +Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. + +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. + +## Problem statement + +The current desktop app uses an OPFS-backed SQLite worker in the renderer and periodically exports to disk through `persist-db/run-export-periodically!`. + +The current logseq-cli starts and connects to db-worker-node, and directly reads and writes SQLite files in data-dir. + +This split creates two write paths with eventual synchronization, and desktop plus CLI concurrent usage depends on export timing instead of one shared lock-governed write path. + +The goal is to make desktop and CLI share db-worker-node semantics and lock behavior so disk SQLite becomes the single source of truth. + +The desktop default DB graphs directory is `~/logseq/graphs`, defined in `deps/cli/src/logseq/cli/common/graph.cljs`, and used by Electron DB file operations in `src/electron/electron/db.cljs`. + +This plan focuses on backend access flow and lifecycle management, and does not change business-level thread-api semantics or SQLite schema. + +## Testing Plan + +I will follow `@test-driven-development` for every phase and write failing tests before implementation changes. + +I will prioritize pure-function and dependency-injected tests so core behavior can be validated without launching a full Electron GUI. + +I will add Electron main db-worker manager lifecycle tests for first graph open, multi-window reuse, last-window close, and app shutdown cleanup. + +I will add remote client transport tests for invoke success, invoke failure propagation, SSE disconnect and reconnect, and missing auth token handling. + +I will extend db-worker-node tests with desktop plus CLI coexistence cases, including lock contention and stale lock recovery. + +I will run targeted tests first and then run `bb dev:lint-and-test`, and I will apply the review checklist in `@prompts/review.md`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Target implementation | +|---|---|---| +| Desktop graph DB runtime | Renderer uses `frontend.persist-db.browser` and an OPFS worker. | Renderer uses a remote `PersistentDB` client against db-worker-node. | +| Desktop persistence model | OPFS acts as source of truth and is periodically exported to disk through Electron IPC. | Disk SQLite in data-dir is the source of truth with no OPFS periodic export flow. | +| CLI persistence model | CLI starts db-worker-node and calls `/v1/invoke`. | Keep unchanged and align desktop with the same daemon semantics. | +| Lock and ownership | Desktop OPFS path bypasses db-worker-node lock semantics during in-memory writes. | Desktop and CLI both go through db-worker-node lock and single-writer enforcement. | +| Process lifecycle | Desktop has no graph-scoped daemon manager in main process. | Electron main manages per-graph daemon start, reuse, health checks, and stop. | + +## Integration sketch + +```text +Desktop Renderer + -> requests graph runtime from Electron Main via `electron.ipc/ipc` (transit-json) + -> receives {base-url, auth-token, repo} for db-worker-node + -> calls /v1/invoke and listens /v1/events through remote PersistentDB client + +Electron Main + -> on graph open: ensure db-worker-node started for graph in data-dir + -> on graph close or app quit: stop graph daemon when the last window exits + -> maintains graph -> daemon state cache + +Logseq CLI + -> uses existing logseq.cli.server ensure/start/stop + -> talks to the same graph data-dir and lock protocol + +db-worker-node + -> provides the single write path to disk SQLite files + -> enforces lock ownership and readiness checks +``` + +## Scope and non-goals + +In scope are Electron main daemon lifecycle management, renderer persistence client switch, OPFS export-path removal, and required tests plus docs. + +Out of scope are thread-api business behavior changes, SQLite schema changes, mobile behavior changes, and broad sync-system redesign. + +## Implementation plan + +### Phase 1: Add failing tests for the new desktop backend contract. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that verifies stable lock-conflict error reporting from daemon orchestration. +2. Add `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` and write a failing test for `ensure-started!` idempotency. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies one daemon is reused across multiple windows for the same graph. +4. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies stop on last-window close. +5. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies stop-all behavior on app quit. +6. Add `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` and write failing tests for invoke success and invoke error propagation. +7. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` for SSE parsing and reconnect behavior. +8. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm new assertions fail first. +9. Run `bb dev:test -v 'frontend.persist-db.remote-test'` and confirm new assertions fail first. + +### Phase 2: Extract shared daemon orchestration for CLI and Electron. + +10. Extract CLI-output-independent daemon orchestration logic from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`. +11. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` and move `spawn`, `wait-ready`, `read-lock`, and `cleanup-stale-lock` core functions into it. +12. Keep command-facing API shape in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` stable and delegate internally to the new helper. +13. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` with regression tests for unchanged CLI behavior. +14. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm green. + +### Phase 3: Implement Electron main db-worker manager. + +15. Add `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs` with graph-to-daemon state cache and reference counting. +16. Implement `ensure-started!` in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs` using the shared daemon helper and return `base-url` plus `auth-token`. +17. Implement `ensure-stopped!` and `stop-all!` in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs`. +18. Add `electron.ipc/ipc` handlers in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` for renderer requests of graph runtime configuration, encoded with transit-json. +19. Hook `stop-all!` into app lifecycle in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/core.cljs` at `before-quit` and `window-all-closed`. +20. Hook graph last-window stop logic into close flow in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/core.cljs`. +21. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm lifecycle tests pass. + +### Phase 4: Add Electron renderer remote PersistentDB client. + +22. Add `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` implementing `protocol/PersistentDB` with HTTP plus SSE transport. +23. Implement browser-safe invoke transport in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` and avoid Node-only `http` dependency in the renderer. +24. Implement event subscription and reconnect policy in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` compatible with current event handler signatures. +25. Extend runtime implementation selection in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db.cljs` to explicitly use remote client for Electron. +26. Add initialization flow in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db.cljs` that fetches runtime config from `electron.ipc/ipc` via transit-json before creating the remote client. +27. Update `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` to green after implementation. +28. Run `bb dev:test -v 'frontend.persist-db.remote-test'` and confirm green. + +### Phase 5: Replace OPFS startup path and remove periodic export workflow. + +29. Remove Electron-path dependency on `frontend.persist-db.browser/start-db-worker!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +30. Start remote `PersistentDB` initialization flow in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +31. Remove Electron-path invocation of `persist-db/run-export-periodically!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +32. Adjust `:graph/save-db-to-disk` behavior in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/events.cljs` to a no-op or user guidance path. +33. Adjust shortcut behavior for `:graph/db-save` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/modules/shortcut/config.cljs` so it no longer triggers legacy export flow. +34. Mark `:db-get` and `:db-export` `electron.ipc/ipc` endpoints in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` as compatibility-only or remove unused entry points. +35. Clean up OPFS-export-only logic in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db.cljs` while preserving needed utility paths. +36. Run `bb dev:test -v 'frontend.handler.route-test'` and `bb dev:test -v 'frontend.handler.common.config-edn-test'` for regression coverage. + +### Phase 6: Add desktop and CLI coexistence verification. + +37. Add concurrency tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` covering desktop plus CLI access to the same graph. +38. Add stale-lock recovery tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs`. +39. Add tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` for CLI reuse or conflict reporting when desktop already started the graph daemon. +40. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm green for coexistence tests. +41. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm green for coexistence tests. + +### Phase 7: Docs and rollout safety. + +42. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document shared db-worker-node semantics between desktop and CLI. +43. Add `/Users/rcmerci/gh-repos/logseq/docs/developers/desktop-db-worker-node.md` describing Electron main lifecycle and renderer remote-client init ordering. +44. Add release notes describing that Electron no longer uses OPFS as the primary database storage path. +45. Add rollback notes for a temporary fallback switch if emergency recovery is required. +46. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +47. Run a final review against `@prompts/review.md` and fix findings before merge. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Desktop opens a graph before daemon readiness completes. | Renderer waits for main-process ready runtime or receives a retryable error, and does not silently fall back to OPFS. | +| Desktop and CLI both attempt first start for the same graph daemon. | Exactly one owner acquires lock and the other path reuses the existing daemon or retries after a `:repo-locked` status check. | +| Graph name contains special characters. | Existing graph-dir encoding resolves to the same on-disk directory for desktop and CLI. | +| SSE connection drops. | Remote client reconnects and keeps event handling consistent, while invoke calls remain independent. | +| App exits abnormally and leaves a stale lock. | Next startup cleans stale lock via existing lock housekeeping logic without manual lock deletion. | +| Version switch from OPFS-backed desktop behavior. | No one-time migration is required because desktop already writes to disk data-dir, and startup verification should check disk DB readability before enabling the new backend. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:test -v 'frontend.persist-db.remote-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +clojure -M:cljs compile db-worker-node +bb dev:lint-and-test +``` + +Each test command should finish with `0 failures, 0 errors`. + +`clojure -M:cljs compile db-worker-node` should produce a runnable `static/db-worker-node.js`. + +In manual smoke tests, desktop and CLI reads and writes for the same graph should be immediately visible to each other without periodic export dependency. + +## Rollout strategy + +Phase one ships behind a feature flag and enables by default in development builds for coexistence and recovery telemetry. + +Phase two enables by default in stable builds and keeps a short-lived rollback switch. + +Phase three removes legacy OPFS export-path code and removes the rollback switch. + +## Clarity required before implementation + +Confirm product-level messaging for desktop and CLI lock-contention conflicts on the same graph. + +Confirm whether `:graph/db-save` is removed or redefined as a checkpoint action in the new model. + +Confirm rollback-switch lifecycle and target removal release. + +## Testing Details + +Tests focus on behavior, not implementation details, by validating daemon lifecycle, invoke responses, and event-stream consistency. + +Core coexistence tests validate lock ownership, recovery, and cross-client visibility for the same graph instead of mock call counts. + +Regression tests ensure existing CLI behavior stays stable and Electron startup plus shutdown does not leave zombie processes or lock files. + +## Implementation Details + +- Reuse and extract daemon orchestration from `logseq.cli.server` to avoid duplicate process-management logic. +- Add a dedicated Electron main db-worker manager with graph-scoped daemon state cache. +- Use a renderer remote `PersistentDB` client against db-worker-node HTTP plus SSE endpoints. +- Use transit-json for frontend and Electron communication through `electron.ipc/ipc` (see also `ldb/write-transit-str`). +- Remove periodic OPFS export workflow in Electron so disk SQLite is the only source of truth. +- Keep thread-api and database schema unchanged to limit application-level regressions. +- Keep lock-file and `:repo-locked` semantics identical for desktop and CLI. +- Add reconnect and stale-lock recovery tests to cover availability risks. +- Roll out with feature-flag phases and a short rollback window. +- Follow `@test-driven-development` and `@prompts/review.md` through implementation and verification. + +## Question + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index bb10461a59..1a611e209e 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -12,6 +12,12 @@ clojure -M:cljs compile logseq-cli db-worker-node `logseq` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. +Desktop + CLI shared semantics: +- Electron desktop and CLI are expected to use the same `db-worker-node` and lock-file protocol for a graph. +- Disk SQLite under `~/logseq/graphs` is the source of truth; OPFS periodic export is not part of the desktop primary write path. +- If a daemon already exists for the graph, CLI reuses it via lock-file discovery instead of starting a second writer. +- If lock ownership is invalid or stale, startup cleans stale lock state before retrying. + ## Run the CLI ```bash diff --git a/src/electron/electron/core.cljs b/src/electron/electron/core.cljs index 659afb3dea..4a856ef13a 100644 --- a/src/electron/electron/core.cljs +++ b/src/electron/electron/core.cljs @@ -294,7 +294,8 @@ :else nil))))) (.on app' "before-quit" (fn [_e] - (reset! win/*quitting? true))) + (reset! win/*quitting? true) + (handler/stop-all-db-workers!))) (.on app' "activate" #(when @*win (.show win))))))) @@ -326,6 +327,7 @@ (.on app "window-all-closed" (fn [] (logger/debug "window-all-closed" "Quitting...") + (handler/stop-all-db-workers!) (.quit app))) (on-app-ready! app)))) diff --git a/src/electron/electron/db.cljs b/src/electron/electron/db.cljs index 1208a14b05..934d338584 100644 --- a/src/electron/electron/db.cljs +++ b/src/electron/electron/db.cljs @@ -19,6 +19,7 @@ graph-dir)) (defn get-db + "Legacy compatibility path for Electron OPFS import bootstrap." [db-name] (let [_ (ensure-graph-dir! db-name) [_db-name db-path] (common-sqlite/get-db-full-path (cli-common-graph/get-db-graphs-dir) db-name)] @@ -26,6 +27,7 @@ (fs/readFileSync db-path)))) (defn save-db! + "Legacy compatibility path for Electron OPFS export." [db-name data] (let [[db-name db-path] (common-sqlite/get-db-full-path (cli-common-graph/get-db-graphs-dir) db-name) old-data (get-db db-name) diff --git a/src/electron/electron/db_worker.cljs b/src/electron/electron/db_worker.cljs new file mode 100644 index 0000000000..dc3e1eed0e --- /dev/null +++ b/src/electron/electron/db_worker.cljs @@ -0,0 +1,184 @@ +(ns electron.db-worker + (:require [logseq.cli.server :as cli-server] + [logseq.db-worker.daemon :as daemon] + [promesa.core :as p])) + +(defn- initial-state + [] + {:repos {} + :window->repo {}}) + +(defn- ensure-state + [state] + (merge (initial-state) state)) + +(defn- dissoc-window + [state window-id] + (update state :window->repo dissoc window-id)) + +(defn- detach-window + [state window-id] + (let [state (ensure-state state) + repo (get-in state [:window->repo window-id])] + (if-not repo + [state nil] + (let [entry (get-in state [:repos repo])] + (if-not entry + [(dissoc-window state window-id) nil] + (let [remaining (disj (:windows entry) window-id) + state' (cond-> (dissoc-window state window-id) + (seq remaining) + (assoc-in [:repos repo :windows] remaining) + + (empty? remaining) + (update :repos dissoc repo))] + [state' (when (empty? remaining) (:runtime entry))])))))) + +(defn- detach-window-from-repo + [state repo window-id] + (let [state (ensure-state state) + entry (get-in state [:repos repo])] + (if-not entry + [state nil] + (let [remaining (disj (:windows entry) window-id) + state' (cond-> state + true + (update :window->repo dissoc window-id) + + (seq remaining) + (assoc-in [:repos repo :windows] remaining) + + (empty? remaining) + (update :repos dissoc repo))] + [state' (when (empty? remaining) (:runtime entry))])))) + +(defn create-manager + [{:keys [start-daemon! stop-daemon! runtime-ready?] :as deps}] + {:deps deps + :start-daemon! start-daemon! + :stop-daemon! stop-daemon! + :runtime-ready? (or runtime-ready? (fn [_runtime] (p/resolved true))) + :state (atom (initial-state))}) + +(defn ensure-window-stopped! + [{:keys [state stop-daemon!]} window-id] + (let [runtime* (atom nil)] + (swap! state + (fn [current] + (let [[next-state runtime] (detach-window current window-id)] + (reset! runtime* runtime) + next-state))) + (if-let [runtime @runtime*] + (p/let [_ (stop-daemon! runtime)] + true) + (p/resolved false)))) + +(defn ensure-started! + [{:keys [state start-daemon! stop-daemon! runtime-ready?] :as manager} repo window-id] + (p/let [current-repo (get-in (ensure-state @state) [:window->repo window-id]) + _ (when (and current-repo (not= current-repo repo)) + (ensure-window-stopped! manager window-id))] + (if-let [entry (get-in (ensure-state @state) [:repos repo])] + (p/let [runtime (:runtime entry) + ready? (runtime-ready? runtime)] + (if ready? + (do + (swap! state (fn [current] + (-> (ensure-state current) + (update-in [:repos repo :windows] (fnil conj #{}) window-id) + (assoc-in [:window->repo window-id] repo)))) + runtime) + (p/let [_ (-> (stop-daemon! runtime) + (p/catch (fn [_] nil))) + runtime' (start-daemon! repo)] + (swap! state + (fn [current] + (let [current' (ensure-state current) + windows (get-in current' [:repos repo :windows] #{})] + (-> current' + (assoc-in [:repos repo] {:runtime runtime' + :windows (conj windows window-id)}) + (assoc-in [:window->repo window-id] repo))))) + runtime'))) + (p/let [runtime (start-daemon! repo)] + (swap! state (fn [current] + (-> (ensure-state current) + (assoc-in [:repos repo] {:runtime runtime + :windows #{window-id}}) + (assoc-in [:window->repo window-id] repo)))) + runtime)))) + +(defn- parse-runtime-lock + [{:keys [base-url]}] + (when (seq base-url) + (try + (let [^js parsed-url (js/URL. base-url) + host (.-hostname parsed-url) + port-str (.-port parsed-url) + port (js/parseInt port-str 10)] + (when (and (seq host) (number? port) (pos-int? port)) + {:host host + :port port})) + (catch :default _ + nil)))) + +(defn- runtime-ready-default? + [runtime] + (if-let [lock (parse-runtime-lock runtime)] + (daemon/ready? lock) + (p/resolved false))) + +(defn ensure-stopped! + [{:keys [state stop-daemon!]} repo window-id] + (if (= repo (get-in (ensure-state @state) [:window->repo window-id])) + (ensure-window-stopped! {:state state :stop-daemon! stop-daemon!} window-id) + (let [runtime* (atom nil)] + (swap! state + (fn [current] + (let [[next-state runtime] (detach-window-from-repo current repo window-id)] + (reset! runtime* runtime) + next-state))) + (if-let [runtime @runtime*] + (p/let [_ (stop-daemon! runtime)] + true) + (p/resolved false))))) + +(defn stop-all! + [{:keys [state stop-daemon!]}] + (let [entries (vals (:repos (ensure-state @state)))] + (-> (p/all (map (fn [{:keys [runtime]}] + (stop-daemon! runtime)) + entries)) + (p/then (fn [_] + (reset! state (initial-state)) + true))))) + +(defn- start-managed-daemon! + [repo] + (p/let [config (cli-server/ensure-server! {} repo)] + {:repo repo + :base-url (:base-url config) + :auth-token nil})) + +(defn- stop-managed-daemon! + [{:keys [repo]}] + (p/let [result (cli-server/stop-server! {} repo)] + (:ok? result))) + +(defonce manager + (create-manager + {:start-daemon! start-managed-daemon! + :stop-daemon! stop-managed-daemon! + :runtime-ready? runtime-ready-default?})) + +(defn ensure-runtime! + [repo window-id] + (ensure-started! manager repo window-id)) + +(defn release-window! + [window-id] + (ensure-window-stopped! manager window-id)) + +(defn stop-all-managed! + [] + (stop-all! manager)) diff --git a/src/electron/electron/handler.cljs b/src/electron/electron/handler.cljs index 6d800b22b7..31daa5f093 100644 --- a/src/electron/electron/handler.cljs +++ b/src/electron/electron/handler.cljs @@ -16,6 +16,7 @@ [electron.backup-file :as backup-file] [electron.configs :as cfgs] [electron.db :as db] + [electron.db-worker :as db-worker] [electron.find-in-page :as find] [electron.handler-interface :refer [handle]] [electron.keychain :as keychain] @@ -219,11 +220,26 @@ ;; DB related IPCs start +(defn stop-all-db-workers! + [] + (db-worker/stop-all-managed!)) + +(defmethod handle :db-worker-runtime [^js window [_ repo]] + (if (string/blank? repo) + (p/rejected (ex-info "repo is required" {:code :missing-repo})) + (db-worker/ensure-runtime! repo (.-id window)))) + (defmethod handle :db-export [_window [_ repo data]] + (logger/warn ::db-export-compat + {:repo repo + :message "legacy db-export IPC path invoked; desktop should use db-worker runtime"}) (db/ensure-graph-dir! repo) (db/save-db! repo data)) (defmethod handle :db-get [_window [_ repo]] + (logger/warn ::db-get-compat + {:repo repo + :message "legacy db-get IPC path invoked; desktop should use db-worker runtime"}) (db/get-db repo)) ;; DB related IPCs End @@ -309,8 +325,14 @@ nil) (defmethod handle :setCurrentGraph [^js window [_ graph-name]] - (when graph-name - (set-current-graph! window (utils/get-graph-dir graph-name)))) + (let [next-graph-path (when graph-name (utils/get-graph-dir graph-name)) + current-graph-path (state/get-window-graph-path window)] + (p/let [_ (when (not= current-graph-path next-graph-path) + (db-worker/release-window! (.-id window)))] + (if next-graph-path + (set-current-graph! window next-graph-path) + (state/close-window! window)) + nil))) (defmethod handle :runCli [window [_ {:keys [command args returnResult]}]] (try @@ -463,21 +485,43 @@ (defmethod handle :window/open-blank-callback [^js win [_ _type]] (win/setup-window-listeners! win) nil) +(defn- decode-main-ipc-message + [args-js] + (if (string? args-js) + (sqlite-util/read-transit-str args-js) + (bean/->clj args-js))) + +(defn- command-name + [message] + (let [command (first message)] + (cond + (keyword? command) (name command) + (string? command) command + :else nil))) + +(defn- clj args-js)] - ;; Be careful with the return values of `handle` defmethods. - ;; Values that are not non-JS objects will cause this - ;; exception - - ;; https://www.electronjs.org/docs/latest/breaking-changes#behavior-changed-sending-non-js-objects-over-ipc-now-throws-an-exception - (bean/->js (handle (or (utils/get-win-from-sender event) window) message))) - (catch :default e - (when-not (contains? #{"mkdir" "stat"} (nth args-js 0)) - (logger/error "IPC error: " {:event event - :args args-js} - e)) - e)))) + (let [message* (volatile! nil)] + (try + (let [message (decode-main-ipc-message args-js) + _ (vreset! message* message) + result (handle (or (utils/get-win-from-sender event) window) message)] + (js args))] - result))) + (p/let [payload (sqlite-util/write-transit-str (vec args)) + result (js/window.apis.doAction payload)] + (maybe-read-transit result)))) (defn invoke [channel & args] diff --git a/src/main/frontend/handler.cljs b/src/main/frontend/handler.cljs index 559b954765..f50041d311 100644 --- a/src/main/frontend/handler.cljs +++ b/src/main/frontend/handler.cljs @@ -151,7 +151,7 @@ (p/do! (-> (p/let [t2 (util/time-ms) - _ (db-browser/start-db-worker!) + _ (persist-db/InBrowser)) (defonce node-db (atom nil)) +(defonce remote-db (atom nil)) +(defonce remote-repo (atom nil)) (defn- node-runtime? [] (and (exists? js/process) (not (exists? js/window)))) +(defn- electron-runtime? + [] + (and (not (node-runtime?)) + (util/electron?))) + +(defn- max-tx @@ -70,9 +138,10 @@ [& {:keys [succ-notification? force-save?]}] (when (util/electron?) (when-let [repo (state/get-current-repo)] - (when (or force-save? + (when (or force-save? (and (graph-has-changed? repo) (state/input-idle? repo :diff 5000))) + (log/debug :event :save-db-to-disk :repo repo) (println :debug :save-db-to-disk repo) (-> (p/do! @@ -83,14 +152,8 @@ [:notification/show {:content "The current db has been saved successfully to the disk." :status :success}]))) (p/catch (fn [^js error] - (js/console.error error) + (log/error :event :save-db-to-disk-failed :repo repo :error error) (state/pub-event! [:notification/show {:content (str (.getMessage error)) :status :error :clear? false}])))))))) - -(defn run-export-periodically! - [] - (js/setInterval export-current-graph! - ;; every 30 seconds - (* 30 1000))) diff --git a/src/main/frontend/persist_db/browser.cljs b/src/main/frontend/persist_db/browser.cljs index 177bc91955..e0b00ac798 100644 --- a/src/main/frontend/persist_db/browser.cljs +++ b/src/main/frontend/persist_db/browser.cljs @@ -159,8 +159,9 @@ (defn {"Content-Type" "application/json" + "Accept" "application/json"} + (seq auth-token) + (assoc "Authorization" (str "Bearer " auth-token)))) + +(defn- parse-response-body + [body] + (cond + (string? body) (js->clj (js/JSON.parse body) :keywordize-keys true) + (map? body) body + :else {})) + +(defn- normalize-code + [code] + (cond + (keyword? code) code + (string? code) (keyword code) + :else :invoke-failed)) + +(defn- decode-event + [{:keys [type payload]}] + (let [decoded (when (some? payload) + (try + (ldb/read-transit-str payload) + (catch :default _ + payload)))] + (if (and (vector? decoded) + (= 2 (count decoded)) + (keyword? (first decoded))) + [(first decoded) (second decoded)] + [(when type (keyword type)) decoded]))) + +(defn- data-line + [event-text] + (some (fn [line] + (when (string/starts-with? line "data: ") + (subs line 6))) + (string/split-lines event-text))) + +(defn create-client + [{:keys [fetch-fn open-sse-fn schedule-fn reconnect-delay-ms] :as opts}] + (let [default-fetch-fn + (fn [{:keys [method url headers body]}] + (p/let [^js res (js/fetch url (clj->js (cond-> {:method method + :headers (or headers {})} + body (assoc :body body)))) + text (.text res)] + {:status (.-status res) + :body text})) + default-open-sse-fn + (fn [{:keys [url on-message on-error]}] + (if (exists? js/EventSource) + (let [es (js/EventSource. url)] + (set! (.-onmessage es) + (fn [event] + (when on-message + (on-message (str "data: " (.-data event) "\n\n"))))) + (set! (.-onerror es) + (fn [event] + (when on-error + (on-error event)))) + {:close! (fn [] + (.close es))}) + {:close! (fn [] nil)}))] + (assoc opts + :fetch-fn (or fetch-fn default-fetch-fn) + :open-sse-fn (or open-sse-fn default-open-sse-fn) + :schedule-fn (or schedule-fn (fn [f delay-ms] + (js/setTimeout f delay-ms))) + :reconnect-delay-ms (or reconnect-delay-ms 1000)))) + +(defn invoke! + [{:keys [base-url auth-token fetch-fn]} method direct-pass? args] + (let [payload (js/JSON.stringify + (clj->js (if direct-pass? + {:method method + :directPass true + :args args} + {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)})))] + (p/let [{:keys [status body]} + (fetch-fn {:method "POST" + :url (invoke-url base-url) + :headers (base-headers auth-token) + :body payload}) + parsed (parse-response-body body)] + (if (<= 200 status 299) + (if direct-pass? + (:result parsed) + (ldb/read-transit-str (:resultTransit parsed))) + (let [error (:error parsed)] + (throw (ex-info (or (:message error) "db-worker invoke failed") + (cond-> {:status status + :code (normalize-code (:code error))} + error (assoc :error error))))))))) + +(defn connect-events! + [{:keys [base-url auth-token event-handler open-sse-fn schedule-fn reconnect-delay-ms]} wrapped-worker] + (let [connected? (atom true) + buffer (atom "") + subscription (atom nil) + dispatch! (fn [event-str] + (when-let [line (data-line event-str)] + (let [event (parse-response-body line) + [event-type payload] (decode-event event)] + (when (and event-handler event-type) + (event-handler event-type wrapped-worker payload)))))] + (letfn [(open! [] + (when @connected? + (reset! subscription + (open-sse-fn + {:url (events-url base-url) + :headers (base-headers auth-token) + :on-message (fn [chunk] + (when @connected? + (swap! buffer str chunk) + (loop [] + (when-let [idx (string/index-of @buffer "\n\n")] + (let [event-str (subs @buffer 0 idx) + next-buffer (subs @buffer (+ idx 2))] + (reset! buffer next-buffer) + (dispatch! event-str) + (recur)))))) + :on-error (fn [_error] + (when @connected? + (schedule-fn open! reconnect-delay-ms)))}))))] + (open!) + {:disconnect! (fn [] + (reset! connected? false) + (when-let [close! (:close! @subscription)] + (close!)) + nil)}))) + +(defn- method->str + [qkw] + (str (namespace qkw) "/" (name qkw))) + +(defrecord InRemote [client wrapped-worker disconnect!] + protocol/PersistentDB + (str qkw) direct-pass? args)) + {:keys [disconnect!]} (connect-events! (assoc client + :event-handler event-handler + :auth-token auth-token) + wrapped-worker)] + (->InRemote client wrapped-worker disconnect!))) + +(defn stop! + [remote-client] + (when-let [disconnect! (:disconnect! remote-client)] + (disconnect!)) + (p/resolved true)) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 4b90791769..8a5667c37f 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -78,8 +78,8 @@ (defn- handle-event! [type payload] - (let [event (js/JSON.stringify (clj->js {:type type} - :payload (encode-event-payload payload))) + (let [event (js/JSON.stringify (clj->js {:type type + :payload (encode-event-payload payload)})) message (str "data: " event "\n\n")] (doseq [^js res @*sse-clients] (try @@ -125,7 +125,30 @@ #{:thread-api/init :thread-api/list-db :thread-api/get-version - :thread-api/set-infer-worker-proxy}) + :thread-api/set-infer-worker-proxy + :thread-api/set-context + :thread-api/sync-app-state + :thread-api/update-thread-atom + :thread-api/mobile-logs + :thread-api/rtc-start + :thread-api/rtc-stop + :thread-api/rtc-toggle-auto-push + :thread-api/rtc-toggle-remote-profile + :thread-api/rtc-grant-graph-access + :thread-api/rtc-get-graphs + :thread-api/rtc-delete-graph + :thread-api/rtc-get-users-info + :thread-api/rtc-get-block-content-versions + :thread-api/rtc-get-debug-state + :thread-api/rtc-request-download-graph + :thread-api/rtc-wait-download-graph-info-ready + :thread-api/rtc-download-graph-from-s3 + :thread-api/get-user-rsa-key-pair + :thread-api/init-user-rsa-key-pair + :thread-api/reset-user-rsa-key-pair + :thread-api/change-e2ee-password + :thread-api/get-e2ee-password + :thread-api/save-e2ee-password}) (def ^:private write-methods #{:thread-api/transact diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 3bea561842..f3d3ab3f36 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -1,13 +1,11 @@ (ns logseq.cli.server "db-worker-node lifecycle orchestration for logseq." - (:require ["child_process" :as child-process] - ["fs" :as fs] - ["http" :as http] + (:require ["fs" :as fs] ["os" :as os] ["path" :as node-path] [clojure.string :as string] - [logseq.cli.log :as cli-log] [frontend.worker.db-worker-node-lock :as db-lock] + [logseq.db-worker.daemon :as daemon] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -59,166 +57,51 @@ [] (node-path/join js/__dirname "../static/db-worker-node.js")) -(defn- pid-status - [pid] - (when (number? pid) - (try - (.kill js/process pid 0) - :alive - (catch :default e - (case (.-code e) - "ESRCH" :not-found - "EPERM" :no-permission - :error))))) - -(defn- read-lock - [path] - (when (and (seq path) (fs/existsSync path)) - (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) - :keywordize-keys true))) - -(defn- remove-lock! - [path] - (when (and (seq path) (fs/existsSync path)) - (fs/unlinkSync path))) - (defn- base-url [{:keys [host port]}] (str "http://" host ":" port)) +(defn- pid-status + [pid] + (daemon/pid-status pid)) + +(defn- read-lock + [path] + (daemon/read-lock path)) + +(defn- remove-lock! + [path] + (daemon/remove-lock! path)) + (defn- http-request - [{:keys [method host port path headers body timeout-ms]}] - (p/create - (fn [resolve reject] - (let [timeout-ms (or timeout-ms 5000) - start-ms (js/Date.now) - req (.request - http - #js {:method method - :hostname host - :port port - :path path - :headers (clj->js (or headers {}))} - (fn [^js res] - (let [chunks (array)] - (.on res "data" (fn [chunk] (.push chunks chunk))) - (.on res "end" (fn [] - (let [buf (js/Buffer.concat chunks) - response {:status (.-statusCode res) - :body (.toString buf "utf8")}] - (log/debug :event :cli.server/http-response - :method method - :path path - :status (:status response) - :elapsed-ms (- (js/Date.now) start-ms) - :body (cli-log/truncate-preview (:body response))) - (resolve response)))) - (.on res "error" reject)))) - timeout-id (js/setTimeout - (fn [] - (.destroy req) - (reject (ex-info "request timeout" {:code :timeout}))) - timeout-ms)] - (log/debug :event :cli.server/http-request - :method method - :host host - :port port - :path path - :body (cli-log/truncate-preview body)) - (.on req "error" (fn [err] - (js/clearTimeout timeout-id) - (reject err))) - (when body - (.write req body)) - (.end req) - (.on req "response" (fn [_] - (js/clearTimeout timeout-id))))))) - -(defn- ready? - [{:keys [host port]}] - (-> (p/let [{:keys [status]} (http-request {:method "GET" - :host host - :port port - :path "/readyz" - :timeout-ms 1000})] - (= 200 status)) - (p/catch (fn [_] false)))) - -(defn- healthy? - [{:keys [host port]}] - (-> (p/let [{:keys [status]} (http-request {:method "GET" - :host host - :port port - :path "/healthz" - :timeout-ms 1000})] - (= 200 status)) - (p/catch (fn [_] false)))) - -(defn- valid-lock? - [lock] - (and (seq (:host lock)) - (pos-int? (:port lock)))) + [opts] + (daemon/http-request opts)) (defn- cleanup-stale-lock! [path lock] - (cond - (nil? lock) - (p/resolved nil) - - (= :not-found (pid-status (:pid lock))) - (do - (remove-lock! path) - (p/resolved nil)) - - (not (valid-lock? lock)) - (do - (remove-lock! path) - (p/resolved nil)) - - :else - (p/let [healthy (healthy? lock)] - (when-not healthy - (remove-lock! path))))) + (daemon/cleanup-stale-lock! path lock)) (defn- wait-for - [pred-fn {:keys [timeout-ms interval-ms] - :or {timeout-ms 8000 - interval-ms 200}}] - (p/create - (fn [resolve reject] - (let [start (js/Date.now) - tick (fn tick [] - (p/let [ok? (pred-fn)] - (if ok? - (resolve true) - (if (> (- (js/Date.now) start) timeout-ms) - (reject (ex-info "timeout" {:code :timeout})) - (js/setTimeout tick interval-ms)))))] - (tick))))) + [pred-fn opts] + (daemon/wait-for pred-fn opts)) (defn- wait-for-lock [path] - (wait-for (fn [] - (p/resolved (and (fs/existsSync path) - (let [lock (read-lock path)] - (pos-int? (:port lock)))))) - {:timeout-ms 8000 - :interval-ms 200})) + (daemon/wait-for-lock path)) (defn- wait-for-ready [lock] - (wait-for (fn [] (ready? lock)) - {:timeout-ms 8000 - :interval-ms 250})) + (daemon/wait-for-ready lock)) + +(defn- ready? + [lock] + (daemon/ready? lock)) (defn- spawn-server! [{:keys [repo data-dir]}] - (let [script (db-worker-script-path) - args #js ["--repo" repo "--data-dir" data-dir] - child (.spawn child-process script args #js {:detached true - :stdio "ignore"})] - (.unref child) - child)) + (daemon/spawn-server! {:script (db-worker-script-path) + :repo repo + :data-dir data-dir})) (defn- ensure-server-started! [config repo] @@ -285,9 +168,16 @@ (defn start-server! [config repo] - (p/let [_ (ensure-server-started! config repo)] - {:ok? true - :data {:repo repo}})) + (-> (p/let [_ (ensure-server-started! config repo)] + {:ok? true + :data {:repo repo}}) + (p/catch (fn [e] + (let [data (ex-data e) + code (or (:code data) :server-start-failed)] + {:ok? false + :error (cond-> {:code code + :message (or (.-message e) "failed to start server")} + (:lock data) (assoc :lock (:lock data)))}))))) (defn restart-server! [config repo] diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs new file mode 100644 index 0000000000..50db7f4943 --- /dev/null +++ b/src/main/logseq/db_worker/daemon.cljs @@ -0,0 +1,153 @@ +(ns logseq.db-worker.daemon + "Shared db-worker-node lifecycle helpers for CLI and Electron." + (:require ["child_process" :as child-process] + ["fs" :as fs] + ["http" :as http] + [lambdaisland.glogi :as log] + [promesa.core :as p])) + +(defn pid-status + [pid] + (when (number? pid) + (try + (.kill js/process pid 0) + :alive + (catch :default e + (case (.-code e) + "ESRCH" :not-found + "EPERM" :no-permission + :error))))) + +(defn read-lock + [path] + (when (and (seq path) (fs/existsSync path)) + (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true))) + +(defn remove-lock! + [path] + (when (and (seq path) (fs/existsSync path)) + (fs/unlinkSync path))) + +(defn http-request + [{:keys [method host port path headers body timeout-ms]}] + (p/create + (fn [resolve reject] + (let [timeout-ms (or timeout-ms 5000) + start-ms (js/Date.now) + req (.request + http + #js {:method method + :hostname host + :port port + :path path + :headers (clj->js (or headers {}))} + (fn [^js res] + (let [chunks (array)] + (.on res "data" (fn [chunk] (.push chunks chunk))) + (.on res "end" (fn [] + (let [buf (js/Buffer.concat chunks)] + (resolve {:status (.-statusCode res) + :body (.toString buf "utf8") + :elapsed-ms (- (js/Date.now) start-ms)})))) + (.on res "error" reject)))) + timeout-id (js/setTimeout + (fn [] + (.destroy req) + (reject (ex-info "request timeout" {:code :timeout}))) + timeout-ms)] + (.on req "error" (fn [err] + (js/clearTimeout timeout-id) + (reject err))) + (when body + (.write req body)) + (.end req) + (.on req "response" (fn [_] + (js/clearTimeout timeout-id))))))) + +(defn ready? + [{:keys [host port]}] + (-> (p/let [{:keys [status]} (http-request {:method "GET" + :host host + :port port + :path "/readyz" + :timeout-ms 1000})] + (= 200 status)) + (p/catch (fn [_] false)))) + +(defn healthy? + [{:keys [host port]}] + (-> (p/let [{:keys [status]} (http-request {:method "GET" + :host host + :port port + :path "/healthz" + :timeout-ms 1000})] + (= 200 status)) + (p/catch (fn [_] false)))) + +(defn valid-lock? + [lock] + (and (seq (:host lock)) + (pos-int? (:port lock)))) + +(defn cleanup-stale-lock! + [path lock] + (cond + (nil? lock) + (p/resolved nil) + + (= :not-found (pid-status (:pid lock))) + (do + (remove-lock! path) + (p/resolved nil)) + + (not (valid-lock? lock)) + (do + (remove-lock! path) + (p/resolved nil)) + + :else + (p/let [healthy (healthy? lock)] + (when-not healthy + (remove-lock! path))))) + +(defn wait-for + [pred-fn {:keys [timeout-ms interval-ms] + :or {timeout-ms 8000 + interval-ms 200}}] + (p/create + (fn [resolve reject] + (let [start (js/Date.now) + tick (fn tick [] + (p/let [ok? (pred-fn)] + (if ok? + (resolve true) + (if (> (- (js/Date.now) start) timeout-ms) + (reject (ex-info "timeout" {:code :timeout})) + (js/setTimeout tick interval-ms)))))] + (tick))))) + +(defn wait-for-lock + [path] + (wait-for (fn [] + (p/resolved (and (fs/existsSync path) + (let [lock (read-lock path)] + (pos-int? (:port lock)))))) + {:timeout-ms 8000 + :interval-ms 200})) + +(defn wait-for-ready + [lock] + (wait-for (fn [] (ready? lock)) + {:timeout-ms 8000 + :interval-ms 250})) + +(defn spawn-server! + [{:keys [script repo data-dir]}] + (let [args #js ["--repo" repo "--data-dir" data-dir] + child (.spawn child-process script args #js {:detached true + :stdio "ignore"})] + (when-not script + (log/warn :db-worker-daemon/missing-script {:repo repo :data-dir data-dir})) + (.unref child) + child)) diff --git a/src/test/electron/db_worker_manager_test.cljs b/src/test/electron/db_worker_manager_test.cljs new file mode 100644 index 0000000000..0c5ca57ded --- /dev/null +++ b/src/test/electron/db_worker_manager_test.cljs @@ -0,0 +1,141 @@ +(ns electron.db-worker-manager-test + (:require [cljs.test :refer [async deftest is]] + [electron.db-worker :as db-worker] + [promesa.core :as p])) + +(defn- runtime + [repo] + {:repo repo + :base-url (str "http://127.0.0.1/" repo) + :auth-token (str "token-" repo)}) + +(deftest ensure-started-is-idempotent-for-same-window + (async done + (let [start-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] + (swap! start-calls conj repo) + (p/resolved (runtime repo))) + :stop-daemon! (fn [_] (p/resolved true))})] + (-> (p/let [a (db-worker/ensure-started! manager "graph-a" :window-1) + b (db-worker/ensure-started! manager "graph-a" :window-1)] + (is (= 1 (count @start-calls))) + (is (= a b))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-started-reuses-daemon-across-windows + (async done + (let [start-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] + (swap! start-calls conj repo) + (p/resolved (runtime repo))) + :stop-daemon! (fn [_] (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-a" :window-2)] + (is (= ["graph-a"] @start-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-started-switches-window-repo-and-stops-previous-daemon + (async done + (let [start-calls (atom []) + stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] + (swap! start-calls conj repo) + (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-b" :window-1)] + (is (= ["graph-a" "graph-b"] @start-calls)) + (is (= ["graph-a"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-stopped-stops-only-on-last-window + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-a" :window-2) + _ (db-worker/ensure-stopped! manager "graph-a" :window-1) + _ (is (empty? @stop-calls)) + _ (db-worker/ensure-stopped! manager "graph-a" :window-2)] + (is (= ["graph-a"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-window-stopped-releases-active-runtime-by-window + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-a" :window-2) + _ (db-worker/ensure-window-stopped! manager :window-1) + _ (is (empty? @stop-calls)) + _ (db-worker/ensure-window-stopped! manager :window-2)] + (is (= ["graph-a"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest stop-all-stops-every-active-graph + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-b" :window-2) + _ (db-worker/stop-all! manager)] + (is (= #{"graph-a" "graph-b"} (set @stop-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-started-restarts-unhealthy-cached-runtime + (async done + (let [start-count (atom 0) + stop-calls (atom []) + created-runtimes (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] + (let [idx (swap! start-count inc) + rt {:repo repo + :base-url (str "http://127.0.0.1:910" idx) + :auth-token (str "token-" idx)}] + (swap! created-runtimes conj rt) + (p/resolved rt))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:base-url rt)) + (p/resolved true)) + :runtime-ready? (fn [rt] + ;; first runtime reported unhealthy, restarted runtime healthy + (p/resolved (not= (:base-url rt) "http://127.0.0.1:9101")))})] + (-> (p/let [rt1 (db-worker/ensure-started! manager "graph-a" :window-1) + rt2 (db-worker/ensure-started! manager "graph-a" :window-1)] + (is (= "http://127.0.0.1:9101" (:base-url rt1))) + (is (= "http://127.0.0.1:9102" (:base-url rt2))) + (is (= 2 @start-count)) + (is (= ["http://127.0.0.1:9101"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) diff --git a/src/test/frontend/electron/ipc_test.cljs b/src/test/frontend/electron/ipc_test.cljs new file mode 100644 index 0000000000..0163e86d4f --- /dev/null +++ b/src/test/frontend/electron/ipc_test.cljs @@ -0,0 +1,46 @@ +(ns frontend.electron.ipc-test + (:require [cljs.test :refer [async deftest is]] + [electron.ipc :as ipc] + [frontend.util :as util] + [logseq.db.sqlite.util :as sqlite-util] + [promesa.core :as p])) + +(deftest ipc-uses-transit-json-for-main-channel + (async done + (let [payloads (atom []) + original-electron? util/electron? + original-apis (.-apis js/window)] + (set! util/electron? (constantly true)) + (set! (.-apis js/window) + #js {:doAction (fn [payload] + (swap! payloads conj payload) + (js/Promise.resolve (sqlite-util/write-transit-str {:ok true})))}) + (-> (p/let [result (ipc/ipc :db-worker-runtime "logseq_db_graph_a")] + (is (= 1 (count @payloads))) + (is (string? (first @payloads))) + (is (= [:db-worker-runtime "logseq_db_graph_a"] + (vec (sqlite-util/read-transit-str (first @payloads))))) + (is (= {:ok true} result))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! util/electron? original-electron?) + (set! (.-apis js/window) original-apis) + (done))))))) + +(deftest ipc-keeps-non-string-response-unchanged + (async done + (let [original-electron? util/electron? + original-apis (.-apis js/window)] + (set! util/electron? (constantly true)) + (set! (.-apis js/window) + #js {:doAction (fn [_payload] + (js/Promise.resolve #js {:legacy true}))}) + (-> (p/let [result (ipc/ipc :system/info)] + (is (= true (aget result "legacy")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! util/electron? original-electron?) + (set! (.-apis js/window) original-apis) + (done))))))) diff --git a/src/test/frontend/modules/shortcut/config_test.cljs b/src/test/frontend/modules/shortcut/config_test.cljs new file mode 100644 index 0000000000..7d1de4535d --- /dev/null +++ b/src/test/frontend/modules/shortcut/config_test.cljs @@ -0,0 +1,14 @@ +(ns frontend.modules.shortcut.config-test + (:require [cljs.test :refer [deftest is testing]] + [frontend.modules.shortcut.config :as shortcut-config] + [frontend.state :as state])) + +(deftest graph-db-save-shortcut-does-not-trigger-legacy-save-event + (testing "mod+s in db graph sends new save-info event" + (let [events* (atom [])] + (with-redefs [state/pub-event! (fn [event] + (swap! events* conj event))] + ((get-in shortcut-config/all-built-in-keyboard-shortcuts + [:graph/db-save :fn])) + (is (= [[:graph/db-save-shortcut]] + @events*)))))) diff --git a/src/test/frontend/persist_db/remote_test.cljs b/src/test/frontend/persist_db/remote_test.cljs new file mode 100644 index 0000000000..20eb48c9cd --- /dev/null +++ b/src/test/frontend/persist_db/remote_test.cljs @@ -0,0 +1,100 @@ +(ns frontend.persist-db.remote-test + (:require [cljs.test :refer [async deftest is]] + [frontend.persist-db.remote :as remote] + [logseq.db :as ldb] + [promesa.core :as p])) + +(deftest invoke-success-returns-decoded-transit-result + (async done + (let [captured (atom nil) + client (remote/create-client + {:base-url "http://127.0.0.1:9101" + :auth-token "token-1" + :fetch-fn (fn [req] + (reset! captured req) + (p/resolved {:status 200 + :body (js/JSON.stringify + #js {:ok true + :resultTransit (ldb/write-transit-str [{:repo "graph-a"}])})}))})] + (-> (p/let [result (remote/invoke! client "thread-api/list-db" false []) + headers (:headers @captured)] + (is (= [{:repo "graph-a"}] result)) + (is (= "Bearer token-1" (get headers "Authorization")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest invoke-error-propagates-status-and-error-code + (async done + (let [client (remote/create-client + {:base-url "http://127.0.0.1:9101" + :fetch-fn (fn [_req] + (p/resolved {:status 409 + :body (js/JSON.stringify + #js {:ok false + :error #js {:code "repo-locked" + :message "graph already locked"}})}))})] + (-> (remote/invoke! client "thread-api/transact" false ["graph-a" [] {}]) + (p/then (fn [_] + (is false "expected invoke! to reject on non-2xx status"))) + (p/catch (fn [e] + (let [data (ex-data e)] + (is (= 409 (:status data))) + (is (= :repo-locked (:code data)))))) + (p/finally (fn [] (done))))))) + +(deftest connect-events-parses-sse-and-reconnects + (async done + (let [events (atom []) + open-count (atom 0) + scheduled (atom nil) + latest-handlers (atom nil) + wrapped-worker (fn [& _] nil) + client (remote/create-client + {:base-url "http://127.0.0.1:9101" + :auth-token "token-1" + :event-handler (fn [event-type wrapped-worker' payload] + (swap! events conj [event-type wrapped-worker' payload])) + :open-sse-fn (fn [{:keys [on-message on-error]}] + (swap! open-count inc) + (reset! latest-handlers {:on-message on-message + :on-error on-error}) + {:close! (fn [] nil)}) + :schedule-fn (fn [f _delay-ms] + (reset! scheduled f) + :scheduled) + :reconnect-delay-ms 1}) + sub (remote/connect-events! client wrapped-worker) + payload (ldb/write-transit-str [:thread-api/persist-db {:repo "graph-a"}])] + (-> (p/let [_ ((:on-message @latest-handlers) + (str "data: " (js/JSON.stringify #js {:type "thread-api/persist-db" + :payload payload}) + "\n\n")) + _ (is (= [[:thread-api/persist-db wrapped-worker {:repo "graph-a"}]] @events)) + _ ((:on-error @latest-handlers) (js/Error. "disconnect")) + scheduled-fn @scheduled + _ (is (fn? scheduled-fn)) + _ (when (fn? scheduled-fn) + (scheduled-fn))] + (is (= 2 @open-count)) + ((:disconnect! sub))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest invoke-without-auth-token-omits-authorization-header + (async done + (let [captured (atom nil) + client (remote/create-client + {:base-url "http://127.0.0.1:9101" + :fetch-fn (fn [req] + (reset! captured req) + (p/resolved {:status 200 + :body (js/JSON.stringify + #js {:ok true + :resultTransit (ldb/write-transit-str [])})}))})] + (-> (p/let [_ (remote/invoke! client "thread-api/list-db" false [])] + (is (nil? (get (:headers @captured) "Authorization")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) diff --git a/src/test/frontend/persist_db_test.cljs b/src/test/frontend/persist_db_test.cljs new file mode 100644 index 0000000000..94538a48c3 --- /dev/null +++ b/src/test/frontend/persist_db_test.cljs @@ -0,0 +1,173 @@ +(ns frontend.persist-db-test + (:require [cljs.test :refer [async deftest is]] + [electron.ipc :as ipc] + [frontend.persist-db.browser :as browser] + [frontend.persist-db :as persist-db] + [frontend.persist-db.protocol :as protocol] + [frontend.persist-db.remote :as remote] + [frontend.state :as state] + [frontend.util :as util] + [promesa.core :as p])) + +(defrecord FakeRemote [repo wrapped-worker] + protocol/PersistentDB + (FakeRemote repo wrapped-worker))) + (set! remote/stop! (fn [_] (p/resolved true))) + (-> (p/let [result (persist-db/FakeRemote repo (fn [& _] nil)))) + (set! remote/stop! (fn [client] + (swap! stop-calls conj (:repo client)) + (p/resolved true))) + (-> (p/let [_ (ensure-remote! "logseq_db_graph_a") + _ (ensure-remote! "logseq_db_graph_a") + _ (ensure-remote! "logseq_db_graph_b")] + (is (= [["db-worker-runtime" "logseq_db_graph_a"] + ["db-worker-runtime" "logseq_db_graph_b"]] + @ipc-calls)) + (is (= ["logseq_db_graph_a" "logseq_db_graph_b"] @start-calls)) + (is (= ["logseq_db_graph_a"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! ipc/ipc original-ipc) + (set! remote/start! original-start!) + (set! remote/stop! original-stop!) + (done))))))) + +(deftest electron-list-db-without-current-repo-does-not-bootstrap-runtime + (async done + (let [ipc-calls (atom []) + start-calls (atom []) + original-electron? util/electron? + original-current-repo state/get-current-repo + original-ipc ipc/ipc + original-start! remote/start! + original-stop! remote/stop!] + (reset-runtime-state!) + (set! util/electron? (constantly true)) + (set! state/get-current-repo (constantly nil)) + (set! ipc/ipc (fn [& args] + (swap! ipc-calls conj args) + (p/resolved {:base-url "http://127.0.0.1:9101" + :auth-token nil + :repo "logseq_db_unused"}))) + (set! remote/start! (fn [_] + (swap! start-calls conj :start) + (->FakeRemote "logseq_db_unused" (fn [& _] nil)))) + (set! remote/stop! (fn [_] (p/resolved true))) + (-> (p/let [repos (persist-db/ (p/let [_ (browser/ raw-message + (string/replace-first #"^data: " "") + (string/replace #"\n\n$" "")) + parsed (js->clj (js/JSON.parse event-json) :keywordize-keys true)] + (is (= "sync-db-changes" (:type parsed))) + (is (= {:repo "graph-a"} + (ldb/read-transit-str (:payload parsed))))) + (finally + (reset! *sse-clients old-clients))))) + (deftest db-worker-node-help-omits-auth-token (let [show-help! #'db-worker-node/show-help! output (binding [style/*color-enabled?* true] @@ -229,6 +252,8 @@ bound-repo "logseq_db_bound"] (is (nil? (repo-error :thread-api/list-db [] bound-repo))) (is (nil? (repo-error "thread-api/list-db" [] bound-repo))) + (is (nil? (repo-error :thread-api/rtc-get-graphs ["token"] bound-repo))) + (is (nil? (repo-error :thread-api/set-context [{:repo "not-a-repo-arg"}] bound-repo))) (is (= {:status 400 :error {:code :missing-repo :message "repo is required"}} @@ -240,6 +265,24 @@ :bound-repo bound-repo}} (repo-error :thread-api/create-or-open-db ["other"] bound-repo))))) +(deftest db-worker-node-set-context-does-not-trigger-repo-mismatch + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-set-context") + repo (str "logseq_db_set_context_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + result (invoke host port "thread-api/set-context" [{:app "desktop"}])] + (is (nil? result))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) @@ -556,3 +599,73 @@ (if-let [stop! (:stop! @daemon)] (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) + +(deftest db-worker-node-start-recovers-stale-lock-before-acquire + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-stale-lock-recover") + repo (str "logseq_db_stale_lock_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo) + stale-lock {:repo repo + :pid 999999 + :host "127.0.0.1" + :port 6553 + :lock-id "stale-lock-id"}] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js stale-lock))) + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + lock' (js->clj (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) + :keywordize-keys true)] + (is (not= 999999 (:pid lock'))) + (is (not= "stale-lock-id" (:lock-id lock')))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) + (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-desktop-and-cli-share-same-graph-daemon + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-desktop-cli") + repo (str "logseq_db_desktop_cli_" (subs (str (random-uuid)) 0 8)) + now (js/Date.now) + page-uuid (random-uuid)] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + _ (invoke host port "thread-api/transact" + [repo + [{:block/uuid page-uuid + :block/title "Desktop+CLI Shared" + :block/name "desktop-cli-shared" + :block/tags #{:logseq.class/Page} + :block/created-at now + :block/updated-at now}] + {} + nil]) + ensured (cli-server/ensure-server! {:data-dir data-dir} repo) + url (js/URL. (:base-url ensured)) + cli-host (.-hostname url) + cli-port (js/parseInt (.-port url) 10) + result (invoke cli-host cli-port "thread-api/q" + [repo + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + "Desktop+CLI Shared"]])] + (is (seq result))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) + (p/finally (fn [] (done)))) + (done)))))))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index bc0179d0d7..e1ce32418e 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -1,9 +1,12 @@ (ns logseq.cli.server-test + {:clj-kondo/config '{:linters {:private-var-access {:level :off}}}} (:require [cljs.test :refer [async deftest is]] [frontend.test.node-helper :as node-helper] [logseq.cli.server :as cli-server] + [logseq.db-worker.daemon :as daemon] [promesa.core :as p] ["fs" :as fs] + ["http" :as http] ["path" :as node-path] ["child_process" :as child-process])) @@ -57,3 +60,75 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest ensure-server-reuses-existing-running-daemon-lock + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-reuse") + repo (str "logseq_db_reuse_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + host "127.0.0.1" + spawn-calls (atom 0) + server (http/createServer + (fn [^js req ^js res] + (case (.-url req) + "/healthz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")) + "/readyz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")) + (do (.writeHead res 404 #js {"Content-Type" "text/plain"}) + (.end res "not-found")))))] + (.listen server 0 host + (fn [] + (let [address (.address server) + port (if (number? address) address (.-port address)) + lock {:repo repo + :pid (.-pid js/process) + :host host + :port port} + original-spawn! daemon/spawn-server!] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (set! daemon/spawn-server! + (fn [_opts] + (swap! spawn-calls inc) + (throw (ex-info "should not spawn when lock is ready" {})))) + (-> (cli-server/ensure-server! {:data-dir data-dir} repo) + (p/then (fn [config] + (is (= (str "http://" host ":" port) (:base-url config))) + (is (= 0 @spawn-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/spawn-server! original-spawn!) + (.close server (fn [] (done)))))))))))) + +(deftest start-server-reports-repo-locked-error-stably + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-repo-locked") + repo (str "logseq_db_locked_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid 999999 + :host "127.0.0.1" + :port 55555} + original-cleanup daemon/cleanup-stale-lock! + original-ready daemon/wait-for-ready] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (set! daemon/cleanup-stale-lock! (fn [_path _lock] (p/resolved nil))) + (set! daemon/wait-for-ready + (fn [_lock] + (p/rejected (ex-info "graph already locked" + {:code :repo-locked + :lock lock})))) + (-> (cli-server/start-server! {:data-dir data-dir} repo) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :repo-locked (get-in result [:error :code]))) + (is (= lock (get-in result [:error :lock]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/cleanup-stale-lock! original-cleanup) + (set! daemon/wait-for-ready original-ready) + (done))))))) diff --git a/src/test/logseq/db_worker/daemon_test.cljs b/src/test/logseq/db_worker/daemon_test.cljs new file mode 100644 index 0000000000..f155d87b7c --- /dev/null +++ b/src/test/logseq/db_worker/daemon_test.cljs @@ -0,0 +1,47 @@ +(ns logseq.db-worker.daemon-test + (:require [cljs.test :refer [async deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.db-worker.daemon :as daemon] + [promesa.core :as p] + ["fs" :as fs] + ["path" :as node-path] + ["child_process" :as child-process])) + +(deftest spawn-server-uses-detached-process-and-no-host-port-args + (let [captured (atom nil) + original-spawn (.-spawn child-process)] + (set! (.-spawn child-process) + (fn [cmd args opts] + (reset! captured {:cmd cmd + :args (vec (js->clj args)) + :opts (js->clj opts :keywordize-keys true)}) + (js-obj "unref" (fn [] nil)))) + (try + (daemon/spawn-server! {:script "/tmp/db-worker-node.js" + :repo "logseq_db_spawn_helper_test" + :data-dir "/tmp/logseq-db-worker"}) + (is (= "/tmp/db-worker-node.js" (:cmd @captured))) + (is (some #{"--repo"} (:args @captured))) + (is (some #{"--data-dir"} (:args @captured))) + (is (not-any? #{"--host" "--port"} (:args @captured))) + (is (= true (get-in @captured [:opts :detached]))) + (finally + (set! (.-spawn child-process) original-spawn))))) + +(deftest cleanup-stale-lock-removes-invalid-lock + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-daemon-helper") + repo (str "logseq_db_helper_stale_" (subs (str (random-uuid)) 0 8)) + path (node-path/join data-dir "db-worker.lock") + invalid-lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0}] + (fs/mkdirSync data-dir #js {:recursive true}) + (fs/writeFileSync path (js/JSON.stringify (clj->js invalid-lock))) + (-> (p/let [_ (daemon/cleanup-stale-lock! path invalid-lock)] + (is (not (fs/existsSync path))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) From 103d9fd07946fdbf21d1a379a450accc9c9800d2 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 12 Feb 2026 20:26:31 +0800 Subject: [PATCH 072/375] update package.json --- package.json | 28 +++++++++++++-------------- src/main/frontend/worker/db_core.cljs | 6 ++++++ 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/package.json b/package.json index 7c23865116..08a1c768c7 100644 --- a/package.json +++ b/package.json @@ -88,29 +88,29 @@ "gulp:buildMobile": "cross-env NODE_ENV=production gulp buildMobile", "css:build": "postcss tailwind.all.css -o static/css/style.css --verbose --env production", "css:watch": "cross-env TAILWIND_MODE=watch postcss tailwind.all.css -o static/css/style.css --verbose --watch", - "cljs:watch": "clojure -M:cljs watch app db-worker inference-worker electron", + "cljs:watch": "clojure -M:cljs watch app db-worker db-worker-node inference-worker electron", "cljs:storybook-watch": "clojure -M:cljs watch stories-dev", "gulp:mobile-watch": "gulp watchMobile", "css:mobile-build": "postcss tailwind.mobile.css -o static/mobile/css/style.css --verbose --env production", "css:mobile-watch": "cross-env TAILWIND_MODE=watch postcss tailwind.mobile.css -o static/mobile/css/style.css --verbose --watch", - "cljs:mobile-watch": "clojure -M:cljs watch mobile db-worker --config-merge \"{:output-dir \\\"./static/mobile/js\\\" :asset-path \\\"/static/mobile/js\\\" :release {:asset-path \\\"http://localhost\\\"}}\"", - "cljs:release-mobile": "clojure -M:cljs release mobile db-worker --config-merge \"{:output-dir \\\"./static/mobile/js\\\" :asset-path \\\"/static/mobile/js\\\" :release {:asset-path \\\"http://localhost\\\"}}\"", - "cljs:dev-watch": "clojure -M:cljs watch app db-worker inference-worker electron mobile", - "cljs:app-watch": "clojure -M:cljs watch app db-worker inference-worker", - "cljs:electron-watch": "clojure -M:cljs watch app db-worker inference-worker electron --config-merge \"{:asset-path \\\"./js\\\"}\"", - "cljs:release": "clojure -M:cljs release app db-worker inference-worker publishing electron", - "cljs:release-electron": "clojure -M:cljs release app db-worker inference-worker electron --debug && clojure -M:cljs release publishing", - "cljs:release-app": "clojure -M:cljs release app db-worker inference-worker", + "cljs:mobile-watch": "clojure -M:cljs watch mobile db-worker db-worker-node --config-merge \"{:output-dir \\\"./static/mobile/js\\\" :asset-path \\\"/static/mobile/js\\\" :release {:asset-path \\\"http://localhost\\\"}}\"", + "cljs:release-mobile": "clojure -M:cljs release mobile db-worker db-worker-node --config-merge \"{:output-dir \\\"./static/mobile/js\\\" :asset-path \\\"/static/mobile/js\\\" :release {:asset-path \\\"http://localhost\\\"}}\"", + "cljs:dev-watch": "clojure -M:cljs watch app db-worker db-worker-node inference-worker electron mobile", + "cljs:app-watch": "clojure -M:cljs watch app db-worker db-worker-node inference-worker", + "cljs:electron-watch": "clojure -M:cljs watch app db-worker db-worker-node inference-worker electron --config-merge \"{:asset-path \\\"./js\\\"}\"", + "cljs:release": "clojure -M:cljs release app db-worker db-worker-node inference-worker publishing electron", + "cljs:release-electron": "clojure -M:cljs release app db-worker db-worker-node inference-worker electron --debug && clojure -M:cljs release publishing", + "cljs:release-app": "clojure -M:cljs release app db-worker db-worker-node inference-worker", "cljs:release-publishing": "clojure -M:cljs release app publishing", "cljs:test": "clojure -M:test compile test", "cljs:run-test": "node static/tests.js -r '^(?!logseq.db-sync.).*' -e fix-me", "cljs:test-no-worker": "clojure -M:test compile test-no-worker", "cljs:run-test-no-worker": "node static/tests-no-worker.js", - "cljs:dev-release-app": "clojure -M:cljs release app db-worker inference-worker --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\"", - "cljs:dev-release-electron": "clojure -M:cljs release app db-worker inference-worker electron --debug --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\" && clojure -M:cljs release publishing", - "cljs:debug": "clojure -M:cljs release app db-worker inference-worker --debug", - "cljs:report": "clojure -M:cljs run shadow.cljs.build-report app db-worker inference-worker report.html", - "cljs:build-electron": "clojure -A:cljs compile app db-worker inference-worker electron", + "cljs:dev-release-app": "clojure -M:cljs release app db-worker db-worker-node inference-worker --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\"", + "cljs:dev-release-electron": "clojure -M:cljs release app db-worker db-worker-node inference-worker electron --debug --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\" && clojure -M:cljs release publishing", + "cljs:debug": "clojure -M:cljs release app db-worker db-worker-node inference-worker --debug", + "cljs:report": "clojure -M:cljs run shadow.cljs.build-report app db-worker db-worker-node inference-worker report.html", + "cljs:build-electron": "clojure -A:cljs compile app db-worker db-worker-node inference-worker electron", "cljs:lint": "clojure -M:clj-kondo --parallel --lint src --cache false", "ios:dev": "cross-env PLATFORM=ios gulp cap", "android:dev": "cross-env PLATFORM=android gulp cap", diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 3fed17c119..aeac170036 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -742,6 +742,12 @@ (let [conn (worker-state/get-datascript-conn repo)] (db-view/get-property-values @conn property-ident option))) +(def-thread-api :thread-api/get-bidirectional-properties + [repo {:keys [target-id]}] + (let [conn (worker-state/get-datascript-conn repo)] + (worker-util/profile "get-bidirectional-properties" + (ldb/get-bidirectional-properties @conn target-id)))) + (def-thread-api :thread-api/build-graph [repo option] (let [conn (worker-state/get-datascript-conn repo)] From 39f7006559f57323feb9ae9b1120b4375c3d516d Mon Sep 17 00:00:00 2001 From: Tienson Qin Date: Thu, 12 Feb 2026 23:53:29 +0800 Subject: [PATCH 073/375] fix: don't use deprecated url/parse --- src/main/logseq/cli/transport.cljs | 40 +++--- src/test/logseq/cli/transport_test.cljs | 181 ++++++++++++++---------- 2 files changed, 127 insertions(+), 94 deletions(-) diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index 1972e63387..6e72f443f5 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -1,15 +1,14 @@ (ns logseq.cli.transport "HTTP transport for communicating with db-worker-node." - (:require [cljs.reader :as reader] - [clojure.string :as string] - [logseq.db :as ldb] - [logseq.cli.log :as cli-log] - [lambdaisland.glogi :as log] - [promesa.core :as p] - ["fs" :as fs] + (:require ["fs" :as fs] ["http" :as http] ["https" :as https] - ["url" :as url])) + [cljs.reader :as reader] + [clojure.string :as string] + [lambdaisland.glogi :as log] + [logseq.cli.log :as cli-log] + [logseq.db :as ldb] + [promesa.core :as p])) (defn- request-module [^js parsed] @@ -22,18 +21,25 @@ {"Content-Type" "application/json" "Accept" "application/json"}) +(defn- request-port + [^js parsed] + (let [port (.-port parsed)] + (if (seq port) + port + (if (= "https:" (.-protocol parsed)) 443 80)))) + (defn- js headers)} (fn [^js res] @@ -50,13 +56,13 @@ (reject (ex-info "request timeout" {:code :timeout}))) timeout-ms)] (.on req "error" (fn [err] - (js/clearTimeout timeout-id) - (reject err))) + (js/clearTimeout timeout-id) + (reject err))) (when body (.write req body)) (.end req) (.on req "response" (fn [_] - (js/clearTimeout timeout-id))))))) + (js/clearTimeout timeout-id))))))) (defn request [{:keys [method url headers body timeout-ms]}] @@ -97,10 +103,10 @@ :args args-preview :url url) (p/let [{:keys [body]} (request {:method "POST" - :url url - :headers (base-headers) - :body body - :timeout-ms timeout-ms}) + :url url + :headers (base-headers) + :body body + :timeout-ms timeout-ms}) {:keys [result resultTransit]} (js->clj (js/JSON.parse body) :keywordize-keys true)] (if direct-pass? (let [response-preview (cli-log/truncate-preview result)] diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index 83297b55db..5402a487ef 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -29,94 +29,121 @@ (resolve {:url (str "http://127.0.0.1:" port) :stop! stop!})))))))) +(deftest test-request-avoids-deprecated-url-parse + (async done + (let [url-module (js/require "url") + original-parse (.-parse url-module) + parse-calls (atom 0)] + (set! (.-parse url-module) + (fn [& args] + (swap! parse-calls inc) + (.apply original-parse url-module (to-array args)))) + (-> (p/let [{:keys [url stop!]} (start-server + (fn [_req ^js res] + (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")))] + (p/let [response (transport/request {:method "GET" + :url (str url "/status") + :timeout-ms 1000})] + (is (= 200 (:status response))) + (is (= 0 @parse-calls)) + (p/let [_ (stop!)] true))) + (p/then (fn [_] + (set! (.-parse url-module) original-parse) + (done))) + (p/catch (fn [e] + (set! (.-parse url-module) original-parse) + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-request-does-not-retry (async done - (let [calls (atom 0)] - (-> (p/let [{:keys [url stop!]} (start-server - (fn [_req ^js res] - (let [attempt (swap! calls inc)] - (if (= attempt 1) - (do - (.writeHead res 500 #js {"Content-Type" "text/plain"}) - (.end res "boom")) - (do - (.writeHead res 200 #js {"Content-Type" "text/plain"}) - (.end res "ok"))))))] - (p/catch - (transport/request {:method "GET" - :url (str url "/retry") - :timeout-ms 1000}) - (fn [e] - (is (= :http-error (-> (ex-data e) :code))) - (is (= 500 (-> (ex-data e) :status))))) - (is (= 1 @calls)) - (p/let [_ (stop!)] true)) - (p/then (fn [_] (done))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [calls (atom 0)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [_req ^js res] + (let [attempt (swap! calls inc)] + (if (= attempt 1) + (do + (.writeHead res 500 #js {"Content-Type" "text/plain"}) + (.end res "boom")) + (do + (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok"))))))] + (p/catch + (transport/request {:method "GET" + :url (str url "/retry") + :timeout-ms 1000}) + (fn [e] + (is (= :http-error (-> (ex-data e) :code))) + (is (= 500 (-> (ex-data e) :status))))) + (is (= 1 @calls)) + (p/let [_ (stop!)] true)) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-request-timeout (async done - (-> (p/let [{:keys [url stop!]} (start-server - (fn [_req _res] - nil))] - (p/catch - (transport/request {:method "GET" - :url (str url "/hang") - :timeout-ms 10}) - (fn [e] - (is (= :timeout (-> (ex-data e) :code))) - (p/let [_ (stop!)] true)))) - (p/then (fn [_] (done))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done)))))) + (-> (p/let [{:keys [url stop!]} (start-server + (fn [_req _res] + nil))] + (p/catch + (transport/request {:method "GET" + :url (str url "/hang") + :timeout-ms 10}) + (fn [e] + (is (= :timeout (-> (ex-data e) :code))) + (p/let [_ (stop!)] true)))) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done)))))) (deftest test-invoke-accepts-keyword-method (async done - (let [received (atom nil)] - (-> (p/let [{:keys [url stop!]} (start-server - (fn [^js req ^js res] - (let [chunks (array)] - (.on req "data" (fn [chunk] (.push chunks chunk))) - (.on req "end" (fn [] - (let [buf (js/Buffer.concat chunks) - payload (js/JSON.parse (.toString buf "utf8"))] - (reset! received (js->clj payload :keywordize-keys true)) - (.writeHead res 200 #js {"Content-Type" "application/json"}) - (.end res (js/JSON.stringify #js {:result "ok"}))))))))] - (p/let [result (transport/invoke {:base-url url} :thread-api/pull true ["repo" [:block/title]])] - (is (= "ok" result)) - (is (= "thread-api/pull" (:method @received))) - (is (= true (:directPass @received))) - (p/let [_ (stop!)] true))) - (p/then (fn [_] (done))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [received (atom nil)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [^js req ^js res] + (let [chunks (array)] + (.on req "data" (fn [chunk] (.push chunks chunk))) + (.on req "end" (fn [] + (let [buf (js/Buffer.concat chunks) + payload (js/JSON.parse (.toString buf "utf8"))] + (reset! received (js->clj payload :keywordize-keys true)) + (.writeHead res 200 #js {"Content-Type" "application/json"}) + (.end res (js/JSON.stringify #js {:result "ok"}))))))))] + (p/let [result (transport/invoke {:base-url url} :thread-api/pull true ["repo" [:block/title]])] + (is (= "ok" result)) + (is (= "thread-api/pull" (:method @received))) + (is (= true (:directPass @received))) + (p/let [_ (stop!)] true))) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-invoke-does-not-send-auth-header (async done - (let [auth-header (atom :unset)] - (-> (p/let [{:keys [url stop!]} (start-server - (fn [^js req ^js res] - (let [headers (.-headers req)] - (reset! auth-header (aget headers "authorization"))) - (.writeHead res 200 #js {"Content-Type" "application/json"}) - (.end res (js/JSON.stringify #js {:result "ok"}))))] - (p/let [result (transport/invoke {:base-url url - :auth-token "secret"} - :thread-api/pull - true - ["repo" [:block/title]])] - (is (= "ok" result)) - (is (nil? @auth-header)) - (p/let [_ (stop!)] true))) - (p/then (fn [_] (done))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [auth-header (atom :unset)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [^js req ^js res] + (let [headers (.-headers req)] + (reset! auth-header (aget headers "authorization"))) + (.writeHead res 200 #js {"Content-Type" "application/json"}) + (.end res (js/JSON.stringify #js {:result "ok"}))))] + (p/let [result (transport/invoke {:base-url url + :auth-token "secret"} + :thread-api/pull + true + ["repo" [:block/title]])] + (is (= "ok" result)) + (is (nil? @auth-header)) + (p/let [_ (stop!)] true))) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest test-read-input (testing "reads edn input" From 6a8e35344bf53847c6c8a06489a7a7adefe9d0ed Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 12 Feb 2026 20:57:37 -0500 Subject: [PATCH 074/375] fix: old cli unable to start Looks like agent confused old cli w/ new cli --- deps/cli/src/logseq/cli.cljs | 59 +++---- deps/cli/src/logseq/cli/common/mcp/tools.cljs | 146 +++++++----------- 2 files changed, 71 insertions(+), 134 deletions(-) diff --git a/deps/cli/src/logseq/cli.cljs b/deps/cli/src/logseq/cli.cljs index f825c71595..cfcd4557fc 100644 --- a/deps/cli/src/logseq/cli.cljs +++ b/deps/cli/src/logseq/cli.cljs @@ -6,35 +6,17 @@ [clojure.string :as string] [logseq.cli.common.graph :as cli-common-graph] [logseq.cli.spec :as cli-spec] - [logseq.cli.style :as style] [logseq.cli.text-util :as cli-text-util] [nbb.error] [promesa.core :as p])) -(defn- escape-regex - [value] - (string/replace value #"[\\.^$|?*+()\\[\\]{}]" "\\\\$&")) - -(defn- bold-command-names - [value commands] - (reduce (fn [acc command] - (let [pattern (re-pattern (str "(?m)^(\\s*)" (escape-regex command) "(\\s+)"))] - (string/replace acc pattern (fn [[_ prefix spacing]] - (str prefix (style/bold command) spacing))))) - value - commands)) - (defn- format-commands [{:keys [table]}] - (let [entries (->> table - (filter (comp seq :cmds))) - rows (mapv (fn [{:keys [cmds desc spec]}] - (cond-> [(str (string/join " " cmds) - (when spec " [options]"))] - desc (conj desc))) - entries) - commands (map (comp #(string/join " " %) :cmds) entries)] - (-> (cli/format-table {:rows rows}) - (bold-command-names commands)))) + (let [table (mapv (fn [{:keys [cmds desc spec]}] + (cond-> [(str (string/join " " cmds) + (when spec " [options]"))] + desc (conj desc))) + (filter (comp seq :cmds) table))] + (cli/format-table {:rows table}))) (def ^:private default-spec {:version {:coerce :boolean @@ -43,10 +25,9 @@ (declare table) (defn- print-general-help [_m] - (println (str "Usage: logseq [command] [options]\n\n" - (style/bold "Options") ":\n" - (style/bold-options (cli/format-opts {:spec default-spec})))) - (println (str "\n" (style/bold "Commands") ":\n" (format-commands {:table table})))) + (println (str "Usage: logseq [command] [options]\n\nOptions:\n" + (cli/format-opts {:spec default-spec}))) + (println (str "\nCommands:\n" (format-commands {:table table})))) (defn- default-command [{{:keys [version]} :opts :as m}] @@ -64,11 +45,10 @@ (str " " (string/join " " (map #(str "[" (name %) "]") (:args->opts cmd-map))))) (when (:spec cmd-map) - (str " [options]\n\n" (style/bold "Options") ":\n" - (style/bold-options (cli/format-opts {:spec (:spec cmd-map)})))) + (str " [options]\n\nOptions:\n" + (cli/format-opts {:spec (:spec cmd-map)}))) (when (:description cmd-map) - (str "\n\n" (style/bold "Description") ":\n" - (cli-text-util/wrap-text (:description cmd-map) 80)))))) + (str "\n\nDescription:\n" (cli-text-util/wrap-text (:description cmd-map) 80)))))) (defn- help-command [{{:keys [command help]} :opts}] (if-let [cmd-map (and command (some #(when (= command (first (:cmds %))) %) table))] @@ -76,7 +56,7 @@ ;; handle help --help (if-let [cmd-map (and help (some #(when (= "help" (first (:cmds %))) %) table))] (print-command-help "help" cmd-map) - (println (style/bold "Command") (pr-str command) "does not exist")))) + (println "Command" (pr-str command) "does not exist")))) (defn- lazy-load-fn "Lazy load fn to speed up start time. After nbb requires ~30 namespaces, start time gets close to 1s. @@ -129,7 +109,7 @@ :args->opts [:args] :require [:args] :coerce {:args []} :spec cli-spec/append} {:cmds ["mcp-server"] :desc "Run a MCP server" - :description "Run a MCP server against a local graph if --repo is given or against the current in-app graph. For the in-app graph, the API server must be on in the app. By default the MCP server runs as a HTTP Streamable server. Use --stdio to run it as a stdio server." + :description "Run a MCP server against a local graph if --graph is given or against the current in-app graph. For the in-app graph, the API server must be on in the app. By default the MCP server runs as a HTTP Streamable server. Use --stdio to run it as a stdio server." :fn (lazy-load-fn 'logseq.cli.commands.mcp-server/start) :spec cli-spec/mcp-server} {:cmds ["validate"] :desc "Validate DB graph" @@ -166,12 +146,9 @@ (if (and (= :org.babashka/cli type') (= :require cause)) (do - (println (style/bold-keywords - (str "Error: Command missing required " - (if (get-in data [:spec option]) "option" "argument") - " " - (style/bold (pr-str (name option)))) - ["command" "option" "argument"])) + (println "Error: Command missing required" + (if (get-in data [:spec option]) "option" "argument") + (pr-str (name option))) (when-let [cmd-m (some #(when (= {:spec (:spec %) :require (:require %)} (select-keys data [:spec :require])) %) table)] @@ -182,4 +159,4 @@ (nbb.error/print-error-report e) (js/process.exit 1)))) -#js {:main -main} +#js {:main -main} \ No newline at end of file diff --git a/deps/cli/src/logseq/cli/common/mcp/tools.cljs b/deps/cli/src/logseq/cli/common/mcp/tools.cljs index 8b75fff3d5..f9f8ef838e 100644 --- a/deps/cli/src/logseq/cli/common/mcp/tools.cljs +++ b/deps/cli/src/logseq/cli/common/mcp/tools.cljs @@ -16,64 +16,50 @@ [malli.core :as m] [malli.error :as me])) -(defn- minimal-list-item - [e] - (cond-> {:db/id (:db/id e) - :block/title (:block/title e) - :block/created-at (:block/created-at e) - :block/updated-at (:block/updated-at e)} - (:db/ident e) (assoc :db/ident (:db/ident e)))) - (defn list-properties "Main fn for ListProperties tool" - [db {:keys [expand include-built-in] :as options}] - (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] - (->> (d/datoms db :avet :block/tags :logseq.class/Property) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-built-in?) - (ldb/built-in? e)))) - #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) - (map (fn [e] - (if expand - (cond-> (into {} e) - true - (dissoc e :block/tags :block/order :block/refs :block/name :db/index - :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) - true - (update :block/uuid str) - (:logseq.property/classes e) - (update :logseq.property/classes #(mapv :db/ident %)) - (:logseq.property/description e) - (update :logseq.property/description db-property/property-value-content)) - (minimal-list-item e))))))) + [db {:keys [expand]}] + (->> (d/datoms db :avet :block/tags :logseq.class/Property) + (map #(d/entity db (:e %))) + #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name :db/index + :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) + true + (update :block/uuid str) + (:logseq.property/classes e) + (update :logseq.property/classes #(mapv :db/ident %)) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + {:block/title (:block/title e) + :block/uuid (str (:block/uuid e))}))))) (defn list-tags "Main fn for ListTags tool" - [db {:keys [expand include-built-in] :as options}] - (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] - (->> (d/datoms db :avet :block/tags :logseq.class/Tag) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-built-in?) - (ldb/built-in? e)))) - (map (fn [e] - (if expand - (cond-> (into {} e) - true - (dissoc e :block/tags :block/order :block/refs :block/name - :logseq.property.embedding/hnsw-label-updated-at) - true - (update :block/uuid str) - (:logseq.property.class/extends e) - (update :logseq.property.class/extends #(mapv :db/ident %)) - (:logseq.property.class/properties e) - (update :logseq.property.class/properties #(mapv :db/ident %)) - (:logseq.property.view/type e) - (assoc :logseq.property.view/type (:db/ident (:logseq.property.view/type e))) - (:logseq.property/description e) - (update :logseq.property/description db-property/property-value-content)) - (minimal-list-item e))))))) + [db {:keys [expand]}] + (->> (d/datoms db :avet :block/tags :logseq.class/Tag) + (map #(d/entity db (:e %))) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name + :logseq.property.embedding/hnsw-label-updated-at) + true + (update :block/uuid str) + (:logseq.property.class/extends e) + (update :logseq.property.class/extends #(mapv :db/ident %)) + (:logseq.property.class/properties e) + (update :logseq.property.class/properties #(mapv :db/ident %)) + (:logseq.property.view/type e) + (assoc :logseq.property.view/type (:db/ident (:logseq.property.view/type e))) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + {:block/title (:block/title e) + :block/uuid (str (:block/uuid e))}))))) (defn- get-page-blocks [db page-id] @@ -105,47 +91,21 @@ (dissoc :block/children :block/page)) (get-page-blocks db (:db/id page)))})) -(defn- parse-time - [value] - (cond - (number? value) value - (string? value) (let [ms (js/Date.parse value)] - (when-not (js/isNaN ms) ms)) - :else nil)) - (defn list-pages "Main fn for ListPages tool" - [db {:keys [expand include-hidden include-journal journal-only created-after updated-after] :as options}] - (let [include-hidden? (boolean include-hidden) - include-journal? (if (contains? options :include-journal) include-journal true) - journal-only? (boolean journal-only) - created-after-ms (parse-time created-after) - updated-after-ms (parse-time updated-after)] - (->> (d/datoms db :avet :block/name) - (map #(d/entity db (:e %))) - (remove (fn [e] - (and (not include-hidden?) - (entity-util/hidden? e)))) - (remove (fn [e] - (let [is-journal? (ldb/journal? e)] - (cond - journal-only? (not is-journal?) - (false? include-journal?) is-journal? - :else false)))) - (remove (fn [e] - (and created-after-ms - (<= (:block/created-at e 0) created-after-ms)))) - (remove (fn [e] - (and updated-after-ms - (<= (:block/updated-at e 0) updated-after-ms)))) - (map (fn [e] - (if expand - (-> e - ;; Until there are options to limit pages, return minimal info to avoid - ;; exceeding max payload size - (select-keys [:db/id :db/ident :block/uuid :block/title :block/created-at :block/updated-at]) - (update :block/uuid str)) - (minimal-list-item e))))))) + [db {:keys [expand]}] + (->> (d/datoms db :avet :block/name) + (map #(d/entity db (:e %))) + (remove entity-util/hidden?) + (map (fn [e] + (if expand + (-> e + ;; Until there are options to limit pages, return minimal info to avoid + ;; exceeding max payload size + (select-keys [:block/uuid :block/title :block/created-at :block/updated-at]) + (update :block/uuid str)) + {:block/title (:block/title e) + :block/uuid (str (:block/uuid e))}))))) ;; upsert-nodes tool ;; ================= @@ -435,4 +395,4 @@ [conn operations* {:keys [dry-run] :as opts}] (let [import-edn (build-upsert-nodes-edn @conn operations*)] (when-not dry-run (import-edn-data conn import-edn)) - (summarize-upsert-operations operations* opts))) + (summarize-upsert-operations operations* opts))) \ No newline at end of file From 0d45ebb1e6f9e9e44d1ac4064a7655d7425e59e8 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Feb 2026 10:14:02 +0800 Subject: [PATCH 075/375] 034-db-worker-node-owner-process-management.md --- .../033-desktop-db-worker-node-backend.md | 2 + ...db-worker-node-owner-process-management.md | 205 ++++++++++++++++++ docs/cli/logseq-cli.md | 8 + src/electron/electron/db_worker.cljs | 32 ++- src/main/frontend/worker/db_worker_node.cljs | 16 +- .../frontend/worker/db_worker_node_lock.cljs | 34 ++- src/main/logseq/cli/format.cljs | 7 +- src/main/logseq/cli/server.cljs | 136 +++++++++--- src/main/logseq/db_worker/daemon.cljs | 120 +++++++++- src/test/electron/db_worker_manager_test.cljs | 16 ++ src/test/frontend/fs_test.cljs | 2 +- .../worker/db_worker_node_lock_test.cljs | 56 ++++- .../frontend/worker/db_worker_node_test.cljs | 40 ++++ src/test/logseq/cli/format_test.cljs | 50 +++++ src/test/logseq/cli/server_test.cljs | 121 +++++++++++ src/test/logseq/db_worker/daemon_test.cljs | 43 ++++ 16 files changed, 827 insertions(+), 61 deletions(-) create mode 100644 docs/agent-guide/034-db-worker-node-owner-process-management.md diff --git a/docs/agent-guide/033-desktop-db-worker-node-backend.md b/docs/agent-guide/033-desktop-db-worker-node-backend.md index c6261231df..592d4ed0ab 100644 --- a/docs/agent-guide/033-desktop-db-worker-node-backend.md +++ b/docs/agent-guide/033-desktop-db-worker-node-backend.md @@ -14,6 +14,8 @@ Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. +Related: Owner-aware lifecycle follow-up is documented in `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + ## Problem statement The current desktop app uses an OPFS-backed SQLite worker in the renderer and periodically exports to disk through `persist-db/run-export-periodically!`. diff --git a/docs/agent-guide/034-db-worker-node-owner-process-management.md b/docs/agent-guide/034-db-worker-node-owner-process-management.md new file mode 100644 index 0000000000..6d9148b9f8 --- /dev/null +++ b/docs/agent-guide/034-db-worker-node-owner-process-management.md @@ -0,0 +1,205 @@ +# DB Worker Node Owner-Aware Process Management Implementation Plan + +Goal: Add owner-aware lock metadata and orphan-process recovery so CLI and Electron can safely share one graph daemon without cross-managing each other. + +Architecture: Keep one `db-worker.lock` per graph directory, but extend lock schema with `owner-source` so lifecycle actions can enforce owner boundaries. +Architecture: Keep read and write traffic reusable across clients, while restricting `stop` and `restart` to the side that originally started the daemon. +Architecture: Add orphan-process detection for lock-missing cases so `logseq server restart` does not hang on timeout when a legacy process is still alive. + +Tech Stack: ClojureScript, Node.js child process APIs, `promesa`, `logseq.cli.server`, `logseq.db-worker.daemon`, Electron main-process daemon manager, db-worker-node lock helpers. + +Related: Builds on `docs/agent-guide/033-desktop-db-worker-node-backend.md`. +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. +Related: Relates to `docs/agent-guide/003-db-worker-node-cli-orchestration.md`. + +## Problem statement + +Current lock payload only records repo, pid, host, and port, so ownership is implicit and lifecycle commands cannot distinguish CLI-started and Electron-started daemons. + +`stop-server!` and `restart-server!` can currently terminate any alive daemon if the lock exists, which violates the requirement that each client should manage only its own process. + +If a db-worker-node process remains alive but lock file is missing, `server restart` can wait until timeout because startup relies on lock appearance and has no orphan recovery path. + +When CLI starts a daemon first, Electron may treat the runtime as managed-by-self and attempt stop or restart logic, which can break graph open flow and produce user-facing errors. + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before each implementation change. + +I will add lock schema and owner compatibility tests in `src/test/frontend/worker/db_worker_node_lock_test.cljs`. + +I will add daemon-owner lifecycle and orphan-recovery tests in `src/test/logseq/cli/server_test.cljs` and `src/test/logseq/db_worker/daemon_test.cljs`. + +I will add Electron manager tests for external-runtime attachment and no-cross-stop behavior in `src/test/electron/db_worker_manager_test.cljs`. + +I will add db-worker-node argument and lock-write tests in `src/test/frontend/worker/db_worker_node_test.cljs`. + +I will run focused red-green loops first, then run `bb dev:lint-and-test`, and finish with a review pass against `@prompts/review.md`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current behavior | Target behavior | +|---|---|---| +| Lock metadata | No owner field in `db-worker.lock`. | Lock includes `owner-source` as `cli` or `electron`, plus versioned metadata. | +| Lifecycle authority | Any caller can stop or restart lock-owned daemon. | `stop` and `restart` are allowed only for matching owner-source. | +| Runtime reuse | Reuse happens, but manager cannot tell owned vs external runtime. | Reuse still happens, and runtime state tracks `owned?` to prevent cross-stop. | +| Lock missing + orphan process | Startup can timeout with no clear recovery path. | Orphan detection and cleanup path runs before or after failed startup wait. | +| Compatibility | Legacy lock without owner is ambiguous. | Legacy lock is treated as `owner-source: unknown` with explicit policy. | + +## Integration sketch + +```text +CLI or Electron request ensure-server(repo, requester-owner) + -> read lock + -> if lock exists and healthy: + return runtime + ownership(owned/external) + -> if lock missing: + scan orphan db-worker-node process for same repo/data-dir + if orphan found: + terminate orphan + spawn new daemon with --owner-source + -> db-worker-node writes lock {repo,pid,host,port,owner-source,lock-id,...} + +stop/restart(requester-owner) + -> read lock owner-source + -> if owner-source matches requester-owner: allow + -> else: deny with :server-owned-by-other +``` + +## Implementation plan + +### Phase 1: Add failing tests for owner-aware lock schema and policies. + +1. Add a failing test in `src/test/frontend/worker/db_worker_node_lock_test.cljs` that lock serialization includes `owner-source`. +2. Add a failing test in `src/test/frontend/worker/db_worker_node_lock_test.cljs` that missing owner metadata is normalized to `:unknown`. +3. Add a failing test in `src/test/frontend/worker/db_worker_node_test.cljs` that `--owner-source cli` is written into the lock. +4. Add a failing test in `src/test/frontend/worker/db_worker_node_test.cljs` that `--owner-source electron` is written into the lock. +5. Add a failing test in `src/test/logseq/cli/server_test.cljs` that `stop-server!` returns `:server-owned-by-other` on owner mismatch. +6. Add a failing test in `src/test/logseq/cli/server_test.cljs` that `restart-server!` does not SIGTERM external-owner daemon. +7. Add a failing test in `src/test/electron/db_worker_manager_test.cljs` that external runtime release does not call `stop-daemon!`. +8. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and confirm failures match new assertions. +9. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm failures match new assertions. + +### Phase 2: Extend lock schema and daemon startup arguments. + +10. Update argument parsing in `src/main/frontend/worker/db_worker_node.cljs` to accept `--owner-source`. +11. Add owner-source validation in `src/main/frontend/worker/db_worker_node.cljs` with allowed values `cli`, `electron`, and fallback `unknown`. +12. Thread owner-source through `start-daemon!` in `src/main/frontend/worker/db_worker_node.cljs` into lock creation. +13. Update `create-lock!` in `src/main/frontend/worker/db_worker_node_lock.cljs` to persist `owner-source`. +14. Update `read-lock` normalization path in `src/main/frontend/worker/db_worker_node_lock.cljs` to inject `:owner-source :unknown` for legacy files. +15. Keep `update-lock!` in `src/main/frontend/worker/db_worker_node_lock.cljs` from mutating existing owner-source during port updates. +16. Add targeted tests for lock read-write roundtrip in `src/test/frontend/worker/db_worker_node_lock_test.cljs`. +17. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and make sure lock owner assertions pass. + +### Phase 3: Make CLI server orchestration owner-aware. + +18. Add requester owner config to `ensure-server!` in `src/main/logseq/cli/server.cljs`. +19. Pass owner-source to daemon spawn args from `spawn-server!` in `src/main/logseq/db_worker/daemon.cljs`. +20. Update `ensure-server-started!` in `src/main/logseq/cli/server.cljs` to return ownership metadata for caller state tracking. +21. Update `stop-server!` in `src/main/logseq/cli/server.cljs` to deny stop when lock owner-source differs from requester owner. +22. Update `restart-server!` in `src/main/logseq/cli/server.cljs` to preserve the same owner check semantics as `stop-server!`. +23. Update `server-status` and `list-servers` in `src/main/logseq/cli/server.cljs` to include owner-source in output payload. +24. Update server command response formatting so `logseq server list` human output includes an `OWNER` column mapped from `owner-source` (and preserves owner metadata in structured output). +25. Add regression tests in `src/test/logseq/cli/server_test.cljs` for owner-aware start, stop, and restart. +26. Run `bb dev:test -v 'logseq.cli.server-test'` and verify no timeout-based flaky failure remains. + +### Phase 4: Add orphan-process detection and recovery for lock-missing start. + +27. Add process listing helper in `src/main/logseq/db_worker/daemon.cljs` that discovers db-worker-node processes by repo and data-dir arguments. +28. Add parser helpers in `src/main/logseq/db_worker/daemon.cljs` to read `repo`, `data-dir`, and `owner-source` from command args. +29. Add `cleanup-orphan-process!` in `src/main/logseq/db_worker/daemon.cljs` to SIGTERM matched orphan pids before new spawn. +30. Call orphan cleanup path in `ensure-server-started!` in `src/main/logseq/cli/server.cljs` when lock is missing before spawn. +31. Add timeout fallback in `ensure-server-started!` in `src/main/logseq/cli/server.cljs` to emit `:server-start-timeout-orphan` with discovered pids. +32. Add unit tests in `src/test/logseq/db_worker/daemon_test.cljs` for process-arg parsing and orphan match logic. +33. Add CLI regression test in `src/test/logseq/cli/server_test.cljs` for lock-missing orphan scenario to avoid raw timeout. +34. Run `bb dev:test -v 'logseq.db-worker.daemon-test'` and `bb dev:test -v 'logseq.cli.server-test'`. + +### Phase 5: Make Electron manager attach external daemon without cross-management. + +35. Pass requester owner as `electron` from `start-managed-daemon!` in `src/electron/electron/db_worker.cljs`. +36. Save ownership flag in manager runtime state in `src/electron/electron/db_worker.cljs`. +37. Update stop flow in `src/electron/electron/db_worker.cljs` so `stop-daemon!` runs only when `owned?` is true. +38. Update unhealthy-runtime branch in `src/electron/electron/db_worker.cljs` to avoid stopping external owner daemon and re-resolve runtime instead. +39. Add tests in `src/test/electron/db_worker_manager_test.cljs` for external runtime reuse plus no-stop-on-release. +40. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm lifecycle behavior. + +### Phase 6: Update docs and error surfaces. + +41. Update CLI docs in `docs/cli/logseq-cli.md` to document owner-aware `server stop` and `server restart` behavior. +42. Update desktop lifecycle docs in `docs/developers/desktop-db-worker-node.md` to explain external runtime attachment semantics. +43. Add explicit error messages for `:server-owned-by-other` and `:server-start-timeout-orphan` in `src/main/logseq/cli/format.cljs`. +44. Add one integration note in `docs/agent-guide/033-desktop-db-worker-node-backend.md` linking to owner-aware behavior. + +### Phase 7: Full verification and review gate. + +45. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'`. +46. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'`. +47. Run `bb dev:test -v 'logseq.db-worker.daemon-test'`. +48. Run `bb dev:test -v 'logseq.cli.server-test'`. +49. Run `bb dev:test -v 'electron.db-worker-manager-test'`. +50. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +51. Perform final review checklist pass against `@prompts/review.md`. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Lock file missing but old CLI daemon still alive. | CLI restart detects orphan by repo and data-dir, cleans it, then starts cleanly. | +| Lock owner is `cli` and Electron calls ensure runtime. | Electron reuses runtime with `owned? false` and never stops it on window close. | +| Lock owner is `electron` and CLI calls `server stop`. | CLI returns `:server-owned-by-other` with owner metadata and no process kill. | +| Legacy lock file has no owner-source field. | System treats owner as `unknown` and allows CLI takeover with owner metadata rewrite. | +| Two owners race to start same graph. | First lock wins and second caller reuses healthy daemon without extra spawn. | +| Owner process crashes and lock remains stale. | Stale lock cleanup still works, and next owner can start daemon normally. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'frontend.worker.db-worker-node-lock-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +bb dev:test -v 'logseq.db-worker.daemon-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +The owner-mismatch tests should return `:server-owned-by-other` instead of timeout or forced stop behavior. + +`logseq server list` human output should include an `OWNER` column. + +The orphan-recovery tests should return deterministic cleanup behavior instead of waiting until generic timeout. + +## Testing Details + +The tests validate behavior by asserting lifecycle authority boundaries, successful runtime reuse across clients, and orphan recovery outcomes. + +The tests avoid mock-only success criteria by asserting returned error codes and process-management side effects observable from public APIs. + +The critical regressions are lock-missing orphan restart and CLI-first then Electron-open graph flow, and both are explicitly covered. + +## Implementation Details + +- Extend lock payload with `owner-source` and preserve it across lock updates. +- Pass `--owner-source` when spawning db-worker-node from both CLI and Electron pathways. +- Return ownership metadata from server orchestration so callers can track `owned?` state. +- Enforce owner check for stop and restart while keeping read and write invoke reuse unchanged. +- Add orphan process discovery by command args for lock-missing recovery. +- Scope orphan process discovery in v1 to macOS and Linux, and use a Windows-safe no-op fallback. +- Keep stale-lock cleanup logic and layer orphan recovery without changing healthy lock reuse flow. +- Add explicit CLI error codes for owner mismatch and orphan timeout contexts. +- Prevent Electron manager from stopping external-owner runtime on release or health fallback. +- Document operator-visible behavior changes in CLI and desktop developer docs. +- Execute full suite and `@prompts/review.md` checks before merge. + +## Question + +Decision: CLI is allowed to take over `owner-source: unknown` and rewrite ownership metadata in v1. + +Decision: when lock file is missing, orphan cleanup terminates all matching repo and data-dir processes. + +Decision: v1 scopes orphan process-scan support to macOS and Linux only, with a Windows-safe no-op fallback. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 1a611e209e..811026221d 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -17,6 +17,9 @@ Desktop + CLI shared semantics: - Disk SQLite under `~/logseq/graphs` is the source of truth; OPFS periodic export is not part of the desktop primary write path. - If a daemon already exists for the graph, CLI reuses it via lock-file discovery instead of starting a second writer. - If lock ownership is invalid or stale, startup cleans stale lock state before retrying. +- Lock metadata includes an `owner-source` value (`cli`, `electron`, `unknown`) and lifecycle actions enforce owner boundaries. +- `server stop` and `server restart` are owner-aware: CLI can only stop/restart servers it owns (or legacy `unknown` ownership). +- If lock is missing but a matching orphan `db-worker-node` process still exists for the same repo/data-dir, startup performs orphan cleanup before retrying. ## Run the CLI @@ -79,6 +82,11 @@ Server commands: - `server restart --repo ` - restart db-worker-node for a graph - `doctor` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness +Server ownership behavior: +- `server stop` and `server restart` can return `server-owned-by-other` if the daemon was started by another owner source. +- `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. +- `server list` human output includes an `OWNER` column, and `server status` / `server list` include owner metadata in structured output (`--output json|edn`). + Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags diff --git a/src/electron/electron/db_worker.cljs b/src/electron/electron/db_worker.cljs index dc3e1eed0e..67ff0087f5 100644 --- a/src/electron/electron/db_worker.cljs +++ b/src/electron/electron/db_worker.cljs @@ -60,6 +60,10 @@ :runtime-ready? (or runtime-ready? (fn [_runtime] (p/resolved true))) :state (atom (initial-state))}) +(defn- owned-runtime? + [runtime] + (not= false (:owned? runtime))) + (defn ensure-window-stopped! [{:keys [state stop-daemon!]} window-id] (let [runtime* (atom nil)] @@ -69,8 +73,10 @@ (reset! runtime* runtime) next-state))) (if-let [runtime @runtime*] - (p/let [_ (stop-daemon! runtime)] - true) + (if (owned-runtime? runtime) + (p/let [_ (stop-daemon! runtime)] + true) + (p/resolved true)) (p/resolved false)))) (defn ensure-started! @@ -88,8 +94,9 @@ (update-in [:repos repo :windows] (fnil conj #{}) window-id) (assoc-in [:window->repo window-id] repo)))) runtime) - (p/let [_ (-> (stop-daemon! runtime) - (p/catch (fn [_] nil))) + (p/let [_ (when (owned-runtime? runtime) + (-> (stop-daemon! runtime) + (p/catch (fn [_] nil)))) runtime' (start-daemon! repo)] (swap! state (fn [current] @@ -139,15 +146,19 @@ (reset! runtime* runtime) next-state))) (if-let [runtime @runtime*] - (p/let [_ (stop-daemon! runtime)] - true) + (if (owned-runtime? runtime) + (p/let [_ (stop-daemon! runtime)] + true) + (p/resolved true)) (p/resolved false))))) (defn stop-all! [{:keys [state stop-daemon!]}] (let [entries (vals (:repos (ensure-state @state)))] (-> (p/all (map (fn [{:keys [runtime]}] - (stop-daemon! runtime)) + (if (owned-runtime? runtime) + (stop-daemon! runtime) + (p/resolved true))) entries)) (p/then (fn [_] (reset! state (initial-state)) @@ -155,14 +166,15 @@ (defn- start-managed-daemon! [repo] - (p/let [config (cli-server/ensure-server! {} repo)] + (p/let [config (cli-server/ensure-server! {:owner-source :electron} repo)] {:repo repo :base-url (:base-url config) - :auth-token nil})) + :auth-token nil + :owned? (:owned? config)})) (defn- stop-managed-daemon! [{:keys [repo]}] - (p/let [result (cli-server/stop-server! {} repo)] + (p/let [result (cli-server/stop-server! {:owner-source :electron} repo)] (:ok? result))) (defonce manager diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 8a5667c37f..b51ca8f897 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -49,11 +49,16 @@ (case flag "--data-dir" (recur remaining (assoc opts :data-dir value)) "--repo" (recur remaining (assoc opts :repo value)) + "--owner-source" (recur remaining (assoc opts :owner-source value)) "--rtc-ws-url" (recur remaining (assoc opts :rtc-ws-url value)) "--log-level" (recur remaining (assoc opts :log-level value)) "--help" (recur remaining (assoc opts :help? true)) (recur remaining opts)))))) +(defn- normalize-owner-source + [owner-source] + (db-lock/normalize-owner-source owner-source)) + (defn- encode-event-payload [payload] (if (string? payload) @@ -339,9 +344,10 @@ file-path)) (defn start-daemon! - [{:keys [data-dir repo rtc-ws-url log-level]}] + [{:keys [data-dir repo rtc-ws-url log-level owner-source]}] (let [host "127.0.0.1" - port 0] + port 0 + owner-source (normalize-owner-source owner-source)] (if-not (seq repo) (p/rejected (ex-info "repo is required" {:code :missing-repo})) (try @@ -363,7 +369,8 @@ {:keys [path lock]} (db-lock/ensure-lock! {:data-dir data-dir :repo repo :host host - :port port}) + :port port + :owner-source owner-source}) _ (reset! *lock-info {:path path :lock lock}) _ (let [method-kw :thread-api/create-or-open-db method-str (normalize-method-str method-kw)] @@ -415,7 +422,7 @@ (defn main [] - (let [{:keys [data-dir repo rtc-ws-url help?] :as opts} + (let [{:keys [data-dir repo rtc-ws-url help? owner-source] :as opts} (parse-args (.-argv js/process))] (when help? (show-help!) @@ -426,6 +433,7 @@ (-> (p/let [{:keys [stop!] :as daemon} (start-daemon! {:data-dir data-dir :repo repo + :owner-source owner-source :rtc-ws-url rtc-ws-url :log-level (:log-level opts)})] (log/info :db-worker-node-ready {:host (:host daemon) :port (:port daemon)}) diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index 77044572bf..cf68a4f7e7 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -55,11 +55,30 @@ "EPERM" :no-permission :error))))) +(def ^:private valid-owner-sources + #{:cli :electron :unknown}) + +(defn normalize-owner-source + [owner-source] + (let [owner-source (cond + (keyword? owner-source) owner-source + (string? owner-source) (keyword owner-source) + :else :unknown)] + (if (contains? valid-owner-sources owner-source) + owner-source + :unknown))) + +(defn- normalize-lock + [lock] + (when lock + (assoc lock :owner-source (normalize-owner-source (:owner-source lock))))) + (defn read-lock [path] (when (and (seq path) (fs/existsSync path)) - (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) - :keywordize-keys true))) + (normalize-lock + (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true)))) (defn remove-lock! [path] @@ -67,7 +86,7 @@ (fs/unlinkSync path))) (defn create-lock! - [{:keys [data-dir repo host port]}] + [{:keys [data-dir repo host port owner-source]}] (p/create (fn [resolve reject] (try @@ -85,6 +104,7 @@ :lock-id (str (random-uuid)) :host host :port port + :owner-source (normalize-owner-source owner-source) :startedAt (.toISOString (js/Date.))}] (try (fs/writeFileSync fd (js/JSON.stringify (clj->js lock))) @@ -106,8 +126,9 @@ (assoc :repo (:repo existing)) (assoc :pid (:pid existing)) (assoc :lock-id (or (:lock-id existing) (:lock-id lock))) + (assoc :owner-source (normalize-owner-source (:owner-source existing))) (assoc :startedAt (:startedAt existing))) - lock)] + (update lock :owner-source normalize-owner-source))] (fs/writeFileSync path (js/JSON.stringify (clj->js lock'))) (resolve lock')) (catch :default e @@ -159,12 +180,13 @@ lock))) (defn ensure-lock! - [{:keys [data-dir repo host port]}] + [{:keys [data-dir repo host port owner-source]}] (let [data-dir (resolve-data-dir data-dir) path (lock-path data-dir repo)] (p/let [lock (create-lock! {:data-dir data-dir :repo repo :host host - :port port})] + :port port + :owner-source owner-source})] {:path path :lock lock}))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index d5d40642aa..ed5cfacb5f 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -93,6 +93,8 @@ :missing-query "Use --query " :unknown-query "Use `logseq query list` to see available queries" :data-dir-permission "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR" + :server-owned-by-other "Retry from the process owner that started the server" + :server-start-timeout-orphan "Check and stop lingering db-worker-node processes, then retry" nil)) (defn- format-error @@ -176,13 +178,14 @@ (defn- format-server-list [servers] (format-counted-table - ["REPO" "STATUS" "HOST" "PORT" "PID"] + ["REPO" "STATUS" "HOST" "PORT" "PID" "OWNER"] (mapv (fn [server] [(:repo server) (:status server) (:host server) (:port server) - (:pid server)]) + (:pid server) + (:owner-source server)]) (or servers [])))) (defn- format-query-results diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index f3d3ab3f36..a7f1d3ccd5 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -61,6 +61,33 @@ [{:keys [host port]}] (str "http://" host ":" port)) +(defn- normalize-owner-source + [owner-source] + (daemon/normalize-owner-source owner-source)) + +(defn- requester-owner-source + [config] + (normalize-owner-source (or (:owner-source config) :cli))) + +(defn- lock-owner-source + [lock] + (normalize-owner-source (:owner-source lock))) + +(defn- owner-manageable? + [requester-owner lock-owner] + (or (= requester-owner lock-owner) + (and (= requester-owner :cli) + (= lock-owner :unknown)))) + +(defn- owner-mismatch-error + [repo requester-owner lock-owner] + {:ok? false + :error {:code :server-owned-by-other + :message "server is owned by another process" + :repo repo + :owner-source lock-owner + :requester-owner-source requester-owner}}) + (defn- pid-status [pid] (daemon/pid-status pid)) @@ -98,31 +125,64 @@ (daemon/ready? lock)) (defn- spawn-server! - [{:keys [repo data-dir]}] + [{:keys [repo data-dir owner-source]}] (daemon/spawn-server! {:script (db-worker-script-path) :repo repo - :data-dir data-dir})) + :data-dir data-dir + :owner-source owner-source})) + +(defn- rewrite-lock-owner-source! + [path lock owner-source] + (let [lock' (assoc lock :owner-source (normalize-owner-source owner-source))] + (fs/writeFileSync path (js/JSON.stringify (clj->js lock'))) + lock')) (defn- ensure-server-started! [config repo] (let [data-dir (resolve-data-dir config) - path (lock-path data-dir repo)] + path (lock-path data-dir repo) + requester-owner (requester-owner-source config)] (ensure-repo-dir! data-dir repo) (p/let [existing (read-lock path) _ (cleanup-stale-lock! path existing) _ (when (not (fs/existsSync path)) - (spawn-server! {:repo repo :data-dir data-dir}) - (wait-for-lock path)) - lock (read-lock path)] + (daemon/cleanup-orphan-processes! {:repo repo + :data-dir data-dir}) + (spawn-server! {:repo repo + :data-dir data-dir + :owner-source requester-owner}) + (-> (wait-for-lock path) + (p/catch (fn [e] + (if (= :timeout (:code (ex-data e))) + (let [orphans (daemon/find-orphan-processes {:repo repo + :data-dir data-dir}) + pids (mapv :pid orphans)] + (throw (ex-info "db-worker-node failed to create lock" + {:code :server-start-timeout-orphan + :repo repo + :pids pids}))) + (throw e)))))) + lock (read-lock path) + lock (if (and lock + (= :cli requester-owner) + (= :unknown (lock-owner-source lock))) + (rewrite-lock-owner-source! path lock :cli) + lock)] (when-not lock (throw (ex-info "db-worker-node failed to start" {:code :server-start-failed}))) (p/let [_ (wait-for-ready lock)] - lock)))) + (let [lock-owner (lock-owner-source lock)] + (assoc lock + :owner-source lock-owner + :owned? (owner-manageable? requester-owner lock-owner))))))) (defn ensure-server! [config repo] (p/let [lock (ensure-server-started! config repo)] - (assoc config :base-url (base-url lock)))) + (assoc config + :base-url (base-url lock) + :owner-source (:owner-source lock) + :owned? (:owned? lock)))) (defn- shutdown! [{:keys [host port]}] @@ -136,55 +196,65 @@ (defn stop-server! [config repo] - (let [data-dir (resolve-data-dir config) + (let [requester-owner (requester-owner-source config) + data-dir (resolve-data-dir config) path (lock-path data-dir repo) lock (read-lock path)] (if-not lock (p/resolved {:ok? false :error {:code :server-not-found :message "server is not running"}}) - (-> (p/let [_ (shutdown! lock)] + (let [lock-owner (lock-owner-source lock)] + (if-not (owner-manageable? requester-owner lock-owner) + (p/resolved (owner-mismatch-error repo requester-owner lock-owner)) + (-> (p/let [_ (shutdown! lock)] (wait-for (fn [] (p/resolved (not (fs/existsSync path)))) {:timeout-ms 5000 :interval-ms 200}) {:ok? true :data {:repo repo}}) - (p/catch (fn [_] - (when (and (= :alive (pid-status (:pid lock))) - (not= (:pid lock) (.-pid js/process))) - (try - (.kill js/process (:pid lock) "SIGTERM") - (catch :default e - (log/warn :cli-server-stop-sigterm-failed e)))) - (when (= :not-found (pid-status (:pid lock))) - (remove-lock! path)) - (if (fs/existsSync path) - {:ok? false - :error {:code :server-stop-timeout - :message "timed out stopping server"}} - {:ok? true - :data {:repo repo}}))))))) + (p/catch (fn [_] + (when (and (= :alive (pid-status (:pid lock))) + (not= (:pid lock) (.-pid js/process))) + (try + (.kill js/process (:pid lock) "SIGTERM") + (catch :default e + (log/warn :cli-server-stop-sigterm-failed e)))) + (when (= :not-found (pid-status (:pid lock))) + (remove-lock! path)) + (if (fs/existsSync path) + {:ok? false + :error {:code :server-stop-timeout + :message "timed out stopping server"}} + {:ok? true + :data {:repo repo}}))))))))) (defn start-server! [config repo] - (-> (p/let [_ (ensure-server-started! config repo)] + (-> (p/let [lock (ensure-server-started! config repo)] {:ok? true - :data {:repo repo}}) + :data {:repo repo + :owner-source (:owner-source lock) + :owned? (:owned? lock)}}) (p/catch (fn [e] (let [data (ex-data e) code (or (:code data) :server-start-failed)] {:ok? false :error (cond-> {:code code :message (or (.-message e) "failed to start server")} - (:lock data) (assoc :lock (:lock data)))}))))) + (:lock data) (assoc :lock (:lock data)) + (:pids data) (assoc :pids (:pids data)) + (:repo data) (assoc :repo (:repo data)))}))))) (defn restart-server! [config repo] - (-> (p/let [_ (stop-server! config repo)] - (start-server! config repo)) - (p/catch (fn [_] - (start-server! config repo))))) + (p/let [stop-result (stop-server! config repo)] + (if (:ok? stop-result) + (start-server! config repo) + (if (= :server-not-found (get-in stop-result [:error :code])) + (start-server! config repo) + stop-result)))) (defn server-status [config repo] @@ -202,6 +272,7 @@ :host (:host lock) :port (:port lock) :pid (:pid lock) + :owner-source (lock-owner-source lock) :started-at (:startedAt lock)}})))) (defn list-servers @@ -222,6 +293,7 @@ :host (:host lock) :port (:port lock) :pid (:pid lock) + :owner-source (lock-owner-source lock) :status (if ready :ready :starting)}))))) (defn list-graphs diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs index 50db7f4943..5a6ac29f7a 100644 --- a/src/main/logseq/db_worker/daemon.cljs +++ b/src/main/logseq/db_worker/daemon.cljs @@ -3,9 +3,119 @@ (:require ["child_process" :as child-process] ["fs" :as fs] ["http" :as http] + ["path" :as node-path] + [clojure.string :as string] [lambdaisland.glogi :as log] [promesa.core :as p])) +(def ^:private valid-owner-sources + #{:cli :electron :unknown}) + +(defn normalize-owner-source + [owner-source] + (let [owner-source (cond + (keyword? owner-source) owner-source + (string? owner-source) (keyword (string/trim owner-source)) + :else :unknown)] + (if (contains? valid-owner-sources owner-source) + owner-source + :unknown))) + +(defn- platform-supports-process-scan? + [] + (contains? #{"darwin" "linux"} (.-platform js/process))) + +(defn- normalize-dir + [path] + (when (seq path) + (node-path/resolve path))) + +(defn- unquote-arg + [value] + (if (and (string? value) + (>= (count value) 2) + (or (and (string/starts-with? value "\"") + (string/ends-with? value "\"")) + (and (string/starts-with? value "'") + (string/ends-with? value "'")))) + (subs value 1 (dec (count value))) + value)) + +(defn- extract-arg + [command flag] + (some-> (re-find (re-pattern (str "(?:^|\\s)" flag "\\s+((?:\"[^\"]+\"|'[^']+'|\\S+))")) command) + second + unquote-arg)) + +(defn parse-process-args + [command] + (let [command (string/trim (or command ""))] + (when (and (seq command) + (re-find #"db-worker-node(?:\.js)?\b" command)) + (let [repo (extract-arg command "--repo") + data-dir (extract-arg command "--data-dir") + owner-source (normalize-owner-source (extract-arg command "--owner-source"))] + (when (and (seq repo) (seq data-dir)) + {:repo repo + :data-dir (normalize-dir data-dir) + :owner-source owner-source}))))) + +(defn- parse-process-line + [line] + (let [line (string/trim (or line ""))] + (when-let [[_ pid-str command] (and (seq line) + (re-matches #"^(\d+)\s+(.*)$" line))] + (let [pid (js/parseInt pid-str 10)] + (when (and (number? pid) (pos-int? pid)) + (when-let [args (parse-process-args command)] + (assoc args + :pid pid + :command command))))))) + +(defn list-db-worker-processes + [] + (if-not (platform-supports-process-scan?) + [] + (try + (let [output (.execFileSync child-process "ps" + #js ["-ax" "-o" "pid=" "-o" "command="] + #js {:encoding "utf8"})] + (->> (string/split-lines (or output "")) + (keep parse-process-line) + (vec))) + (catch :default e + (log/warn :db-worker-daemon/process-scan-failed e) + [])))) + +(defn find-orphan-processes + [{:keys [repo data-dir]}] + (let [data-dir (normalize-dir data-dir)] + (->> (list-db-worker-processes) + (filter (fn [process] + (and (= repo (:repo process)) + (= data-dir (:data-dir process))))) + (vec)))) + +(defn cleanup-orphan-processes! + [{:keys [repo data-dir]}] + (let [orphans (find-orphan-processes {:repo repo :data-dir data-dir}) + current-pid (.-pid js/process) + killed-pids (reduce (fn [result {:keys [pid]}] + (if (= current-pid pid) + result + (try + (.kill js/process pid "SIGTERM") + (conj result pid) + (catch :default e + (when-not (= "ESRCH" (.-code e)) + (log/warn :db-worker-daemon/orphan-kill-failed + {:pid pid :error e})) + result)))) + [] + orphans)] + {:orphans orphans + :killed-pids killed-pids})) + (defn pid-status [pid] (when (number? pid) @@ -21,8 +131,9 @@ (defn read-lock [path] (when (and (seq path) (fs/existsSync path)) - (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) - :keywordize-keys true))) + (let [lock (js->clj (js/JSON.parse (.toString (fs/readFileSync path) "utf8")) + :keywordize-keys true)] + (assoc lock :owner-source (normalize-owner-source (:owner-source lock)))))) (defn remove-lock! [path] @@ -143,8 +254,9 @@ :interval-ms 250})) (defn spawn-server! - [{:keys [script repo data-dir]}] - (let [args #js ["--repo" repo "--data-dir" data-dir] + [{:keys [script repo data-dir owner-source]}] + (let [owner-source (normalize-owner-source owner-source) + args #js ["--repo" repo "--data-dir" data-dir "--owner-source" (name owner-source)] child (.spawn child-process script args #js {:detached true :stdio "ignore"})] (when-not script diff --git a/src/test/electron/db_worker_manager_test.cljs b/src/test/electron/db_worker_manager_test.cljs index 0c5ca57ded..77c8650a37 100644 --- a/src/test/electron/db_worker_manager_test.cljs +++ b/src/test/electron/db_worker_manager_test.cljs @@ -139,3 +139,19 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] (done))))))) + +(deftest ensure-window-stopped-does-not-stop-external-runtime + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] + (p/resolved (assoc (runtime repo) :owned? false))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-window-stopped! manager :window-1)] + (is (empty? @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) diff --git a/src/test/frontend/fs_test.cljs b/src/test/frontend/fs_test.cljs index fed880a2c6..5ecd98e21f 100644 --- a/src/test/frontend/fs_test.cljs +++ b/src/test/frontend/fs_test.cljs @@ -1,5 +1,5 @@ (ns frontend.fs-test - (:require [clojure.test :refer [is use-fixtures]] + (:require [cljs.test :refer [is use-fixtures]] [frontend.test.node-fixtures :as node-fixtures] [frontend.test.helper :as test-helper :include-macros true :refer [deftest-async]] [frontend.test.node-helper :as test-node-helper] diff --git a/src/test/frontend/worker/db_worker_node_lock_test.cljs b/src/test/frontend/worker/db_worker_node_lock_test.cljs index e024d15c98..37283c556b 100644 --- a/src/test/frontend/worker/db_worker_node_lock_test.cljs +++ b/src/test/frontend/worker/db_worker_node_lock_test.cljs @@ -2,9 +2,10 @@ (:require ["fs" :as fs] ["os" :as os] ["path" :as node-path] - [cljs.test :refer [deftest is testing]] + [cljs.test :refer [async deftest is testing]] [frontend.test.node-helper :as node-helper] - [frontend.worker.db-worker-node-lock :as db-lock])) + [frontend.worker.db-worker-node-lock :as db-lock] + [promesa.core :as p])) (deftest repo-dir-canonicalizes-db-prefixed-repo (testing "db-prefixed repo name resolves to prefix-free graph directory key" @@ -35,3 +36,54 @@ expected-lock-path (node-path/join expected-data-dir "demo" "db-worker.lock")] (is (= expected-data-dir default-data-dir)) (is (= expected-lock-path (db-lock/lock-path default-data-dir "logseq_db_demo")))))) + +(deftest create-lock-persists-owner-source + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-owner") + repo (str "logseq_db_lock_owner_" (subs (str (random-uuid)) 0 8)) + path (db-lock/lock-path data-dir repo)] + (-> (p/let [_ (db-lock/create-lock! {:data-dir data-dir + :repo repo + :host "127.0.0.1" + :port 9101 + :owner-source :cli}) + lock (db-lock/read-lock path)] + (is (= :cli (:owner-source lock)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (db-lock/remove-lock! path) + (done))))))) + +(deftest read-lock-normalizes-missing-owner-source-to-unknown + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-legacy-owner") + repo (str "logseq_db_lock_legacy_" (subs (str (random-uuid)) 0 8)) + path (db-lock/lock-path data-dir repo) + legacy-lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9101 + :startedAt (.toISOString (js/Date.))}] + (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) + (fs/writeFileSync path (js/JSON.stringify (clj->js legacy-lock))) + (is (= :unknown (:owner-source (db-lock/read-lock path)))))) + +(deftest update-lock-preserves-existing-owner-source + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-update-owner") + repo (str "logseq_db_lock_update_owner_" (subs (str (random-uuid)) 0 8)) + path (db-lock/lock-path data-dir repo)] + (-> (p/let [{:keys [lock]} (db-lock/ensure-lock! {:data-dir data-dir + :repo repo + :host "127.0.0.1" + :port 9101 + :owner-source :cli}) + _ (db-lock/update-lock! path (assoc lock :port 9200 :owner-source :electron)) + updated (db-lock/read-lock path)] + (is (= :cli (:owner-source updated))) + (is (= 9200 (:port updated)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (db-lock/remove-lock! path) + (done))))))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 85dc5ad21d..8cc651c1ce 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -211,6 +211,46 @@ (is (nil? (:auth-token result))) (is (= "/tmp/db-worker" (:data-dir result))))) +(deftest db-worker-node-owner-source-cli-is-written-into-lock + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-owner-source-cli") + repo (str "logseq_db_owner_cli_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo)] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :owner-source :cli}) + _ (reset! daemon {:stop! stop!}) + lock-json (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8"))] + (is (= "cli" (gobj/get lock-json "owner-source")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + +(deftest db-worker-node-owner-source-electron-is-written-into-lock + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-owner-source-electron") + repo (str "logseq_db_owner_electron_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo)] + (-> (p/let [{:keys [stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :owner-source :electron}) + _ (reset! daemon {:stop! stop!}) + lock-json (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8"))] + (is (= "electron" (gobj/get lock-json "owner-source")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-handle-event-encodes-sse-json-payload (let [handle-event! #'db-worker-node/handle-event! *sse-clients @#'db-worker-node/*sse-clients diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index bfd3aec065..1374fbffd3 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -177,6 +177,36 @@ "Host: 127.0.0.1 Port: 1234") result))))) +(deftest test-human-output-server-list-includes-owner + (testing "server list shows owner column and value" + (let [result (format/format-result {:status :ok + :command :server-list + :data {:servers [{:repo "demo-repo" + :status :ready + :host "127.0.0.1" + :port 1234 + :pid 9876 + :owner-source :cli}]}} + {:output-format nil})] + (is (= (str "REPO STATUS HOST PORT PID OWNER\n" + "demo-repo :ready 127.0.0.1 1234 9876 :cli\n" + "Count: 1") + result)))) + + (testing "server list falls back to placeholder when owner is missing" + (let [result (format/format-result {:status :ok + :command :server-list + :data {:servers [{:repo "demo-repo" + :status :ready + :host "127.0.0.1" + :port 1234 + :pid 9876}]}} + {:output-format nil})] + (is (= (str "REPO STATUS HOST PORT PID OWNER\n" + "demo-repo :ready 127.0.0.1 1234 9876 -\n" + "Count: 1") + result))))) + (deftest test-human-output-show (testing "show renders text payloads directly" (let [result (format/format-result {:status :ok @@ -335,6 +365,26 @@ {:output-format nil})] (is (= (str "Error (missing-graph): graph name is required\n" "Hint: Use --repo ") + result)))) + + (testing "owner mismatch includes ownership hint" + (let [result (format/format-result {:status :error + :command :server-stop + :error {:code :server-owned-by-other + :message "server is owned by another process"}} + {:output-format nil})] + (is (= (str "Error (server-owned-by-other): server is owned by another process\n" + "Hint: Retry from the process owner that started the server") + result)))) + + (testing "orphan timeout includes recovery hint" + (let [result (format/format-result {:status :error + :command :server-start + :error {:code :server-start-timeout-orphan + :message "db-worker-node failed to create lock"}} + {:output-format nil})] + (is (= (str "Error (server-start-timeout-orphan): db-worker-node failed to create lock\n" + "Hint: Check and stop lingering db-worker-node processes, then retry") result))))) (deftest test-human-output-doctor diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index e1ce32418e..e8ef5f7f10 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -132,3 +132,124 @@ (set! daemon/cleanup-stale-lock! original-cleanup) (set! daemon/wait-for-ready original-ready) (done))))))) + +(deftest stop-server-denies-owner-mismatch + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-stop") + repo (str "logseq_db_owner_stop_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9101 + :owner-source :electron} + original-http daemon/http-request + original-wait daemon/wait-for] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (set! daemon/http-request (fn [_] (p/resolved {:status 200 :body ""}))) + (set! daemon/wait-for (fn [_ _] (p/resolved true))) + (-> (cli-server/stop-server! {:data-dir data-dir + :owner-source :cli} + repo) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-owned-by-other (get-in result [:error :code]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/http-request original-http) + (set! daemon/wait-for original-wait) + (done))))))) + +(deftest restart-server-does-not-sigterm-external-owner-daemon + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-restart") + repo (str "logseq_db_owner_restart_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid 424242 + :host "127.0.0.1" + :port 9102 + :owner-source :electron} + start-calls (atom 0) + kill-calls (atom []) + original-start cli-server/start-server! + original-http daemon/http-request + original-wait daemon/wait-for + original-pid-status daemon/pid-status + original-kill (.-kill js/process)] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (set! daemon/http-request (fn [_] (p/resolved {:status 200 :body ""}))) + (set! daemon/wait-for (fn [_ _] (p/rejected (ex-info "timeout" {:code :timeout})))) + (set! daemon/pid-status (fn [_] :alive)) + (set! (.-kill js/process) + (fn [pid signal] + (swap! kill-calls conj [pid signal]) + true)) + (set! cli-server/start-server! + (fn [_ _] + (swap! start-calls inc) + (p/resolved {:ok? true + :data {:repo repo}}))) + (-> (cli-server/restart-server! {:data-dir data-dir + :owner-source :cli} + repo) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-owned-by-other (get-in result [:error :code]))) + (is (zero? @start-calls)) + (is (empty? @kill-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/http-request original-http) + (set! daemon/wait-for original-wait) + (set! daemon/pid-status original-pid-status) + (set! (.-kill js/process) original-kill) + (set! cli-server/start-server! original-start) + (done))))))) + +(deftest start-server-returns-timeout-orphan-error-with-pids + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-orphan-timeout") + repo (str "logseq_db_orphan_timeout_" (subs (str (random-uuid)) 0 8)) + cleanup-calls (atom 0) + spawn-calls (atom 0) + original-cleanup-stale daemon/cleanup-stale-lock! + original-cleanup-orphans daemon/cleanup-orphan-processes! + original-spawn daemon/spawn-server! + original-wait-lock daemon/wait-for-lock + original-find-orphans daemon/find-orphan-processes] + (set! daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil))) + (set! daemon/cleanup-orphan-processes! (fn [_] + (swap! cleanup-calls inc) + {:killed-pids [111]})) + (set! daemon/spawn-server! (fn [_] + (swap! spawn-calls inc) + nil)) + (set! daemon/wait-for-lock (fn [_] + (p/rejected (ex-info "timeout" + {:code :timeout})))) + (set! daemon/find-orphan-processes (fn [_] + [{:pid 111} + {:pid 222}])) + (-> (cli-server/start-server! {:data-dir data-dir + :owner-source :cli} + repo) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-start-timeout-orphan (get-in result [:error :code]))) + (is (= [111 222] (get-in result [:error :pids]))) + (is (= 1 @cleanup-calls)) + (is (= 1 @spawn-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/cleanup-stale-lock! original-cleanup-stale) + (set! daemon/cleanup-orphan-processes! original-cleanup-orphans) + (set! daemon/spawn-server! original-spawn) + (set! daemon/wait-for-lock original-wait-lock) + (set! daemon/find-orphan-processes original-find-orphans) + (done))))))) diff --git a/src/test/logseq/db_worker/daemon_test.cljs b/src/test/logseq/db_worker/daemon_test.cljs index f155d87b7c..32ea68e444 100644 --- a/src/test/logseq/db_worker/daemon_test.cljs +++ b/src/test/logseq/db_worker/daemon_test.cljs @@ -45,3 +45,46 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)) (done))))))) + +(deftest parse-process-args-reads-db-worker-flags + (let [command "node /tmp/db-worker-node.js --repo logseq_db_demo --data-dir /tmp/logseq/graphs --owner-source electron" + parsed (daemon/parse-process-args command)] + (is (= "logseq_db_demo" (:repo parsed))) + (is (= (node-path/resolve "/tmp/logseq/graphs") (:data-dir parsed))) + (is (= :electron (:owner-source parsed))))) + +(deftest find-orphan-processes-matches-repo-and-data-dir + (let [original-list daemon/list-db-worker-processes + target-dir (node-path/resolve "/tmp/logseq/graphs") + processes [{:pid 101 :repo "logseq_db_demo" :data-dir target-dir :owner-source :cli} + {:pid 102 :repo "logseq_db_demo" :data-dir (node-path/resolve "/tmp/other") :owner-source :cli} + {:pid 103 :repo "logseq_db_other" :data-dir target-dir :owner-source :electron}]] + (set! daemon/list-db-worker-processes (fn [] processes)) + (try + (let [orphans (daemon/find-orphan-processes {:repo "logseq_db_demo" + :data-dir "/tmp/logseq/graphs"})] + (is (= [101] (mapv :pid orphans)))) + (finally + (set! daemon/list-db-worker-processes original-list))))) + +(deftest cleanup-orphan-processes-kills-matched-pids + (let [original-find daemon/find-orphan-processes + original-kill (.-kill js/process) + kill-calls (atom [])] + (set! daemon/find-orphan-processes + (fn [_] + [{:pid 90001} {:pid 90002}])) + (set! (.-kill js/process) + (fn [pid signal] + (swap! kill-calls conj [pid signal]) + true)) + (try + (let [{:keys [killed-pids]} (daemon/cleanup-orphan-processes! {:repo "logseq_db_demo" + :data-dir "/tmp/logseq/graphs"})] + (is (= [90001 90002] killed-pids)) + (is (= [[90001 "SIGTERM"] + [90002 "SIGTERM"]] + @kill-calls))) + (finally + (set! daemon/find-orphan-processes original-find) + (set! (.-kill js/process) original-kill))))) From 03cdac109c7cbd2e6787b9887aec53c12c1093a9 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Feb 2026 15:04:57 +0800 Subject: [PATCH 076/375] 035-logseq-cli-db-worker-deps-cli-decoupling.md --- ...ogseq-cli-db-worker-deps-cli-decoupling.md | 167 +++++++ src/main/logseq/cli/common/mcp/tools.cljs | 442 ++++++++++++++++++ .../logseq/cli/mcp_tools_contract_test.cljs | 156 +++++++ 3 files changed, 765 insertions(+) create mode 100644 docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md create mode 100644 src/main/logseq/cli/common/mcp/tools.cljs create mode 100644 src/test/logseq/cli/mcp_tools_contract_test.cljs diff --git a/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md b/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md new file mode 100644 index 0000000000..0ad05a9d67 --- /dev/null +++ b/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md @@ -0,0 +1,167 @@ +# Logseq CLI and db-worker-node deps/cli Decoupling Implementation Plan + +Goal: Make `logseq-cli` behavior independent from old `deps/cli` regressions while restoring list behavior that regressed after commit `0de3c337e` on February 12, 2026. + +Architecture: Apply a CLI-scoped decoupling only for the runtime path used by `logseq-cli` (`src/main/logseq/cli/*` plus db-worker API path it invokes), and defer non-CLI namespace migration to follow-up work. + +Tech Stack: ClojureScript, Datascript, `shadow-cljs` node-script builds (`:logseq-cli` and `:db-worker-node`), babashka test workflow. + +Related: Builds on `docs/agent-guide/003-db-worker-node-cli-orchestration.md`, `docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md`, and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +Developer note (CLI-scoped): The implementation for this plan only migrates the `logseq-cli -> db-worker-node -> thread-api` runtime path by adding `src/main/logseq/cli/common/mcp/tools.cljs`. It does not migrate or alter export/Electron/non-CLI namespaces still resolving from `deps/cli`. + +## Problem statement + +`deps/cli` is the old CLI codebase, but current runtime paths under `src/` still require namespaces from `deps/cli`, so old CLI commits can silently change new CLI behavior. + +Commit `0de3c337e` changed `deps/cli/src/logseq/cli/common/mcp/tools.cljs` and removed fields and filters that new CLI still depends on through db-worker thread-api calls. + +Current regressions are reproducible with existing tests. + +`bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` currently fails because list items no longer include numeric `:id`. + +`bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` currently fails because default tag/property lists no longer include built-in entities needed for ID-based add flows. + +The dependency path that causes this is shown below. + +```text +logseq-cli (src/main/logseq/cli/*) + -> HTTP invoke to db-worker-node + -> thread-api methods in frontend.worker.db-core + -> implementation helper logseq.cli.common.mcp.tools (currently loaded from deps/cli) +``` + +## Current dependency inventory + +| Namespace currently resolved from `deps/cli` | `src/` consumers | Build impact | Status in this plan | +| --- | --- | --- | --- | +| `logseq.cli.common.mcp.tools` | `src/main/frontend/worker/db_core.cljs`, `src/main/logseq/api/db_based/cli.cljs`, `src/main/logseq/sdk/utils.cljs` | `:db-worker-node`, app API, SDK | In scope now | +| `logseq.cli.common.file` | `src/main/frontend/worker/export.cljs` | `:db-worker-node` export path | Out of scope now | +| `logseq.cli.common.util` | `src/main/frontend/extensions/zip.cljs`, export handlers | app export and zip features | Out of scope now | +| `logseq.cli.common.export.common` | `src/main/frontend/handler/export/common.cljs`, `src/main/frontend/handler/export/html.cljs`, `src/main/frontend/handler/export/opml.cljs`, `src/main/frontend/handler/export/text.cljs` | app export features | Out of scope now | +| `logseq.cli.common.export.text` | `src/main/frontend/handler/export/text.cljs` | app markdown export | Out of scope now | +| `logseq.cli.common.graph` | `src/electron/electron/utils.cljs`, `src/electron/electron/db.cljs`, `src/electron/electron/handler.cljs` | Electron graph directory behavior | Out of scope now | +| `logseq.cli.common.mcp.server` | `src/electron/electron/server.cljs` | Electron MCP HTTP endpoint | Out of scope now | +| `logseq.cli.text-util` | `src/main/frontend/util/text.cljs` | frontend text helpers | Out of scope now | + +## Scope + +This plan only changes logseq-cli related runtime behavior. + +This plan migrates the `logseq-cli -> db-worker-node -> thread-api` path where the implementation currently resolves to `deps/cli`. + +In this PR, that means migrating `logseq.cli.common.mcp.tools` out of `deps/cli` because it is the thread-api-side implementation used by logseq-cli list/add flows. + +This plan does not modify any code under `deps/cli`. + +This plan does not migrate frontend export namespaces, Electron namespaces, or unrelated old CLI modules in `deps/cli`. + +This plan does not redesign CLI command UX or old CLI feature parity, and old CLI under `deps/cli` remains frozen unless a compatibility fix is required for release safety. + +Implementation must follow @test-driven-development, and any unexpected test failure while migrating must follow @clojure-debug before changing behavior. + +## Testing Plan + +I will first lock the current regressions by running the two failing integration tests as baseline checks and recording failure reasons in the PR notes. + +I will add focused tests for list data contract behavior so `:id`, built-in inclusion defaults, and page filtering options are guaranteed independent of old CLI code. + +I will run compile checks for both node builds to ensure the CLI path behaves correctly after the targeted namespace move. + +I will not include export/Electron migration tests in this PR because those modules are explicitly out of scope. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +1. Add a migration checklist section in the PR description that references this document and lists the two known failing integration tests. + +2. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` and confirm it fails before code changes. + +3. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` and confirm it fails before code changes. + +4. Add a focused test file at `src/test/logseq/cli/mcp_tools_contract_test.cljs` for `list-pages`, `list-tags`, and `list-properties` contract behavior used by new CLI. + +5. In that test file, add a case asserting non-expanded list output includes `:db/id`, `:block/title`, `:block/created-at`, and `:block/updated-at`. + +6. In that test file, add a case asserting `include-built-in` defaults to true and explicitly false excludes built-in tags and properties. + +7. In that test file, add a case asserting page list filtering honors `include-hidden`, `include-journal`, `journal-only`, `created-after`, and `updated-after`. + +8. Add `src/main/logseq/cli/common/mcp/tools.cljs` by porting the pre-`0de3c337e` behavior as baseline thread-api implementation and keeping API signatures used by `db-core` and SDK. + +9. Update `src/` callsites to resolve the new `src/main` implementation and keep `deps/cli` source files untouched. + +10. Re-run `bb dev:test -v 'logseq.cli.mcp-tools-contract-test'` and make it pass with no test skips. + +11. Re-run the two integration regressions and make both pass. + +12. Keep all files under `deps/cli` unchanged in this PR. + +13. Do not remove `logseq/cli` from `deps.edn` in this PR because unrelated frontend and Electron runtime modules still depend on it. + +14. Run `clojure -M:cljs compile logseq-cli` and verify compile success. + +15. Run `clojure -M:cljs compile db-worker-node` and verify compile success. + +16. Run `bb dev:test -v 'logseq.cli.commands-test/test-list-subcommand-parse'` to verify list option parsing remains stable. + +17. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` to verify ID contract restoration end-to-end. + +18. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` to verify built-in tag/property ID flows end-to-end. + +19. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-add-show-remove'` as a smoke test for normal create/list/show/remove flow. + +20. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-query-recent-updated'` as a smoke test for list timestamps and query interplay. + +21. Run `bb dev:lint-and-test` before merge to satisfy repository review checklist. + +## Edge cases to validate during implementation + +If `created-after` or `updated-after` is an invalid date string, filtering should not crash and should behave as no time filter. + +When `include-journal` is omitted, journals should remain included by default to preserve existing CLI expectations. + +When `include-built-in` is omitted, built-in tags and properties must remain included so ID resolution in add/update flows still works. + +Non-expanded list output must contain stable numeric IDs even if UUID and title are present. + +Expanded list output must keep UUID string conversion and keep relationship fields (`classes`, `extends`, `properties`) in expected shapes. + +Non-CLI frontend export and Electron behavior must remain untouched in this PR. + +## Open questions requiring clarity + +Should we do a second PR for non-CLI namespace migration (`export`, `electron`, `text-util`) immediately after this thread-api decoupling PR, or wait until after release. + +## Testing Details + +The key behavior tests are integration-first and validate user-observable outcomes, not internal helper implementation. + +`test-cli-list-outputs-include-id` verifies that real CLI JSON output includes usable IDs for list operations. + +`test-cli-add-tags-and-properties-by-id` verifies that list output can feed directly into add operations using IDs and complete a write/read roundtrip. + +`mcp-tools-contract-test` verifies option semantics and filtering behavior at the db-worker API boundary to prevent future regressions from unrelated old CLI edits. + +## Implementation Details + +- Keep migration atomic by changing only logseq-cli related runtime pieces in this PR. +- Preserve public function names and argument shapes to minimize callsite churn. +- Restore pre-`0de3c337e` list semantics for IDs and built-in filtering. +- Do not edit any file under `deps/cli`; all migration changes must happen in `src/` and `src/test/`. +- Do not change command parsing behavior in `src/main/logseq/cli/command/list.cljs` unless tests show incompatibility. +- Keep Electron/export/frontend non-CLI require lines unchanged in this PR. +- Treat `deps/cli` as frozen legacy code for non-CLI modules in this PR. +- Add a short developer note in `README.md` or `docs` explaining this PR is CLI-scoped only. +- Use smallest possible commits per namespace group to simplify rollback. +- Ensure all touched files remain ASCII and follow existing formatting conventions. +- Finish with `bb dev:lint-and-test` and include command outputs in the PR summary. + +## Decision + +This migration will keep `logseq.cli.common.*` namespace names stable and only move CLI-path runtime implementation needed for `logseq-cli`. + +Namespace renaming will be done in a follow-up PR after decoupling and regression fixes are complete. + +--- diff --git a/src/main/logseq/cli/common/mcp/tools.cljs b/src/main/logseq/cli/common/mcp/tools.cljs new file mode 100644 index 0000000000..3f2cbc528a --- /dev/null +++ b/src/main/logseq/cli/common/mcp/tools.cljs @@ -0,0 +1,442 @@ +(ns logseq.cli.common.mcp.tools + "MCP tool related fns shared between CLI and frontend" + (:require [clojure.string :as string] + [datascript.core :as d] + [logseq.common.util :as common-util] + [logseq.common.util.date-time :as date-time-util] + [logseq.db :as ldb] + [logseq.db.frontend.class :as db-class] + [logseq.db.frontend.content :as db-content] + [logseq.db.frontend.entity-util :as entity-util] + [logseq.db.frontend.property :as db-property] + [logseq.db.frontend.property.type :as db-property-type] + [logseq.db.sqlite.export :as sqlite-export] + [logseq.outliner.tree :as otree] + [logseq.outliner.validate :as outliner-validate] + [malli.core :as m] + [malli.error :as me])) + +(defn- minimal-list-item + [e] + (cond-> {:db/id (:db/id e) + :block/title (:block/title e) + :block/created-at (:block/created-at e) + :block/updated-at (:block/updated-at e)} + (:db/ident e) (assoc :db/ident (:db/ident e)))) + +(defn list-properties + "Main fn for ListProperties tool" + [db {:keys [expand include-built-in] :as options}] + (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] + (->> (d/datoms db :avet :block/tags :logseq.class/Property) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) + #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name :db/index + :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) + true + (update :block/uuid str) + (:logseq.property/classes e) + (update :logseq.property/classes #(mapv :db/ident %)) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + (minimal-list-item e))))))) + +(defn list-tags + "Main fn for ListTags tool" + [db {:keys [expand include-built-in] :as options}] + (let [include-built-in? (if (contains? options :include-built-in) include-built-in true)] + (->> (d/datoms db :avet :block/tags :logseq.class/Tag) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) + (map (fn [e] + (if expand + (cond-> (into {} e) + true + (dissoc e :block/tags :block/order :block/refs :block/name + :logseq.property.embedding/hnsw-label-updated-at) + true + (update :block/uuid str) + (:logseq.property.class/extends e) + (update :logseq.property.class/extends #(mapv :db/ident %)) + (:logseq.property.class/properties e) + (update :logseq.property.class/properties #(mapv :db/ident %)) + (:logseq.property.view/type e) + (assoc :logseq.property.view/type (:db/ident (:logseq.property.view/type e))) + (:logseq.property/description e) + (update :logseq.property/description db-property/property-value-content)) + (minimal-list-item e))))))) + +(defn- get-page-blocks + [db page-id] + (let [datoms (d/datoms db :avet :block/page page-id) + block-eids (mapv :e datoms) + block-ents (map #(d/entity db %) block-eids) + blocks (map #(assoc % :block/title (db-content/recur-replace-uuid-in-block-title %)) block-ents)] + (->> (otree/blocks->vec-tree db blocks page-id) + (map #(update % :block/uuid str))))) + +(defn ^:api remove-hidden-properties + "Given an entity map, remove properties that shouldn't be returned in api calls" + [m] + (->> (remove (fn [[k _v]] + (or (= "block.temp" (namespace k)) + (contains? #{:logseq.property.embedding/hnsw-label-updated-at :block/tx-id} k))) m) + (into {}))) + +(defn get-page-data + "Get page data for GetPage tool including the page's entity and its blocks" + [db page-name-or-uuid] + (when-let [page (ldb/get-page db page-name-or-uuid)] + {:entity (-> (remove-hidden-properties page) + (dissoc :block/tags :block/refs) + (update :block/uuid str)) + :blocks (map #(-> % + remove-hidden-properties + ;; remove unused and untranslated attrs + (dissoc :block/children :block/page)) + (get-page-blocks db (:db/id page)))})) + +(defn- parse-time + [value] + (cond + (number? value) value + (string? value) (let [ms (js/Date.parse value)] + (when-not (js/isNaN ms) ms)) + :else nil)) + +(defn list-pages + "Main fn for ListPages tool" + [db {:keys [expand include-hidden include-journal journal-only created-after updated-after] :as options}] + (let [include-hidden? (boolean include-hidden) + include-journal? (if (contains? options :include-journal) include-journal true) + journal-only? (boolean journal-only) + created-after-ms (parse-time created-after) + updated-after-ms (parse-time updated-after)] + (->> (d/datoms db :avet :block/name) + (map #(d/entity db (:e %))) + (remove (fn [e] + (and (not include-hidden?) + (entity-util/hidden? e)))) + (remove (fn [e] + (let [is-journal? (ldb/journal? e)] + (cond + journal-only? (not is-journal?) + (false? include-journal?) is-journal? + :else false)))) + (remove (fn [e] + (and created-after-ms + (<= (:block/created-at e 0) created-after-ms)))) + (remove (fn [e] + (and updated-after-ms + (<= (:block/updated-at e 0) updated-after-ms)))) + (map (fn [e] + (if expand + (-> e + ;; Until there are options to limit pages, return minimal info to avoid + ;; exceeding max payload size + (select-keys [:db/id :db/ident :block/uuid :block/title :block/created-at :block/updated-at]) + (update :block/uuid str)) + (minimal-list-item e))))))) + +;; upsert-nodes tool +;; ================= +(defn- import-edn-data + [conn export-map] + (let [{:keys [init-tx block-props-tx misc-tx error] :as _txs} + (sqlite-export/build-import export-map @conn {})] + ;; (cljs.pprint/pprint _txs) + (when error + (throw (ex-info (str "Error while building import data: " error) {}))) + (let [tx-meta {::sqlite-export/imported-data? true}] + (ldb/transact! conn (vec (concat init-tx block-props-tx misc-tx)) tx-meta)))) + +(defn- get-ident [idents title] + (or (get idents title) + (throw (ex-info (str "No ident found for " (pr-str title)) {})))) + +(defn- build-add-block [op {:keys [class-idents]}] + (cond-> {:block/title (get-in op [:data :title])} + (get-in op [:data :tags]) + (assoc :build/tags (mapv #(get-ident class-idents %) (get-in op [:data :tags]))))) + +(defn- ops->existing-pages-and-blocks + "Converts block operations for existing pages and prepares them for :pages-and-blocks" + [db operations idents] + (let [new-blocks-for-existing-pages + (->> (filter #(and (= "block" (:entityType %)) + (= "add" (:operation %)) + (common-util/uuid-string? (get-in % [:data :page-id]))) operations) + (map (fn [op] (assoc op ::page-id (uuid (get-in op [:data :page-id])))))) + edit-blocks + (->> (filter #(and (= "block" (:entityType %)) (= "edit" (:operation %))) operations) + (map (fn [op] + (let [block-uuid (uuid (:id op)) + ent (d/entity db [:block/uuid block-uuid])] + (when-not (:block/page ent) + (throw (ex-info "Block edit operation requires a block to have a page." {}))) + (assoc op ::page-id (get-in ent [:block/page :block/uuid]))))))] + (->> (concat new-blocks-for-existing-pages edit-blocks) + (group-by ::page-id) + (map (fn [[page-id ops]] + {:page {:block/uuid page-id} + :blocks (mapv (fn [op] + (if (= "add" (:operation op)) + (build-add-block op idents) + ;; edit :block + (cond-> {:block/uuid (uuid (:id op))} + (get-in op [:data :title]) + (assoc :block/title (get-in op [:data :title]))))) + ops)}))))) + +(defn- ops->pages-and-blocks + [db operations idents] + (let [new-blocks-by-page + (group-by #(get-in % [:data :page-id]) + (filter #(and (= "block" (:entityType %)) (= "add" (:operation %))) operations)) + new-pages (filter #(and (= "page" (:entityType %)) (= "add" (:operation %))) operations) + pages-and-blocks + (into (mapv (fn [op] + (cond-> {:page (if-let [journal-day (date-time-util/journal-title->int + (get-in op [:data :title]) + ;; consider user's date-formatter as needed + (date-time-util/safe-journal-title-formatters nil))] + {:build/journal journal-day} + {:block/title (get-in op [:data :title])})} + (some->> (:id op) (get new-blocks-by-page)) + (assoc :blocks + (mapv #(build-add-block % idents) (get new-blocks-by-page (:id op)))))) + new-pages) + (ops->existing-pages-and-blocks db operations idents))] + pages-and-blocks)) + +(defn- ops->classes + [operations {:keys [property-idents class-idents existing-classes]}] + (let [new-classes (filter #(and (= "tag" (:entityType %)) (= "add" (:operation %))) operations) + classes (merge + (into {} (keep (fn [[k v]] + ;; Removing existing until edits are supported + (when-not (existing-classes v) [v {:block/title k}])) + class-idents)) + (->> new-classes + (map (fn [{:keys [data] :as op}] + (let [title (get-in op [:data :title]) + class-m (cond-> {:block/title title} + (:class-extends data) + (assoc :build/class-extends (mapv #(get-ident class-idents %) (:class-extends data))) + (:class-properties data) + (assoc :build/class-properties (mapv #(get-ident property-idents %) (:class-properties data))))] + [(get-ident class-idents title) class-m]))) + (into {})))] + classes)) + +(defn- ops->properties + [operations {:keys [property-idents class-idents existing-properties]}] + (let [new-properties (filter #(and (= "property" (:entityType %)) (= "add" (:operation %))) operations) + properties + (merge + existing-properties + (->> new-properties + (map (fn [{:keys [data] :as op}] + (let [title (get-in op [:data :title]) + prop-m (cond-> {:block/title title} + (some->> (:property-type data) keyword (contains? (set db-property-type/user-built-in-property-types))) + (assoc :logseq.property/type (keyword (:property-type data))) + (= "many" (:property-cardinality data)) + (assoc :db/cardinality :db.cardinality/many) + (:property-classes data) + (assoc :build/property-classes + (mapv #(get-ident class-idents %) (:property-classes data)) + :logseq.property/type :node))] + [(get-ident property-idents title) prop-m]))) + (into {})))] + properties)) + +(defn- operations->idents + "Creates property and class idents from all uses of them in operations" + [db operations] + (let [existing-classes (atom #{}) + existing-properties (atom {}) + property-idents + (->> (filter #(and (= "property" (:entityType %)) (= "add" (:operation %))) + operations) + (map #(get-in % [:data :title])) + (into (mapcat #(get-in % [:data :class-properties]) + (filter #(and (= "tag" (:entityType %)) (= "add" (:operation %))) + operations))) + distinct + (map #(vector % (if (common-util/uuid-string? %) + (let [ent (d/entity db [:block/uuid (uuid %)]) + ident (:db/ident ent)] + (when-not (entity-util/property? ent) + (throw (ex-info (str (pr-str (:block/title ent)) + " is not a property and can't be used as one") + {}))) + (swap! existing-properties assoc ident (select-keys ent [:db/cardinality :logseq.property/type])) + ident) + (db-property/create-user-property-ident-from-name %)))) + (into {})) + class-idents + (->> (filter #(and (= "tag" (:entityType %)) (= "add" (:operation %))) operations) + (mapcat (fn [op] + (into [(get-in op [:data :title])] (get-in op [:data :class-extends])))) + (into (mapcat #(get-in % [:data :property-classes]) + (filter #(and (= "property" (:entityType %)) (= "add" (:operation %))) + operations))) + (into (mapcat #(get-in % [:data :tags]) + (filter #(and (= "block" (:entityType %)) (= "add" (:operation %))) + operations))) + distinct + (map #(vector % (if (common-util/uuid-string? %) + (let [ent (d/entity db [:block/uuid (uuid %)]) + ident (:db/ident ent)] + (when-not (entity-util/class? ent) + (throw (ex-info (str (pr-str (:block/title ent)) + " is not a tag and can't be used as one") + {}))) + (swap! existing-classes conj ident) + ident) + (db-class/create-user-class-ident-from-name db %)))) + (into {}))] + {:property-idents property-idents + :class-idents class-idents + :existing-classes @existing-classes + :existing-properties @existing-properties})) + +(def ^:private add-non-block-schema + [:map + [:data [:map + [:title :string]]]]) + +(def ^:private uuid-string + [:and :string [:fn {:error/message "Must be a uuid string"} common-util/uuid-string?]]) + +(def ^:private upsert-nodes-operation-schema + [:and + ;; Base schema. Has some overlap with inputSchema + [:map + {:closed true} + [:operation [:enum "add" "edit"]] + [:entityType [:enum "block" "page" "tag" "property"]] + [:id {:optional true} [:or :string :nil]] + [:data [:map + [:title {:optional true} :string] + [:page-id {:optional true} :string] + [:tags {:optional true} [:sequential uuid-string]] + [:property-type {:optional true} :string] + [:property-cardinality {:optional true} [:enum "many" "one"]] + [:property-classes {:optional true} [:sequential :string]] + [:class-extends {:optional true} [:sequential :string]] + [:class-properties {:optional true} [:sequential :string]]]]] + ;; Validate special cases of operation and entityType e.g. required keys and uuid strings + [:multi {:dispatch (juxt :operation :entityType)} + [["add" "block"] [:map + [:data [:map {:closed true} + [:tags {:optional true} [:sequential uuid-string]] + [:title :string] + [:page-id :string]]]]] + [["add" "page"] add-non-block-schema] + [["add" "tag"] add-non-block-schema] + [["add" "property"] add-non-block-schema] + [["edit" "block"] [:map + [:id uuid-string] + ;; :tags not supported yet + [:data [:map {:closed true} + [:title :string]]]]] + ;; other edit's + [::m/default [:map [:id uuid-string]]]]]) + +(def ^:private Upsert-nodes-operations-schema + [:sequential upsert-nodes-operation-schema]) + +(defn- validate-import-edn + "Validates everything as coming from add operations, failing fast on first invalid + node. Will need to adjust add operation assumption when supporting editing pages" + [{:keys [pages-and-blocks properties classes]}] + (try + (doseq [{:block/keys [title] :as m} + ;; Only validate new properties + (filter :block/title (vals properties))] + (outliner-validate/validate-property-title title {:entity-type :property :title title :entity-map m}) + (outliner-validate/validate-page-title-characters title {:entity-type :property :title title :entity-map m}) + (outliner-validate/validate-page-title title {:entity-type :property :title title :entity-map m})) + (doseq [{:block/keys [title] :as m} (vals classes)] + (outliner-validate/validate-page-title-characters title {:entity-type :tag :title title :entity-map m}) + (outliner-validate/validate-page-title title {:entity-type :tag :title title :entity-map m})) + (doseq [{:block/keys [title] :as m} (map :page pages-and-blocks)] + ;; title is only present for new pages + (when title + (outliner-validate/validate-page-title-characters title {:entity-type :page :title title :entity-map m}) + (outliner-validate/validate-page-title title {:entity-type :page :title title :entity-map m}))) + (catch :default e + (js/console.error e) + (throw (ex-info (str (string/capitalize (name (get (ex-data e) :entity-type :page))) + " " (pr-str (:title (ex-data e))) " is invalid: " (ex-message e)) + (ex-data e)))))) + +(defn ^:api summarize-upsert-operations [operations {:keys [dry-run]}] + (let [counts (reduce (fn [acc op] + (let [entity-type (keyword (:entityType op)) + operation-type (keyword (:operation op))] + (update-in acc [operation-type entity-type] (fnil inc 0)))) + {} + operations)] + (str (if dry-run "Dry run: " "") + (when (counts :add) + (str "Added: " (pr-str (counts :add)) ".")) + (when (counts :edit) + (str " Edited: " (pr-str (counts :edit)) "."))))) + +(declare upsert-nodes) + +(defn ^:api build-upsert-nodes-edn + "Given llm generated operations, builds the import EDN, validates it and returns it. It fails + fast on anything invalid" + [db operations*] + ;; Only support these operations with appropriate outliner validations + (when (seq (filter #(and (#{"page" "tag" "property"} (:entityType %)) (= "edit" (:operation %))) operations*)) + (throw (ex-info "Editing a page, tag or property isn't supported yet" {}))) + (let [;; Keep an explicit var reference so carve treats this namespace-level API as in use. + _upsert-nodes-api-ref upsert-nodes + operations + (->> operations* + ;; normalize classes as they sometimes have titles in :name + (map #(if (and (= "tag" (:entityType %)) (= "add" (:operation %))) + (assoc-in % [:data :title] + (or (get-in % [:data :name]) (get-in % [:data :title]))) + %))) + ;; _ (prn :ops operations) + _ (when-let [errors (m/explain Upsert-nodes-operations-schema operations)] + (throw (ex-info (str "Tool arguments are invalid:\n" (me/humanize errors)) + {:errors errors}))) + idents (operations->idents db operations) + pages-and-blocks (ops->pages-and-blocks db operations idents) + classes (ops->classes operations idents) + properties (ops->properties operations idents) + import-edn + (cond-> {} + (seq pages-and-blocks) + (assoc :pages-and-blocks pages-and-blocks) + (seq classes) + (assoc :classes classes) + (seq properties) + (assoc :properties properties))] + (prn :debug-import-edn import-edn) + (validate-import-edn import-edn) + import-edn)) + +(defn ^:api upsert-nodes + "Builds import-edn from llm generated operations and then imports resulting data. Only + used for CLI. See logseq.api/upsert_nodes for API equivalent" + [conn operations* {:keys [dry-run] :as opts}] + (let [import-edn (build-upsert-nodes-edn @conn operations*)] + (when-not dry-run (import-edn-data conn import-edn)) + (summarize-upsert-operations operations* opts))) diff --git a/src/test/logseq/cli/mcp_tools_contract_test.cljs b/src/test/logseq/cli/mcp_tools_contract_test.cljs new file mode 100644 index 0000000000..1ec1ef77b7 --- /dev/null +++ b/src/test/logseq/cli/mcp_tools_contract_test.cljs @@ -0,0 +1,156 @@ +(ns logseq.cli.mcp-tools-contract-test + (:require [cljs.test :refer [deftest is testing]] + [clojure.set :as set] + [datascript.core :as d] + [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] + [logseq.db.test.helper :as db-test])) + +(defn- create-test-db + [] + (let [conn (db-test/create-conn-with-blocks + {:classes {:custom-tag {}} + :properties {:custom-property {:logseq.property/type :default}} + :pages-and-blocks [{:page {:block/title "Visible Page" + :block/created-at 1000 + :block/updated-at 2000}} + {:page {:block/title "Hidden Page" + :block/created-at 1500 + :block/updated-at 2500 + :build/properties {:logseq.property/hide? true}}} + {:page {:build/journal 20260201 + :block/created-at 3000 + :block/updated-at 4000}} + {:page {:block/title "Late Page" + :block/created-at 5000 + :block/updated-at 6000}}]})] + @conn)) + +(defn- list-item-ids + [items] + (->> items (map :db/id) set)) + +(defn- first-user-tag-entity + [db] + (->> (d/datoms db :avet :block/tags :logseq.class/Tag) + (map :e) + (map #(d/entity db %)) + (remove :logseq.property/built-in?) + first)) + +(defn- first-user-property-entity + [db] + (->> (d/datoms db :avet :block/tags :logseq.class/Property) + (map :e) + (map #(d/entity db %)) + (remove :logseq.property/built-in?) + first)) + +(deftest test-list-non-expanded-contract + (let [db (create-test-db) + required-keys #{:db/id :block/title :block/created-at :block/updated-at} + visible-page (some #(when (= "Visible Page" (:block/title %)) %) + (cli-common-mcp-tools/list-pages db {})) + custom-tag-entity (first-user-tag-entity db) + custom-tag-title (:block/title custom-tag-entity) + custom-tag (some #(when (= custom-tag-title (:block/title %)) %) + (cli-common-mcp-tools/list-tags db {})) + custom-property-entity (first-user-property-entity db) + custom-property-title (:block/title custom-property-entity) + custom-property (some #(when (= custom-property-title (:block/title %)) %) + (cli-common-mcp-tools/list-properties db {}))] + (testing "list-pages non-expanded includes stable id and timestamps" + (is (some? visible-page)) + (is (set/subset? required-keys (set (keys visible-page))))) + + (testing "list-tags non-expanded includes stable id and timestamps" + (is (some? custom-tag)) + (is (set/subset? required-keys (set (keys custom-tag))))) + + (testing "list-properties non-expanded includes stable id and timestamps" + (is (some? custom-property)) + (is (set/subset? required-keys (set (keys custom-property))))))) + +(deftest test-list-tags-and-properties-include-built-in-default + (let [db (create-test-db) + built-in-tag-ids (->> (d/datoms db :avet :block/tags :logseq.class/Tag) + (map :e) + (map #(d/entity db %)) + (filter :logseq.property/built-in?) + (map :db/id) + set) + built-in-property-ids (->> (d/datoms db :avet :block/tags :logseq.class/Property) + (map :e) + (map #(d/entity db %)) + (filter :logseq.property/built-in?) + (map :db/id) + set) + custom-tag-id (:db/id (first-user-tag-entity db)) + custom-property-id (:db/id (first-user-property-entity db)) + default-tag-ids (list-item-ids (cli-common-mcp-tools/list-tags db {})) + default-property-ids (list-item-ids (cli-common-mcp-tools/list-properties db {})) + no-built-in-tag-ids (list-item-ids (cli-common-mcp-tools/list-tags db {:include-built-in false})) + no-built-in-property-ids (list-item-ids (cli-common-mcp-tools/list-properties db {:include-built-in false}))] + (testing "built-ins are included by default" + (is (seq built-in-tag-ids)) + (is (seq built-in-property-ids)) + (is (seq (set/intersection default-tag-ids built-in-tag-ids))) + (is (seq (set/intersection default-property-ids built-in-property-ids)))) + + (testing "include-built-in=false excludes built-ins but keeps user entities" + (is (contains? no-built-in-tag-ids custom-tag-id)) + (is (contains? no-built-in-property-ids custom-property-id)) + (is (empty? (set/intersection no-built-in-tag-ids built-in-tag-ids))) + (is (empty? (set/intersection no-built-in-property-ids built-in-property-ids)))))) + +(deftest test-list-pages-filter-contract + (let [db (create-test-db) + hidden-id (->> (d/q '[:find [?e ...] + :in $ ?title + :where + [?e :block/title ?title] + [?e :logseq.property/hide? true]] + db "Hidden Page") + first) + journal-id (->> (d/datoms db :avet :block/journal-day 20260201) + first + :e) + visible-id (->> (d/q '[:find [?e ...] + :in $ ?title + :where [?e :block/title ?title]] + db "Visible Page") + first) + late-id (->> (d/q '[:find [?e ...] + :in $ ?title + :where [?e :block/title ?title]] + db "Late Page") + first) + default-ids (list-item-ids (cli-common-mcp-tools/list-pages db {})) + include-hidden-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:include-hidden true})) + exclude-journal-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:include-journal false})) + journal-only-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:journal-only true})) + created-after-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:created-after 2500})) + updated-after-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:updated-after "1970-01-01T00:00:03.500Z"})) + invalid-created-after-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:created-after "not-a-date"})) + invalid-updated-after-ids (list-item-ids (cli-common-mcp-tools/list-pages db {:updated-after "still-not-a-date"}))] + (testing "journals are included by default and hidden pages are excluded by default" + (is (contains? default-ids journal-id)) + (is (not (contains? default-ids hidden-id)))) + + (testing "include-hidden includes hidden pages" + (is (contains? include-hidden-ids hidden-id))) + + (testing "include-journal and journal-only page filters" + (is (not (contains? exclude-journal-ids journal-id))) + (is (= #{journal-id} journal-only-ids))) + + (testing "created-after and updated-after filters" + (is (contains? created-after-ids journal-id)) + (is (contains? created-after-ids late-id)) + (is (not (contains? created-after-ids visible-id))) + (is (contains? updated-after-ids journal-id)) + (is (contains? updated-after-ids late-id)) + (is (not (contains? updated-after-ids visible-id)))) + + (testing "invalid date filters behave as no-op" + (is (= default-ids invalid-created-after-ids)) + (is (= default-ids invalid-updated-after-ids))))) From de22f37f4565dfa5a69b6a907fd68f6f418ca94b Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 16 Feb 2026 17:38:55 -0500 Subject: [PATCH 077/375] fix: graph list showing non-db graphs Also shows Unlinked graphs dir --- src/main/logseq/cli/server.cljs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index a7f1d3ccd5..02a474b032 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -5,6 +5,7 @@ ["path" :as node-path] [clojure.string :as string] [frontend.worker.db-worker-node-lock :as db-lock] + [logseq.common.config :as common-config] [logseq.db-worker.daemon :as daemon] [lambdaisland.glogi :as log] [promesa.core :as p])) @@ -306,4 +307,7 @@ (map (fn [^js dirent] (db-lock/decode-canonical-graph-dir-key (.-name dirent)))) (filter some?) + (remove (fn [s] + (or (= s common-config/unlinked-graphs-dir) + (string/starts-with? s common-config/file-version-prefix)))) (vec)))) From aece4c80c0ec305fbb0772269344c4bed6a28f3b Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 16 Feb 2026 17:40:03 -0500 Subject: [PATCH 078/375] fix lint and mark long-running cli tests Long tests should be marked with ^:long so that most unit tests can still run in under 30s --- ...logseq-cli-db-graph-default-dir-locking.md | 2 +- src/test/logseq/cli/integration_test.cljs | 66 +++++++++---------- 2 files changed, 34 insertions(+), 34 deletions(-) diff --git a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md index 2f09deb083..1f139e372d 100644 --- a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md +++ b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md @@ -8,7 +8,7 @@ Architecture: Centralize graph directory resolution and lock ownership checks so Tech Stack: ClojureScript, Node.js `fs` and `path`, promesa, logseq-cli command pipeline, db-worker-node daemon, existing lock file protocol. Related: Builds on `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`. -Related: Supercedes `docs/agent-guide/020-logseq-cli-default-paths-move.md`. +Related: Supersedes `docs/agent-guide/020-logseq-cli-default-paths-move.md`. Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. ## Problem statement diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9ef690009f..dfb2eeb2fc 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -184,7 +184,7 @@ (when (= title (item-title item)) item))) item-id)) -(deftest test-cli-graph-list +(deftest ^:long test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -198,7 +198,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-data-dir-permission-error +(deftest ^:long test-cli-data-dir-permission-error (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly")] (fs/chmodSync data-dir 365) @@ -215,7 +215,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-graph-create-readonly-graph-dir +(deftest ^:long test-cli-graph-create-readonly-graph-dir (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-graph-readonly") repo "readonly-graph" @@ -236,7 +236,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-graph-create-and-info +(deftest ^:long test-cli-graph-create-and-info (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -259,7 +259,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-list-add-show-remove +(deftest ^:long test-cli-list-add-show-remove (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -302,7 +302,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-block-rewrites-page-ref +(deftest ^:long test-cli-add-block-rewrites-page-ref (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -347,7 +347,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-block-keeps-uuid-ref +(deftest ^:long test-cli-add-block-keeps-uuid-ref (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-uuid-ref")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -398,7 +398,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-block-missing-uuid-ref-errors +(deftest ^:long test-cli-add-block-missing-uuid-ref-errors (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-missing-uuid-ref")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -423,7 +423,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-tags-and-properties-by-name +(deftest ^:long test-cli-add-tags-and-properties-by-name (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) @@ -498,7 +498,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-tags-and-properties-by-id +(deftest ^:long test-cli-add-tags-and-properties-by-id (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-id")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) @@ -547,7 +547,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-properties-human-output +(deftest ^:long test-cli-show-properties-human-output (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-show-properties") repo "show-properties-graph" @@ -605,7 +605,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-verbose-logs-to-stderr +(deftest ^:long test-cli-verbose-logs-to-stderr (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-verbose") repo "verbose-graph"] @@ -626,7 +626,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-update-tags-and-properties +(deftest ^:long test-cli-update-tags-and-properties (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-update-tags")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) @@ -668,7 +668,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-tags-rejects-missing-tag +(deftest ^:long test-cli-add-tags-rejects-missing-tag (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-missing")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -697,7 +697,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query +(deftest ^:long test-cli-query (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query") query-text "[:find ?e :in $ ?title :where [?e :block/title ?title]]"] @@ -735,7 +735,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query-task-search +(deftest ^:long test-cli-query-task-search (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-task-query")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -803,7 +803,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query-list-status-priority +(deftest ^:long test-cli-query-list-status-priority (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-status-query")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -856,7 +856,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query-recent-updated +(deftest ^:long test-cli-query-recent-updated (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-recent-updated")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -953,7 +953,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-resolve-nested-uuid-refs +(deftest ^:long test-cli-show-resolve-nested-uuid-refs (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -989,7 +989,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-linked-references-json +(deftest ^:long test-cli-show-linked-references-json (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1033,7 +1033,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-update-block-move +(deftest ^:long test-cli-update-block-move (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1066,7 +1066,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-add-block-pos-ordering +(deftest ^:long test-cli-add-block-pos-ordering (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1095,7 +1095,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-output-formats-graph-list +(deftest ^:long test-cli-output-formats-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1114,7 +1114,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-list-outputs-include-id +(deftest ^:long test-cli-list-outputs-include-id (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1141,7 +1141,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-list-page-human-output +(deftest ^:long test-cli-list-page-human-output (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1159,7 +1159,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-page-block-by-id-and-uuid +(deftest ^:long test-cli-show-page-block-by-id-and-uuid (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1198,7 +1198,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-multi-id +(deftest ^:long test-cli-show-multi-id (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1270,7 +1270,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-multi-id-filters-contained +(deftest ^:long test-cli-show-multi-id-filters-contained (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id-contained")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1324,7 +1324,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query-human-output-pipes-to-show +(deftest ^:long test-cli-query-human-output-pipes-to-show (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query-pipe")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1398,7 +1398,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-query-human-output-pipes-to-show-stdin +(deftest ^:long test-cli-query-human-output-pipes-to-show-stdin (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query-stdin")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1460,7 +1460,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-show-linked-references +(deftest ^:long test-cli-show-linked-references (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1501,7 +1501,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-graph-export-import-edn +(deftest ^:long test-cli-graph-export-import-edn (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-export-edn")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -1540,7 +1540,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-graph-export-import-sqlite +(deftest ^:long test-cli-graph-export-import-sqlite (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-export-sqlite")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") From 969251bff765389da49eefc0e1c0598f934cb005 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 17 Feb 2026 22:36:59 +0800 Subject: [PATCH 079/375] 036-db-worker-node-ncc-bundling --- .gitignore | 5 + dist/db-worker-node.js | 6 - .../036-db-worker-node-ncc-bundling.md | 191 ++++++++++++++++++ docs/cli/logseq-cli.md | 9 +- package.json | 14 +- scripts/build-db-worker-node-bundle.mjs | 122 +++++++++++ src/main/frontend/worker/db_worker_node.cljs | 23 ++- src/main/logseq/cli/command/doctor.cljs | 16 +- src/main/logseq/cli/commands.cljs | 2 +- src/main/logseq/cli/server.cljs | 4 + src/test/logseq/cli/command/doctor_test.cljs | 23 ++- src/test/logseq/cli/commands_test.cljs | 18 +- src/test/logseq/cli/server_test.cljs | 4 + .../logseq/db_worker/ncc_bundle_test.cljs | 163 +++++++++++++++ yarn.lock | 5 + 15 files changed, 582 insertions(+), 23 deletions(-) delete mode 100755 dist/db-worker-node.js create mode 100644 docs/agent-guide/036-db-worker-node-ncc-bundling.md create mode 100644 scripts/build-db-worker-node-bundle.mjs create mode 100644 src/test/logseq/db_worker/ncc_bundle_test.cljs diff --git a/.gitignore b/.gitignore index 44c09b6fab..888c1ce302 100644 --- a/.gitignore +++ b/.gitignore @@ -83,3 +83,8 @@ clj-e2e/e2e-dump .projectile deps/db-sync/data *.map +/dist/db-worker-node.js +/dist/build/ +/dist/db-worker-node-assets.json +/dist/*.wasm +/dist/cljs-runtime/ diff --git a/dist/db-worker-node.js b/dist/db-worker-node.js deleted file mode 100755 index 4dabd0bf17..0000000000 --- a/dist/db-worker-node.js +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env node -"use strict"; - -const path = require("path"); - -require(path.resolve(__dirname, "../static/db-worker-node.js")); diff --git a/docs/agent-guide/036-db-worker-node-ncc-bundling.md b/docs/agent-guide/036-db-worker-node-ncc-bundling.md new file mode 100644 index 0000000000..3d052cae0c --- /dev/null +++ b/docs/agent-guide/036-db-worker-node-ncc-bundling.md @@ -0,0 +1,191 @@ +# db-worker-node ncc Standalone Bundle Implementation Plan + +Goal: Build `db-worker-node.js` with `@vercel/ncc` so the runtime can run without `node_modules` present next to the executable. + +Architecture: Keep `shadow-cljs` as the source compiler for `:db-worker-node`, then run `ncc` on the generated entry and publish a single runtime artifact in `dist/` that is used by CLI daemon orchestration. +Architecture: Preserve local development ergonomics by keeping `static/db-worker-node.js` for fast dev loops, while production and package paths resolve to the ncc artifact first. + +Tech Stack: ClojureScript, `shadow-cljs` `:node-script`, `@vercel/ncc`, Node.js 22, `yarn` scripts in `package.json`, existing CLI daemon and doctor checks. + +Related: Builds on `docs/agent-guide/031-logseq-cli-doctor-command.md`. +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. +Related: Relates to `docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md`. + +## Problem statement + +`/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js` is currently generated by `shadow-cljs`, and runtime behavior assumes dependencies are available from `node_modules`. + +The CLI server startup path in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` points to `../dist/db-worker-node.js`, which is currently a thin wrapper that forwards to `../static/db-worker-node.js`. + +This wrapper model keeps runtime coupled to workspace layout and to `node_modules`, so the daemon script is not independently portable. + +We need a deterministic packaging path that produces one runnable artifact plus copied native assets, and we need to verify that artifact works when `node_modules` is absent. + +The solution must keep existing CLI and Electron daemon orchestration behavior unchanged, including lock-file semantics, owner-source semantics, and health endpoint behavior. + +## Current packaging map + +| Area | Current behavior | Limitation | +| --- | --- | --- | +| Build output | `yarn db-worker-node:compile` writes `/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js`. | Output is not a standalone distribution artifact. | +| Dist entry | `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` only `require`s `../static/db-worker-node.js`. | Runtime still depends on static output and installed dependencies. | +| Daemon spawn | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` spawns `../dist/db-worker-node.js`. | Spawn path is stable, but executable is not standalone. | +| Doctor check | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` checks `../static/db-worker-node.js` by default. | Diagnostic target does not match the intended distributable runtime. | +| Package manifest | `/Users/rcmerci/gh-repos/logseq/package.json` includes `static/db-worker-node.js` in `files`. | Published package does not guarantee standalone daemon artifact contract. | + +## Target packaging map + +| Area | Target behavior | Verification signal | +| --- | --- | --- | +| Bundle output | `ncc` emits a standalone `db-worker-node` runtime in `/Users/rcmerci/gh-repos/logseq/dist/` with required runtime assets copied adjacent to entrypoint. | Daemon starts and serves `/healthz` and `/readyz` without `node_modules`. | +| Spawn path | CLI server keeps spawning `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` as canonical runtime. | Existing `logseq.cli.server-test` assertions remain green with updated contract. | +| Doctor check | Doctor defaults to the same packaged runtime path used for spawn, and does not auto-fallback to static runtime. | Doctor check path matches runtime path in tests and manual runs. | +| Dev flow | Fast local dev command remains available using `static/db-worker-node.js` for watch and debug workflows. | `yarn db-worker-node:compile` and `node ./static/db-worker-node.js` still work during development. | +| Publish flow | Package `files` include standalone runtime assets required by ncc output. | Installed package can execute daemon without extra dependency install. | + +## Integration sketch + +```text +shadow-cljs (:db-worker-node) + -> /static/db-worker-node.js + -> ncc build step + -> /dist/db-worker-node.js + -> /dist/ + +logseq-cli runtime + -> logseq.cli.server/spawn-server! + -> /dist/db-worker-node.js + -> db-worker daemon HTTP + SSE API +``` + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before implementation changes in each phase. + +I will add behavior tests for runtime path resolution so spawn and doctor point to the same canonical bundle target. + +I will add a standalone smoke test that launches the bundled daemon from a temporary directory without `node_modules` and verifies `/healthz`, `/readyz`, and shutdown behavior. + +I will keep existing daemon lifecycle tests green to ensure no regression in lock cleanup, owner checks, and timeout error semantics. + +I will run focused tests first, then full validation with `yarn cljs:lint && yarn test`, and if any unexpected failures appear I will use `@clojure-debug` before changing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: RED for runtime artifact contract. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` asserting canonical db-worker runtime path resolves to `dist/db-worker-node.js` as the production target. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` asserting default doctor script check points to the same canonical runtime target as server spawn. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` asserting optional dev-mode check can still validate `static/db-worker-node.js` when explicitly requested. +4. Add a new failing bundle smoke test file at `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` that expects daemon startup success from a bundle-only temp directory. +5. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and confirm failures occur on the new path-contract assertions. +6. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm failures occur on the new default-path expectations. +7. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm standalone smoke test fails before implementation. + +### Phase 2: Add ncc build pipeline. + +8. Add `@vercel/ncc` to `/Users/rcmerci/gh-repos/logseq/package.json` as a dev dependency. +9. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to build `:db-worker-node` in release mode before bundling. +10. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to run `ncc` against `/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js`. +11. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to normalize ncc output into `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` with adjacent assets preserved. +12. If script complexity is non-trivial, add `/Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs` to encapsulate output normalization and deterministic cleanup. +13. Add or update `yarn` scripts in `/Users/rcmerci/gh-repos/logseq/package.json` for one-command bundle build and optional local run of the bundled artifact. +14. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` is regenerated with executable permissions preserved. + +### Phase 3: Align runtime path and diagnostics. + +15. Refactor runtime path helpers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so there is one canonical function for packaged runtime and one explicit dev fallback function. +16. Keep `spawn-server!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` bound to the packaged runtime path to avoid ambiguity. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` to default-check the same packaged runtime path used by spawn. +18. Add an explicit action option in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` for fallback static-path diagnostics used only for development troubleshooting. +19. Ensure doctor failure codes remain stable as `:doctor-script-missing` and `:doctor-script-unreadable`. +20. Re-run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` to make the new path contract green. + +### Phase 4: Package manifest and docs alignment. + +21. Update `files` in `/Users/rcmerci/gh-repos/logseq/package.json` so packaged runtime includes `dist/db-worker-node.js` and ncc-emitted adjacent assets. +22. Keep `static/db-worker-node.js` inclusion only if required for development workflows, and document that distinction explicitly. +23. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build instructions to include the ncc bundle command and standalone runtime expectations. +24. Update daemon runtime notes in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` so `doctor` references packaged runtime as primary target. +25. Add troubleshooting notes for native module asset copy behavior from ncc output. + +### Phase 5: Standalone runtime behavior verification. + +26. Implement bundle smoke test setup in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to copy only bundle artifacts into a temp directory with no `node_modules`. +27. In that test, spawn `node ./db-worker-node.js --repo --data-dir ` from the bundle-only directory. +28. In that test, poll `/healthz` and `/readyz` and assert both return HTTP 200 after startup. +29. In that test, invoke `/v1/shutdown` and assert process exits and lock file is cleaned or becomes stale-removable. +30. In that test, assert failure output is actionable if native binary asset is missing, to guard accidental packaging regressions. +31. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and make it green. + +### Phase 6: Final validation and review checklist. + +32. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and confirm zero failures and zero errors. +33. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm zero failures and zero errors. +34. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm zero failures and zero errors. +35. Run `yarn db-worker-node:compile` and verify `static/db-worker-node.js` remains valid for local dev flow. +36. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` starts successfully with `node dist/db-worker-node.js --help`. +37. Run `yarn cljs:lint && yarn test` and confirm repository review checklist passes. +38. Validate changed code against `@prompts/review.md` before merge. + +## Edge cases to validate during implementation + +| Scenario | Expected behavior | +| --- | --- | +| `ncc` emits native `.node` assets for `better-sqlite3`. | Bundle output keeps those assets adjacent to entrypoint and runtime loads without `node_modules`. | +| Bundle is copied to another directory without static files. | Daemon still starts because packaged runtime no longer `require`s `../static/db-worker-node.js`. | +| Developer runs doctor in source workspace before bundle build. | Doctor reports missing packaged artifact by default, and only checks static runtime when explicitly requested. | +| `dist/db-worker-node.js` exists but is not readable or is a directory. | Doctor returns `:doctor-script-unreadable` with path detail. | +| Bundle build is run twice. | Build output remains deterministic and stale ncc artifacts are cleaned safely. | +| CLI and Electron share lock for same repo under bundled runtime. | Existing ownership and lock semantics remain unchanged from current behavior. | + +## Verification commands and expected outputs + +```bash +yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test' +yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test' +yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' +yarn db-worker-node:compile +yarn db-worker-node:release:bundle +node dist/db-worker-node.js --help +yarn cljs:lint && yarn test +``` + +All test commands should finish with `0 failures, 0 errors`. + +The bundle command should finish without `MODULE_NOT_FOUND` for runtime dependencies. + +`node dist/db-worker-node.js --help` should print daemon help text and exit with code `0`. + +## Testing Details + +Behavior-focused tests will validate that runtime path resolution, doctor diagnostics, and daemon startup behavior match user-visible expectations. + +The standalone smoke test will verify real process startup and HTTP readiness in a bundle-only filesystem layout, rather than asserting internal helper calls. + +Regression safety is provided by existing CLI server and doctor tests to ensure lock lifecycle and error-code contracts remain stable. + +## Implementation Details + +- Keep packaged runtime path as one canonical helper shared by spawn and doctor. +- Keep dev runtime path explicit and opt-in for local diagnostics only. +- Introduce ncc build scripts with deterministic output normalization into `dist/`. +- Preserve local `shadow-cljs` development flow and avoid slowing watch mode. +- Add a dedicated standalone bundle smoke test namespace for runtime validation. +- Keep error code contracts stable for doctor and server command callers. +- Ensure package `files` include all ncc runtime assets required at execution time. +- Document build and troubleshooting steps in CLI docs for contributors and release workflows. +- Use `@test-driven-development` for every behavior change and follow `@clojure-debug` for unexpected failures. +- Finish with full lint and test validation and checklist review from `@prompts/review.md`. + +## Question + +Resolved: choose option 1. + +`doctor` defaults to strict packaged-runtime validation only. + +`doctor` does not auto-fallback to static runtime without an explicit flag. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 811026221d..03eec18d6f 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -6,8 +6,11 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to ```bash clojure -M:cljs compile logseq-cli db-worker-node +yarn db-worker-node:release:bundle ``` +`yarn db-worker-node:release:bundle` compiles and bundles `db-worker-node` with `@vercel/ncc`, and writes a standalone runtime to `dist/db-worker-node.js` plus adjacent runtime assets (for example `dist/build/Release/better_sqlite3.node`). + ## db-worker-node lifecycle `logseq` manages `db-worker-node` automatically. You should not start the server manually. The server binds to localhost on a random port and records that port in the repo lock file. @@ -80,7 +83,7 @@ Server commands: - `server start --repo ` - start db-worker-node for a graph - `server stop --repo ` - stop db-worker-node for a graph - `server restart --repo ` - restart db-worker-node for a graph -- `doctor` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness +- `doctor [--dev-script]` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness (`--dev-script` checks `static/db-worker-node.js` explicitly) Server ownership behavior: - `server stop` and `server restart` can return `server-owned-by-other` if the daemon was started by another owner source. @@ -137,10 +140,11 @@ Output formats: - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - `doctor` output includes overall status (`ok`, `warning`, `error`) and per-check rows for `db-worker-script`, `data-dir`, and `running-servers`. For scripting, `--output json|edn` keeps the structured check payload. - Common doctor failures: - - `doctor-script-missing`: `db-worker-node.js` runtime target is missing (typically `static/db-worker-node.js`; `dist/db-worker-node.js` is only the wrapper entry). + - `doctor-script-missing`: `db-worker-node.js` runtime target is missing (default target: `dist/db-worker-node.js`; use `doctor --dev-script` to check `static/db-worker-node.js`). - `doctor-script-unreadable`: script path exists but is not a readable file. - `data-dir-permission`: configured data dir is not readable or writable. - `doctor-server-not-ready`: one or more lock-discovered servers are still in `:starting` state (warning). + - If bundled runtime startup fails with native module load errors, rebuild with `yarn db-worker-node:release:bundle` and confirm `dist/db-worker-node-assets.json` and listed assets are present next to `dist/db-worker-node.js`. - `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. - Built-in named queries currently include `block-search`, `task-search`, `recent-updated`, `list-status`, and `list-priority`. Use `query list` to see the full set for your config. - Show and search outputs resolve block reference UUIDs inside text, replacing `[[]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. @@ -169,5 +173,6 @@ node ./dist/logseq.js search "hello" node ./dist/logseq.js show --page TestPage --output json node ./dist/logseq.js server list node ./dist/logseq.js doctor +node ./dist/logseq.js doctor --dev-script node ./dist/logseq.js doctor --output json ``` diff --git a/package.json b/package.json index 08a1c768c7..79e6420dcb 100644 --- a/package.json +++ b/package.json @@ -7,8 +7,7 @@ "logseq": "dist/logseq.js" }, "files": [ - "dist/logseq.js", - "static/db-worker-node.js", + "dist/", "static/logseq-cli.js" ], "engines": { @@ -20,6 +19,7 @@ "@capacitor/cli": "7.2.0", "@jcesarmobile/ssl-skip": "^0.4.0", "@playwright/test": "=1.51.0", + "@vercel/ncc": "0.38.3", "@tailwindcss/aspect-ratio": "0.4.2", "@tailwindcss/forms": "0.5.3", "@tailwindcss/typography": "0.5.7", @@ -104,8 +104,14 @@ "cljs:release-publishing": "clojure -M:cljs release app publishing", "cljs:test": "clojure -M:test compile test", "cljs:run-test": "node static/tests.js -r '^(?!logseq.db-sync.).*' -e fix-me", - "cljs:test-no-worker": "clojure -M:test compile test-no-worker", - "cljs:run-test-no-worker": "node static/tests-no-worker.js", + "cljs:test-no-worker": "clojure -M:test compile test-no-worker", + "cljs:run-test-no-worker": "node static/tests-no-worker.js", + "db-worker-node:compile": "clojure -M:cljs compile db-worker-node", + "db-worker-node:release": "clojure -M:cljs release db-worker-node", + "db-worker-node:ncc": "ncc build static/db-worker-node.js --out dist/.db-worker-node-ncc --asset-builds", + "db-worker-node:bundle:normalize": "node ./scripts/build-db-worker-node-bundle.mjs", + "db-worker-node:release:bundle": "run-s db-worker-node:release db-worker-node:ncc db-worker-node:bundle:normalize", + "db-worker-node:compile:bundle": "run-s db-worker-node:compile db-worker-node:ncc db-worker-node:bundle:normalize", "cljs:dev-release-app": "clojure -M:cljs release app db-worker db-worker-node inference-worker --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\"", "cljs:dev-release-electron": "clojure -M:cljs release app db-worker db-worker-node inference-worker electron --debug --config-merge \"{:closure-defines {frontend.config/DEV-RELEASE true}}\" && clojure -M:cljs release publishing", "cljs:debug": "clojure -M:cljs release app db-worker db-worker-node inference-worker --debug", diff --git a/scripts/build-db-worker-node-bundle.mjs b/scripts/build-db-worker-node-bundle.mjs new file mode 100644 index 0000000000..21908d801c --- /dev/null +++ b/scripts/build-db-worker-node-bundle.mjs @@ -0,0 +1,122 @@ +#!/usr/bin/env node + +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = path.dirname(__filename); +const repoRoot = path.resolve(__dirname, ".."); +const distDir = path.join(repoRoot, "dist"); +const nccOutDir = path.join(distDir, ".db-worker-node-ncc"); +const bundleEntry = path.join(distDir, "db-worker-node.js"); +const manifestPath = path.join(distDir, "db-worker-node-assets.json"); + +async function exists(targetPath) { + try { + await fs.access(targetPath); + return true; + } catch { + return false; + } +} + +async function listFilesRecursive(baseDir, dir = baseDir) { + const entries = await fs.readdir(dir, { withFileTypes: true }); + const files = []; + for (const entry of entries) { + const absolute = path.join(dir, entry.name); + if (entry.isDirectory()) { + files.push(...(await listFilesRecursive(baseDir, absolute))); + } else if (entry.isFile()) { + files.push(path.relative(baseDir, absolute)); + } + } + return files.sort(); +} + +async function removeIfExists(targetPath) { + if (await exists(targetPath)) { + await fs.rm(targetPath, { recursive: true, force: true }); + } +} + +async function cleanupPreviousBundle() { + await removeIfExists(bundleEntry); + + if (!(await exists(manifestPath))) { + return; + } + + let manifest; + try { + manifest = JSON.parse(await fs.readFile(manifestPath, "utf8")); + } catch (error) { + throw new Error(`failed to read ${manifestPath}: ${error.message}`); + } + + const assets = Array.isArray(manifest.assets) ? manifest.assets : []; + for (const relativePath of assets) { + if (typeof relativePath !== "string" || relativePath.length === 0) { + continue; + } + const assetPath = path.join(distDir, relativePath); + await removeIfExists(assetPath); + } + + await removeIfExists(manifestPath); +} + +async function copyBundle() { + if (!(await exists(nccOutDir))) { + throw new Error(`missing ncc output directory: ${nccOutDir}`); + } + + const files = await listFilesRecursive(nccOutDir); + if (!files.includes("index.js")) { + throw new Error(`ncc output missing index.js in ${nccOutDir}`); + } + + await cleanupPreviousBundle(); + + const copiedAssets = []; + for (const relativePath of files) { + const sourcePath = path.join(nccOutDir, relativePath); + const destinationPath = + relativePath === "index.js" + ? bundleEntry + : path.join(distDir, relativePath); + + await fs.mkdir(path.dirname(destinationPath), { recursive: true }); + await fs.copyFile(sourcePath, destinationPath); + + if (relativePath === "index.js") { + const stat = await fs.stat(sourcePath); + await fs.chmod(destinationPath, stat.mode); + } else { + copiedAssets.push(relativePath); + } + } + + await fs.writeFile( + manifestPath, + `${JSON.stringify( + { + assets: copiedAssets, + }, + null, + 2 + )}\n`, + "utf8" + ); +} + +async function main() { + await copyBundle(); + await removeIfExists(nccOutDir); +} + +main().catch((error) => { + console.error(`[db-worker-node-bundle] ${error.message}`); + process.exit(1); +}); diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index b51ca8f897..2a0b4f6e11 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -3,6 +3,7 @@ (:require ["fs" :as fs] ["http" :as http] ["path" :as node-path] + [clojure.string :as string] [frontend.worker.db-core :as db-core] [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.platform.node :as platform-node] @@ -446,8 +447,24 @@ (.on js/process "SIGTERM" shutdown))) (p/catch (fn [error] (let [data (ex-data error) + code (:code data) message (or (.-message error) (str error))] - (when (= :data-dir-permission (:code data)) + (cond + (= :data-dir-permission code) (.error js/console message) - (.exit js/process 1)) - (throw error))))))) + + (or (string/includes? message ".node") + (string/includes? message "Cannot find module") + (string/includes? message "bindings file")) + (.error js/console + (str "db-worker-node failed to start: missing native bundle asset. " + "Rebuild with `yarn db-worker-node:release:bundle` and ensure " + "`dist/db-worker-node-assets.json` assets are next to `db-worker-node.js`. " + "Root error: " + message)) + + :else + (.error js/console (str "db-worker-node failed to start: " message))) + (when-let [stack (.-stack error)] + (.error js/console stack)) + (.exit js/process 1))))))) diff --git a/src/main/logseq/cli/command/doctor.cljs b/src/main/logseq/cli/command/doctor.cljs index 001545c896..0f3adb8f55 100644 --- a/src/main/logseq/cli/command/doctor.cljs +++ b/src/main/logseq/cli/command/doctor.cljs @@ -8,12 +8,20 @@ [promesa.core :as p])) (def entries - [(core/command-entry ["doctor"] :doctor "Run runtime diagnostics" {})]) + [(core/command-entry ["doctor"] + :doctor + "Run runtime diagnostics" + {:dev-script {:desc "Check static/db-worker-node.js instead of bundled dist runtime" + :coerce :boolean}})]) (defn build-action - [] - {:ok? true - :action {:type :doctor}}) + ([] + (build-action {})) + ([options] + {:ok? true + :action (cond-> {:type :doctor} + (:dev-script options) + (assoc :script-path (cli-server/db-worker-dev-script-path)))})) (defn- doctor-error [checks code message] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index cde338c4ec..3e8a778dcb 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -378,7 +378,7 @@ (show-command/build-action options repo) :doctor - (doctor-command/build-action) + (doctor-command/build-action options) {:ok? false :error {:code :unknown-command diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 02a474b032..3b6f981e51 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -55,6 +55,10 @@ (node-path/join js/__dirname "../dist/db-worker-node.js")) (defn db-worker-runtime-script-path + [] + (db-worker-script-path)) + +(defn db-worker-dev-script-path [] (node-path/join js/__dirname "../static/db-worker-node.js")) diff --git a/src/test/logseq/cli/command/doctor_test.cljs b/src/test/logseq/cli/command/doctor_test.cljs index b785083125..497a9fd904 100644 --- a/src/test/logseq/cli/command/doctor_test.cljs +++ b/src/test/logseq/cli/command/doctor_test.cljs @@ -116,7 +116,7 @@ (set! cli-server/list-servers orig-list-servers) (done))))))) -(deftest test-execute-doctor-default-script-checks-static-runtime-target +(deftest test-execute-doctor-default-script-checks-packaged-runtime-target (async done (let [orig-ensure-data-dir! data-dir/ensure-data-dir! orig-list-servers cli-server/list-servers] @@ -126,6 +126,27 @@ {:data-dir "/tmp/logseq-doctor"}) checked-path (get-in result [:data :checks 0 :path])] (is (= :ok (:status result))) + (is (= (cli-server/db-worker-script-path) checked-path)) + (is (string/ends-with? checked-path "/dist/db-worker-node.js"))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) + (set! cli-server/list-servers orig-list-servers) + (done))))))) + +(deftest test-execute-doctor-explicit-script-path-checks-static-runtime-target + (async done + (let [orig-ensure-data-dir! data-dir/ensure-data-dir! + orig-list-servers cli-server/list-servers] + (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) + (set! cli-server/list-servers (fn [_] (p/resolved []))) + (-> (p/let [result (commands/execute {:type :doctor + :script-path (cli-server/db-worker-dev-script-path)} + {:data-dir "/tmp/logseq-doctor"}) + checked-path (get-in result [:data :checks 0 :path])] + (is (= :ok (:status result))) + (is (= (cli-server/db-worker-dev-script-path) checked-path)) (is (string/ends-with? checked-path "/static/db-worker-node.js"))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 4fa96bcbad..08d2f11393 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -290,7 +290,13 @@ (testing "doctor command parses" (let [result (commands/parse-args ["doctor"])] (is (true? (:ok? result))) - (is (= :doctor (:command result)))))) + (is (= :doctor (:command result))))) + + (testing "doctor command parses explicit dev script option" + (let [result (commands/parse-args ["doctor" "--dev-script"])] + (is (true? (:ok? result))) + (is (= :doctor (:command result))) + (is (= true (get-in result [:options :dev-script])))))) (deftest test-tree->text-format (testing "show tree text uses db/id with tree glyphs" @@ -1157,7 +1163,15 @@ (let [parsed {:ok? true :command :doctor :options {}} result (commands/build-action parsed {})] (is (true? (:ok? result))) - (is (= :doctor (get-in result [:action :type])))))) + (is (= :doctor (get-in result [:action :type]))))) + + (testing "doctor dev script option builds explicit static runtime action" + (let [parsed {:ok? true :command :doctor :options {:dev-script true}} + result (commands/build-action parsed {})] + (is (true? (:ok? result))) + (is (= :doctor (get-in result [:action :type]))) + (is (= (cli-server/db-worker-dev-script-path) + (get-in result [:action :script-path])))))) (deftest test-build-action-inspect-edit (testing "list page requires repo" diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index e8ef5f7f10..47e607fd37 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -40,6 +40,10 @@ expected (node-path/join data-dir "demo" "db-worker.lock")] (is (= expected (cli-server/lock-path data-dir repo))))) +(deftest db-worker-runtime-script-path-defaults-to-packaged-dist-target + (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") + (cli-server/db-worker-runtime-script-path)))) + (deftest ensure-server-repairs-stale-lock (async done diff --git a/src/test/logseq/db_worker/ncc_bundle_test.cljs b/src/test/logseq/db_worker/ncc_bundle_test.cljs new file mode 100644 index 0000000000..88afa676f8 --- /dev/null +++ b/src/test/logseq/db_worker/ncc_bundle_test.cljs @@ -0,0 +1,163 @@ +(ns logseq.db-worker.ncc-bundle-test + (:require [cljs.test :refer [async deftest is]] + [clojure.string :as string] + [frontend.test.node-helper :as node-helper] + [frontend.worker.db-worker-node-lock :as db-lock] + [logseq.db-worker.daemon :as daemon] + [promesa.core :as p] + ["child_process" :as child-process] + ["fs" :as fs] + ["path" :as node-path])) + +(defonce ^:private bundle-built? (atom false)) + +(defn- repo-root + [] + (.cwd js/process)) + +(defn- dist-path + [& segments] + (apply node-path/join (repo-root) "dist" segments)) + +(defn- absolute-path + [path] + (node-path/resolve path)) + +(defn- run-bundle-build! + [] + (.execFileSync child-process + "yarn" + #js ["db-worker-node:release:bundle"] + #js {:cwd (repo-root) + :encoding "utf8"})) + +(defn- ensure-bundle-built! + [] + (when-not @bundle-built? + (run-bundle-build!) + (reset! bundle-built? true))) + +(defn- read-asset-manifest + [] + (let [manifest-path (dist-path "db-worker-node-assets.json")] + (js->clj (js/JSON.parse (.toString (fs/readFileSync manifest-path) "utf8")) + :keywordize-keys true))) + +(defn- copy-file! + [source destination] + (fs/mkdirSync (node-path/dirname destination) #js {:recursive true}) + (fs/copyFileSync source destination)) + +(defn- copy-bundle-to-temp! + [] + (let [runtime-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-runtime")) + manifest (read-asset-manifest) + assets (vec (:assets manifest)) + entry-source (dist-path "db-worker-node.js") + entry-destination (node-path/join runtime-dir "db-worker-node.js")] + (copy-file! entry-source entry-destination) + (doseq [asset assets] + (copy-file! (dist-path asset) + (node-path/join runtime-dir asset))) + {:runtime-dir runtime-dir + :assets assets})) + +(defn- lock-path + [data-dir repo] + (node-path/join (db-lock/repo-dir data-dir repo) "db-worker.lock")) + +(defn- spawn-daemon! + [runtime-dir repo data-dir] + (let [child (.spawn child-process + "node" + #js ["./db-worker-node.js" + "--repo" repo + "--data-dir" data-dir + "--owner-source" "cli"] + #js {:cwd runtime-dir}) + stdout (atom "") + stderr (atom "")] + (.on (.-stdout child) + "data" + (fn [chunk] + (swap! stdout str (.toString chunk "utf8")))) + (.on (.-stderr child) + "data" + (fn [chunk] + (swap! stderr str (.toString chunk "utf8")))) + {:child child + :stdout stdout + :stderr stderr})) + +(deftest bundle-only-daemon-startup-smoke-test + (async done + (let [child* (atom nil)] + (-> (p/let [_ (ensure-bundle-built!) + {:keys [runtime-dir]} (copy-bundle-to-temp!) + data-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-data")) + repo (str "logseq_db_ncc_smoke_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo) + {:keys [child]} (spawn-daemon! runtime-dir repo data-dir) + _ (reset! child* child) + _ (daemon/wait-for-lock lock-file) + lock (daemon/read-lock lock-file) + _ (is (some? lock)) + health (daemon/http-request {:method "GET" + :host (:host lock) + :port (:port lock) + :path "/healthz" + :timeout-ms 1000}) + ready (daemon/http-request {:method "GET" + :host (:host lock) + :port (:port lock) + :path "/readyz" + :timeout-ms 1000}) + shutdown (daemon/http-request {:method "POST" + :host (:host lock) + :port (:port lock) + :path "/v1/shutdown" + :headers {"Content-Type" "application/json"} + :timeout-ms 2000}) + _ (is (= 200 (:status health))) + _ (is (= 200 (:status ready))) + _ (is (= 200 (:status shutdown))) + _ (daemon/wait-for (fn [] + (p/resolved (not (fs/existsSync lock-file)))) + {:timeout-ms 10000 + :interval-ms 200})] + (is (not (fs/existsSync lock-file))) + (done)) + (p/catch (fn [e] + (when-let [^js child @child*] + (try + (.kill child "SIGTERM") + (catch :default _))) + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest bundle-missing-native-asset-has-actionable-error + (let [_ (ensure-bundle-built!) + {:keys [runtime-dir assets]} (copy-bundle-to-temp!) + native-asset (first (filter #(string/ends-with? % ".node") assets))] + (is (some? native-asset)) + (when native-asset + (let [missing-path (node-path/join runtime-dir native-asset) + _ (fs/unlinkSync missing-path) + data-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-missing-asset")) + repo (str "logseq_db_ncc_missing_" (subs (str (random-uuid)) 0 8)) + result (.spawnSync child-process + "node" + #js ["./db-worker-node.js" + "--repo" repo + "--data-dir" data-dir + "--owner-source" "cli"] + #js {:cwd runtime-dir + :encoding "utf8"}) + status (.-status result) + output (str (or (.-stderr result) "") + (or (.-stdout result) "")) + missing-file (node-path/basename missing-path)] + (is (not= 0 status)) + (is (or (string/includes? output missing-file) + (string/includes? output "Cannot find module") + (string/includes? output "could not locate the bindings file"))))))) diff --git a/yarn.lock b/yarn.lock index b14f4298e9..45f41e8e5d 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1546,6 +1546,11 @@ "@types/expect" "^1.20.4" "@types/node" "*" +"@vercel/ncc@0.38.3": + version "0.38.3" + resolved "https://registry.yarnpkg.com/@vercel/ncc/-/ncc-0.38.3.tgz#5475eeee3ac0f1a439f237596911525a490a88b5" + integrity sha512-rnK6hJBS6mwc+Bkab+PGPs9OiS0i/3kdTO+CkI8V0/VrW3vmz7O2Pxjw/owOlmo6PKEIxRSeZKv/kuL9itnpYA== + "@webassemblyjs/ast@1.14.1", "@webassemblyjs/ast@^1.14.1": version "1.14.1" resolved "https://registry.yarnpkg.com/@webassemblyjs/ast/-/ast-1.14.1.tgz#a9f6a07f2b03c95c8d38c4536a1fdfb521ff55b6" From c155be440a766923509d29770a42bf65ab87bcd6 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 23 Feb 2026 21:09:55 +0800 Subject: [PATCH 080/375] 036-db-worker-node-ncc-bundling.md (2) --- .../036-db-worker-node-ncc-bundling.md | 48 +++++++++------ package.json | 2 +- resources/forge.config.js | 1 + resources/forge.config.test.js | 61 +++++++++++++++++++ src/main/logseq/db_worker/daemon.cljs | 18 +++--- src/test/logseq/cli/server_test.cljs | 4 +- src/test/logseq/db_worker/daemon_test.cljs | 4 +- 7 files changed, 108 insertions(+), 30 deletions(-) create mode 100644 resources/forge.config.test.js diff --git a/docs/agent-guide/036-db-worker-node-ncc-bundling.md b/docs/agent-guide/036-db-worker-node-ncc-bundling.md index 3d052cae0c..fb3fcfb79b 100644 --- a/docs/agent-guide/036-db-worker-node-ncc-bundling.md +++ b/docs/agent-guide/036-db-worker-node-ncc-bundling.md @@ -21,6 +21,8 @@ This wrapper model keeps runtime coupled to workspace layout and to `node_module We need a deterministic packaging path that produces one runnable artifact plus copied native assets, and we need to verify that artifact works when `node_modules` is absent. +Electron release packaging also needs to include the db-worker standalone bundle step, so `yarn release-electron` always ships the same packaged runtime artifact. + The solution must keep existing CLI and Electron daemon orchestration behavior unchanged, including lock-file semantics, owner-source semantics, and health endpoint behavior. ## Current packaging map @@ -32,6 +34,7 @@ The solution must keep existing CLI and Electron daemon orchestration behavior u | Daemon spawn | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` spawns `../dist/db-worker-node.js`. | Spawn path is stable, but executable is not standalone. | | Doctor check | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` checks `../static/db-worker-node.js` by default. | Diagnostic target does not match the intended distributable runtime. | | Package manifest | `/Users/rcmerci/gh-repos/logseq/package.json` includes `static/db-worker-node.js` in `files`. | Published package does not guarantee standalone daemon artifact contract. | +| Electron release | `yarn release-electron` does not guarantee db-worker bundle refresh before packaging. | Desktop release artifact can drift from standalone db-worker bundle contract. | ## Target packaging map @@ -42,6 +45,7 @@ The solution must keep existing CLI and Electron daemon orchestration behavior u | Doctor check | Doctor defaults to the same packaged runtime path used for spawn, and does not auto-fallback to static runtime. | Doctor check path matches runtime path in tests and manual runs. | | Dev flow | Fast local dev command remains available using `static/db-worker-node.js` for watch and debug workflows. | `yarn db-worker-node:compile` and `node ./static/db-worker-node.js` still work during development. | | Publish flow | Package `files` include standalone runtime assets required by ncc output. | Installed package can execute daemon without extra dependency install. | +| Electron release | `yarn release-electron` runs db-worker bundle build before Electron packaging steps. | Electron release artifact includes the same standalone db-worker runtime contract. | ## Integration sketch @@ -92,7 +96,8 @@ NOTE: I will write *all* tests before I add any implementation behavior. 11. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to normalize ncc output into `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` with adjacent assets preserved. 12. If script complexity is non-trivial, add `/Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs` to encapsulate output normalization and deterministic cleanup. 13. Add or update `yarn` scripts in `/Users/rcmerci/gh-repos/logseq/package.json` for one-command bundle build and optional local run of the bundled artifact. -14. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` is regenerated with executable permissions preserved. +14. Update `yarn release-electron` in `/Users/rcmerci/gh-repos/logseq/package.json` so it includes `db-worker-node:release:bundle` before Electron packaging. +15. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` is regenerated with executable permissions preserved. ### Phase 3: Align runtime path and diagnostics. @@ -101,34 +106,35 @@ NOTE: I will write *all* tests before I add any implementation behavior. 17. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` to default-check the same packaged runtime path used by spawn. 18. Add an explicit action option in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` for fallback static-path diagnostics used only for development troubleshooting. 19. Ensure doctor failure codes remain stable as `:doctor-script-missing` and `:doctor-script-unreadable`. -20. Re-run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` to make the new path contract green. +21. Re-run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` to make the new path contract green. ### Phase 4: Package manifest and docs alignment. -21. Update `files` in `/Users/rcmerci/gh-repos/logseq/package.json` so packaged runtime includes `dist/db-worker-node.js` and ncc-emitted adjacent assets. -22. Keep `static/db-worker-node.js` inclusion only if required for development workflows, and document that distinction explicitly. -23. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build instructions to include the ncc bundle command and standalone runtime expectations. -24. Update daemon runtime notes in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` so `doctor` references packaged runtime as primary target. -25. Add troubleshooting notes for native module asset copy behavior from ncc output. +22. Update `files` in `/Users/rcmerci/gh-repos/logseq/package.json` so packaged runtime includes `dist/db-worker-node.js` and ncc-emitted adjacent assets. +23. Keep `static/db-worker-node.js` inclusion only if required for development workflows, and document that distinction explicitly. +24. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build instructions to include the ncc bundle command and standalone runtime expectations. +25. Update daemon runtime notes in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` so `doctor` references packaged runtime as primary target. +26. Add troubleshooting notes for native module asset copy behavior from ncc output. ### Phase 5: Standalone runtime behavior verification. -26. Implement bundle smoke test setup in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to copy only bundle artifacts into a temp directory with no `node_modules`. -27. In that test, spawn `node ./db-worker-node.js --repo --data-dir ` from the bundle-only directory. -28. In that test, poll `/healthz` and `/readyz` and assert both return HTTP 200 after startup. -29. In that test, invoke `/v1/shutdown` and assert process exits and lock file is cleaned or becomes stale-removable. -30. In that test, assert failure output is actionable if native binary asset is missing, to guard accidental packaging regressions. -31. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and make it green. +27. Implement bundle smoke test setup in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to copy only bundle artifacts into a temp directory with no `node_modules`. +28. In that test, spawn `node ./db-worker-node.js --repo --data-dir ` from the bundle-only directory. +29. In that test, poll `/healthz` and `/readyz` and assert both return HTTP 200 after startup. +30. In that test, invoke `/v1/shutdown` and assert process exits and lock file is cleaned or becomes stale-removable. +31. In that test, assert failure output is actionable if native binary asset is missing, to guard accidental packaging regressions. +32. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and make it green. ### Phase 6: Final validation and review checklist. -32. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and confirm zero failures and zero errors. -33. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm zero failures and zero errors. -34. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm zero failures and zero errors. -35. Run `yarn db-worker-node:compile` and verify `static/db-worker-node.js` remains valid for local dev flow. -36. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` starts successfully with `node dist/db-worker-node.js --help`. -37. Run `yarn cljs:lint && yarn test` and confirm repository review checklist passes. -38. Validate changed code against `@prompts/review.md` before merge. +33. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'` and confirm zero failures and zero errors. +34. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm zero failures and zero errors. +35. Run `yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm zero failures and zero errors. +36. Run `yarn db-worker-node:compile` and verify `static/db-worker-node.js` remains valid for local dev flow. +37. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` starts successfully with `node dist/db-worker-node.js --help`. +38. Run `yarn release-electron` and verify the script execution includes `db-worker-node:release:bundle` before Electron packaging steps. +39. Run `yarn cljs:lint && yarn test` and confirm repository review checklist passes. +40. Validate changed code against `@prompts/review.md` before merge. ## Edge cases to validate during implementation @@ -139,6 +145,7 @@ NOTE: I will write *all* tests before I add any implementation behavior. | Developer runs doctor in source workspace before bundle build. | Doctor reports missing packaged artifact by default, and only checks static runtime when explicitly requested. | | `dist/db-worker-node.js` exists but is not readable or is a directory. | Doctor returns `:doctor-script-unreadable` with path detail. | | Bundle build is run twice. | Build output remains deterministic and stale ncc artifacts are cleaned safely. | +| `yarn release-electron` is run directly. | Release flow still builds db-worker standalone bundle before Electron packaging artifacts are produced. | | CLI and Electron share lock for same repo under bundled runtime. | Existing ownership and lock semantics remain unchanged from current behavior. | ## Verification commands and expected outputs @@ -149,6 +156,7 @@ yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test' yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' yarn db-worker-node:compile yarn db-worker-node:release:bundle +yarn release-electron node dist/db-worker-node.js --help yarn cljs:lint && yarn test ``` diff --git a/package.json b/package.json index 79e6420dcb..cfe5ce71de 100644 --- a/package.json +++ b/package.json @@ -70,7 +70,7 @@ "release-mobile": "run-s gulp:buildMobile cljs:release-mobile webpack-mobile-build", "dev-release-app": "run-s gulp:build cljs:dev-release-app webpack-app-build", "dev-electron-app": "gulp electron", - "release-electron": "run-s gulp:build && yarn webpack-app-build && gulp electronMaker", + "release-electron": "run-s gulp:build db-worker-node:release:bundle webpack-app-build && gulp electronMaker", "debug-electron": "cd static/ && yarn electron:debug", "webpack-watch": "webpack --watch", "webpack-app-watch": "npx webpack --watch --config-name app", diff --git a/resources/forge.config.js b/resources/forge.config.js index 00da941ff1..4ee1545a86 100644 --- a/resources/forge.config.js +++ b/resources/forge.config.js @@ -5,6 +5,7 @@ module.exports = { packagerConfig: { name: 'Logseq', icon: './icons/logseq_big_sur.icns', + extraResource: [path.resolve(__dirname, '../dist')], buildVersion: "88", appBundleId: "com.logseq.logseq", protocols: [ diff --git a/resources/forge.config.test.js b/resources/forge.config.test.js new file mode 100644 index 0000000000..1da63bf832 --- /dev/null +++ b/resources/forge.config.test.js @@ -0,0 +1,61 @@ +const test = require('node:test') +const assert = require('node:assert/strict') +const fs = require('fs') +const path = require('path') + +const config = require('./forge.config') + +const repoRoot = path.resolve(__dirname, '..') + +function normalizeResourcePath(resourcePath) { + return path.resolve(resourcePath) +} + +function collectExtraResources() { + return (config.packagerConfig?.extraResource || []).map(normalizeResourcePath) +} + +function isCoveredByExtraResources(targetPath, extraResources) { + const normalizedTarget = path.resolve(targetPath) + return extraResources.some((resource) => + normalizedTarget === resource || normalizedTarget.startsWith(resource + path.sep) + ) +} + +test('packager includes all JavaScript files under dist/', () => { + const distDir = path.resolve(repoRoot, 'dist') + const distJsFiles = fs + .readdirSync(distDir) + .filter((name) => name.endsWith('.js')) + .map((name) => path.join(distDir, name)) + + assert.ok( + distJsFiles.length > 0, + 'Expected dist/ to contain at least one JavaScript file for packaging checks' + ) + + const extraResources = collectExtraResources() + + for (const jsFile of distJsFiles) { + assert.ok( + isCoveredByExtraResources(jsFile, extraResources), + `Expected extraResource to include or cover ${jsFile}` + ) + } +}) + +test('packager includes dist/build directory', () => { + const distBuildDir = path.resolve(repoRoot, 'dist/build') + + assert.ok( + fs.existsSync(distBuildDir), + 'Expected dist/build directory to exist for packaging checks' + ) + + const extraResources = collectExtraResources() + + assert.ok( + isCoveredByExtraResources(distBuildDir, extraResources), + `Expected extraResource to include or cover ${distBuildDir}` + ) +}) diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs index 5a6ac29f7a..985d919a94 100644 --- a/src/main/logseq/db_worker/daemon.cljs +++ b/src/main/logseq/db_worker/daemon.cljs @@ -256,10 +256,14 @@ (defn spawn-server! [{:keys [script repo data-dir owner-source]}] (let [owner-source (normalize-owner-source owner-source) - args #js ["--repo" repo "--data-dir" data-dir "--owner-source" (name owner-source)] - child (.spawn child-process script args #js {:detached true - :stdio "ignore"})] - (when-not script - (log/warn :db-worker-daemon/missing-script {:repo repo :data-dir data-dir})) - (.unref child) - child)) + args #js [script "--repo" repo "--data-dir" data-dir "--owner-source" (name owner-source)] + env (js/Object.assign #js {} (.-env js/process) #js {:ELECTRON_RUN_AS_NODE "1"})] + (if-not script + (do + (log/warn :db-worker-daemon/missing-script {:repo repo :data-dir data-dir}) + nil) + (let [child (.spawn child-process (.-execPath js/process) args #js {:detached true + :stdio "ignore" + :env env})] + (.unref child) + child)))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index 47e607fd37..92918fb799 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -25,8 +25,10 @@ (.chdir js/process "/") (spawn-server! {:repo "logseq_db_spawn_test" :data-dir "/tmp/logseq-db-worker"}) - (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") + (is (= (.-execPath js/process) (:cmd @captured))) + (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") + (first (:args @captured)))) (is (some #{"--repo"} (:args @captured))) (is (some #{"--data-dir"} (:args @captured))) (is (not-any? #{"--host" "--port"} (:args @captured))) diff --git a/src/test/logseq/db_worker/daemon_test.cljs b/src/test/logseq/db_worker/daemon_test.cljs index 32ea68e444..07242aa6e7 100644 --- a/src/test/logseq/db_worker/daemon_test.cljs +++ b/src/test/logseq/db_worker/daemon_test.cljs @@ -20,11 +20,13 @@ (daemon/spawn-server! {:script "/tmp/db-worker-node.js" :repo "logseq_db_spawn_helper_test" :data-dir "/tmp/logseq-db-worker"}) - (is (= "/tmp/db-worker-node.js" (:cmd @captured))) + (is (= (.-execPath js/process) (:cmd @captured))) + (is (= "/tmp/db-worker-node.js" (first (:args @captured)))) (is (some #{"--repo"} (:args @captured))) (is (some #{"--data-dir"} (:args @captured))) (is (not-any? #{"--host" "--port"} (:args @captured))) (is (= true (get-in @captured [:opts :detached]))) + (is (= "1" (get-in @captured [:opts :env :ELECTRON_RUN_AS_NODE]))) (finally (set! (.-spawn child-process) original-spawn))))) From 1af88b2b6ac802e7873039f9e5239c4ff7558377 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 24 Feb 2026 16:37:16 +0800 Subject: [PATCH 081/375] 037-db-worker-node-node-sqlite.md --- .../037-db-worker-node-node-sqlite.md | 166 ++++++++++++++++ .../task--db-worker-nodejs-compatible.md | 24 +-- docs/cli/logseq-cli.md | 4 +- package.json | 1 - src/main/frontend/worker/db_worker_node.cljs | 6 +- src/main/frontend/worker/platform/node.cljs | 127 +++++++++--- .../frontend/worker/platform_node_test.cljs | 184 ++++++++++++++++++ .../logseq/db_worker/ncc_bundle_test.cljs | 141 +++++++++++--- yarn.lock | 8 - 9 files changed, 582 insertions(+), 79 deletions(-) create mode 100644 docs/agent-guide/037-db-worker-node-node-sqlite.md create mode 100644 src/test/frontend/worker/platform_node_test.cljs diff --git a/docs/agent-guide/037-db-worker-node-node-sqlite.md b/docs/agent-guide/037-db-worker-node-node-sqlite.md new file mode 100644 index 0000000000..e23e65fd33 --- /dev/null +++ b/docs/agent-guide/037-db-worker-node-node-sqlite.md @@ -0,0 +1,166 @@ +# db-worker-node Node Built-in SQLite Migration Implementation Plan + +Goal: Replace `better-sqlite3` in `db-worker-node` with Node.js built-in `node:sqlite` while keeping `logseq-cli` behavior and db-worker thread-api contracts unchanged. + +Architecture: Keep the existing platform adapter boundary in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` and swap only the Node SQLite backend implementation to a compatibility wrapper around `DatabaseSync` and `StatementSync`. +Architecture: Preserve daemon lifecycle and lock ownership semantics, then update bundle/test/doc assumptions that currently depend on native `.node` assets from `better-sqlite3`. + +Tech Stack: ClojureScript, `shadow-cljs` `:node-script`, Node.js `>=22.20.0`, `node:sqlite`, `@vercel/ncc`, `logseq-cli` HTTP transport. + +Related: Builds on `docs/agent-guide/033-desktop-db-worker-node-backend.md` and `docs/agent-guide/036-db-worker-node-ncc-bundling.md`. +Related: Relates to `docs/agent-guide/task--db-worker-nodejs-compatible.md` and `docs/cli/logseq-cli.md`. + +## Problem statement + +`db-worker-node` currently requires `better-sqlite3` from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. + +`logseq-cli` and Electron desktop both depend on this daemon runtime through `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`, so backend driver replacement must not change daemon API behavior. + +The current ncc bundle plan and tests assume native `.node` assets are emitted and copied next to `dist/db-worker-node.js`, which is specific to `better-sqlite3` and must be revised after migration. + +Node.js in this repository is already pinned to `>=22.20.0` in `/Users/rcmerci/gh-repos/logseq/package.json`, so `node:sqlite` is available, but it is still experimental and emits runtime warnings that we need to account for. + +## Current implementation map + +| Area | Current implementation | Migration impact | +| --- | --- | --- | +| Node sqlite adapter | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` wraps `better-sqlite3` with custom `exec` and `transaction` behavior. | Replace constructor and statement execution with `node:sqlite` API while preserving wrapper contract. | +| Daemon runtime contract | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` exposes `/healthz`, `/readyz`, `/v1/invoke`, `/v1/events`. | No protocol change allowed. | +| CLI runtime spawn | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` spawns `dist/db-worker-node.js`. | Behavior must remain unchanged. | +| Bundle smoke tests | `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` expects a missing native `.node` asset failure mode. | Replace asset assertions to match built-in sqlite runtime with zero native assets. | +| Dependency declaration | `/Users/rcmerci/gh-repos/logseq/package.json` includes `better-sqlite3`. | Remove dependency and refresh lockfile. | +| CLI documentation | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` references `dist/build/Release/better_sqlite3.node`. | Update build output description and troubleshooting text. | + +## Target architecture + +```text +logseq-cli / electron + -> db-worker-node HTTP API + -> frontend.worker.db-core + -> frontend.worker.platform.node sqlite wrapper + -> node:sqlite DatabaseSync + -> graph-dir/*.sqlite files +``` + +The wrapper contract consumed by db-core remains `open-db`, `exec`, `transaction`, and `close`. + +The wrapper implementation changes from `better-sqlite3` to `node:sqlite` internals only. + +## Testing Plan + +I will use `@test-driven-development` and add failing tests first for adapter behavior and bundle assumptions before modifying runtime code. + +I will add focused Node adapter tests for parameter binding, array-row reads, commit behavior, and rollback behavior so the compatibility wrapper is behavior-locked. + +I will keep existing daemon smoke tests and CLI sqlite import/export integration tests as end-to-end regression guards. + +I will update ncc bundle tests so they validate standalone runtime startup without relying on native `.node` artifacts. + +I will run `bb dev:test` for focused namespaces first, then run `bb dev:lint-and-test` for the repository checklist. + +I will use `@clojure-debug` if any ClojureScript test fails unexpectedly while porting the adapter. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Lock behavior with failing tests. + +1. Add a new test namespace at `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/platform_node_test.cljs` to cover Node sqlite wrapper behavior in isolation. +2. Add a failing test that `exec` with SQL string creates schema and writes data through the wrapper. +3. Add a failing test that `exec` with `{:sql ... :bind ... :rowMode "array"}` returns array rows in the same shape used by `restore-data-from-addr`. +4. Add a failing test that named bindings with `$name` and `:name` styles are both accepted by the wrapper. +5. Add a failing test that wrapper `transaction` commits writes on success. +6. Add a failing test that wrapper `transaction` rolls back writes when callback throws. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` asserting bundled runtime can start even when bundle manifest has zero assets. +8. Replace the current missing-native-asset expectation test with a failing test that checks bundle manifest format and actionable errors for missing manifest or missing entry script instead. +9. Run `bb dev:test -v 'frontend.worker.platform-node-test'` and confirm failures before implementation. +10. Run `bb dev:test -v 'logseq.db-worker.ncc-bundle-test'` and confirm failures before implementation. + +### Phase 2: Port Node sqlite adapter to node:sqlite. + +11. Edit `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` to replace `"better-sqlite3"` require with `"node:sqlite"`. +12. Build a `DatabaseSync` constructor resolver compatible with Shadow-CLJS interop and avoid default-export assumptions. +13. Keep `open-sqlite-db` async shape unchanged and create `DatabaseSync` after ensuring parent directory exists. +14. Re-implement statement execution so `:rowMode "array"` maps to `StatementSync#setReturnArrays(true)`. +15. Re-implement positional and named parameter passing for array and object binds without changing db-core callsites. +16. Preserve existing bind key normalization behavior for `$name` and `:name` forms to avoid hidden regressions. +17. Re-implement wrapper `transaction` semantics using explicit SQL transaction control with rollback on exceptions. +18. Add nested-transaction safety via savepoint naming or equivalent deterministic strategy to avoid partial writes from nested calls. +19. Keep wrapper `close` idempotent and compatible with existing shutdown paths in `db-core` and daemon stop. +20. Keep all public platform map keys unchanged in `node-platform`. + +### Phase 3: Update dependency and bundle assumptions. + +21. Remove `better-sqlite3` from `/Users/rcmerci/gh-repos/logseq/package.json` dependencies. +22. Run `yarn install` to refresh `/Users/rcmerci/gh-repos/logseq/yarn.lock` and verify `better-sqlite3` is removed from runtime dependency graph. +23. Verify no remaining runtime require for `better-sqlite3` via `rg -n "better-sqlite3" /Users/rcmerci/gh-repos/logseq/src /Users/rcmerci/gh-repos/logseq/package.json /Users/rcmerci/gh-repos/logseq/yarn.lock`. +24. Keep `/Users/rcmerci/gh-repos/logseq/scripts/package.json` unchanged in this task because scope is limited to `logseq-cli` and `db-worker-node` runtime paths. +25. Update `/Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs` only if manifest handling needs explicit support for empty asset arrays. + +### Phase 4: Refresh tests and docs around runtime packaging. + +26. Update `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to stop asserting a required `.node` asset exists. +27. Keep startup smoke test that runs copied `db-worker-node.js` from a temporary runtime directory with no `node_modules`. +28. Add assertions that `/healthz`, `/readyz`, and `/v1/shutdown` still work under bundled runtime. +29. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build section to remove `better_sqlite3.node` runtime-asset example. +30. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` troubleshooting text to describe expected bundle output when native assets are absent. +31. Update references in `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/task--db-worker-nodejs-compatible.md` to reflect that Node runtime now uses built-in sqlite. + +### Phase 5: Run regression and verification commands. + +32. Run `bb dev:test -v 'frontend.worker.platform-node-test'` and expect `0 failures, 0 errors`. +33. Run `bb dev:test -v 'frontend.worker.db-worker-node-test/db-worker-node-daemon-smoke-test'` and expect daemon startup and query path to pass. +34. Run `bb dev:test -v 'frontend.worker.db-worker-node-test/db-worker-node-import-db-base64'` and expect sqlite export/import behavior to pass. +35. Run `bb dev:test -v 'logseq.db-worker.ncc-bundle-test'` and expect standalone bundle smoke tests to pass. +36. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-graph-export-import-sqlite'` and expect end-to-end CLI sqlite flow to pass. +37. Run `clojure -M:cljs compile db-worker-node logseq-cli` and expect successful node-script builds. +38. Run `yarn db-worker-node:release:bundle` and verify `dist/db-worker-node.js` still starts with `node ./dist/db-worker-node.js --help`. +39. Run `bb dev:lint-and-test` and expect full lint and test checks to pass. +40. Review changed files against `/Users/rcmerci/gh-repos/logseq/prompts/review.md` checklist before merge. + +## Edge cases to validate during implementation + +| Scenario | Expected behavior | +| --- | --- | +| Named parameter binding uses `$name` keys from current callsites. | Statement executes without bind-key mismatch errors. | +| Named parameter binding uses `:name` keys from normalized callsites. | Statement executes and returns same data as before. | +| `rowMode` is `"array"` for kv restore reads. | First row remains index-addressable for existing `first` and destructuring logic. | +| Transaction callback throws mid-write. | All writes in that transaction scope are rolled back. | +| Nested transaction callback occurs inside outer transaction. | Inner failure does not commit partial data and outer behavior is deterministic. | +| ncc bundle emits zero extra assets. | Bundle tests and CLI docs still treat runtime as valid standalone output. | +| Daemon starts under Node 22 and emits experimental sqlite warning. | Warning does not break health/readiness checks or CLI invoke flow. | +| Graph sqlite import/export uses large payloads. | Base64 transport and file writes still preserve binary integrity. | + +## Decisions confirmed + +1. Keep Node experimental `node:sqlite` warning output as-is for this migration phase; warning suppression and logging policy changes are out of scope. +2. Scope is limited to `logseq-cli` and `db-worker-node`; do not modify `/Users/rcmerci/gh-repos/logseq/scripts/package.json` in this task. +3. No temporary fallback flag to `better-sqlite3`; enforce one-way migration to built-in `node:sqlite`. +4. Treat Node.js `>=22.20.0` as a hard prerequisite for local and CI runtime to ensure `node:sqlite` availability. + +## Testing Details + +The adapter unit tests will validate observable behavior for SQL execution, parameter binding, row shape, and transaction semantics instead of testing internal helper structure. + +The daemon smoke tests will validate real process startup and thread-api calls so platform wiring and lock behavior stay stable. + +The CLI integration sqlite export/import test will verify user-visible behavior from command surface to db-worker storage backend. + +The bundle tests will validate standalone runtime packaging assumptions that changed because native `.node` assets are no longer required. + +## Implementation Details + +- Keep `db-worker-node` HTTP and SSE API contracts unchanged. +- Keep platform adapter keys unchanged to avoid db-core callsite churn. +- Implement a compatibility wrapper over `DatabaseSync` instead of refactoring db-core. +- Preserve bind normalization semantics for backward compatibility. +- Implement explicit rollback-safe transaction handling with nested safety. +- Remove `better-sqlite3` only from main runtime dependency declarations. +- Update bundle tests to assert behavior, not driver-specific artifact names. +- Update CLI docs and historical planning notes that mention native sqlite assets. +- Require Node.js `>=22.20.0` in local and CI verification environments. +- Run focused tests first, then repository-wide lint and tests. +- Follow `@test-driven-development` and `@clojure-debug` for implementation and debugging workflow. + +--- diff --git a/docs/agent-guide/task--db-worker-nodejs-compatible.md b/docs/agent-guide/task--db-worker-nodejs-compatible.md index 45595e5b4a..3057c4798b 100644 --- a/docs/agent-guide/task--db-worker-nodejs-compatible.md +++ b/docs/agent-guide/task--db-worker-nodejs-compatible.md @@ -18,7 +18,7 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No - Implement `frontend.worker.platform.node` using `fs/promises`, `path`, `crypto`, and `ws`. 3. Abstract sqlite storage and VFS specifics. - Browser: keep OPFS SAH pool implementation. - - Node: use file-backed sqlite storage via `better-sqlite3` (no OPFS, no sqlite-wasm). + - Node: use file-backed sqlite storage via Node built-in `node:sqlite` (no OPFS, no sqlite-wasm). - Route db path resolution through the platform adapter (data dir, per-repo paths). 4. Replace `importScripts` bootstrap with an explicit init entrypoint. - Browser build still uses `:web-worker`, but entrypoint should call `init!` with a browser platform adapter. @@ -41,13 +41,13 @@ Make `frontend.worker.db-worker` and its dependencies run in both browser and No 10. Build config changes. - Add a Node build target in `shadow-cljs.edn` for db-worker (e.g. `:db-worker-node`). - Ensure shared code compiles for `:node-script` or `:node-library` with the correct externs. - - Add `better-sqlite3` dependency and ensure Node target treats it as a native external. + - Use Node built-in `node:sqlite` and keep Node runtime on `>=22.20.0`. 11. Tests and fixtures. - Add unit tests for platform adapters and storage abstraction. - Add a minimal integration test that starts the Node daemon and exercises a small RPC call. -## Node.js sqlite Implementation (better-sqlite3) -Node runtime must not use OPFS or sqlite-wasm. Instead, use `better-sqlite3` as the direct file-backed sqlite engine. +## Node.js sqlite Implementation (node:sqlite) +Node runtime must not use OPFS or sqlite-wasm. Instead, use Node built-in `node:sqlite` as the direct file-backed sqlite engine. ### Concrete Refactor Items (File + Function + Summary) - `src/main/frontend/worker/db_core.cljs` (`init-sqlite-module!`, `]]` with the referenced block content. Nested references are resolved recursively up to 10 levels to avoid excessive expansion. For example: `[[]]` → `[[some text [[]]]]` and then `` is also replaced. diff --git a/package.json b/package.json index cfe5ce71de..033a117ce5 100644 --- a/package.json +++ b/package.json @@ -164,7 +164,6 @@ "@tabler/icons-react": "^2.47.0", "@tabler/icons-webfont": "^2.47.0", "@tippyjs/react": "4.2.5", - "better-sqlite3": "12.6.0", "bignumber.js": "^9.0.2", "chokidar": "3.5.1", "chrono-node": "2.2.4", diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 2a0b4f6e11..a26df43785 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -455,11 +455,13 @@ (or (string/includes? message ".node") (string/includes? message "Cannot find module") + (string/includes? message "MODULE_NOT_FOUND") (string/includes? message "bindings file")) (.error js/console - (str "db-worker-node failed to start: missing native bundle asset. " + (str "db-worker-node failed to start: bundled runtime files are missing or incomplete. " "Rebuild with `yarn db-worker-node:release:bundle` and ensure " - "`dist/db-worker-node-assets.json` assets are next to `db-worker-node.js`. " + "`dist/db-worker-node.js` exists and assets listed in " + "`dist/db-worker-node-assets.json` are next to it. " "Root error: " message)) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index 19c7b385b0..f37a2dfd48 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -1,6 +1,6 @@ (ns frontend.worker.platform.node "Node.js platform adapter for db-worker." - (:require ["better-sqlite3" :as sqlite3] + (:require ["node:sqlite" :as node-sqlite] ["fs/promises" :as fs] ["os" :as os] ["path" :as node-path] @@ -11,8 +11,19 @@ [lambdaisland.glogi :as log] [promesa.core :as p])) -(def ^:private sqlite - (or (aget sqlite3 "default") sqlite3)) +(defn- resolve-database-sync-ctor + [] + (or (gobj/get node-sqlite "DatabaseSync") + (some-> (gobj/get node-sqlite "default") + (gobj/get "DatabaseSync")) + (let [default-export (gobj/get node-sqlite "default")] + (when (fn? default-export) + default-export)) + (throw (ex-info "node:sqlite DatabaseSync constructor missing" + {:module-keys (js->clj (js/Object.keys node-sqlite))})))) + +(def ^:private DatabaseSync + (resolve-database-sync-ctor)) (defn- expand-home [path] @@ -70,6 +81,36 @@ (p/then (fn [_] true)) (p/catch (fn [_] false))))) +(defn- normalize-bind + [bind] + (cond + (array? bind) bind + (and bind (object? bind)) + (let [out (js-obj)] + (doseq [key (js/Object.keys bind)] + (let [value (gobj/get bind key) + normalized (cond + (string/starts-with? key "$") (subs key 1) + (string/starts-with? key ":") (subs key 1) + :else key)] + (gobj/set out normalized value))) + out) + :else bind)) + +(defn- stmt-all + [^js stmt bind] + (cond + (array? bind) (.apply (.-all stmt) stmt bind) + (some? bind) (.all stmt bind) + :else (.all stmt))) + +(defn- stmt-run + [^js stmt bind] + (cond + (array? bind) (.apply (.-run stmt) stmt bind) + (some? bind) (.run stmt bind) + :else (.run stmt))) + (defn- exec-sql [^js db opts-or-sql] (if (string? opts-or-sql) @@ -77,47 +118,75 @@ (let [sql (gobj/get opts-or-sql "sql") bind (gobj/get opts-or-sql "bind") row-mode (gobj/get opts-or-sql "rowMode") - bind' (cond - (array? bind) bind - (and bind (object? bind)) - (let [out (js-obj)] - (doseq [key (js/Object.keys bind)] - (let [value (gobj/get bind key) - normalized (cond - (string/starts-with? key "$") (subs key 1) - (string/starts-with? key ":") (subs key 1) - :else key)] - (gobj/set out normalized value))) - out) - :else bind) + bind' (normalize-bind bind) ^js stmt (.prepare db sql)] (if (= row-mode "array") (do - (.raw stmt) - (if (some? bind') - (.all stmt bind') - (.all stmt))) + (.setReturnArrays stmt true) + (stmt-all stmt bind')) (do - (if (some? bind') - (.run stmt bind') - (.run stmt)) + (stmt-run stmt bind') nil))))) -(defn- wrap-better-db +(defn- with-transaction + [^js db tx-depth savepoint-seq tx-body] + (let [outermost? (zero? @tx-depth) + savepoint (when-not outermost? + (str "__logseq_tx_" (swap! savepoint-seq inc)))] + (if outermost? + (.exec db "BEGIN") + (.exec db (str "SAVEPOINT " savepoint))) + (swap! tx-depth inc) + (try + (let [result (tx-body)] + (if outermost? + (.exec db "COMMIT") + (.exec db (str "RELEASE SAVEPOINT " savepoint))) + result) + (catch :default e + (if outermost? + (try + (.exec db "ROLLBACK") + (catch :default _)) + (do + (try + (.exec db (str "ROLLBACK TO SAVEPOINT " savepoint)) + (catch :default _)) + (try + (.exec db (str "RELEASE SAVEPOINT " savepoint)) + (catch :default _)))) + (throw e)) + (finally + (swap! tx-depth dec))))) + +(defn- wrap-node-sqlite-db [db] - (let [wrapper (js-obj)] + (let [wrapper (js-obj) + closed? (atom false) + tx-depth (atom 0) + savepoint-seq (atom 0)] (set! (.-exec wrapper) (fn [opts-or-sql] (exec-sql db opts-or-sql))) (set! (.-transaction wrapper) (fn [f] - (let [run-tx (.transaction db (fn [] (f wrapper)))] - (run-tx)))) - (set! (.-close wrapper) (fn [] (.close db))) + (with-transaction db tx-depth savepoint-seq + (fn [] + (f wrapper))))) + (set! (.-close wrapper) + (fn [] + (when-not @closed? + (reset! closed? true) + (try + (.close db) + (catch :default e + (when-not (string/includes? (str e) "database is not open") + (throw e))))) + nil)) wrapper)) (defn- open-sqlite-db [{:keys [path]}] (p/let [_ (ensure-dir! (node-path/dirname path))] - (wrap-better-db (new sqlite path)))) + (wrap-node-sqlite-db (new DatabaseSync path)))) (defn- install-opfs-pool [data-dir _sqlite pool-name] diff --git a/src/test/frontend/worker/platform_node_test.cljs b/src/test/frontend/worker/platform_node_test.cljs new file mode 100644 index 0000000000..faedfea3e0 --- /dev/null +++ b/src/test/frontend/worker/platform_node_test.cljs @@ -0,0 +1,184 @@ +(ns frontend.worker.platform-node-test + (:require ["fs" :as fs] + ["path" :as node-path] + [cljs.test :refer [async deftest is testing]] + [clojure.string :as string] + [frontend.test.node-helper :as node-helper] + [frontend.worker.platform.node :as platform-node] + [goog.object :as gobj] + [promesa.core :as p])) + +(defn- node-platform-source + [] + (let [source-path (node-path/join (.cwd js/process) + "src" + "main" + "frontend" + "worker" + "platform" + "node.cljs")] + (.toString (fs/readFileSync source-path) "utf8"))) + +(defn- js-bind + [pairs] + (let [bind (js-obj)] + (doseq [[k v] pairs] + (gobj/set bind k v)) + bind)) + +(defn- (p/let [{:keys [sqlite db] :as conn} (clj rows))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) + +(deftest exec-row-mode-array-returns-index-addressable-rows + (async done + (let [conn* (atom nil)] + (-> (p/let [{:keys [sqlite db] :as conn} (clj rows))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) + +(deftest exec-accepts-dollar-and-colon-bind-key-styles + (async done + (let [conn* (atom nil)] + (-> (p/let [{:keys [sqlite db] :as conn} (clj rows))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) + +(deftest transaction-commits-on-success + (async done + (let [conn* (atom nil)] + (-> (p/let [{:keys [sqlite db] :as conn} (clj rows))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) + +(deftest transaction-rolls-back-when-callback-throws + (async done + (let [conn* (atom nil)] + (-> (p/let [{:keys [sqlite db] :as conn} (clj rows))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) + +(deftest nested-transactions-keep-outer-writes-after-inner-rollback + (async done + (let [conn* (atom nil)] + (-> (p/let [{:keys [sqlite db] :as conn} (clj rows)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (close-db! @conn*) + (done))))))) diff --git a/src/test/logseq/db_worker/ncc_bundle_test.cljs b/src/test/logseq/db_worker/ncc_bundle_test.cljs index 88afa676f8..2b374beee5 100644 --- a/src/test/logseq/db_worker/ncc_bundle_test.cljs +++ b/src/test/logseq/db_worker/ncc_bundle_test.cljs @@ -3,6 +3,7 @@ [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker.db-worker-node-lock :as db-lock] + [logseq.db :as ldb] [logseq.db-worker.daemon :as daemon] [promesa.core :as p] ["child_process" :as child-process] @@ -43,6 +44,12 @@ (js->clj (js/JSON.parse (.toString (fs/readFileSync manifest-path) "utf8")) :keywordize-keys true))) +(defn- write-asset-manifest! + [manifest] + (let [manifest-path (dist-path "db-worker-node-assets.json") + payload (str (js/JSON.stringify (clj->js manifest) nil 2) "\n")] + (fs/writeFileSync manifest-path payload "utf8"))) + (defn- copy-file! [source destination] (fs/mkdirSync (node-path/dirname destination) #js {:recursive true}) @@ -89,6 +96,20 @@ :stdout stdout :stderr stderr})) +(defn- invoke + [host port method args] + (let [payload (js/JSON.stringify + (clj->js {:method method + :directPass false + :argsTransit (ldb/write-transit-str args)}))] + (daemon/http-request {:method "POST" + :host host + :port port + :path "/v1/invoke" + :headers {"Content-Type" "application/json"} + :timeout-ms 5000 + :body payload}))) + (deftest bundle-only-daemon-startup-smoke-test (async done (let [child* (atom nil)] @@ -135,29 +156,99 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest bundle-missing-native-asset-has-actionable-error +(deftest bundle-daemon-starts-with-empty-asset-manifest + (async done + (let [child* (atom nil) + original-manifest* (atom nil)] + (-> (p/let [_ (ensure-bundle-built!) + original-manifest (read-asset-manifest) + _ (reset! original-manifest* original-manifest) + _ (write-asset-manifest! (assoc original-manifest :assets [])) + {:keys [runtime-dir]} (copy-bundle-to-temp!) + data-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-empty-assets")) + repo (str "logseq_db_ncc_empty_assets_" (subs (str (random-uuid)) 0 8)) + lock-file (lock-path data-dir repo) + {:keys [child]} (spawn-daemon! runtime-dir repo data-dir) + _ (reset! child* child) + _ (daemon/wait-for-lock lock-file) + lock (daemon/read-lock lock-file) + _ (is (some? lock)) + health (daemon/http-request {:method "GET" + :host (:host lock) + :port (:port lock) + :path "/healthz" + :timeout-ms 1000}) + ready (daemon/http-request {:method "GET" + :host (:host lock) + :port (:port lock) + :path "/readyz" + :timeout-ms 1000}) + create-db (invoke (:host lock) + (:port lock) + "thread-api/create-or-open-db" + [repo {}]) + create-db-body (js->clj (js/JSON.parse (:body create-db)) + :keywordize-keys true) + shutdown (daemon/http-request {:method "POST" + :host (:host lock) + :port (:port lock) + :path "/v1/shutdown" + :headers {"Content-Type" "application/json"} + :timeout-ms 2000}) + _ (is (= 200 (:status health))) + _ (is (= 200 (:status ready))) + _ (is (= 200 (:status create-db))) + _ (is (:ok create-db-body)) + _ (is (= 200 (:status shutdown))) + _ (daemon/wait-for (fn [] + (p/resolved (not (fs/existsSync lock-file)))) + {:timeout-ms 10000 + :interval-ms 200})] + (is (not (fs/existsSync lock-file)))) + (p/catch (fn [e] + (when-let [^js child @child*] + (try + (.kill child "SIGTERM") + (catch :default _))) + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (when-let [manifest @original-manifest*] + (write-asset-manifest! manifest)) + (done))))))) + +(deftest bundle-errors-are-actionable-when-manifest-or-entry-is-missing (let [_ (ensure-bundle-built!) - {:keys [runtime-dir assets]} (copy-bundle-to-temp!) - native-asset (first (filter #(string/ends-with? % ".node") assets))] - (is (some? native-asset)) - (when native-asset - (let [missing-path (node-path/join runtime-dir native-asset) - _ (fs/unlinkSync missing-path) - data-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-missing-asset")) - repo (str "logseq_db_ncc_missing_" (subs (str (random-uuid)) 0 8)) - result (.spawnSync child-process - "node" - #js ["./db-worker-node.js" - "--repo" repo - "--data-dir" data-dir - "--owner-source" "cli"] - #js {:cwd runtime-dir - :encoding "utf8"}) - status (.-status result) - output (str (or (.-stderr result) "") - (or (.-stdout result) "")) - missing-file (node-path/basename missing-path)] - (is (not= 0 status)) - (is (or (string/includes? output missing-file) - (string/includes? output "Cannot find module") - (string/includes? output "could not locate the bindings file"))))))) + manifest-path (dist-path "db-worker-node-assets.json") + manifest-backup-path (dist-path "db-worker-node-assets.json.bak") + _ (when (fs/existsSync manifest-backup-path) + (fs/unlinkSync manifest-backup-path)) + _ (fs/renameSync manifest-path manifest-backup-path) + missing-manifest-error (try + (read-asset-manifest) + nil + (catch :default e + e)) + _ (fs/renameSync manifest-backup-path manifest-path) + {:keys [runtime-dir]} (copy-bundle-to-temp!) + missing-entry-path (node-path/join runtime-dir "db-worker-node.js") + _ (fs/unlinkSync missing-entry-path) + data-dir (absolute-path (node-helper/create-tmp-dir "db-worker-node-bundle-missing-entry")) + repo (str "logseq_db_ncc_missing_entry_" (subs (str (random-uuid)) 0 8)) + result (.spawnSync child-process + "node" + #js ["./db-worker-node.js" + "--repo" repo + "--data-dir" data-dir + "--owner-source" "cli"] + #js {:cwd runtime-dir + :encoding "utf8"}) + status (.-status result) + output (str (or (.-stderr result) "") + (or (.-stdout result) ""))] + (is (some? missing-manifest-error)) + (is (string/includes? (str missing-manifest-error) "db-worker-node-assets.json")) + (is (string/includes? (str missing-manifest-error) "ENOENT")) + (is (not= 0 status)) + (is (string/includes? output "db-worker-node.js")) + (is (or (string/includes? output "Cannot find module") + (string/includes? output "MODULE_NOT_FOUND"))))) diff --git a/yarn.lock b/yarn.lock index 45f41e8e5d..c9d4454f2b 100644 --- a/yarn.lock +++ b/yarn.lock @@ -2216,14 +2216,6 @@ base@^0.11.1: mixin-deep "^1.2.0" pascalcase "^0.1.1" -better-sqlite3@^12.6.2: - version "12.6.2" - resolved "https://registry.yarnpkg.com/better-sqlite3/-/better-sqlite3-12.6.2.tgz#770649f28a62e543a360f3dfa1afe4cc944b1937" - integrity sha512-8VYKM3MjCa9WcaSAI3hzwhmyHVlH8tiGFwf0RlTsZPWJ1I5MkzjiudCo4KC4DxOaL/53A5B1sI/IbldNFDbsKA== - dependencies: - bindings "^1.5.0" - prebuild-install "^7.1.1" - big-integer@1.6.x: version "1.6.52" resolved "https://registry.yarnpkg.com/big-integer/-/big-integer-1.6.52.tgz#60a887f3047614a8e1bffe5d7173490a97dc8c85" From 67b1f256938346040b6de4eea8778fef11f5e84a Mon Sep 17 00:00:00 2001 From: Tienson Qin Date: Tue, 24 Feb 2026 17:24:17 +0800 Subject: [PATCH 082/375] fix: record log on electron --- src/electron/electron/logger.cljs | 15 ++++++++++++--- src/main/logseq/db_worker/daemon.cljs | 4 ++-- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/src/electron/electron/logger.cljs b/src/electron/electron/logger.cljs index e94fe76610..35806df4cb 100644 --- a/src/electron/electron/logger.cljs +++ b/src/electron/electron/logger.cljs @@ -1,7 +1,7 @@ (ns electron.logger "Electron logger, do not depends other libs" - (:require ["electron-log" :as logger])) - + (:require ["electron-log" :as logger] + [lambdaisland.glogi :as log])) (defn- transform-args [args] (map #(cond @@ -12,7 +12,6 @@ %) args)) - (defn debug [& args] (apply (.-debug logger) (transform-args args))) @@ -29,3 +28,13 @@ [& args] (apply (.-error logger) (transform-args args))) +(log/add-handler (fn [{:keys [level message exception]}] + (let [f (case level + :warn + warn + :error + error + :debug + debug + info)] + (f message exception)))) diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs index 985d919a94..ddacb1cb49 100644 --- a/src/main/logseq/db_worker/daemon.cljs +++ b/src/main/logseq/db_worker/daemon.cljs @@ -263,7 +263,7 @@ (log/warn :db-worker-daemon/missing-script {:repo repo :data-dir data-dir}) nil) (let [child (.spawn child-process (.-execPath js/process) args #js {:detached true - :stdio "ignore" - :env env})] + :stdio "inherit" + :env env})] (.unref child) child)))) From ce103fea48ead81cfd1fd8769695ba4a5f1bbfb9 Mon Sep 17 00:00:00 2001 From: Tienson Qin Date: Tue, 24 Feb 2026 22:25:09 +0800 Subject: [PATCH 083/375] fix: add missing closure --- shadow-cljs.edn | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 2d5dc7210a..33645efdfc 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -192,7 +192,8 @@ :externs ["datascript/externs.js" "externs.js"] :warnings {:fn-deprecated false - :redef false}}} + :redef false}} + :closure-defines {goog.debug.LOGGING_ENABLED true}} :test {:target :node-test :output-to "static/tests.js" From 4ac10a144c7290230eef3e0bbddbab66ec8430cd Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 25 Feb 2026 16:10:51 +0800 Subject: [PATCH 084/375] 038-electron-db-worker-switch-graph.md --- .../038-electron-db-worker-switch-graph.md | 187 ++++++++++++++++++ src/electron/electron/db_worker.cljs | 2 +- src/electron/electron/graph_switch_flow.cljs | 13 ++ src/electron/electron/handler.cljs | 8 +- src/test/electron/db_worker_manager_test.cljs | 44 +++++ src/test/electron/graph_switch_flow_test.cljs | 15 ++ src/test/frontend/persist_db_test.cljs | 47 +++++ 7 files changed, 313 insertions(+), 3 deletions(-) create mode 100644 docs/agent-guide/038-electron-db-worker-switch-graph.md create mode 100644 src/electron/electron/graph_switch_flow.cljs create mode 100644 src/test/electron/graph_switch_flow_test.cljs diff --git a/docs/agent-guide/038-electron-db-worker-switch-graph.md b/docs/agent-guide/038-electron-db-worker-switch-graph.md new file mode 100644 index 0000000000..eed3e7acb3 --- /dev/null +++ b/docs/agent-guide/038-electron-db-worker-switch-graph.md @@ -0,0 +1,187 @@ +# Electron Db Worker Switch Graph Implementation Plan + +Goal: Fix Electron graph switching with db-worker-node so switching to a new graph never leaves the renderer bound to a stopped runtime. + +Architecture: Keep db-worker runtime lifecycle in the `db-worker-runtime` path and window close path, and treat `setCurrentGraph` as window graph metadata synchronization only. + +Architecture: Remove or guard the duplicate release path in `setCurrentGraph` that can stop the newly started runtime after `persist-db/ persist-db/ ipc "db-worker-runtime" B. + -> main db-worker manager switches A -> B. + -> state/set-current-repo! B. + -> ipc "setCurrentGraph" B. + -> handler calls release-window! again. + -> B runtime may be stopped unexpectedly. + +Target sequence. +Renderer restore graph B. + -> persist-db/ ipc "db-worker-runtime" B. + -> main db-worker manager switches A -> B. + -> state/set-current-repo! B. + -> ipc "setCurrentGraph" B. + -> handler only updates window graph path. + -> B runtime stays available. +``` + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before each behavior change. + +I will add a failing regression test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that encodes the expected post-switch invariant that the new runtime remains active until explicit release on window close. + +I will add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db_test.cljs` for the graph switch flow to assert runtime rebinding happens once per target repo and is not reinitialized by graph metadata sync. + +I will add a focused unit test file `/Users/rcmerci/gh-repos/logseq/src/test/electron/graph_switch_flow_test.cljs` for extracted pure graph-switch decision logic so the release/no-release condition is testable without Electron GUI dependencies. + +I will run focused tests after each phase and finish with `bb dev:lint-and-test`. + +I will perform manual Electron smoke checks that open graph A, switch to graph B, execute thread-api reads and writes, and then switch back to graph A. + +I will review changes against `@prompts/review.md` before merge. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests that reproduce the switch-order regression. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that simulates A -> B switch and asserts no stop is triggered for B before explicit release. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db_test.cljs` for the sequence ` B -> A and invoking read and write actions after each switch. +26. Validate no immediate invoke failures after switch and confirm runtime stays available for the active graph. +27. Remove temporary logs that are not needed for long-term maintainability. + +### Phase 7: Final verification and review gate. + +28. Run `bb dev:test -v 'electron.graph-switch-flow-test'`. +29. Run `bb dev:test -v 'electron.db-worker-manager-test'`. +30. Run `bb dev:test -v 'frontend.persist-db-test'`. +31. Run `bb dev:lint-and-test`. +32. Confirm each command reports `0 failures, 0 errors`. +33. Run final review checklist pass against `@prompts/review.md`. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Switch A -> B in one window where both graphs are local db graphs. | Runtime for B remains active and calls succeed immediately after switch. | +| Switch A -> B -> A quickly before all async handlers settle. | Late `setCurrentGraph` sync does not stop the currently active runtime. | +| Two windows share graph A and one window switches to graph B. | Graph A runtime remains alive for the other window and graph B runtime starts for the switching window. | +| Re-select current graph B from UI without actual graph change. | No runtime restart and no runtime release occurs. | +| Window closes right after switch to B. | Runtime release happens exactly once via close flow and does not leave stale window mapping. | +| Runtime ownership is external (`:owned? false`). | Graph switch does not attempt to stop external runtime unexpectedly. | +| Restore fails after runtime bind but before UI route redirect. | Failure handling does not silently stop the newly bound runtime unless explicit cleanup path runs. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'electron.graph-switch-flow-test' +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:test -v 'frontend.persist-db-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +Manual Electron switch checks should show successful read and write operations immediately after each switch. + +No `db-worker invoke failed` errors should appear during normal A -> B -> A switching. + +## Testing Details + +The new tests verify behavior around event ordering and runtime lifecycle boundaries instead of implementation details. + +The manager tests validate that stale or duplicate release operations cannot terminate the active repo runtime. + +The frontend tests validate that runtime rebinding is driven by repo changes and not by metadata synchronization calls. + +Manual smoke checks validate real Electron runtime behavior that unit tests cannot fully represent. + +## Implementation Details + +- Keep `setCurrentGraph` focused on graph-path synchronization and remove lifecycle side effects from that path. +- Keep runtime start and stop orchestration centralized in `electron.db-worker` manager APIs. +- Use a small pure helper for release decision logic so regression tests do not depend on Electron runtime modules. +- Preserve existing `db-worker-runtime` IPC contract from renderer to main process. +- Keep old graph cleanup tied to explicit switch lifecycle and window close lifecycle only. +- Validate switch behavior with both single-window and multi-window tests. +- Use `@test-driven-development` for red-green implementation order. +- Follow `@prompts/review.md` checks before merging. + +## Question + +Decision: On failed graph restore, keep the newly bound runtime alive for fast retry in the same window. + +Decision: Do not add a short-lived debug flag for switch sequencing logs in development builds. + +Decision: Add an E2E scenario in `/Users/rcmerci/gh-repos/logseq/clj-e2e` for switch graph with db-worker-node and treat it as a release gate. + +--- diff --git a/src/electron/electron/db_worker.cljs b/src/electron/electron/db_worker.cljs index 67ff0087f5..3b0709bb19 100644 --- a/src/electron/electron/db_worker.cljs +++ b/src/electron/electron/db_worker.cljs @@ -42,7 +42,7 @@ [state nil] (let [remaining (disj (:windows entry) window-id) state' (cond-> state - true + (= repo (get-in state [:window->repo window-id])) (update :window->repo dissoc window-id) (seq remaining) diff --git a/src/electron/electron/graph_switch_flow.cljs b/src/electron/electron/graph_switch_flow.cljs new file mode 100644 index 0000000000..a91b405302 --- /dev/null +++ b/src/electron/electron/graph_switch_flow.cljs @@ -0,0 +1,13 @@ +(ns electron.graph-switch-flow) + +(defn release-runtime-on-set-current-graph? + "Decides whether `setCurrentGraph` should release db-worker runtime. + + Returns `false` by design. + + In Electron, `setCurrentGraph` is metadata synchronization for window graph + path only. Runtime lifecycle is handled by the `db-worker-runtime` IPC path + (switch/start) and window close path (release). Releasing here reintroduces + the stale double-release bug that can stop the newly bound runtime." + [_switch] + false) diff --git a/src/electron/electron/handler.cljs b/src/electron/electron/handler.cljs index 31daa5f093..9379140af4 100644 --- a/src/electron/electron/handler.cljs +++ b/src/electron/electron/handler.cljs @@ -27,6 +27,7 @@ [electron.state :as state] [electron.utils :as utils] [electron.window :as win] + [electron.graph-switch-flow :as graph-switch-flow] [logseq.cli.common.graph :as cli-common-graph] [logseq.common.graph :as common-graph] [logseq.db.sqlite.util :as sqlite-util] @@ -326,8 +327,11 @@ (defmethod handle :setCurrentGraph [^js window [_ graph-name]] (let [next-graph-path (when graph-name (utils/get-graph-dir graph-name)) - current-graph-path (state/get-window-graph-path window)] - (p/let [_ (when (not= current-graph-path next-graph-path) + current-graph-path (state/get-window-graph-path window) + release-runtime? (graph-switch-flow/release-runtime-on-set-current-graph? + {:previous-graph-path current-graph-path + :next-graph-path next-graph-path})] + (p/let [_ (when release-runtime? (db-worker/release-window! (.-id window)))] (if next-graph-path (set-current-graph! window next-graph-path) diff --git a/src/test/electron/db_worker_manager_test.cljs b/src/test/electron/db_worker_manager_test.cljs index 77c8650a37..34048e9daa 100644 --- a/src/test/electron/db_worker_manager_test.cljs +++ b/src/test/electron/db_worker_manager_test.cljs @@ -77,6 +77,50 @@ (is false (str "unexpected error: " e)))) (p/finally (fn [] (done))))))) +(deftest ensure-stopped-stale-repo-does-not-clear-new-window-mapping + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-a" :window-2) + _ (db-worker/ensure-started! manager "graph-b" :window-1) + ;; simulate late/stale release for previous repo after window-1 already moved to graph-b + _ (db-worker/ensure-stopped! manager "graph-a" :window-1) + manager-state @(:state manager)] + (is (= "graph-b" (get-in manager-state [:window->repo :window-1]))) + (is (= #{:window-2} (get-in manager-state [:repos "graph-a" :windows]))) + (is (= #{:window-1} (get-in manager-state [:repos "graph-b" :windows]))) + (is (empty? @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest ensure-stopped-stale-intermediate-repo-after-switch-back-keeps-current-repo + (async done + (let [stop-calls (atom []) + manager (db-worker/create-manager + {:start-daemon! (fn [repo] (p/resolved (runtime repo))) + :stop-daemon! (fn [rt] + (swap! stop-calls conj (:repo rt)) + (p/resolved true))})] + (-> (p/let [_ (db-worker/ensure-started! manager "graph-a" :window-1) + _ (db-worker/ensure-started! manager "graph-b" :window-1) + _ (db-worker/ensure-started! manager "graph-a" :window-1) + ;; stale cleanup for graph-b arrives after the window is already back on graph-a + _ (db-worker/ensure-stopped! manager "graph-b" :window-1) + manager-state @(:state manager)] + (is (= "graph-a" (get-in manager-state [:window->repo :window-1]))) + (is (= #{:window-1} (get-in manager-state [:repos "graph-a" :windows]))) + (is (nil? (get-in manager-state [:repos "graph-b"]))) + (is (= ["graph-a" "graph-b"] @stop-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + (deftest ensure-window-stopped-releases-active-runtime-by-window (async done (let [stop-calls (atom []) diff --git a/src/test/electron/graph_switch_flow_test.cljs b/src/test/electron/graph_switch_flow_test.cljs new file mode 100644 index 0000000000..eebeb437ac --- /dev/null +++ b/src/test/electron/graph_switch_flow_test.cljs @@ -0,0 +1,15 @@ +(ns electron.graph-switch-flow-test + (:require [cljs.test :refer [deftest is]] + [electron.graph-switch-flow :as graph-switch-flow])) + +(deftest set-current-graph-switch-does-not-release-runtime + (is (false? + (graph-switch-flow/release-runtime-on-set-current-graph? + {:previous-graph-path "graph-a" + :next-graph-path "graph-b"})))) + +(deftest set-current-graph-reselect-does-not-release-runtime + (is (false? + (graph-switch-flow/release-runtime-on-set-current-graph? + {:previous-graph-path "graph-a" + :next-graph-path "graph-a"})))) diff --git a/src/test/frontend/persist_db_test.cljs b/src/test/frontend/persist_db_test.cljs index 94538a48c3..37a74dd78d 100644 --- a/src/test/frontend/persist_db_test.cljs +++ b/src/test/frontend/persist_db_test.cljs @@ -5,6 +5,7 @@ [frontend.persist-db :as persist-db] [frontend.persist-db.protocol :as protocol] [frontend.persist-db.remote :as remote] + [frontend.storage :as storage] [frontend.state :as state] [frontend.util :as util] [promesa.core :as p])) @@ -112,6 +113,52 @@ (set! remote/stop! original-stop!) (done))))))) +(deftest electron-fetch-init-data-then-set-current-repo-does-not-rebind-runtime + (async done + (let [ipc-calls (atom []) + start-calls (atom []) + stop-calls (atom []) + wrapped-worker (fn [& _] nil) + original-state @state/state + original-electron? util/electron? + original-ipc ipc/ipc + original-start! remote/start! + original-stop! remote/stop! + original-storage-set storage/set + original-storage-remove storage/remove] + (reset-runtime-state!) + (set! util/electron? (constantly true)) + (set! ipc/ipc (fn [channel repo] + (swap! ipc-calls conj [channel repo]) + (p/resolved nil))) + (set! remote/start! (fn [{:keys [repo]}] + (swap! start-calls conj repo) + (->FakeRemote repo wrapped-worker))) + (set! remote/stop! (fn [client] + (swap! stop-calls conj (:repo client)) + (p/resolved true))) + (set! storage/set (fn [& _] nil)) + (set! storage/remove (fn [& _] nil)) + (-> (p/let [repo "logseq_db_graph_a" + _ (persist-db/ Date: Wed, 25 Feb 2026 23:20:17 +0800 Subject: [PATCH 085/375] rebase master --- src/main/frontend/worker/db_core.cljs | 218 ++++++++++++------ src/main/frontend/worker/db_worker_node.cljs | 18 +- src/test/frontend/worker/db_core_test.cljs | 16 ++ .../frontend/worker/db_worker_node_test.cljs | 11 +- yarn.lock | 13 +- 5 files changed, 192 insertions(+), 84 deletions(-) create mode 100644 src/test/frontend/worker/db_core_test.cljs diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index aeac170036..194200d867 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -2,7 +2,6 @@ "Core db-worker logic without host-specific bootstrap." (:require [cljs-bean.core :as bean] [cljs.cache :as cache] - [clojure.edn :as edn] [clojure.set] [clojure.string :as string] [datascript.core :as d] @@ -11,28 +10,25 @@ [frontend.common.graph-view :as graph-view] [frontend.common.missionary :as c.m] [frontend.common.thread-api :as thread-api :refer [def-thread-api]] - [frontend.worker.graph-dir :as graph-dir] - [frontend.worker.platform :as platform] [frontend.worker-common.util :as worker-util] [frontend.worker.db-listener :as db-listener] - [frontend.worker.db-metadata :as worker-db-metadata] [frontend.worker.db.fix :as db-fix] [frontend.worker.db.migrate :as db-migrate] [frontend.worker.db.validate :as worker-db-validate] [frontend.worker.embedding :as embedding] [frontend.worker.export :as worker-export] + [frontend.worker.graph-dir :as graph-dir] [frontend.worker.handler.page :as worker-page] [frontend.worker.pipeline :as worker-pipeline] + [frontend.worker.platform :as platform] [frontend.worker.publish] - [frontend.worker.rtc.asset-db-listener] - [frontend.worker.rtc.client-op :as client-op] - [frontend.worker.rtc.core :as rtc.core] - [frontend.worker.rtc.db-listener] - [frontend.worker.rtc.debug-log :as rtc-debug-log] - [frontend.worker.rtc.migrate :as rtc-migrate] [frontend.worker.search :as search] [frontend.worker.shared-service :as shared-service] [frontend.worker.state :as worker-state] + [frontend.worker.sync :as db-sync] + [frontend.worker.sync.client-op :as client-op] + [frontend.worker.sync.crypt :as sync-crypt] + [frontend.worker.sync.log-and-state :as sync-log-and-state] [frontend.worker.thread-atom] [lambdaisland.glogi :as log] [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] @@ -45,7 +41,9 @@ [logseq.db.common.sqlite :as common-sqlite] [logseq.db.common.view :as db-view] [logseq.db.frontend.class :as db-class] + [logseq.db.frontend.entity-util :as entity-util] [logseq.db.frontend.property :as db-property] + [logseq.db.frontend.schema :as db-schema] [logseq.db.sqlite.create-graph :as sqlite-create-graph] [logseq.db.sqlite.export :as sqlite-export] [logseq.db.sqlite.gc :as sqlite-gc] @@ -54,9 +52,7 @@ [logseq.outliner.op :as outliner-op] [me.tonsky.persistent-sorted-set :as set :refer [BTSet]] [missionary.core :as m] - [promesa.core :as p] - [logseq.db.frontend.schema :as db-schema] - [logseq.db.frontend.entity-util :as entity-util])) + [promesa.core :as p])) (defonce *sqlite worker-state/*sqlite) (defonce *sqlite-conns worker-state/*sqlite-conns) @@ -113,7 +109,6 @@ nil))) (def repo-path "/db.sqlite") -(def debug-log-path "/debug-log/db.sqlite") (defn- resolve-db-path [repo pool path] @@ -126,10 +121,10 @@ ([repo] (sqlite-binds + [rows] + (mapv (fn [[addr content addresses]] + #js {:$addr addr + :$content content + :$addresses addresses}) + rows)) + +(defn- ensure-db-sync-import-db! + [repo reset?] + (if-let [sqlite @*sqlite] + (p/let [db (platform/sqlite-open (platform/current) + {:sqlite sqlite + :path ":memory:" + :mode "c"})] + (common-sqlite/create-kvs-table! db) + (when reset? + (.exec db "delete from kvs")) + db) + (db-sync/fail-fast :db-sync/missing-field {:repo repo + :field :sqlite}))) + (defn restore-data-from-addr "Update sqlite-cli/restore-data-from-addr when making changes" [db addr] @@ -238,7 +255,6 @@ db-path (resolve-db-path repo pool repo-path) search-path (resolve-db-path repo pool (str "search" repo-path)) client-ops-path (resolve-db-path repo pool (str "client-ops-" repo-path)) - debug-log-db-path (resolve-db-path repo pool (str "debug-log" repo-path)) _ (log/info :db-worker/get-dbs-open {:repo repo :db-path db-path}) db (platform/sqlite-open (platform/current) {:sqlite @*sqlite @@ -253,13 +269,8 @@ client-ops-db (platform/sqlite-open (platform/current) {:sqlite @*sqlite :pool pool - :path client-ops-path}) - _ (log/info :db-worker/get-dbs-open {:repo repo :debug-log-db-path debug-log-db-path}) - debug-log-db (platform/sqlite-open (platform/current) - {:sqlite @*sqlite - :pool pool - :path debug-log-db-path})] - [db search-db client-ops-db debug-log-db]))) + :path client-ops-path})] + [db search-db client-ops-db]))) (defn- enable-sqlite-wal-mode! [^Object db] @@ -268,7 +279,7 @@ (defn- gc-sqlite-dbs! "Gc main db weekly and rtc ops db each time when opening it" - [sqlite-db client-ops-db debug-log-db datascript-conn {:keys [full-gc?]}] + [sqlite-db client-ops-db datascript-conn {:keys [full-gc?]}] (let [last-gc-at (:kv/value (d/entity @datascript-conn :logseq.kv/graph-last-gc-at))] (when (or full-gc? (nil? last-gc-at) @@ -278,7 +289,6 @@ (doseq [db (if @*publishing? [sqlite-db] [sqlite-db client-ops-db])] (sqlite-gc/gc-kvs-table! db {:full-gc? full-gc?}) (.exec db "VACUUM")) - (rtc-debug-log/gc! debug-log-db) (ldb/transact! datascript-conn [{:db/ident :logseq.kv/graph-last-gc-at :kv/value (common-util/time-ms)}])))) @@ -288,20 +298,18 @@ (log/info :db-worker/create-or-open-start {:repo repo :has-datoms? (boolean datoms) :import-type (:import-type opts)}) - (p/let [[db search-db client-ops-db debug-log-db :as dbs] (get-dbs repo) + (p/let [[db search-db client-ops-db :as dbs] (get-dbs repo) storage (new-sqlite-storage db) client-ops-storage (when-not @*publishing? (new-sqlite-storage client-ops-db)) db-based? true] (swap! *sqlite-conns assoc repo {:db db :search search-db - :client-ops client-ops-db - :debug-log debug-log-db}) + :client-ops client-ops-db}) (doseq [db' dbs] (enable-sqlite-wal-mode! db')) (common-sqlite/create-kvs-table! db) (when-not @*publishing? (common-sqlite/create-kvs-table! client-ops-db)) - (rtc-debug-log/create-tables! debug-log-db) (search/create-tables-and-triggers! search-db) (ldb/register-transact-pipeline-fn! worker-pipeline/transact-pipeline) (let [conn (common-sqlite/get-storage-conn storage db-schema/schema) @@ -337,12 +345,9 @@ config (select-keys opts [:import-type :graph-git-sha]))] (ldb/transact! conn initial-data {:initial-db? true}))) - (gc-sqlite-dbs! db client-ops-db debug-log-db conn {}) + (gc-sqlite-dbs! db client-ops-db conn {}) - (let [migration-result (db-migrate/migrate conn)] - (when (client-op/rtc-db-graph? repo) - (let [client-ops (rtc-migrate/migration-results=>client-ops migration-result)] - (client-op/add-ops! repo client-ops)))) + (db-migrate/migrate conn) (db-listener/listen-db-changes! repo (get @*datascript-conns repo)))))) @@ -351,10 +356,8 @@ (p/let [storage (platform/storage (platform/current)) graph-names ((:list-graphs storage))] (p/all (map (fn [graph-name] - (p/let [repo (str sqlite-util/db-version-prefix graph-name) - metadata (worker-db-metadata/ + (p/do! + (sync-log-and-state/rtc-log :rtc.log/download + {:sub-type :download-progress + :graph-uuid graph-id + :message "Saving data to DB"}) + ((@thread-api/*thread-apis :thread-api/create-or-open-db) + repo + {:close-other-db? true + :datoms datoms}) + (db-sync/rehydrate-large-titles-from-db! repo graph-id) + (sync-log-and-state/rtc-log :rtc.log/download + {:sub-type :download-completed + :graph-uuid graph-id + :message "Graph is ready!"}) + ((@thread-api/*thread-apis :thread-api/export-db) repo) + (client-op/update-local-tx repo remote-tx) + (shared-service/broadcast-to-clients! :add-repo {:repo repo})) + (p/catch (fn [error] + (js/console.error error))))) + +(def-thread-api :thread-api/db-sync-import-kvs-rows + [repo rows reset? graph-id remote-tx graph-e2ee?] + (let [graph-e2ee? (if (nil? graph-e2ee?) true (true? graph-e2ee?))] + (p/let [_ (when reset? + (close-db! repo)) + aes-key (when graph-e2ee? + (sync-crypt/sqlite-binds rows-batch)) + (sync-log-and-state/rtc-log :rtc.log/download + {:sub-type :download-progress + :graph-uuid graph-id + :message (str (if graph-e2ee? + "Decrypting data" + "Importing data") + " " + (inc index) + "/" + total-batches)}))) + (let [storage (new-sqlite-storage db) + conn (common-sqlite/get-storage-conn storage db-schema/schema) + datoms (vec (d/datoms @conn :eavt))] + (.close db) + (import-datoms-to-db! repo graph-id remote-tx datoms))))) + (def-thread-api :thread-api/release-access-handles [repo] (when-let [^js pool (get-storage-pool repo)] @@ -594,23 +692,6 @@ (js/Buffer.from data))] (.toString buffer "base64"))))) -(def-thread-api :thread-api/export-debug-log-db - [repo] - (when-let [^js db (worker-state/get-sqlite-conn repo :debug-log)] - (.exec db "PRAGMA wal_checkpoint(2)")) - (-> (p/let [data ( (default ~/logseq/graphs)")) (println (str " " (style/bold "--repo") " (required)")) - (println (str " " (style/bold "--rtc-ws-url") " (optional)")) (println (str " " (style/bold "--log-level") " (default info)")) (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)")) @@ -345,7 +348,7 @@ file-path)) (defn start-daemon! - [{:keys [data-dir repo rtc-ws-url log-level owner-source]}] + [{:keys [data-dir repo log-level owner-source]}] (let [host "127.0.0.1" port 0 owner-source (normalize-owner-source owner-source)] @@ -366,7 +369,7 @@ :event-fn handle-event! :write-guard-fn write-guard-fn}) proxy (db-core/init-core! platform) - _ ( Date: Thu, 5 Mar 2026 08:43:55 -0500 Subject: [PATCH 086/375] 039-worker-platform-abstraction-cleanup.md, rebase master fix (2) --- ...039-worker-platform-abstraction-cleanup.md | 213 ++++++++++++++++++ package.json | 2 +- src/main/frontend/worker/sync.cljs | 4 +- src/main/frontend/worker/sync/crypt.cljs | 16 +- src/test/frontend/worker/db_sync_test.cljs | 27 +++ src/test/frontend/worker/sync/crypt_test.cljs | 154 +++++++++++++ 6 files changed, 406 insertions(+), 10 deletions(-) create mode 100644 docs/agent-guide/039-worker-platform-abstraction-cleanup.md diff --git a/docs/agent-guide/039-worker-platform-abstraction-cleanup.md b/docs/agent-guide/039-worker-platform-abstraction-cleanup.md new file mode 100644 index 0000000000..2bc4e3762c --- /dev/null +++ b/docs/agent-guide/039-worker-platform-abstraction-cleanup.md @@ -0,0 +1,213 @@ +# Worker Platform Abstraction Cleanup Implementation Plan + +Goal: Route shared db-worker sync code through `frontend.worker.platform` wrappers so browser and node runtimes both work without runtime-specific branches in shared modules. + +Architecture: Keep runtime-specific APIs inside `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/browser.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. +Call platform capabilities from shared modules via `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` using `platform/current` at the call site. +Preserve existing key names and payload shapes to avoid data migration. + +Tech Stack: ClojureScript, promesa, cljs.test, clojure-lsp diagnostics, db-worker platform adapters. + +Related: Relates to `docs/agent-guide/038-electron-db-worker-switch-graph.md` and `docs/agent-guide/db-sync/db-sync-guide.md`. + +## Problem statement + +`frontend.worker.platform` currently exposes public wrappers that clojure-lsp reports as unused. + +The reported vars are `kv-get`, `kv-set!`, `read-text!`, `write-text!`, and `websocket-connect`. + +Shared worker modules still contain runtime-specific calls that bypass those wrappers. + +The bypasses are concentrated in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +This creates duplicated runtime assumptions and weakens node and browser parity. + +| Symptom | Current location | Impact | +|---|---|---| +| `clojure-lsp/unused-public-var` on `platform/kv-get` and `platform/kv-set!` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals shared code is not using the adapter path for kv persistence. | +| `clojure-lsp/unused-public-var` on `platform/read-text!` and `platform/write-text!` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals text file I/O in shared code can still hardcode a backend. | +| `clojure-lsp/unused-public-var` on `platform/websocket-connect` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals websocket creation in shared sync code may bypass node adapter (`ws`). | +| Direct `js/WebSocket.` in shared sync module | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` | Couples shared sync lifecycle to browser global API. | +| Direct `opfs/ sync.cljs -> js/WebSocket. +db_core -> sync/crypt.cljs -> opfs/idb-keyval. + +Target shared path. +db_core -> sync.cljs -> platform/websocket-connect. +db_core -> sync/crypt.cljs -> platform/read-text!/write-text!/kv-get/kv-set!. + +Runtime adapter ownership. +browser adapter -> OPFS + IndexedDB + browser WebSocket. +node adapter -> fs + JSON kv file + ws package. +``` + +## Testing Plan + +I will add a unit test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that asserts `#'db-sync/connect!` creates sockets through `platform/websocket-connect` and not direct globals. + +I will write the test by stubbing `platform/current`, `platform/websocket-connect`, and `#'db-sync/attach-ws-handlers!` to verify the adapter call receives the tokenized URL. + +I will add unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that assert non-native password read and write paths call `platform/read-text!` and `platform/write-text!`. + +I will add a unit test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that asserts native write fallback uses `platform/write-text!` when main-thread persistence fails. + +I will add unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that assert encrypted AES key cache I/O flows through `platform/kv-get`, `platform/kv-set!`, and `platform/current`. + +I will run each new test case first and confirm failure before implementation changes using `@test-driven-development`. + +I will then run targeted suites and expect all tests to pass. + +I will run `bb dev:test -v frontend.worker.db-sync-test` and expect `0 failures, 0 errors`. + +I will run `bb dev:test -v frontend.worker.sync.crypt-test` and expect `0 failures, 0 errors` for the selected non-`:fix-me` cases. + +I will run `bb dev:lint-and-test` and expect lint plus unit test completion without new warnings in touched namespaces. + +I will verify clojure-lsp diagnostics no longer report `clojure-lsp/unused-public-var` for the five wrapper vars in `frontend.worker.platform`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +This plan changes shared worker modules that should remain runtime-agnostic. + +This plan does not change browser or node adapter implementations except where signature alignment is required. + +This plan does not redesign db-sync protocol or e2ee crypto flow. + +This plan does not migrate persisted data keys. + +This plan is intentionally limited to the exact five wrapper warnings first. + +This plan includes migrating encrypted AES key cache access in `sync/crypt.cljs` to `platform/kv-get` and `platform/kv-set!` because it is in scope of those warnings. + +This plan does not include unrelated broader idb migration outside these touched shared worker paths. + +## Implementation steps + +1. Add a failing websocket adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for `#'db-sync/connect!`. + +2. Run `bb dev:test -v frontend.worker.db-sync-test` and confirm the new test fails because the code still uses direct `js/WebSocket.`. + +3. Add `frontend.worker.platform` require alias in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs`. + +4. Replace the socket constructor in `connect!` with `(platform/websocket-connect (platform/current) ...)`. + +5. Run `bb dev:test -v frontend.worker.db-sync-test` and confirm the websocket adapter test passes. + +6. Add a failing non-native read and write adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs`. + +7. Add a failing native fallback test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that forces native save failure and expects `platform/write-text!`. + +8. Add a failing kv adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` for AES key cache persistence path. + +9. Run `bb dev:test -v frontend.worker.sync.crypt-test` and confirm new tests fail before implementation. + +10. Add `frontend.worker.platform` require alias in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +11. Replace direct password file I/O calls with `platform/read-text!` and `platform/write-text!` against `(platform/current)`. + +12. Replace direct encrypted AES cache idb calls with `platform/kv-get` and `platform/kv-set!` against `(platform/current)`. + +13. Keep key format exactly as `rtc-encrypted-aes-key###` to preserve existing browser data compatibility. + +14. Remove now-unused direct OPFS and idb-keyval requirements from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` if no longer referenced. + +15. Re-run `bb dev:test -v frontend.worker.sync.crypt-test` and confirm all new tests pass. + +16. Re-run `bb dev:test -v frontend.worker.db-sync-test` and confirm no regressions from sync namespace changes. + +17. Run `bb dev:lint-and-test` and confirm there are no new lint or test regressions. + +18. Verify editor or CI diagnostics no longer show the five `frontend.worker.platform` unused-public-var warnings. + +19. If any wrapper remains unused, decide whether to add a real call site or convert that wrapper to private in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs`. + +20. Document verification evidence and remaining caveats in the PR description. + +## Edge cases and risk controls + +Native worker password flow must keep current behavior that first attempts main-thread persistence and falls back on local storage write. + +Node runtime may not expose browser globals, so all shared websocket construction must pass through the adapter. + +Key-value persistence must keep existing key naming to avoid cache misses for already stored encrypted graph AES keys. + +The adapter functions may return promises, so call sites must preserve existing `p/let` sequencing and error paths. + +If `platform/current` is unset in tests, failures should be explicit and tests should set a minimal platform map. + +## Clarified decisions before coding + +Encrypted AES key cache in `sync/crypt.cljs` will migrate from direct idb store to platform kv now. + +Removal of the five `unused-public-var` warnings is mandatory acceptance criteria. + +Tests remain colocated in `db_sync_test.cljs` and `sync/crypt_test.cljs` instead of adding a dedicated platform test namespace. + +## Verification commands + +```bash +bb dev:test -v frontend.worker.db-sync-test +``` + +Expected output contains the new websocket adapter test name and ends with zero failures. + +```bash +bb dev:test -v frontend.worker.sync.crypt-test +``` + +Expected output contains new adapter usage tests and ends with zero failures for executed tests. + +```bash +bb dev:lint-and-test +``` + +Expected output finishes lint plus test pipeline without new errors in touched files. + +## Skills to apply during implementation + +Use `@test-driven-development` for all behavior changes. + +Use `@clojure-debug` immediately when any new test fails unexpectedly. + +Use `@clojure-paren-repair` if Clojure delimiter errors occur while editing touched namespaces. + +## Testing Details + +Tests focus on externally visible behavior of shared worker modules choosing runtime behavior through adapter calls. + +The websocket test validates the constructor path and tokenized URL input instead of testing internal locals. + +The crypt tests validate fallback and persistence behavior through adapter interaction and returned outcomes. + +The kv cache tests validate read and write behavior by key and value flow rather than implementation-specific helpers. + +## Implementation Details + +- Touch `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to route socket creation through `platform/websocket-connect`. +- Touch `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` to route text and kv persistence through platform wrappers. +- Touch `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` to add websocket adapter behavior tests. +- Touch `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` to add adapter-backed storage and kv behavior tests. +- Preserve existing e2ee key naming and payload shapes. +- Keep adapter interface unchanged unless tests prove a missing capability. +- Prefer removing obsolete direct dependencies once wrappers are adopted. +- Keep all new logic promise-safe with existing `promesa` flow. +- Validate clojure-lsp warning cleanup for all five wrapper vars. +- Keep PR scoped to abstraction usage cleanup and tests only. + +## Question + +Confirmed scope: limit this effort to the exact five wrapper warnings first. + +Confirmed decision: migrate encrypted AES key cache usage in `sync/crypt.cljs` to platform kv in this pass. + +Confirmed quality gate: the five `unused-public-var` warnings must be cleared. + +Confirmed test placement: keep new tests in existing `db_sync_test.cljs` and `sync/crypt_test.cljs`. + +--- diff --git a/package.json b/package.json index 033a117ce5..296d217187 100644 --- a/package.json +++ b/package.json @@ -210,7 +210,7 @@ "threads": "1.6.5", "url": "^0.11.0", "util": "^0.12.5", - "ws": "8.19.0", + "ws": "^8.19.0", "yargs-parser": "20.2.4" }, "resolutions": { diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index 15d0e8da19..9cf77a0e88 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -8,6 +8,7 @@ [frontend.common.crypt :as crypt] [frontend.worker-common.util :as worker-util] [frontend.worker.handler.page :as worker-page] + [frontend.worker.platform :as platform] [frontend.worker.shared-service :as shared-service] [frontend.worker.state :as worker-state] [frontend.worker.sync.client-op :as client-op] @@ -1783,7 +1784,8 @@ (stop-client! client)) ;; use cache token for faster websocket connection (when-let [token' (or token (auth-token))] - (let [ws (js/WebSocket. (append-token url token')) + (let [ws (platform/websocket-connect (platform/current) + (append-token url token')) updated (assoc client :ws ws)] (attach-ws-handlers! repo updated ws url) (set! (.-onopen ws) diff --git a/src/main/frontend/worker/sync/crypt.cljs b/src/main/frontend/worker/sync/crypt.cljs index 670cc8a07c..5f6b41aa23 100644 --- a/src/main/frontend/worker/sync/crypt.cljs +++ b/src/main/frontend/worker/sync/crypt.cljs @@ -3,9 +3,9 @@ (:require ["/frontend/idbkv" :as idb-keyval] [clojure.string :as string] [frontend.common.crypt :as crypt] - [frontend.common.file.opfs :as opfs] [frontend.common.thread-api :refer [def-thread-api]] [frontend.worker-common.util :as worker-util] + [frontend.worker.platform :as platform] [frontend.worker.state :as worker-state] [frontend.worker.sync.const :as sync-const] [lambdaisland.glogi :as log] @@ -48,14 +48,14 @@ nil) (p/catch (fn [e] (log/error :native-save-e2ee-password {:error e}) - (opfs/ r (js->clj :keywordize-keys true)))) (defn- latest-remote-tx latest-prev) (done)))))))))) +(deftest connect-uses-platform-websocket-adapter-test + (let [ws-ctor-prev js/WebSocket + platform-map {:runtime :test} + ws-calls (atom []) + attach-calls (atom [])] + (set! js/WebSocket (js* "(function(_url){ this.readyState = 1; })")) + (try + (with-redefs [worker-state/get-id-token (fn [] "token-123") + platform/current (fn [] platform-map) + platform/websocket-connect (fn [platform' url] + (swap! ws-calls conj {:platform platform' :url url}) + (js-obj)) + db-sync/attach-ws-handlers! (fn [repo _client ws url] + (swap! attach-calls conj {:repo repo :ws ws :url url}))] + (let [connected (#'db-sync/connect! test-repo {:repo test-repo} "wss://example.com/sync/graph-1") + ws (:ws connected)] + (is (= [{:platform platform-map + :url "wss://example.com/sync/graph-1?token=token-123"}] + @ws-calls)) + (is (= [{:repo test-repo + :ws ws + :url "wss://example.com/sync/graph-1"}] + @attach-calls)))) + (finally + (set! js/WebSocket ws-ctor-prev))))) + (deftest reaction-add-enqueues-pending-sync-tx-test (testing "adding a reaction should enqueue tx for db-sync" (let [{:keys [conn client-ops-conn parent]} (setup-parent-child)] diff --git a/src/test/frontend/worker/sync/crypt_test.cljs b/src/test/frontend/worker/sync/crypt_test.cljs index 25e8c6c56b..1e5f931301 100644 --- a/src/test/frontend/worker/sync/crypt_test.cljs +++ b/src/test/frontend/worker/sync/crypt_test.cljs @@ -1,6 +1,12 @@ (ns frontend.worker.sync.crypt-test (:require [cljs.test :refer [deftest is async]] + ["/frontend/idbkv" :as idb-keyval] + [clojure.string :as string] [frontend.common.crypt :as crypt] + [frontend.common.file.opfs :as opfs] + [frontend.worker-common.util :as worker-util] + [frontend.worker.platform :as platform] + [frontend.worker.state :as worker-state] [frontend.worker.sync.crypt :as sync-crypt] [logseq.db :as ldb] [promesa.core :as p])) @@ -10,6 +16,154 @@ (p/let [encrypted (crypt/ (p/with-redefs [sync-crypt/native-worker? (fn [] false) + crypt/ (p/with-redefs [sync-crypt/native-worker? (fn [] false) + platform/current (fn [] platform-map) + platform/read-text! (fn [platform' path] + (swap! read-calls conj {:platform platform' + :path path}) + (ldb/write-transit-str {:cipher "payload"})) + opfs/ (p/with-redefs [sync-crypt/native-worker? (fn [] true) + sync-crypt/ (p/with-redefs [sync-crypt/graph-e2ee? (fn [_repo] true) + sync-crypt/e2ee-base (fn [] "https://example.com") + worker-state/get-id-token (fn [] "token") + worker-util/parse-jwt (fn [_] {:sub "user-1"}) + worker-state/ Date: Thu, 26 Feb 2026 19:21:41 +0800 Subject: [PATCH 087/375] 040-hide-db-prefix-in-user-visible-graph-names.md --- deps/cli/src/logseq/cli/commands/graph.cljs | 4 +- deps/cli/src/logseq/cli/common/graph.cljs | 6 +- deps/common/src/logseq/common/config.cljs | 19 ++ ...e-db-prefix-in-user-visible-graph-names.md | 204 ++++++++++++++++++ src/electron/electron/handler.cljs | 45 ++-- src/electron/electron/utils.cljs | 11 +- src/main/frontend/config.cljs | 6 +- src/main/frontend/db/conn.cljs | 5 +- src/main/frontend/db/persist.cljs | 6 +- src/main/frontend/handler/db_based/sync.cljs | 15 +- src/main/frontend/util/text.cljs | 2 +- src/main/logseq/cli/command/core.cljs | 9 +- src/test/frontend/db/persist_test.cljs | 31 +++ .../frontend/handler/db_based/sync_test.cljs | 85 ++++++++ src/test/frontend/util/text_test.cljs | 8 +- src/test/logseq/cli/commands_test.cljs | 25 ++- src/test/logseq/cli/common/graph_test.cljs | 20 ++ src/test/logseq/cli/format_test.cljs | 29 ++- 18 files changed, 477 insertions(+), 53 deletions(-) create mode 100644 docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md create mode 100644 src/test/frontend/db/persist_test.cljs create mode 100644 src/test/logseq/cli/common/graph_test.cljs diff --git a/deps/cli/src/logseq/cli/commands/graph.cljs b/deps/cli/src/logseq/cli/commands/graph.cljs index 73e5a0b00c..eb764d5525 100644 --- a/deps/cli/src/logseq/cli/commands/graph.cljs +++ b/deps/cli/src/logseq/cli/commands/graph.cljs @@ -38,6 +38,6 @@ (defn list-graphs [] (let [db-graphs (->> (cli-common-graph/get-db-based-graphs) - (map #(string/replace-first % common-config/db-version-prefix "")) + (map common-config/strip-leading-db-version-prefix) sort)] - (println (string/join "\n" db-graphs)))) \ No newline at end of file + (println (string/join "\n" db-graphs)))) diff --git a/deps/cli/src/logseq/cli/common/graph.cljs b/deps/cli/src/logseq/cli/common/graph.cljs index 87644110f2..d8f636cf09 100644 --- a/deps/cli/src/logseq/cli/common/graph.cljs +++ b/deps/cli/src/logseq/cli/common/graph.cljs @@ -27,5 +27,7 @@ (remove (fn [s] (= s common-config/unlinked-graphs-dir))) (map graph-name->path) (keep (fn [s] - (when-not (string/starts-with? s common-config/file-version-prefix) - (str common-config/db-version-prefix s))))))) + (when (and (string? s) + (not (string/starts-with? s common-config/file-version-prefix))) + (common-config/canonicalize-db-version-repo s)))) + distinct))) diff --git a/deps/common/src/logseq/common/config.cljs b/deps/common/src/logseq/common/config.cljs index f1b297a624..565a05feca 100644 --- a/deps/common/src/logseq/common/config.cljs +++ b/deps/common/src/logseq/common/config.cljs @@ -33,6 +33,25 @@ (defonce db-version-prefix "logseq_db_") (defonce file-version-prefix "logseq_local_") +(defn strip-leading-db-version-prefix + "Strip exactly one leading db prefix for user-facing display values." + [s] + (if (and (string? s) + (string/starts-with? s db-version-prefix)) + (subs s (count db-version-prefix)) + s)) + +(defn canonicalize-db-version-repo + "Normalize any repo/graph name to exactly one leading db prefix." + [s] + (when (seq s) + (let [s (str s) + stripped (loop [name s] + (if (string/starts-with? name db-version-prefix) + (recur (subs name (count db-version-prefix))) + name))] + (str db-version-prefix stripped)))) + (defonce local-assets-dir "assets") (defonce unlinked-graphs-dir "Unlinked graphs") diff --git a/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md new file mode 100644 index 0000000000..1b338148c3 --- /dev/null +++ b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md @@ -0,0 +1,204 @@ +# Hide Db Prefix In User Visible Graph Names Implementation Plan + +Goal: Ensure user-visible graph names strip exactly one leading `logseq_db_` prefix while preventing new multi-prefix repos from being created. + +Architecture: Keep display normalization as single-pass prefix stripping for web, Electron, and CLI user-facing fields. + +Architecture: Add shared canonicalization at ingestion and graph-discovery boundaries so internal repo identifiers are normalized to exactly one leading prefix before persistence and routing. + +Architecture: Follow `@test-driven-development` with failing tests first across frontend, Electron boundary code, RTC ingestion, and CLI paths. + +Tech Stack: ClojureScript, Rum, Electron IPC, db-worker-node runtime, Logseq CLI formatting pipeline, Babashka tests. + +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +Related: Builds on `docs/agent-guide/038-electron-db-worker-switch-graph.md`. + +## Problem statement + +Graph names shown in the web graph list can expose `logseq_db_` when the stored repo value has multiple leading prefixes. + +Current rendering logic intentionally strips only one leading prefix, so malformed values like `logseq_db_logseq_db_demo` still render as `logseq_db_demo`. + +The product decision is to keep one-layer stripping behavior and fix the upstream causes that create multi-prefix repo identifiers. + +Multi-prefix data can enter through legacy disk graph discovery, Electron graph mapping, RTC remote metadata ingestion, and CLI graph-name conversion paths. + +The required outcome is stable single-prefix internal repo identifiers plus single-pass display normalization, so normal graphs render as `demo` and malformed legacy doubles render as `logseq_db_demo`. + +## Current and target normalization path + +```text +Current path with multi-prefix source data. +input repo: logseq_db_logseq_db_demo + -> display helper strips once + -> output: logseq_db_demo + -> user-visible prefix leak occurs. + +Target path with source canonicalization + single-pass display stripping. +ingress repo: logseq_db_logseq_db_demo + -> canonicalize to one internal prefix: logseq_db_demo + -> display helper strips once + -> output: demo + +Legacy persisted double-prefix fallback path (no migration). +input repo from old state: logseq_db_logseq_db_demo + -> display helper strips once + -> output: logseq_db_demo +``` + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before implementation. + +I will add frontend unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/util/text_test.cljs` to assert single-pass prefix stripping behavior. + +I will add frontend persist tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/db/persist_test.cljs` to assert merged graph sources are canonicalized to one internal prefix before UI state usage. + +I will add RTC tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handler/db_based/rtc_test.cljs` to assert remote graph payload ingestion cannot create local double-prefix repos. + +I will extend CLI formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to assert user-facing fields strip exactly one prefix and never introduce additional prefixes. + +I will extend CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `graph list` and server output paths with unprefixed and prefix-like graph names, and assert prefix-like `--repo` values are treated as graph-name content instead of invalid input. + +I will add legacy graph discovery tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` to verify old directory names do not produce multi-prefix repo ids. + +I will run focused tests after RED and GREEN phases, then run `bb dev:lint-and-test` before completion. + +I will review changes against `@prompts/review.md` before merge. + +NOTE: I will write all tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests for one-layer display stripping and multi-prefix prevention. + +1. Add failing unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/util/text_test.cljs` for `logseq_db_demo -> demo`, `logseq_db_logseq_db_demo -> logseq_db_demo`, and middle-substring preservation. +2. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/db/persist_test.cljs` to verify worker and Electron graph sources are canonicalized to one internal prefix. +3. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handler/db_based/rtc_test.cljs` for remote graph mapping and download paths with prefixed and double-prefixed payload names. +4. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` with failing cases where `:repo` or `:graph` includes one or two prefixes and output uses one-layer stripping only. +5. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with failing cases for graph list and server output using unprefixed and prefix-like `--repo` values, and assert prefix-like values are not rejected by argument validation. +6. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` for legacy directory names that already contain `logseq_db_`. +7. Run focused tests and confirm failures reflect behavior gaps rather than setup errors. + +### Phase 2: Implement shared helpers for display normalization and repo canonicalization. + +8. Add shared helpers in `/Users/rcmerci/gh-repos/logseq/deps/common/src/logseq/common/config.cljs` for single-pass display stripping and exact-one-prefix canonicalization. +9. Keep display helper semantics strict to remove only one leading prefix and preserve all middle substrings. +10. Keep canonicalization helper semantics strict to collapse any number of leading prefixes to exactly one. +11. Use function names that clearly separate display behavior from internal repo id normalization. +12. Keep helpers pure and dependency-light so they can be reused in frontend, Electron, and CLI namespaces. + +### Phase 3: Apply web app fixes with one-layer display semantics. + +13. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/util/text.cljs` so `get-graph-name-from-path` calls the shared single-pass display helper. +14. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` so `db-graph-name` remains one-layer stripping and does not over-strip. +15. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/db/conn.cljs` so `get-short-repo-name` uses shared one-layer display normalization. +16. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/components/repo.cljs` rendering paths only if needed to ensure all user-visible labels share one-layer stripping behavior. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/db/persist.cljs` so merged graph sources are canonicalized to one internal prefix before entering repo state. +18. Keep `data-testid` and internal routing keys unchanged unless canonicalization is required to prevent multi-prefix key creation. + +### Phase 4: Fix Electron and graph discovery paths that can create multi-prefix repos. + +19. Update `/Users/rcmerci/gh-repos/logseq/deps/cli/src/logseq/cli/common/graph.cljs` so `get-db-based-graphs` canonicalizes discovered graph repo names to one prefix. +20. Update `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` and `/Users/rcmerci/gh-repos/logseq/src/electron/electron/utils.cljs` to canonicalize repo identifiers at IPC mapping boundaries. +21. Verify Electron IPC `getGraphs` keeps stable internal repo identifiers and no path emits new double-prefixed repos. + +### Phase 5: Fix RTC ingestion paths that can reintroduce multi-prefix repos. + +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/db_based/rtc.cljs` so remote graph payload mapping canonicalizes incoming graph names before repo/url construction. +23. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/rtc/full_upload_download_graph.cljs` so download naming canonicalizes to one internal prefix before local repo creation. +24. Ensure existing graphs with prefixed remote names remain accessible after normalization. +25. Ensure new uploads and downloads cannot create `logseq_db_logseq_db_*` local repo ids. + +### Phase 6: Apply CLI fixes for one-layer display and one-prefix internal ids. + +26. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` so `repo->graph` strips exactly one leading prefix for user-visible output. +27. Verify `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` uses one-layer display normalization consistently for human, JSON, and EDN user-facing graph fields. +28. Update `/Users/rcmerci/gh-repos/logseq/deps/cli/src/logseq/cli/commands/graph.cljs` to canonicalize internal repo identifiers before list rendering and server-target resolution. +29. Ensure CLI parsing and command execution treat `--repo` as a graph-name string, so leading `logseq_db_` is interpreted as part of graph name when present and is not rejected by input validation. + +### Phase 7: Verification and release gate. + +30. Run `bb dev:test -v 'frontend.util.text-test'` and confirm one-layer strip behavior and canonicalization tests pass. +31. Run `bb dev:test -v 'frontend.db.persist-test'` and confirm merged-source canonicalization behavior is stable. +32. Run `bb dev:test -v 'frontend.handler.db-based.rtc-test'` and confirm remote ingestion cannot produce multi-prefix repos. +33. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm CLI display fields apply one-layer strip behavior. +34. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm graph command behavior remains stable with unprefixed and prefix-like `--repo` input. +35. Run `bb dev:test -v 'logseq.cli.common.graph-test'` and confirm legacy graph discovery does not emit double-prefixed repo names. +36. Run `bb dev:lint-and-test` and confirm `0 failures, 0 errors`. +37. Perform a manual graph-list smoke check on web and Electron to confirm normal graphs display without prefix and legacy doubles display with one remaining prefix. +38. Perform a manual CLI smoke check with `logseq graph list`, `logseq server status --repo demo`, and `logseq server status --repo logseq_db_demo` to confirm both inputs are accepted as graph names. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Internal repo is `logseq_db_demo`. | User-visible name is `demo`. | +| Internal repo is `logseq_db_logseq_db_demo`. | User-visible name is `logseq_db_demo`. | +| Graph name contains middle substring like `my_logseq_db_notes`. | Middle substring is preserved and only one leading prefix is removed when present. | +| Legacy disk directory is named `logseq_db_demo`. | Discovery canonicalization keeps exactly one prefix and does not create `logseq_db_logseq_db_demo`. | +| Legacy disk directory is named `logseq_db_logseq_db_demo`. | Discovery canonicalization collapses to internal repo `logseq_db_demo`. | +| Remote graph payload returns `graph-name` as `logseq_db_demo`. | RTC mapping keeps internal repo `logseq_db_demo` and user-visible name `demo`. | +| Remote graph payload returns `graph-name` as `logseq_db_logseq_db_demo`. | RTC mapping canonicalizes to internal repo `logseq_db_demo`, and user-visible name is `demo` after one-layer display strip. | +| CLI receives `--repo demo`. | Command works and output graph name is `demo`. | +| CLI receives `--repo logseq_db_demo`. | Command treats `logseq_db_` as part of graph name and does not fail argument validation. | +| CLI receives `--repo logseq_db_logseq_db_demo`. | Command treats the full value as graph name content and does not fail argument validation. | +| Non-user-visible fields like `data-testid` include repo id. | Existing selectors remain unchanged unless canonicalization is required to prevent duplicate graph entries. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'frontend.util.text-test' +bb dev:test -v 'frontend.db.persist-test' +bb dev:test -v 'frontend.handler.db-based.rtc-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.commands-test' +bb dev:test -v 'logseq.cli.common.graph-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +Web and Electron manual checks should show no new multi-prefix repo entries. + +Web and Electron display should strip one prefix only at render time. + +CLI human output should match one-layer strip semantics, and CLI `--repo` should treat prefix-like values as normal graph names. + +## Testing Details + +The tests verify user-visible behavior and repo-id canonicalization at ingress and discovery boundaries. + +Frontend tests assert display output and merged-state behavior rather than helper internals alone. + +RTC and Electron tests assert that incoming prefixed names cannot generate additional prefixes in local repo ids. + +CLI tests assert command behavior and output formatting remain stable for unprefixed and prefix-like graph-name input. + +## Implementation Details + +- Keep display normalization as single-pass prefix stripping. +- Add one shared helper for exact-one-prefix repo canonicalization. +- Canonicalize ingress and discovery data before it reaches persistent repo state. +- Reuse shared helpers across frontend, Electron, and CLI. +- Preserve `data-testid` compatibility unless canonicalization makes key updates unavoidable. +- Avoid one-time metadata migration for existing persisted graphs. +- Treat CLI `--repo` input as raw graph name where `logseq_db_` may be part of the name. +- Keep internal thread-api contracts based on prefixed repo ids. +- Follow `@test-driven-development` for RED, GREEN, and REFACTOR order. +- Validate final patch with `@prompts/review.md` checklist. + +## Question + +Decision: Display normalization removes only one leading prefix, so `logseq_db_logseq_db_xxxx` displays as `logseq_db_xxxx` in app surfaces that read legacy uncanonicalized values. + +Decision: The implementation focus is to identify and fix all paths that can create multi-prefix repo identifiers, so newly produced data stays canonical. + +Decision: This change does not include a one-time metadata migration for existing persisted legacy values. + +Decision: CLI `--repo` option treats leading `logseq_db_` as graph-name content, not as forbidden prefix. + +Decision: Treat `data-testid` stability as a strict compatibility requirement for `clj-e2e`. + +--- diff --git a/src/electron/electron/handler.cljs b/src/electron/electron/handler.cljs index 9379140af4..3453f88f8c 100644 --- a/src/electron/electron/handler.cljs +++ b/src/electron/electron/handler.cljs @@ -29,6 +29,7 @@ [electron.window :as win] [electron.graph-switch-flow :as graph-switch-flow] [logseq.cli.common.graph :as cli-common-graph] + [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] [logseq.db.sqlite.util :as sqlite-util] [promesa.core :as p])) @@ -200,24 +201,30 @@ [] (distinct (cli-common-graph/get-db-based-graphs))) +(defn- canonical-repo + [graph] + (common-config/canonicalize-db-version-repo graph)) + ;; TODO support alias mechanism (defn get-graph-name "Given a graph's name of string, returns the graph's fullname. For example, given `cat`, returns `logseq_db_cat`. Returns `nil` if no such graph exists." [graph-identifier] - (->> (get-graphs) - (some #(when (or - (= (utils/normalize-lc %) (utils/normalize-lc (str sqlite-util/db-version-prefix graph-identifier))) - (string/ends-with? (utils/normalize-lc %) - (str "/" (utils/normalize-lc graph-identifier)))) - %)))) + (when-let [repo (canonical-repo graph-identifier)] + (let [graph-name (common-config/strip-leading-db-version-prefix repo)] + (->> (get-graphs) + (some #(when (or + (= (utils/normalize-lc %) (utils/normalize-lc repo)) + (string/ends-with? (utils/normalize-lc %) + (str "/" (utils/normalize-lc graph-name)))) + %)))))) (defmethod handle :getGraphs [_window [_]] (get-graphs)) (defmethod handle :deleteGraph [_window [_ graph]] - (when graph - (db/unlink-graph! graph))) + (when-let [repo (canonical-repo graph)] + (db/unlink-graph! repo))) ;; DB related IPCs start @@ -228,20 +235,22 @@ (defmethod handle :db-worker-runtime [^js window [_ repo]] (if (string/blank? repo) (p/rejected (ex-info "repo is required" {:code :missing-repo})) - (db-worker/ensure-runtime! repo (.-id window)))) + (db-worker/ensure-runtime! (canonical-repo repo) (.-id window)))) (defmethod handle :db-export [_window [_ repo data]] - (logger/warn ::db-export-compat - {:repo repo - :message "legacy db-export IPC path invoked; desktop should use db-worker runtime"}) - (db/ensure-graph-dir! repo) - (db/save-db! repo data)) + (when-let [repo (canonical-repo repo)] + (logger/warn ::db-export-compat + {:repo repo + :message "legacy db-export IPC path invoked; desktop should use db-worker runtime"}) + (db/ensure-graph-dir! repo) + (db/save-db! repo data))) (defmethod handle :db-get [_window [_ repo]] - (logger/warn ::db-get-compat - {:repo repo - :message "legacy db-get IPC path invoked; desktop should use db-worker runtime"}) - (db/get-db repo)) + (when-let [repo (canonical-repo repo)] + (logger/warn ::db-get-compat + {:repo repo + :message "legacy db-get IPC path invoked; desktop should use db-worker runtime"}) + (db/get-db repo))) ;; DB related IPCs End diff --git a/src/electron/electron/utils.cljs b/src/electron/electron/utils.cljs index cbb26caf46..8d9f26f596 100644 --- a/src/electron/electron/utils.cljs +++ b/src/electron/electron/utils.cljs @@ -7,7 +7,7 @@ [electron.configs :as cfgs] [electron.logger :as logger] [logseq.cli.common.graph :as cli-common-graph] - [logseq.db.sqlite.util :as sqlite-util] + [logseq.common.config :as common-config] [promesa.core :as p])) (defonce *win (atom nil)) ;; The main window @@ -240,14 +240,17 @@ (defn get-graph-dir "required by all internal state in the electron section" [graph-name] - (when (string/starts-with? graph-name sqlite-util/db-version-prefix) - (node-path/join (cli-common-graph/get-db-graphs-dir) (string/replace-first graph-name sqlite-util/db-version-prefix "")))) + (when (and (string? graph-name) + (string/starts-with? graph-name common-config/db-version-prefix)) + (let [repo (common-config/canonicalize-db-version-repo graph-name)] + (node-path/join (cli-common-graph/get-db-graphs-dir) + (common-config/strip-leading-db-version-prefix repo))))) (comment (defn get-graph-name "Reverse `get-graph-dir`" [graph-dir] - (str sqlite-util/db-version-prefix (node-path/basename graph-dir)))) + (str common-config/db-version-prefix (node-path/basename graph-dir)))) (defn decode-protected-assets-schema-path [schema-path] diff --git a/src/main/frontend/config.cljs b/src/main/frontend/config.cljs index 832c98732f..9396d9c4c3 100644 --- a/src/main/frontend/config.cljs +++ b/src/main/frontend/config.cljs @@ -238,7 +238,7 @@ (defn db-graph-name [repo-with-prefix] - (string/replace-first repo-with-prefix db-version-prefix "")) + (common-config/strip-leading-db-version-prefix repo-with-prefix)) (defn db-based-graph? ([] @@ -257,7 +257,7 @@ (path/path-join (get-in @state/state [:system/info :home-dir]) "logseq" "graphs" - (string/replace repo db-version-prefix ""))) + (db-graph-name repo))) (defn get-electron-backup-dir [repo] @@ -269,7 +269,7 @@ (if (util/electron?) (get-local-dir repo-url) (str "memory:///" - (string/replace-first repo-url db-version-prefix ""))))) + (db-graph-name repo-url))))) (defn get-repo-config-path [] diff --git a/src/main/frontend/db/conn.cljs b/src/main/frontend/db/conn.cljs index f74f76967b..3ef5ab6f58 100644 --- a/src/main/frontend/db/conn.cljs +++ b/src/main/frontend/db/conn.cljs @@ -1,7 +1,6 @@ (ns frontend.db.conn "Contains db connections." - (:require [clojure.string :as string] - [datascript.core :as d] + (:require [datascript.core :as d] [frontend.config :as config] [frontend.db.conn-state :as db-conn-state] [frontend.mobile.util :as mobile-util] @@ -52,7 +51,7 @@ :else repo-name)] (if (config/db-based-graph? repo-name') - (string/replace-first repo-name' config/db-version-prefix "") + (config/db-graph-name repo-name') repo-name'))) (defn remove-conn! diff --git a/src/main/frontend/db/persist.cljs b/src/main/frontend/db/persist.cljs index 156c3333fb..7e170a63a0 100644 --- a/src/main/frontend/db/persist.cljs +++ b/src/main/frontend/db/persist.cljs @@ -3,7 +3,6 @@ (:require [cljs-bean.core :as bean] [clojure.string :as string] [electron.ipc :as ipc] - [frontend.config :as config] [frontend.persist-db :as persist-db] [frontend.util :as util] [logseq.common.config :as common-config] @@ -23,12 +22,13 @@ (map (fn [{:keys [name] :as repo}] (assoc repo :name - (str config/db-version-prefix name))))) + (common-config/canonicalize-db-version-repo name))))) electron-disk-graphs (when (util/electron?) (ipc/ipc "getGraphs"))] (distinct (concat repos' - (map (fn [repo-name] {:name repo-name}) + (map (fn [repo-name] + {:name (common-config/canonicalize-db-version-repo repo-name)}) (some-> electron-disk-graphs bean/->clj)))))) (defn delete-graph! diff --git a/src/main/frontend/handler/db_based/sync.cljs b/src/main/frontend/handler/db_based/sync.cljs index 0443e72b92..7d4e0544b3 100644 --- a/src/main/frontend/handler/db_based/sync.cljs +++ b/src/main/frontend/handler/db_based/sync.cljs @@ -9,6 +9,7 @@ [frontend.state :as state] [frontend.util :as util] [lambdaisland.glogi :as log] + [logseq.common.config :as common-config] [logseq.db :as ldb] [logseq.db-sync.malli-schema :as db-sync-schema] [logseq.db.sqlite.util :as sqlite-util] @@ -286,7 +287,7 @@ (if base (p/let [_ (js/Promise. user-handler/task--ensure-id&access-token) body (coerce-http-request :graphs/create - {:graph-name (string/replace repo config/db-version-prefix "") + {:graph-name (common-config/strip-leading-db-version-prefix repo) :schema-version schema-version :graph-e2ee? graph-e2ee?}) result (if (nil? body) @@ -334,15 +335,15 @@ ([graph-name graph-uuid graph-e2ee?] (state/set-state! :rtc/downloading-graph-uuid graph-uuid) (state/pub-event! - [:rtc/log {:type :rtc.log/download + [:rtc/log {:type :rtc.log/download :sub-type :download-progress :graph-uuid graph-uuid :message "Preparing graph snapshot download"}]) (let [graph-e2ee? (normalize-graph-e2ee? graph-e2ee?) + graph (common-config/canonicalize-db-version-repo graph-name) base (http-base)] (-> (if (and graph-uuid base) (-> (p/let [_ (js/Promise. user-handler/task--ensure-id&access-token) - graph (str config/db-version-prefix graph-name) pull-resp (fetch-json (str base "/sync/" graph-uuid "/pull") {:method "GET"} {:response-schema :sync/pull}) @@ -398,12 +399,14 @@ {:response-schema :graphs/list}) graphs (:graphs resp) result (mapv (fn [graph] - (let [graph-e2ee? (if (contains? graph :graph-e2ee?) + (let [repo (common-config/canonicalize-db-version-repo (:graph-name graph)) + graph-name (common-config/strip-leading-db-version-prefix repo) + graph-e2ee? (if (contains? graph :graph-e2ee?) (normalize-graph-e2ee? (:graph-e2ee? graph)) true)] (merge - {:url (str config/db-version-prefix (:graph-name graph)) - :GraphName (:graph-name graph) + {:url repo + :GraphName graph-name :GraphSchemaVersion (:schema-version graph) :GraphUUID (:graph-id graph) :rtc-graph? true diff --git a/src/main/frontend/util/text.cljs b/src/main/frontend/util/text.cljs index c9a9135059..7c3255d564 100644 --- a/src/main/frontend/util/text.cljs +++ b/src/main/frontend/util/text.cljs @@ -101,4 +101,4 @@ On iOS, repo-url might be nil" [repo-url] (when (not-empty repo-url) - (string/replace-first repo-url config/db-version-prefix ""))) + (config/db-graph-name repo-url))) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 94ab0db180..351b7e06f6 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -210,15 +210,14 @@ (defn graph->repo [graph] - (when (seq graph) - (if (string/starts-with? graph common-config/db-version-prefix) - graph - (str common-config/db-version-prefix graph)))) + (some-> graph + string/trim + common-config/canonicalize-db-version-repo)) (defn repo->graph [repo] (when (seq repo) - (string/replace-first repo common-config/db-version-prefix ""))) + (common-config/strip-leading-db-version-prefix repo))) (defn resolve-repo [graph] diff --git a/src/test/frontend/db/persist_test.cljs b/src/test/frontend/db/persist_test.cljs new file mode 100644 index 0000000000..6fce792a8a --- /dev/null +++ b/src/test/frontend/db/persist_test.cljs @@ -0,0 +1,31 @@ +(ns frontend.db.persist-test + (:require [cljs.test :refer [async deftest is]] + [electron.ipc :as ipc] + [frontend.db.persist :as db-persist] + [frontend.persist-db :as persist-db] + [frontend.util :as util] + [promesa.core :as p])) + +(deftest get-all-graphs-canonicalizes-db-prefixes-from-all-sources + (async done + (-> (p/with-redefs [persist-db/ (p/with-redefs [db-sync/http-base (constantly "http://base") + user-handler/task--ensure-id&access-token (fn [resolve _reject] + (resolve true)) + db-sync/fetch-json (fn [url _opts _schema] + (if (string/ends-with? url "/graphs") + (p/resolved {:graphs [{:graph-name "logseq_db_demo" + :graph-id "graph-1" + :schema-version 1} + {:graph-name "logseq_db_logseq_db_legacy" + :graph-id "graph-2" + :schema-version 1}]}) + (p/rejected (ex-info "unexpected fetch-json URL" + {:url url})))) + state/set-state! (fn [k v] + (swap! set-state-calls conj [k v])) + repo-handler/refresh-repos! (fn [] nil)] + (p/let [result (db-sync/ (p/let [gzip-bytes ( (p/with-redefs [db-sync/http-base (constantly "http://base") + db-sync/fetch-json (fn [url _opts _schema] + (cond + (string/ends-with? url "/pull") + (p/resolved {:t 8}) + + :else + (p/rejected (ex-info "unexpected fetch-json URL" + {:url url})))) + user-handler/task--ensure-id&access-token (fn [resolve _reject] + (resolve true)) + state/ (p/let [result (commands/execute {:type :graph-list} {})] (is (= :ok (:status result))) - (is (= ["demo" "other"] (get-in result [:data :graphs])))) + (is (= ["demo" "logseq_db_other" "my_logseq_db_notes"] + (get-in result [:data :graphs])))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] diff --git a/src/test/logseq/cli/common/graph_test.cljs b/src/test/logseq/cli/common/graph_test.cljs new file mode 100644 index 0000000000..1d205a8a39 --- /dev/null +++ b/src/test/logseq/cli/common/graph_test.cljs @@ -0,0 +1,20 @@ +(ns logseq.cli.common.graph-test + (:require [cljs.test :refer [deftest is]] + [clojure.string :as string] + [frontend.test.node-helper :as node-helper] + [logseq.cli.common.graph :as cli-common-graph] + ["fs" :as fs] + ["path" :as node-path])) + +(deftest get-db-based-graphs-canonicalizes-legacy-prefixed-directory-names + (let [graphs-dir (node-helper/create-tmp-dir "cli-common-graph") + _ (doseq [dir ["demo" + "logseq_db_demo" + "logseq_db_logseq_db_demo" + "logseq_local_file-graph" + "Unlinked graphs"]] + (fs/mkdirSync (node-path/join graphs-dir dir) #js {:recursive true}))] + (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] + (let [graphs (cli-common-graph/get-db-based-graphs)] + (is (= #{"logseq_db_demo"} (set graphs))) + (is (not-any? #(string/starts-with? % "logseq_db_logseq_db_") graphs)))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 1374fbffd3..8f8d3239f3 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -175,7 +175,34 @@ {:output-format nil})] (is (= (str "Server ready: demo-repo\n" "Host: 127.0.0.1 Port: 1234") - result))))) + result)))) + + (testing "server status strips only one leading db prefix and keeps middle substrings" + (let [double-prefixed (format/format-result {:status :ok + :command :server-status + :data {:repo "logseq_db_logseq_db_demo" + :status :ready}} + {:output-format nil}) + middle-substring (format/format-result {:status :ok + :command :server-status + :data {:repo "my_logseq_db_notes" + :status :ready}} + {:output-format nil})] + (is (= "Server ready: logseq_db_demo" double-prefixed)) + (is (= "Server ready: my_logseq_db_notes" middle-substring))))) + +(deftest test-json-output-normalizes-graph-fields-with-single-leading-strip-only + (let [result (format/format-result {:status :ok + :data {:repo "logseq_db_logseq_db_demo" + :graph "my_logseq_db_notes" + :graphs ["logseq_db_logseq_db_demo" + "my_logseq_db_notes"]}} + {:output-format :edn}) + parsed (reader/read-string result)] + (is (= "logseq_db_demo" (get-in parsed [:data :repo]))) + (is (= "my_logseq_db_notes" (get-in parsed [:data :graph]))) + (is (= ["logseq_db_demo" "my_logseq_db_notes"] + (get-in parsed [:data :graphs]))))) (deftest test-human-output-server-list-includes-owner (testing "server list shows owner column and value" From b9b21e9ecec0ed1a2e1429284b3ed060e5a6be82 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 26 Feb 2026 20:35:12 +0800 Subject: [PATCH 088/375] fix: ignore daemon process stdio if cli spawn it --- src/main/logseq/db_worker/daemon.cljs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs index ddacb1cb49..4885c2cce8 100644 --- a/src/main/logseq/db_worker/daemon.cljs +++ b/src/main/logseq/db_worker/daemon.cljs @@ -263,7 +263,9 @@ (log/warn :db-worker-daemon/missing-script {:repo repo :data-dir data-dir}) nil) (let [child (.spawn child-process (.-execPath js/process) args #js {:detached true - :stdio "inherit" + :stdio (if (= owner-source :cli) + "ignore" + "inherit") :env env})] (.unref child) child)))) From 48c1f5374eba3700eb62a9e68f227e482e48944f Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 26 Feb 2026 21:37:45 +0800 Subject: [PATCH 089/375] 041-logseq-cli-add-block-json-identifiers.md --- ...1-logseq-cli-add-block-json-identifiers.md | 171 ++++++++++++++++++ docs/cli/logseq-cli.md | 13 ++ src/main/logseq/cli/command/add.cljs | 148 ++++++++++++--- src/main/logseq/cli/format.cljs | 12 +- src/test/logseq/cli/command/add_test.cljs | 45 +++++ src/test/logseq/cli/format_test.cljs | 15 +- src/test/logseq/cli/integration_test.cljs | 160 ++++++++++++++++ 7 files changed, 531 insertions(+), 33 deletions(-) create mode 100644 docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md create mode 100644 src/test/logseq/cli/command/add_test.cljs diff --git a/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md b/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md new file mode 100644 index 0000000000..c3b0e28b86 --- /dev/null +++ b/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md @@ -0,0 +1,171 @@ +# Logseq CLI Add Command Entity Id Output Implementation Plan + +Goal: Make `add page` and `add block` return newly created entity `db/id` in `--output human`, `--output json`, and `--output edn`. + +Architecture: Keep existing add command write paths and add a post-write identifier resolution step in the CLI command layer. +Architecture: Return a unified structured payload for both add commands where `:result` is a vector of created entity `db/id`. +Architecture: Update human formatter for add commands to include created entity ids while preserving existing success semantics. + +Tech Stack: ClojureScript, Logseq CLI command layer, db-worker-node thread API, Logseq CLI formatter. + +Related: Builds on docs/agent-guide/027-logseq-cli-update-command.md and docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md. + +Document naming follows @planning-documents using the next available sequence number `041`. + +## Problem statement + +Current behavior for `add block --output json` is `{"status":"ok","data":{"result":null}}` because `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` sets `:result nil` in `execute-add-block`. + +Current add command outputs are not consistent for machine and human flows when users need the new entity id immediately. + +Users now require `db/id` in all output formats for both `add page` and `add block`. + +Without this output, automation must run extra commands to locate newly created entities before `update` or `remove`. + +## Testing Plan + +I will follow @test-driven-development and complete all RED tests before implementation. + +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `add page --output json` and `add block --output json` asserting returned `db/id`. + +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `add page --output edn` and `add block --output edn` asserting returned `:db/id`. + +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting human output lines for add page and add block include new ids. + +I will add one integration chain test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that uses returned add ids directly in `update` and `remove` commands. + +I will add a focused unit test namespace at `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/add_test.cljs` for helper logic that builds the created-entity result payload. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +``` +add page/add block + -> /src/main/logseq/cli/command/add.cljs execute-add-* + -> thread-api write call + -> resolve created entity ids in CLI command layer + -> result payload {:result [id1 id2 ...]} + -> /src/main/logseq/cli/format.cljs human renderer includes id information +``` + +## Output contract + +JSON output for add page returns created page id vector in `data.result`. + +JSON output for add block returns created block ids in `data.result`. + +EDN output mirrors the same data shape using keyword keys. + +Human output includes created ids for both commands. + +Example human output for add page is: + +```text +Added page: +[123] +``` + +Example human output for add block is: + +```text +Added blocks: +[101 102] +``` + +## Plan + +1. Read `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and confirm current add page and add block result payload shapes. +2. Read `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` and confirm current human formatter paths for add commands. +3. Write RED integration test A in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add page JSON result containing created page `db/id`. +4. Write RED integration test B in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add block JSON result containing created block `db/id` list. +5. Write RED integration test C in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add page and add block EDN output containing `:db/id`. +6. Write RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting human add output includes ids. +7. Write RED unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/add_test.cljs` for id vector normalization and deterministic ordering. +8. Run focused tests and verify failures are caused by missing behavior and not incorrect test setup. +9. Implement add block result enrichment in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` so it returns all created block ids including nested children. +10. Implement add page result enrichment in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` so it returns created page id as a one-element vector. +11. Keep result field naming stable with a shape `:data {:result [101 102]}`. +12. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` add page and add block human renderers to include id data from command result. +13. Ensure JSON and EDN formatter paths continue to serialize the enriched result without special-case regressions. +14. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` with add page and add block structured output examples that include `db/id`. +15. Run focused tests again and verify GREEN behavior for all newly added tests. +16. Refactor helper naming and duplicate mapping code in add command execution if needed without behavior change. +17. Re-run focused tests after refactor to verify tests stay green. +18. Run `bb dev:lint-and-test` for regression verification. + +## Edge cases + +Add block from `--content` must still return generated block id when UUID was not supplied in input. + +Add block from `--blocks` with multiple nested blocks must return all created ids and use deterministic ordering in returned payload. + +Add page should return the created page id even when tags and properties are added in the same command. + +Add block human output should remain readable when returning many ids including nested children. + +When id resolution fails unexpectedly after a successful write, command should return an error rather than a misleading success payload without ids. + +Error output format and exit codes must remain unchanged for invalid input and missing target cases. + +## Testing commands and expected output + +Run focused RED tests. + +```bash +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-json-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-block-json-output-returns-ids' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-block-edn-output-returns-id' +bb dev:test -v 'logseq.cli.format-test/test-human-output-add-remove' +``` + +Expected RED output includes assertion failures indicating missing id fields in add outputs. + +Run focused GREEN tests after implementation. + +```bash +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-json-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-block-json-output-returns-ids' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-block-edn-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-identifiers-chain-update-remove' +bb dev:test -v 'logseq.cli.command.add-test' +bb dev:test -v 'logseq.cli.format-test/test-human-output-add-remove' +``` + +Expected GREEN output includes zero failures and zero errors for the focused namespaces. + +Run full verification. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful lint and tests with exit code `0`. + +## Testing Details + +Integration tests will assert actual CLI command output payloads and real persisted graph behavior for add page and add block. + +Formatter tests will assert exact human output strings for add page and add block including id fragments. + +Unit tests will validate helper behavior for id list assembly and ordering. + +## Implementation Details + +- Enrich `execute-add-block` result so `:result` is a vector of all created block ids including nested children. +- Enrich `execute-add-page` result so `:result` is a one-element vector containing created page id. +- Keep result payload stable across JSON and EDN output paths by using Clojure maps with keyword keys. +- Update human formatters for add page and add block to include id information: + - add page rendered as `Added page:` on one line and `[123]` on the next line. + - add block rendered as `Added blocks:` on one line and `[101 102]` on the next line. +- Preserve existing command success and error semantics besides the new id output fields. +- Add integration coverage for add outputs plus id-based `update` and `remove` chaining. +- Update CLI docs to show the new add page and add block result examples with ids. + +## Decisions + +Human output for add block must display all created ids including nested children. + +For add block and add page, `:result` must be an id vector such as `[101 102]`. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 07b0a2e5c6..41cfe0ba11 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -138,6 +138,19 @@ Output formats: - Global `--output ` applies to all commands - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- `add page` and `add block` return created entity ids in `data.result` for JSON/EDN output, and include ids in human output. + - Human example: + ```text + Added page: + [123] + ``` + - Human example: + ```text + Added blocks: + [201 202] + ``` + - JSON example: `{"status":"ok","data":{"result":[123]}}` + - EDN example: `{:status :ok, :data {:result [123]}}` - `doctor` output includes overall status (`ok`, `warning`, `error`) and per-check rows for `db-worker-script`, `data-dir`, and `running-servers`. For scripting, `--output json|edn` keeps the structured check payload. - Common doctor failures: - `doctor-script-missing`: `db-worker-node.js` runtime target is missing (default target: `dist/db-worker-node.js`; use `doctor --dev-script` to check `static/db-worker-node.js`). diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 6e89244c2f..90c858923b 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -99,19 +99,119 @@ (defn- ensure-block-uuids [blocks] - (mapv (fn [block] - (let [current (:block/uuid block)] - (cond - (some? current) - (update block :block/uuid (fn [value] - (if (and (string? value) (common-util/uuid-string? value)) - (uuid value) - value))) + (mapv (fn ensure-block-uuid [block] + (let [current (:block/uuid block) + block (cond + (some? current) + (update block :block/uuid (fn [value] + (if (and (string? value) (common-util/uuid-string? value)) + (uuid value) + value))) - :else - (assoc block :block/uuid (common-uuid/gen-uuid))))) + :else + (assoc block :block/uuid (common-uuid/gen-uuid)))] + (if (seq (:block/children block)) + (update block :block/children ensure-block-uuids) + block))) blocks)) +(defn- normalize-created-ids + [ids] + (->> ids + (remove nil?) + distinct + vec)) + +(defn- normalized-uuid + [value] + (cond + (uuid? value) value + (and (string? value) (common-util/uuid-string? (string/trim value))) + (uuid (string/trim value)) + :else nil)) + +(defn- collect-created-block-uuids + [blocks] + (letfn [(walk [acc block] + (let [block-uuid (normalized-uuid (:block/uuid block)) + acc (if block-uuid + (conj acc block-uuid) + acc) + children (:block/children block)] + (if (seq children) + (reduce walk acc children) + acc)))] + (->> (reduce walk [] blocks) + distinct + vec))) + +(defn- flatten-block-tree + [blocks] + (letfn [(walk [parent-uuid block] + (let [children (:block/children block) + block (cond-> (dissoc block :block/children) + parent-uuid + (assoc :block/parent [:block/uuid parent-uuid])) + block-uuid (normalized-uuid (:block/uuid block))] + (into [block] + (mapcat #(walk block-uuid %) children))))] + (vec (mapcat #(walk nil %) blocks)))) + +(defn- created-ids-in-order + [ordered-uuids entities entity-kind] + (let [id-by-uuid (reduce (fn [acc {:keys [db/id block/uuid]}] + (if-let [entity-uuid (normalized-uuid uuid)] + (if (some? id) + (assoc acc entity-uuid id) + acc) + acc)) + {} + entities) + missing-uuids (->> ordered-uuids + (remove #(contains? id-by-uuid %)) + vec)] + (when (seq missing-uuids) + (throw (ex-info "unable to resolve created ids" + {:code :add-id-resolution-failed + :entity-kind entity-kind + :missing-uuids missing-uuids}))) + (normalize-created-ids (map #(get id-by-uuid %) ordered-uuids)))) + +(defn- resolve-created-block-ids + [config repo blocks insert-result] + (let [ordered-uuids (or (seq (collect-created-block-uuids (:tx-data insert-result))) + (seq (collect-created-block-uuids blocks)) + (seq (collect-created-block-uuids (:blocks insert-result))))] + (if-not (seq ordered-uuids) + (p/rejected (ex-info "unable to resolve created block ids" + {:code :add-id-resolution-failed + :entity-kind :block + :reason :missing-created-uuids})) + (p/let [entities (p/all + (map (fn [block-uuid] + (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] [:block/uuid block-uuid]])) + ordered-uuids))] + (created-ids-in-order ordered-uuids entities :block))))) + +(defn- resolve-created-page-ids + [config repo page create-result] + (let [page-uuid (some-> create-result second normalized-uuid)] + (if page-uuid + (p/let [page-entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] [:block/uuid page-uuid]])] + (created-ids-in-order [page-uuid] [page-entity] :page)) + (p/let [page-entity (transport/invoke config :thread-api/pull false + [repo [:db/id :block/uuid] + [:block/name (common-util/page-name-sanity-lc page)]]) + page-id (:db/id page-entity)] + (if (some? page-id) + [page-id] + (throw (ex-info "unable to resolve created page id" + {:code :add-id-resolution-failed + :entity-kind :page + :page page}))))))) + (defn- extract-page-refs [title] (when (string? title) @@ -845,8 +945,7 @@ tags-result (parse-tags-option (:tags options)) properties-result (parse-properties-option (:properties options)) tags (:value tags-result) - properties (:value properties-result) - ensure-uuids? (or status (seq tags) (seq properties))] + properties (:value properties-result)] (cond (and (seq status-text) (nil? status)) {:ok? false @@ -865,9 +964,7 @@ (let [vector-result (ensure-blocks (:value blocks-result))] (if-not (:ok? vector-result) vector-result - (let [blocks (cond-> (:value vector-result) - ensure-uuids? - ensure-block-uuids)] + (let [blocks (ensure-block-uuids (:value vector-result))] {:ok? true :action {:type :add-block :repo repo @@ -921,11 +1018,12 @@ blocks (if (seq refs) (normalize-block-title-refs (:blocks action) refs) (:blocks action)) + blocks-for-insert (flatten-block-tree blocks) status (:status action) tags (resolve-tags cfg (:repo action) (:tags action)) properties (resolve-properties cfg (:repo action) (:properties action)) pos (:pos action) - keep-uuid? (or status (seq tags) (seq properties)) + keep-uuid? true opts (case pos "last-child" {:sibling? false :bottom? true} "sibling" {:sibling? true} @@ -933,11 +1031,11 @@ opts (cond-> opts keep-uuid? (assoc :keep-uuid? true)) - ops [[:insert-blocks [blocks + ops [[:insert-blocks [blocks-for-insert target-id (assoc opts :outliner-op :insert-blocks)]]] - _ (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) - block-ids (->> blocks + insert-result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) + block-ids (->> blocks-for-insert (map :block/uuid) (remove nil?) vec) @@ -963,9 +1061,10 @@ [(:repo action) [[:batch-set-property [block-ids k v {}]]] {}])) - properties)))] + properties))) + created-ids (resolve-created-block-ids cfg (:repo action) blocks-for-insert insert-result)] {:status :ok - :data {:result nil}}))) + :data {:result created-ids}}))) (defn execute-add-page [action config] @@ -977,7 +1076,7 @@ options (cond-> {} (seq properties) (assoc :properties properties)) ops [[:create-page [(:page action) options]]] - result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) + create-result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) _ (when (seq tag-ids) (p/let [page-name (common-util/page-name-sanity-lc (:page action)) page (pull-entity cfg (:repo action) [:db/id :block/uuid] [:block/name page-name]) @@ -990,6 +1089,7 @@ [(:repo action) [[:batch-set-property [[page-uuid] :block/tags tag-id {}]]] {}])) - tag-ids))))] + tag-ids)))) + created-ids (resolve-created-page-ids cfg (:repo action) (:page action) create-result)] {:status :ok - :data {:result result}}))) + :data {:result created-ids}}))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index ed5cfacb5f..f70756b0a0 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -236,12 +236,12 @@ (and host port) (conj (str "Host: " host " Port: " port)))))) (defn- format-add-block - [{:keys [repo blocks]}] - (str "Added blocks: " (count blocks) " (repo: " repo ")")) + [_context ids] + (str "Added blocks:\n" (pr-str (vec (or ids []))))) (defn- format-add-page - [{:keys [repo page]}] - (str "Added page: " page " (repo: " repo ")")) + [_context ids] + (str "Added page:\n" (pr-str (vec (or ids []))))) (defn- format-remove [{:keys [repo page uuid id ids]}] @@ -311,8 +311,8 @@ (format-server-action command data) :list-page (format-list-page (:items data) now-ms) (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) - :add-block (format-add-block context) - :add-page (format-add-page context) + :add-block (format-add-block context (:result data)) + :add-page (format-add-page context (:result data)) :remove (format-remove context) :update-block (format-update-block context) :graph-export (format-graph-export context) diff --git a/src/test/logseq/cli/command/add_test.cljs b/src/test/logseq/cli/command/add_test.cljs new file mode 100644 index 0000000000..ac10df43e7 --- /dev/null +++ b/src/test/logseq/cli/command/add_test.cljs @@ -0,0 +1,45 @@ +(ns logseq.cli.command.add-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.command.add :as add-command])) + +(deftest test-collect-created-block-uuids + (testing "collects uuids depth-first and removes duplicates" + (let [root-uuid (random-uuid) + child-uuid (random-uuid) + grandchild-uuid (random-uuid) + sibling-uuid (random-uuid) + blocks [{:block/uuid root-uuid + :block/children [{:block/uuid child-uuid} + {:block/title "without uuid" + :block/children [{:block/uuid grandchild-uuid}]}]} + {:block/uuid sibling-uuid} + {:block/uuid child-uuid}]] + (is (= [root-uuid child-uuid grandchild-uuid sibling-uuid] + (#'add-command/collect-created-block-uuids blocks)))))) + +(deftest test-created-ids-in-order + (testing "normalizes created ids in deterministic uuid order" + (let [uuid-a (random-uuid) + uuid-b (random-uuid) + uuid-c (random-uuid) + ordered-uuids [uuid-c uuid-a uuid-b] + entities [{:block/uuid uuid-a :db/id 101} + {:block/uuid uuid-b :db/id 202} + {:block/uuid uuid-c :db/id 303}]] + (is (= [303 101 202] + (#'add-command/created-ids-in-order ordered-uuids entities :block)))))) + +(deftest test-created-ids-in-order-errors-on-missing-entity + (testing "throws when any created uuid cannot be resolved to db/id" + (let [uuid-a (random-uuid) + uuid-b (random-uuid) + error (try + (#'add-command/created-ids-in-order + [uuid-a uuid-b] + [{:block/uuid uuid-a :db/id 11}] + :block) + nil + (catch :default e e))] + (is (some? error)) + (is (= :add-id-resolution-failed (-> error ex-data :code))) + (is (= [uuid-b] (-> error ex-data :missing-uuids)))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 8f8d3239f3..ba581c4a9f 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -91,14 +91,23 @@ result))))) (deftest test-human-output-add-remove - (testing "add block renders a succinct success line" + (testing "add block renders ids in two lines" (let [result (format/format-result {:status :ok :command :add-block :context {:repo "demo-repo" :blocks ["a" "b"]} - :data {:result {:ok true}}} + :data {:result [201 202]}} {:output-format nil})] - (is (= "Added blocks: 2 (repo: demo-repo)" result)))) + (is (= "Added blocks:\n[201 202]" result)))) + + (testing "add page renders ids in two lines" + (let [result (format/format-result {:status :ok + :command :add-page + :context {:repo "demo-repo" + :page "Home"} + :data {:result [123]}} + {:output-format nil})] + (is (= "Added page:\n[123]" result)))) (testing "remove page renders a succinct success line" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index dfb2eeb2fc..b8f2a1b8ba 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -184,6 +184,10 @@ (when (= title (item-title item)) item))) item-id)) +(defn- first-result-id + [payload] + (first (get-in payload [:data :result]))) + (deftest ^:long test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] @@ -302,6 +306,162 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-add-page-json-output-returns-id + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-page-json-id") + repo "add-page-json-id-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + add-page-result (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + add-page-payload (parse-json-output add-page-result) + page-ids (get-in add-page-payload [:data :result]) + page-id (first page-ids) + query-payload (run-query data-dir cfg-path repo + "[:find ?id . :in $ ?page-name :where [?id :block/name ?page-name]]" + (pr-str [(common-util/page-name-sanity-lc "Home")])) + queried-page-id (get-in query-payload [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (vector? page-ids)) + (is (= 1 (count page-ids))) + (is (number? page-id)) + (is (= page-id queried-page-id)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-add-block-json-output-returns-ids + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-block-json-ids") + repo "add-block-json-ids-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + blocks-edn (pr-str [{:block/title "Parent" + :block/children [{:block/title "Child"}]} + {:block/title "Sibling"}]) + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + _ (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + add-block-result (run-cli ["--repo" repo + "add" "block" + "--target-page-name" "Home" + "--blocks" blocks-edn] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + block-ids (get-in add-block-payload [:data :result]) + title-query-payload (run-query data-dir cfg-path repo + "[:find ?title :in $ [?id ...] :where [?id :block/title ?title]]" + (pr-str [block-ids])) + block-titles (->> (get-in title-query-payload [:data :result]) + (map first) + set) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (vector? block-ids)) + (is (= 3 (count block-ids))) + (is (= 3 (count (distinct block-ids)))) + (is (every? number? block-ids)) + (is (= #{"Parent" "Child" "Sibling"} block-titles)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-add-page-block-edn-output-returns-id + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-edn-id") + repo "add-edn-id-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + add-page-result (run-cli ["--repo" repo + "--output" "edn" + "add" "page" + "--page" "Home"] + data-dir cfg-path) + add-page-payload (parse-edn-output add-page-result) + page-ids (get-in add-page-payload [:data :result]) + add-block-result (run-cli ["--repo" repo + "--output" "edn" + "add" "block" + "--target-page-name" "Home" + "--content" "EDN block"] + data-dir cfg-path) + add-block-payload (parse-edn-output add-block-result) + block-ids (get-in add-block-payload [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-page-result))) + (is (= :ok (:status add-page-payload))) + (is (vector? page-ids)) + (is (= 1 (count page-ids))) + (is (number? (first page-ids))) + (is (= 0 (:exit-code add-block-result))) + (is (= :ok (:status add-block-payload))) + (is (vector? block-ids)) + (is (= 1 (count block-ids))) + (is (number? (first block-ids))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-add-identifiers-chain-update-remove + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-id-chain") + repo "add-id-chain-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + add-page-result (run-cli ["--repo" repo "add" "page" "--page" "ChainPage"] data-dir cfg-path) + add-page-payload (parse-json-output add-page-result) + page-id (first-result-id add-page-payload) + add-block-result (run-cli ["--repo" repo + "add" "block" + "--target-id" (str page-id) + "--content" "Chain block"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + block-id (first-result-id add-block-payload) + update-result (run-cli ["--repo" repo + "update" + "--id" (str block-id) + "--update-properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + update-payload (parse-json-output update-result) + remove-result (run-cli ["--repo" repo "remove" "--id" (str block-id)] data-dir cfg-path) + remove-payload (parse-json-output remove-result) + query-after-remove (run-query data-dir cfg-path repo + "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" + (pr-str ["Chain block"])) + removed-id (get-in query-after-remove [:data :result]) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-page-result))) + (is (= "ok" (:status add-page-payload))) + (is (number? page-id)) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (number? block-id)) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status update-payload))) + (is (= 0 (:exit-code remove-result))) + (is (= "ok" (:status remove-payload))) + (is (nil? removed-id)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-add-block-rewrites-page-ref (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] From ccbbd3c88ee99cdd25d83963f4c0ec9e5c2224bb Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 26 Feb 2026 22:59:23 +0800 Subject: [PATCH 090/375] 042-logseq-cli-add-tag-command.md --- .../042-logseq-cli-add-tag-command.md | 138 +++++++++++++++ src/main/logseq/cli/command/add.cljs | 73 +++++++- src/main/logseq/cli/commands.cljs | 15 ++ src/main/logseq/cli/format.cljs | 6 + src/test/logseq/cli/commands_test.cljs | 161 ++++++++++++++++-- src/test/logseq/cli/format_test.cljs | 9 + src/test/logseq/cli/integration_test.cljs | 105 ++++++++++++ 7 files changed, 494 insertions(+), 13 deletions(-) create mode 100644 docs/agent-guide/042-logseq-cli-add-tag-command.md diff --git a/docs/agent-guide/042-logseq-cli-add-tag-command.md b/docs/agent-guide/042-logseq-cli-add-tag-command.md new file mode 100644 index 0000000000..f5d3656573 --- /dev/null +++ b/docs/agent-guide/042-logseq-cli-add-tag-command.md @@ -0,0 +1,138 @@ +# Logseq CLI Add Tag Subcommand Implementation Plan + +Goal: Add `logseq add tag` so CLI users can create a tag entity before using that tag in `add block`, `add page`, and `update`. + +Architecture: Reuse the existing CLI to db-worker-node write path by sending `:create-page` with `{:class? true}` through `:thread-api/apply-outliner-ops`. +Architecture: Keep db-worker-node protocol and HTTP endpoints unchanged because the feature composes existing worker operations. +Architecture: Validate the result after write by pulling the page entity and asserting the page has `:logseq.class/Tag` semantics. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, db-worker-node, outliner ops. + +Related: Builds on docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md and docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md. + +## Problem statement + +Current CLI behavior can attach only existing tags because `resolve-tag-entity` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` fails with `:tag-not-found` when a tag is missing. + +Current CLI behavior has no `add tag` command entry in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, so users cannot create custom tags from CLI. + +Current db-worker-node flow already supports page and class creation through `:thread-api/apply-outliner-ops` and `:create-page`, so the missing capability is command orchestration in CLI instead of worker transport. + +`list tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` returns entities tagged with `:logseq.class/Tag`, so `add tag` must create that exact class shape instead of a plain page. + +## Testing Plan + +I will follow `@test-driven-development` and write parsing tests before changing command implementation code. + +I will add failing action-building tests that verify required options, normalized action payload, and explicit errors for invalid input. + +I will add failing execution tests that stub transport calls and verify emitted outliner ops include `:create-page` with `{:class? true}`. + +I will add failing format tests for human output so `:add-tag` has a stable success message contract. + +I will add one integration test that creates a new tag and then uses that tag in `add block` to prove end to end behavior through db-worker-node. + +I will add one integration test that confirms failure when the same title already exists as a non-tag page, so the command does not report false success. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +This plan adds only `add tag` under the existing `add` command group. + +This plan does not add new db-worker-node endpoints or thread-api methods. + +This plan does not add editing features such as setting class extends, class properties, or tag description during creation. + +This plan does not change existing `add block`, `add page`, or `update` option syntax. + +`add tag` accepts `--name` only, and does not support `--tag` alias. + +## Integration overview + +``` +logseq add tag --name "Quote" + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> db-worker-node /v1/invoke + -> :thread-api/apply-outliner-ops + -> :create-page [title {:class? true}] + -> Datascript entity tagged as :logseq.class/Tag +``` + +## Detailed implementation plan + +1. Add a failing parser help test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that expects `add` group help to include `add tag`. +2. Add a failing parser test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `["add" "tag" "--name" "Quote"]` to produce `:add-tag`. +3. Add a failing parser validation test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that missing `--name` returns `:missing-tag-name`. +4. Add a failing build-action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that ensures `:add-tag` action contains `:type`, `:repo`, `:graph`, and normalized `:name`. +5. Add a failing execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that stubs transport and verifies `:create-page` options include `:class? true`. +6. Add a failing execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that simulates existing non-tag page conflict and expects a deterministic CLI error code. +7. Add a failing format test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for `:add-tag` human output. +8. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that runs `add tag`, validates `list tag` contains the new tag, and confirms `add block --tags` with that tag succeeds. +9. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for duplicate title where a normal page exists and command must fail with clear error. +10. Run focused tests and confirm all new tests fail for behavior reasons, and use `@clojure-debug` only if failures are caused by test setup mistakes. +11. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` with a new `add-tag` command spec, entry, action builder, and executor. +12. In `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, implement execution via `:thread-api/apply-outliner-ops` and `[:create-page [name {:class? true}]]`. +13. In `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, add a post-create pull check that verifies the resulting entity is class-tagged and raise an explicit error if not. +14. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` parse validation with `:missing-tag-name` handling for `:add-tag`. +15. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` action dispatch and execute dispatch for `:add-tag`. +16. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` context propagation to include the new tag field used by formatter output. +17. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with `format-add-tag` and command routing for `:add-tag`. +18. Run focused unit and integration tests and confirm they pass without changing unrelated command behavior. +19. Run `bb dev:lint-and-test` and confirm the repository remains green after the feature. +20. Refactor only local helper naming and shared logic inside `add.cljs` while preserving behavior and keeping tests green. + +## Edge cases to cover + +Tag names with leading `#` should be normalized consistently, or rejected consistently, with one documented behavior. + +Tag names containing namespace separators like `A/B` should produce deterministic behavior aligned with existing page creation rules. + +Duplicate tag creation should be idempotent when the existing entity is already a tag class. + +If a page with the same name exists but is not a tag class, `add tag` should fail with a dedicated error instead of silently succeeding. + +Built-in tag names should remain valid and should return existing ids without creating duplicate entities. + +The command should reject blank names after trim. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` | New `add tag` appears in add group help assertions. | +| `bb dev:test -v logseq.cli.commands-test/test-verb-subcommand-parse-add` | New parse and validation tests for `add tag` pass. | +| `bb dev:test -v logseq.cli.commands-test/test-build-action-inspect-edit` | Build action includes `:add-tag` cases and passes. | +| `bb dev:test -v logseq.cli.commands-test/test-execute-add-tag-builds-create-page-op` | Outliner op assertions pass with `{:class? true}`. | +| `bb dev:test -v logseq.cli.format-test/test-human-output-add-remove` | Human output for `:add-tag` matches expected string. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-add-tag-create-and-use` | End to end creation and usage of a new tag passes. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-add-tag-rejects-existing-non-tag-page` | Conflict behavior test passes with explicit error. | +| `bb dev:lint-and-test` | Full lint and unit suite pass. | + +## Testing Details + +The tests validate user-visible behavior at parser, action, executor, formatter, and integration boundaries. + +The tests assert command success and failure contracts, and they verify persisted graph behavior with CLI reads such as `list tag` and `show`. + +The tests avoid asserting implementation details that are not externally observable. + +## Implementation Details + +- Add `add tag` spec and entry in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`. +- Add `build-add-tag-action` and `execute-add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`. +- Use existing server bootstrap and transport invoke path from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs`. +- Create tag through `:create-page` with `{:class? true}` and verify resulting entity semantics. +- Add parse validation and missing error mapping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Add build and execute routing for `:add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Add human formatter branch for `:add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +- Add parser and executor unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +- Add integration coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. + +## Question + +No open questions. + +--- diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 90c858923b..b8017988ab 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -34,9 +34,13 @@ :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."}}) +(def ^:private add-tag-spec + {:name {:desc "Tag name"}}) + (def entries [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) - (core/command-entry ["add" "page"] :add-page "Create page" add-page-spec)]) + (core/command-entry ["add" "page"] :add-page "Create page" add-page-spec) + (core/command-entry ["add" "tag"] :add-tag "Create tag" add-tag-spec)]) (defn- today-page-title [config repo] @@ -1007,6 +1011,43 @@ :error {:code :missing-page-name :message "page name is required"}})))) +(defn- normalize-tag-name-option + [value] + (let [normalized (normalize-tag-value value)] + (when (string? normalized) + (let [name (string/trim normalized)] + (when (seq name) + name))))) + +(defn build-add-tag-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for add"}} + (let [name (normalize-tag-name-option (:name options))] + (if (seq name) + {:ok? true + :action {:type :add-tag + :repo repo + :graph (core/repo->graph repo) + :name name}} + {:ok? false + :error {:code :missing-tag-name + :message "tag name is required"}})))) + +(defn- pull-page-by-name + [config repo page-name] + (pull-entity config repo + [:db/id :block/name :block/title :block/uuid + {:block/tags [:db/id :db/ident :block/name :block/title]}] + [:block/name (common-util/page-name-sanity-lc page-name)])) + +(defn- tag-entity? + [entity] + (some #(= :logseq.class/Tag (:db/ident %)) + (:block/tags entity))) + (defn execute-add-block [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -1093,3 +1134,33 @@ created-ids (resolve-created-page-ids cfg (:repo action) (:page action) create-result)] {:status :ok :data {:result created-ids}}))) + +(defn execute-add-tag + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + existing (pull-page-by-name cfg (:repo action) (:name action)) + existing-id (:db/id existing) + _ (when (and existing-id (not (tag-entity? existing))) + (throw (ex-info "tag already exists as a page and is not a tag" + {:code :tag-name-conflict + :name (:name action)}))) + _ (when-not existing-id + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:create-page [(:name action) {:class? true}]]] + {}])) + page (or (when existing-id existing) + (pull-page-by-name cfg (:repo action) (:name action))) + page-id (:db/id page) + _ (when-not page-id + (throw (ex-info "tag not found after create" + {:code :tag-not-found + :name (:name action)}))) + _ (when-not (tag-entity? page) + (throw (ex-info "created entity is not tagged as :logseq.class/Tag" + {:code :tag-create-not-tag + :name (:name action) + :id page-id}))) + created-ids (normalize-created-ids [page-id])] + {:status :ok + :data {:result created-ids}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 3e8a778dcb..b5668c728f 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -62,6 +62,13 @@ :message "page name is required"} :summary summary}) +(defn- missing-tag-name-result + [summary] + {:ok? false + :error {:code :missing-tag-name + :message "tag name is required"} + :summary summary}) + (defn- missing-type-result [summary] {:ok? false @@ -167,6 +174,9 @@ (and (= command :add-page) (not (seq (:page opts)))) (missing-page-name-result summary) + (and (= command :add-tag) (not (seq (some-> (:name opts) string/trim)))) + (missing-tag-name-result summary) + (and (= command :remove) (seq args)) (command-core/invalid-options-result summary "remove does not accept subcommands") @@ -362,6 +372,9 @@ :add-page (add-command/build-add-page-action options repo) + :add-tag + (add-command/build-add-tag-action options repo) + :update-block (update-command/build-action options repo) @@ -410,6 +423,7 @@ :list-property (list-command/execute-list-property action config) :add-block (add-command/execute-add-block action config) :add-page (add-command/execute-add-page action config) + :add-tag (add-command/execute-add-tag action config) :update-block (update-command/execute-update action config) :remove (remove-command/execute-remove action config) :query (query-command/execute-query action config) @@ -427,6 +441,7 @@ (assoc result :command (or (:command action) (:type action)) :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks + :name :source :target :update-tags :update-properties :remove-tags :remove-properties :export-type :output :import-type :input]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index f70756b0a0..3ff9292e45 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -90,6 +90,7 @@ :missing-graph "Use --repo " :missing-repo "Use --repo " :missing-content "Use --content or pass content as args" + :missing-tag-name "Use --name " :missing-query "Use --query " :unknown-query "Use `logseq query list` to see available queries" :data-dir-permission "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR" @@ -243,6 +244,10 @@ [_context ids] (str "Added page:\n" (pr-str (vec (or ids []))))) +(defn- format-add-tag + [_context ids] + (str "Added tag:\n" (pr-str (vec (or ids []))))) + (defn- format-remove [{:keys [repo page uuid id ids]}] (cond @@ -313,6 +318,7 @@ (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) :add-block (format-add-block context (:result data)) :add-page (format-add-page context (:result data)) + :add-tag (format-add-tag context (:result data)) :remove (format-remove context) :update-block (format-update-block context) :graph-export (format-graph-export context) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 0443c99ce1..252997f9e9 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -73,6 +73,7 @@ (is (contains-bold? summary "list property")) (is (contains-bold? summary "add block")) (is (contains-bold? summary "add page")) + (is (contains-bold? summary "add tag")) (is (contains-bold? summary "remove")) (is (contains-bold? summary "update")) (is (contains-bold? summary "query")) @@ -133,17 +134,6 @@ (is (string/includes? plain-summary "Global options:")) (is (string/includes? plain-summary "Command options:")))) - (testing "add group shows subcommands" - (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["add"])) - summary (:summary result) - plain-summary (strip-ansi summary)] - (is (true? (:help? result))) - (is (string/includes? plain-summary "add block")) - (is (string/includes? plain-summary "add page")) - (is (contains-bold? summary "add block")) - (is (contains-bold? summary "add page")))) - (testing "remove command shows help" (let [result (binding [style/*color-enabled?* true] (commands/parse-args ["remove" "--help"])) @@ -202,6 +192,20 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-help-add-group + (testing "add group shows subcommands" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["add"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "add block")) + (is (string/includes? plain-summary "add page")) + (is (string/includes? plain-summary "add tag")) + (is (contains-bold? summary "add block")) + (is (contains-bold? summary "add page")) + (is (contains-bold? summary "add tag"))))) + (deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" (let [result (binding [style/*color-enabled?* true] @@ -967,7 +971,23 @@ (is (true? (:ok? result))) (is (= :add-page (:command result))) (is (= "[\"TagA\"]" (get-in result [:options :tags]))) - (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties])))))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) + + (testing "add tag requires name" + (let [result (commands/parse-args ["add" "tag"])] + (is (false? (:ok? result))) + (is (= :missing-tag-name (get-in result [:error :code]))))) + + (testing "add tag parses with name" + (let [result (commands/parse-args ["add" "tag" "--name" "Quote"])] + (is (true? (:ok? result))) + (is (= :add-tag (:command result))) + (is (= "Quote" (get-in result [:options :name]))))) + + (testing "add tag rejects blank name" + (let [result (commands/parse-args ["add" "tag" "--name" " "])] + (is (false? (:ok? result))) + (is (= :missing-tag-name (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-update-target-page (testing "update parses with target page" @@ -1212,6 +1232,22 @@ (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) + (testing "add tag requires name" + (let [parsed {:ok? true :command :add-tag :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-tag-name (get-in result [:error :code]))))) + + (testing "add tag builds normalized action" + (let [parsed {:ok? true :command :add-tag :options {:name " #Quote "}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= {:type :add-tag + :repo "logseq_db_demo" + :graph "demo" + :name "Quote"} + (:action result))))) + (testing "remove requires target" (let [parsed {:ok? true :command :remove :options {}} result (commands/build-action parsed {:repo "demo"})] @@ -1350,6 +1386,107 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-execute-add-tag-builds-create-page-op + (async done + (let [ops* (atom nil) + created?* (atom false) + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :add-tag + :repo "demo" + :name "Quote"}] + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "quote"]) + (if @created?* + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {}) + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! created?* true) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (add-command/execute-add-tag action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= [[:create-page ["Quote" {:class? true}]]] + @ops*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-add-tag-rejects-existing-non-tag-page + (async done + (let [action {:type :add-tag + :repo "demo" + :name "Home"} + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "home"]) + {:db/id 99 + :block/name "home" + :block/title "Home" + :block/tags [{:db/ident :logseq.class/Page}]} + {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not create tag" {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (add-command/execute-add-tag action {}) + (p/then (fn [_] + (is false "expected add tag conflict error"))) + (p/catch (fn [e] + (is (= :tag-name-conflict (:code (ex-data e)))))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-add-tag-idempotent-when-tag-exists + (async done + (let [apply-calls* (atom 0) + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :add-tag + :repo "demo" + :name "Quote"}] + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "quote"]) + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (add-command/execute-add-tag action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-update-builds-batch-ops (async done (let [ops* (atom nil) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index ba581c4a9f..d6b0601446 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -109,6 +109,15 @@ {:output-format nil})] (is (= "Added page:\n[123]" result)))) + (testing "add tag renders ids in two lines" + (let [result (format/format-result {:status :ok + :command :add-tag + :context {:repo "demo-repo" + :name "Quote"} + :data {:result [321]}} + {:output-format nil})] + (is (= "Added tag:\n[321]" result)))) + (testing "remove page renders a succinct success line" (let [result (format/format-result {:status :ok :command :remove diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index b8f2a1b8ba..36f9aa0ed9 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -857,6 +857,111 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-add-tag-create-and-use + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-create")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "add-tag-create-graph" + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + _ (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + add-tag-result (run-cli ["--repo" repo + "add" "tag" + "--name" "CliQuote"] + data-dir cfg-path) + add-tag-payload (parse-json-output add-tag-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + tag-names (->> (get-in list-tag-payload [:data :items]) + (map #(or (:block/title %) (:title %) (:name %))) + set) + add-block-result (run-cli ["--repo" repo + "add" "block" + "--target-page-name" "Home" + "--content" "Tagged by add tag" + "--tags" "[\"CliQuote\"]"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + block-tag-names (query-tags data-dir cfg-path repo "Tagged by add tag") + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code add-tag-result)) + (pr-str (:error add-tag-payload))) + (is (= "ok" (:status add-tag-payload))) + (is (= "ok" (:status list-tag-payload))) + (is (contains? tag-names "CliQuote")) + (is (= 0 (:exit-code add-block-result)) + (pr-str (:error add-block-payload))) + (is (= "ok" (:status add-block-payload))) + (is (contains? block-tag-names "CliQuote")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-add-tag-rejects-existing-non-tag-page + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-conflict")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "add-tag-conflict-graph" + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + _ (run-cli ["--repo" repo "add" "page" "--page" "ConflictPage"] data-dir cfg-path) + add-tag-result (run-cli ["--repo" repo + "add" "tag" + "--name" "ConflictPage"] + data-dir cfg-path) + add-tag-payload (parse-json-output add-tag-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + tag-names (->> (get-in list-tag-payload [:data :items]) + (map #(or (:block/title %) (:title %) (:name %))) + set) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 1 (:exit-code add-tag-result))) + (is (= "error" (:status add-tag-payload))) + (is (string/includes? (get-in add-tag-payload [:error :message]) + "already exists as a page and is not a tag")) + (is (not (contains? tag-names "ConflictPage"))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-add-tag-idempotent-existing-tag + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-idempotent")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "add-tag-idempotent-graph" + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + first-add-result (run-cli ["--repo" repo "add" "tag" "--name" "StableTag"] + data-dir cfg-path) + first-add-payload (parse-json-output first-add-result) + second-add-result (run-cli ["--repo" repo "add" "tag" "--name" "StableTag"] + data-dir cfg-path) + second-add-payload (parse-json-output second-add-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + stable-tags (->> (get-in list-tag-payload [:data :items]) + (filter #(= "StableTag" (or (:block/title %) (:title %) (:name %))))) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code first-add-result))) + (is (= "ok" (:status first-add-payload))) + (is (= 0 (:exit-code second-add-result))) + (is (= "ok" (:status second-add-payload))) + (is (= 1 (count stable-tags))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-query (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query") From dfa80b084eba84c3f451476af92232e213622803 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 27 Feb 2026 12:21:24 -0500 Subject: [PATCH 091/375] fix: common lint and outdated lockfile --- deps/common/src/logseq/common/config.cljs | 8 ++--- yarn.lock | 41 +++++------------------ 2 files changed, 12 insertions(+), 37 deletions(-) diff --git a/deps/common/src/logseq/common/config.cljs b/deps/common/src/logseq/common/config.cljs index 565a05feca..da4fe27bd6 100644 --- a/deps/common/src/logseq/common/config.cljs +++ b/deps/common/src/logseq/common/config.cljs @@ -46,10 +46,10 @@ [s] (when (seq s) (let [s (str s) - stripped (loop [name s] - (if (string/starts-with? name db-version-prefix) - (recur (subs name (count db-version-prefix))) - name))] + stripped (loop [name' s] + (if (string/starts-with? name' db-version-prefix) + (recur (subs name' (count db-version-prefix))) + name'))] (str db-version-prefix stripped)))) (defonce local-assets-dir "assets") diff --git a/yarn.lock b/yarn.lock index 6c832ce757..a7ddebee12 100644 --- a/yarn.lock +++ b/yarn.lock @@ -9630,7 +9630,7 @@ streamx@^2.15.0, streamx@^2.21.0: optionalDependencies: bare-events "^2.2.0" -"string-width-cjs@npm:string-width@^4.2.0": +"string-width-cjs@npm:string-width@^4.2.0", "string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: version "4.2.3" resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== @@ -9648,15 +9648,6 @@ string-width@^1.0.1, string-width@^1.0.2: is-fullwidth-code-point "^1.0.0" strip-ansi "^3.0.0" -"string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: - version "4.2.3" - resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" - integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== - dependencies: - emoji-regex "^8.0.0" - is-fullwidth-code-point "^3.0.0" - strip-ansi "^6.0.1" - string-width@^2.0.0, string-width@^2.1.1: version "2.1.1" resolved "https://registry.yarnpkg.com/string-width/-/string-width-2.1.1.tgz#ab93f27a8dc13d28cac815c462143a6d9012ae9e" @@ -9730,7 +9721,7 @@ string_decoder@~1.1.1: dependencies: safe-buffer "~5.1.0" -"strip-ansi-cjs@npm:strip-ansi@^6.0.1": +"strip-ansi-cjs@npm:strip-ansi@^6.0.1", strip-ansi@^6.0.0, strip-ansi@^6.0.1: version "6.0.1" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== @@ -9751,13 +9742,6 @@ strip-ansi@^4.0.0: dependencies: ansi-regex "^3.0.0" -strip-ansi@^6.0.0, strip-ansi@^6.0.1: - version "6.0.1" - resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" - integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== - dependencies: - ansi-regex "^5.0.1" - strip-ansi@^7.0.1: version "7.1.0" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-7.1.0.tgz#d5b6568ca689d8561370b0707685d22434faff45" @@ -10995,7 +10979,7 @@ wordwrap@^1.0.0: resolved "https://registry.yarnpkg.com/wordwrap/-/wordwrap-1.0.0.tgz#27584810891456a4171c8d0226441ade90cbcaeb" integrity sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q== -"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0": +"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0", wrap-ansi@^7.0.0: version "7.0.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== @@ -11021,15 +11005,6 @@ wrap-ansi@^6.2.0: string-width "^4.1.0" strip-ansi "^6.0.0" -wrap-ansi@^7.0.0: - version "7.0.0" - resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" - integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== - dependencies: - ansi-styles "^4.0.0" - string-width "^4.1.0" - strip-ansi "^6.0.0" - wrap-ansi@^8.1.0: version "8.1.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-8.1.0.tgz#56dc22368ee570face1b49819975d9b9a5ead214" @@ -11054,16 +11029,16 @@ write-file-atomic@^3.0.3: signal-exit "^3.0.2" typedarray-to-buffer "^3.1.5" -ws@8.19.0: - version "8.19.0" - resolved "https://registry.yarnpkg.com/ws/-/ws-8.19.0.tgz#ddc2bdfa5b9ad860204f5a72a4863a8895fd8c8b" - integrity sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg== - ws@^7.4.6: version "7.5.10" resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9" integrity sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ== +ws@^8.19.0: + version "8.19.0" + resolved "https://registry.yarnpkg.com/ws/-/ws-8.19.0.tgz#ddc2bdfa5b9ad860204f5a72a4863a8895fd8c8b" + integrity sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg== + ws@~8.17.1: version "8.17.1" resolved "https://registry.yarnpkg.com/ws/-/ws-8.17.1.tgz#9293da530bb548febc95371d90f9c878727d919b" From 85ebeb976e95e5de63dfcd8564f82d4893753767 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 1 Mar 2026 20:46:20 +0800 Subject: [PATCH 092/375] 043-logseq-cli-tag-property-management.md --- .../043-logseq-cli-tag-property-management.md | 182 ++++++++ src/main/logseq/cli/command/add.cljs | 73 +--- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/remove.cljs | 377 +++++++++++++--- src/main/logseq/cli/command/upsert.cljs | 227 ++++++++++ src/main/logseq/cli/commands.cljs | 61 ++- src/main/logseq/cli/format.cljs | 55 ++- src/test/logseq/cli/commands_test.cljs | 408 ++++++++++++++---- src/test/logseq/cli/format_test.cljs | 65 ++- src/test/logseq/cli/integration_test.cljs | 145 +++++-- 10 files changed, 1319 insertions(+), 276 deletions(-) create mode 100644 docs/agent-guide/043-logseq-cli-tag-property-management.md create mode 100644 src/main/logseq/cli/command/upsert.cljs diff --git a/docs/agent-guide/043-logseq-cli-tag-property-management.md b/docs/agent-guide/043-logseq-cli-tag-property-management.md new file mode 100644 index 0000000000..906a892bcb --- /dev/null +++ b/docs/agent-guide/043-logseq-cli-tag-property-management.md @@ -0,0 +1,182 @@ +# Logseq CLI Tag and Property Management Implementation Plan + +Goal: Add first class CLI support for `upsert tag`, `upsert property`, `remove tag`, and `remove property`, and restructure existing remove behavior into `remove block` and `remove page`. + +Architecture: Replace current `add tag` command entry with `upsert tag` so tag creation and idempotent update semantics are unified under one verb. +Architecture: Expand `remove` into typed subcommands (`block`, `page`, `tag`, `property`) to make deletion intent explicit and prevent mixed selector ambiguity. +Architecture: Reuse `:thread-api/apply-outliner-ops` in db-worker-node so no new HTTP route or transport protocol is introduced. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, db-worker-node, outliner ops. + +Related: Builds on docs/agent-guide/042-logseq-cli-add-tag-command.md and docs/agent-guide/029-logseq-cli-show-properties.md. + +## Problem statement + +Current CLI supports tag creation through `add tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, but does not expose an `upsert` verb for tags and properties. + +Current `update` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` updates tag and property values on blocks, but does not upsert or remove tag/property entities. + +Current `remove` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` mixes block and page deletion under one command, and has no explicit tag/property deletion path. + +db-worker-node already supports required mutation primitives through `:thread-api/apply-outliner-ops` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and outliner ops in `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/op.cljs`. + +The implementation gap is CLI command surface design, parser wiring, validation, and output contract coverage. + +## Testing Plan + +I will follow `@test-driven-development` and write failing parser, action, executor, formatter, and integration tests before adding behavior. + +I will add parser tests for the new `upsert` verb and the new typed `remove` subcommands. + +I will add action builder tests that verify repo propagation, normalized names, schema coercion, and typed action payloads for each new command. + +I will add executor tests that stub transport and assert exact outliner ops emitted for each command path. + +I will add formatter tests for human and json output so command responses are stable for scripts. + +I will add integration tests against db-worker-node that verify graph state changes through `list`, `show`, and entity queries. + +I will use `@clojure-debug` only when failures are caused by test setup rather than behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Command contract + +The final command surface is top level verbs without a `manage` group. + +| Command | Required options | Optional options | Behavior | +| --- | --- | --- | --- | +| `logseq upsert tag` | `--name` | none in v1 | Creates tag when missing, returns existing tag when already present, and errors if same title exists as non-tag page. | +| `logseq upsert property` | `--name` | `--type`, `--cardinality`, `--hide`, `--public` | Creates property when missing, updates schema for existing property, and validates type or cardinality compatibility. | +| `logseq remove tag` | one of `--name`, `--id` | none | Deletes a tag entity after validating target type and removability, and fails with a candidate list when `--name` matches multiple tags. | +| `logseq remove property` | one of `--name`, `--id` | none | Deletes a property entity after validating target type and removability, and fails with a candidate list when `--name` matches multiple properties. | +| `logseq remove block` | one of `--id`, `--uuid` | none | Deletes one or more blocks with existing block remove behavior. | +| `logseq remove page` | `--name` | none | Deletes a page with existing page remove behavior. | + +`add tag` will be removed from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and replaced by `upsert tag`. + +Bare `remove` without a subcommand will be rejected with a clear parse error. + +No compatibility aliases or deprecation shims will be kept for removed commands. + +## Integration overview + +```text +logseq upsert property --name "owner" --type node --cardinality many + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs (/v1/invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs (:thread-api/apply-outliner-ops) + -> /Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/op.cljs (:upsert-property) +``` + +```text +logseq remove tag --name "Quote" + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs + -> remove tag resolver validates :logseq.class/Tag + -> :thread-api/apply-outliner-ops with [:delete-page [tag-uuid]] + -> outliner page delete flow applies class cleanup and persistence +``` + +## Detailed implementation plan + +1. Add failing parse help tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that expect top level `upsert` and `remove` subcommands (`block`, `page`, `tag`, `property`). +2. Add failing parse tests for `['upsert' 'tag' '--name' 'Quote']` and `['upsert' 'property' '--name' 'owner' '--type' 'node' '--cardinality' 'many']` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +3. Add failing parse tests for `['remove' 'tag' '--name' 'Quote']` and `['remove' 'property' '--id' '123']` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +4. Add failing parse tests for `['remove' 'block' '--id' '1']` and `['remove' 'page' '--name' 'Home']` to preserve old delete behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +5. Add failing parse validation tests that reject bare `remove` and old `add tag` command usage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +6. Add failing parse validation tests for invalid property type and cardinality values in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +7. Add failing build action tests for `upsert tag` name normalization and `#` prefix stripping in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +8. Add failing build action tests for `upsert property` schema coercion to `:logseq.property/type` and `:db/cardinality` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +9. Add failing executor tests that `upsert tag` emits `[:create-page [name {:class? true}]]` only when the tag is missing in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +10. Add failing executor tests that `upsert tag` is idempotent for existing tag entities and rejects non-tag title conflicts in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +11. Add failing executor tests that `upsert property` emits `[:upsert-property [property-id schema opts]]` and passes `{:property-name name}` when creating a new property in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +12. Add failing executor tests that `remove tag` and `remove property` resolve entities by `--name` or `--id` and emit `[:delete-page [uuid]]` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +13. Add failing executor tests that `remove tag --name` and `remove property --name` fail on multiple matches, return all matched candidates in output, and require rerun with `--id` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +14. Add failing executor tests that built in or hidden tag or property targets are rejected with explicit error codes in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +15. Add failing formatter tests for `upsert tag`, `upsert property`, `remove tag`, and `remove property` outputs in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +16. Add failing formatter tests that `remove block` and `remove page` outputs remain backward compatible in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +17. Add failing integration tests for `upsert tag` create and idempotent behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +18. Add failing integration tests for `upsert property` create and schema update behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +19. Add failing integration tests for `remove tag` and `remove property` ensuring entities disappear from `list tag` and `list property` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +20. Add failing integration tests for `remove tag --name` and `remove property --name` ambiguous matches to assert candidate list output and `--id` guidance in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +21. Add failing integration tests for `remove block` and `remove page` command migration behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +22. Run focused tests and confirm all new tests fail for behavior reasons before implementation. +23. Create `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` with command specs, entries, validation helpers, action builders, and executors for tag and property upsert. +24. Refactor shared tag resolver helpers from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` into `upsert.cljs` or a shared helper namespace to avoid duplication. +25. Remove `add tag` command entry and related build or execute dispatch from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +26. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` to register `remove block`, `remove page`, `remove tag`, and `remove property` entries. +27. Keep existing block and page delete execution logic and map it behind `remove block` and `remove page` subcommands. +28. Implement new remove tag and remove property resolver and execution paths in `remove.cljs` using `:delete-page` outliner op after entity type validation and ambiguity failure behavior for `--name`. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` table, parse validation, action builder, context propagation, and execute dispatch for new upsert and remove contracts. +30. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to format results for `upsert tag`, `upsert property`, `remove tag`, and `remove property`, including ambiguous candidate lists. +31. Run focused unit and integration tests, then run `bb dev:lint-and-test`, and keep only behavior preserving refactors. +32. Update CLI help text snapshots and any docs references to remove `add tag` and bare `remove` usage. + +## Edge cases to cover + +Tag names with leading `#` should normalize consistently in `upsert tag`. + +Tag and property names that collide by title or case must fail resolution for `--name`, list all candidate matches, and require explicit `--id`. + +Remove selectors must reject ambiguous name matches with a clear error, include all candidate ids and names in output, and require `--id`. + +Built in tags and built in properties must not be removable. + +Property type and cardinality updates must reject invalid transitions already enforced by outliner validation. + +`remove block` multi id behavior must stay unchanged from current implementation. + +`remove page` must preserve current not found and built in page deletion behavior. + +Commands must reject blank names after trim. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` | Help output includes `upsert` and typed `remove` subcommands. | +| `bb dev:test -v logseq.cli.commands-test/test-verb-subcommand-parse-upsert-remove` | Parse and validation tests for new command contracts pass. | +| `bb dev:test -v logseq.cli.commands-test/test-build-action-upsert-remove` | Action payload tests pass for all new paths. | +| `bb dev:test -v logseq.cli.commands-test/test-execute-upsert-remove-tag-property` | Outliner op emission tests pass for upsert and entity removal flows. | +| `bb dev:test -v logseq.cli.format-test/test-human-output-upsert-remove` | Human output formatting for new commands is stable. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-and-remove-tag-property` | End to end behavior through db-worker-node passes for all four new commands. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-remove-block-page-subcommands` | Block and page deletion still work through new remove subcommands. | +| `bb dev:lint-and-test` | Full lint and unit suite pass. | + +## Migration and compatibility + +No db-worker-node route migration is required because this feature reuses `/v1/invoke` and existing thread-api methods. + +No schema migration is required because tag and property entities are created and deleted through existing outliner operations. + +This is a CLI breaking change because `add tag` is removed and bare `remove` is replaced by typed `remove block` and `remove page` commands. + +This breaking change will be applied directly with no compatibility alias period. + +## Testing Details + +Tests validate parser, action, execution, formatter, and integration behavior for `upsert tag`, `upsert property`, `remove tag`, and `remove property`. + +Tests also validate migration behavior for `remove block` and `remove page` so previous block and page deletion semantics are preserved. + +Tests focus on command contracts and graph outcomes instead of helper internals. + +## Implementation Details + +- Add new upsert command module at `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Register upsert entries and dispatch hooks in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Remove `add tag` command entry and related dispatch from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Refactor remove command entries in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` to `remove block`, `remove page`, `remove tag`, and `remove property`. +- Reuse `:thread-api/apply-outliner-ops` with `:create-page`, `:upsert-property`, and `:delete-page` for all entity mutations. +- Add formatter branches in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` for new commands. +- Add parser and executor unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +- Add end to end coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +- Use `@clojure-debug` only when failures indicate fixture or async wiring issues. + +## Question + +No open questions. + +--- diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index b8017988ab..90c858923b 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -34,13 +34,9 @@ :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."}}) -(def ^:private add-tag-spec - {:name {:desc "Tag name"}}) - (def entries [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) - (core/command-entry ["add" "page"] :add-page "Create page" add-page-spec) - (core/command-entry ["add" "tag"] :add-tag "Create tag" add-tag-spec)]) + (core/command-entry ["add" "page"] :add-page "Create page" add-page-spec)]) (defn- today-page-title [config repo] @@ -1011,43 +1007,6 @@ :error {:code :missing-page-name :message "page name is required"}})))) -(defn- normalize-tag-name-option - [value] - (let [normalized (normalize-tag-value value)] - (when (string? normalized) - (let [name (string/trim normalized)] - (when (seq name) - name))))) - -(defn build-add-tag-action - [options repo] - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for add"}} - (let [name (normalize-tag-name-option (:name options))] - (if (seq name) - {:ok? true - :action {:type :add-tag - :repo repo - :graph (core/repo->graph repo) - :name name}} - {:ok? false - :error {:code :missing-tag-name - :message "tag name is required"}})))) - -(defn- pull-page-by-name - [config repo page-name] - (pull-entity config repo - [:db/id :block/name :block/title :block/uuid - {:block/tags [:db/id :db/ident :block/name :block/title]}] - [:block/name (common-util/page-name-sanity-lc page-name)])) - -(defn- tag-entity? - [entity] - (some #(= :logseq.class/Tag (:db/ident %)) - (:block/tags entity))) - (defn execute-add-block [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -1134,33 +1093,3 @@ created-ids (resolve-created-page-ids cfg (:repo action) (:page action) create-result)] {:status :ok :data {:result created-ids}}))) - -(defn execute-add-tag - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - existing (pull-page-by-name cfg (:repo action) (:name action)) - existing-id (:db/id existing) - _ (when (and existing-id (not (tag-entity? existing))) - (throw (ex-info "tag already exists as a page and is not a tag" - {:code :tag-name-conflict - :name (:name action)}))) - _ (when-not existing-id - (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) - [[:create-page [(:name action) {:class? true}]]] - {}])) - page (or (when existing-id existing) - (pull-page-by-name cfg (:repo action) (:name action))) - page-id (:db/id page) - _ (when-not page-id - (throw (ex-info "tag not found after create" - {:code :tag-not-found - :name (:name action)}))) - _ (when-not (tag-entity? page) - (throw (ex-info "created entity is not tagged as :logseq.class/Tag" - {:code :tag-create-not-tag - :name (:name action) - :id page-id}))) - created-ids (normalize-created-ids [page-id])] - {:status :ok - :data {:result created-ids}}))) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 351b7e06f6..cb9ac273d3 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -93,7 +93,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "remove" "update" "query" "show"}} + :commands #{"list" "add" "upsert" "remove" "update" "query" "show"}} {:title "Graph Management" :commands #{"graph" "server" "doctor"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/remove.cljs b/src/main/logseq/cli/command/remove.cljs index 3bbc3d82c3..79ec672627 100644 --- a/src/main/logseq/cli/command/remove.cljs +++ b/src/main/logseq/cli/command/remove.cljs @@ -8,27 +8,66 @@ [logseq.common.util :as common-util] [promesa.core :as p])) -(def ^:private remove-spec +(def ^:private remove-block-spec {:id {:desc "Block db/id or EDN vector of ids"} - :uuid {:desc "Block UUID"} - :page {:desc "Page name"}}) + :uuid {:desc "Block UUID"}}) + +(def ^:private remove-page-spec + {:name {:desc "Page name"}}) + +(def ^:private remove-entity-spec + {:id {:desc "Entity db/id" + :coerce :long} + :name {:desc "Entity name"}}) (def entries - [(core/command-entry ["remove"] :remove "Remove blocks or pages" remove-spec)]) + [(core/command-entry ["remove" "block"] :remove-block "Remove blocks" remove-block-spec) + (core/command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec) + (core/command-entry ["remove" "tag"] :remove-tag "Remove tag" remove-entity-spec) + (core/command-entry ["remove" "property"] :remove-property "Remove property" remove-entity-spec)]) (defn invalid-options? - [opts] - (let [id-result (id-command/parse-id-option (:id opts))] - (cond - (and (some? (:id opts)) (not (:ok? id-result))) - (:message id-result) + [command opts] + (case command + :remove-block + (let [id-result (id-command/parse-id-option (:id opts)) + selectors (filter some? [(:id opts) (some-> (:uuid opts) string/trim)])] + (cond + (and (some? (:id opts)) (not (:ok? id-result))) + (:message id-result) - :else - nil))) + (> (count selectors) 1) + "only one of --id or --uuid is allowed" + + :else + nil)) + + (:remove-tag :remove-property) + (let [name (some-> (:name opts) string/trim) + selectors (filter some? [(:id opts) name])] + (cond + (> (count selectors) 1) + "only one of --id or --name is allowed" + + (and (contains? opts :name) (string/blank? (or (:name opts) ""))) + "name must be non-empty" + + :else + nil)) + + nil)) (def ^:private block-id-selector [:db/id :block/uuid]) +(def ^:private page-id-selector + [:db/id :block/uuid :block/name :block/title]) + +(def ^:private entity-selector + [:db/id :db/ident :block/uuid :block/name :block/title + :logseq.property/type :logseq.property/public? :logseq.property/built-in? + {:block/tags [:db/id :db/ident :block/title :block/name]}]) + (defn- fetch-block-by-id [config repo id] (transport/invoke config :thread-api/pull false @@ -48,6 +87,11 @@ (transport/invoke config :thread-api/apply-outliner-ops false [repo [[:delete-blocks [ids {}]]] {}])) +(defn- delete-page-by-uuid + [config repo page-uuid] + (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:delete-page [page-uuid]]] {}])) + (defn- remove-block-id [config repo id] (p/let [entity (fetch-block-by-id config repo id)] @@ -74,8 +118,8 @@ :missing-ids missing-ids :result result})) -(defn- perform-remove - [config {:keys [repo ids multi-id? uuid page]}] +(defn- perform-remove-block + [config {:keys [repo ids multi-id? uuid]}] (cond (and (seq ids) multi-id?) (remove-block-ids-best-effort config repo ids) @@ -91,59 +135,288 @@ (delete-block-ids config repo [id]) (throw (ex-info "block not found" {:code :block-not-found}))))) - (seq page) - (p/let [entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid] [:block/name page]])] - (if-let [page-uuid (:block/uuid entity)] - (transport/invoke config :thread-api/apply-outliner-ops false - [repo [[:delete-page [page-uuid]]] {}]) - (throw (ex-info "page not found" {:code :page-not-found})))) + :else + (p/rejected (ex-info "block is required" {:code :missing-target})))) + +(defn- resolve-page-by-name + [config repo name] + (transport/invoke config :thread-api/pull false + [repo page-id-selector [:block/name (common-util/page-name-sanity-lc name)]])) + +(defn- item-id + [item] + (or (:db/id item) (:id item))) + +(defn- item-name + [item] + (or (:block/title item) (:title item) (:name item) (:block/name item))) + +(defn- normalize-name + [value] + (common-util/page-name-sanity-lc (or value ""))) + +(defn- tag-entity? + [entity] + (some #(= :logseq.class/Tag (:db/ident %)) + (:block/tags entity))) + +(defn- property-entity? + [entity] + (some? (:logseq.property/type entity))) + +(defn- list-matches-by-name + [config repo method name] + (let [normalized (normalize-name name)] + (p/let [items (transport/invoke config method false [repo {:include-built-in true :expand true}]) + matches (->> (or items []) + (filter (fn [item] + (= normalized (normalize-name (item-name item))))) + vec)] + matches))) + +(defn- ambiguous-error + [code label name matches] + (let [candidates (->> matches + (map (fn [item] + {:id (item-id item) + :name (item-name item)})) + (filter :id) + vec)] + {:code code + :message (str "multiple " label "s match name: " name "; rerun with --id") + :candidates candidates})) + +(defn- resolve-target + [config repo {:keys [id name]} {:keys [list-method not-found-code ambiguous-code label]}] + (cond + (some? id) + (p/resolved {:ok? true + :lookup id + :id id}) + + (seq name) + (p/let [matches (list-matches-by-name config repo list-method name)] + (cond + (empty? matches) + {:ok? false + :error {:code not-found-code + :message (str label " not found")}} + + (> (count matches) 1) + {:ok? false + :error (ambiguous-error ambiguous-code label name matches)} + + :else + {:ok? true + :lookup [:block/name (normalize-name (or (item-name (first matches)) name))] + :id (item-id (first matches)) + :name (item-name (first matches))})) :else - (p/rejected (ex-info "block or page required" {:code :missing-target})))) + (p/resolved {:ok? false + :error {:code :missing-target + :message (str label " name or id is required")}}))) + +(defn- validate-tag-target + [entity] + (cond + (nil? (:db/id entity)) + {:ok? false + :error {:code :tag-not-found + :message "tag not found"}} + + (not (tag-entity? entity)) + {:ok? false + :error {:code :invalid-tag-target + :message "target is not a tag"}} + + (true? (:logseq.property/built-in? entity)) + {:ok? false + :error {:code :tag-built-in + :message "built-in tag cannot be removed"}} + + (false? (:logseq.property/public? entity)) + {:ok? false + :error {:code :tag-hidden + :message "hidden tag cannot be removed"}} + + (nil? (:block/uuid entity)) + {:ok? false + :error {:code :tag-not-found + :message "tag uuid not found"}} + + :else + {:ok? true + :entity entity})) + +(defn- validate-property-target + [entity] + (cond + (nil? (:db/id entity)) + {:ok? false + :error {:code :property-not-found + :message "property not found"}} + + (not (property-entity? entity)) + {:ok? false + :error {:code :invalid-property-target + :message "target is not a property"}} + + (true? (:logseq.property/built-in? entity)) + {:ok? false + :error {:code :property-built-in + :message "built-in property cannot be removed"}} + + (false? (:logseq.property/public? entity)) + {:ok? false + :error {:code :property-hidden + :message "hidden property cannot be removed"}} + + (nil? (:block/uuid entity)) + {:ok? false + :error {:code :property-not-found + :message "property uuid not found"}} + + :else + {:ok? true + :entity entity})) + +(defn- perform-remove-entity + [config action {:keys [list-method not-found-code ambiguous-code label validate-fn]}] + (p/let [resolved (resolve-target config (:repo action) action + {:list-method list-method + :not-found-code not-found-code + :ambiguous-code ambiguous-code + :label label})] + (if-not (:ok? resolved) + {:status :error + :error (:error resolved)} + (p/let [entity (transport/invoke config :thread-api/pull false + [(:repo action) entity-selector (:lookup resolved)]) + validation (validate-fn entity)] + (if-not (:ok? validation) + {:status :error + :error (:error validation)} + (p/let [result (delete-page-by-uuid config (:repo action) (:block/uuid entity))] + {:status :ok + :data {:result result + :id (:db/id entity) + :name (or (:name resolved) (:block/title entity) (:block/name entity))}})))))) (defn build-action - [options repo] + [command options repo] (if-not (seq repo) {:ok? false :error {:code :missing-repo :message "repo is required for remove"}} - (let [id-result (id-command/parse-id-option (:id options)) - ids (:value id-result) - multi-id? (:multi? id-result) - uuid (some-> (:uuid options) string/trim) - page (some-> (:page options) string/trim) - selectors (filter some? [(:id options) uuid page])] - (cond - (empty? selectors) - {:ok? false - :error {:code :missing-target - :message "block or page is required"}} + (case command + :remove-block + (let [id-result (id-command/parse-id-option (:id options)) + ids (:value id-result) + multi-id? (:multi? id-result) + uuid (some-> (:uuid options) string/trim) + selectors (filter some? [(:id options) uuid])] + (cond + (empty? selectors) + {:ok? false + :error {:code :missing-target + :message "block is required"}} - (> (count selectors) 1) - {:ok? false - :error {:code :invalid-options - :message "only one of --id, --uuid, or --page is allowed"}} + (> (count selectors) 1) + {:ok? false + :error {:code :invalid-options + :message "only one of --id or --uuid is allowed"}} - (and (some? (:id options)) (not (:ok? id-result))) - {:ok? false - :error {:code :invalid-options - :message (:message id-result)}} + (and (some? (:id options)) (not (:ok? id-result))) + {:ok? false + :error {:code :invalid-options + :message (:message id-result)}} - :else - {:ok? true - :action {:type :remove - :repo repo - :id (when (and (seq ids) (not multi-id?)) (first ids)) - :ids ids - :multi-id? multi-id? - :uuid uuid - :page page}})))) + :else + {:ok? true + :action {:type :remove-block + :repo repo + :graph (core/repo->graph repo) + :id (when (and (seq ids) (not multi-id?)) (first ids)) + :ids ids + :multi-id? multi-id? + :uuid uuid}})) -(defn execute-remove + :remove-page + (let [name (some-> (:name options) string/trim)] + (if (seq name) + {:ok? true + :action {:type :remove-page + :repo repo + :graph (core/repo->graph repo) + :name name}} + {:ok? false + :error {:code :missing-page-name + :message "page name is required"}})) + + (:remove-tag :remove-property) + (let [name (some-> (:name options) string/trim) + id (:id options) + selectors (filter some? [id name])] + (cond + (empty? selectors) + {:ok? false + :error {:code :missing-target + :message "name or id is required"}} + + (> (count selectors) 1) + {:ok? false + :error {:code :invalid-options + :message "only one of --id or --name is allowed"}} + + :else + {:ok? true + :action {:type command + :repo repo + :graph (core/repo->graph repo) + :id id + :name name}})) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown remove command: " command)}}))) + +(defn execute-remove-block [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - result (perform-remove cfg action)] + result (perform-remove-block cfg action)] {:status :ok :data (cond-> {:result result} (map? result) (merge (dissoc result :result)))}))) + +(defn execute-remove-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + entity (resolve-page-by-name cfg (:repo action) (:name action))] + (if-let [page-uuid (:block/uuid entity)] + (p/let [result (delete-page-by-uuid cfg (:repo action) page-uuid)] + {:status :ok + :data {:result result}}) + {:status :error + :error {:code :page-not-found + :message "page not found"}})))) + +(defn execute-remove-tag + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action))] + (perform-remove-entity cfg action + {:list-method :thread-api/api-list-tags + :not-found-code :tag-not-found + :ambiguous-code :ambiguous-tag-name + :label "tag" + :validate-fn validate-tag-target})))) + +(defn execute-remove-property + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action))] + (perform-remove-entity cfg action + {:list-method :thread-api/api-list-properties + :not-found-code :property-not-found + :ambiguous-code :ambiguous-property-name + :label "property" + :validate-fn validate-property-target})))) diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs new file mode 100644 index 0000000000..94b91d5a6b --- /dev/null +++ b/src/main/logseq/cli/command/upsert.cljs @@ -0,0 +1,227 @@ +(ns logseq.cli.command.upsert + "Upsert-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [logseq.common.util :as common-util] + [promesa.core :as p])) + +(def ^:private upsert-tag-spec + {:name {:desc "Tag name"}}) + +(def ^:private upsert-property-spec + {:name {:desc "Property name"} + :type {:desc "Property type (default, number, date, datetime, checkbox, url, node, json, string)"} + :cardinality {:desc "Property cardinality (one, many)"} + :hide {:desc "Hide property" + :coerce :boolean} + :public {:desc "Set property public visibility" + :coerce :boolean}}) + +(def entries + [(core/command-entry ["upsert" "tag"] :upsert-tag "Upsert tag" upsert-tag-spec) + (core/command-entry ["upsert" "property"] :upsert-property "Upsert property" upsert-property-spec)]) + +(def ^:private property-types + #{"default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"}) + +(def ^:private property-cardinalities + #{"one" "many"}) + +(defn- normalize-tag-name + [value] + (let [text (some-> value string/trim (string/replace #"^#+" ""))] + (when (seq text) + text))) + +(defn- normalize-property-name + [value] + (let [text (some-> value string/trim)] + (when (seq text) + text))) + +(defn- normalize-property-type + [value] + (some-> value string/trim string/lower-case)) + +(defn- normalize-property-cardinality + [value] + (let [v (some-> value string/trim string/lower-case)] + (case v + "db.cardinality/one" "one" + "db.cardinality/many" "many" + v))) + +(defn invalid-options? + [command opts] + (case command + :upsert-property + (let [type' (normalize-property-type (:type opts)) + cardinality' (normalize-property-cardinality (:cardinality opts))] + (cond + (and (seq (:type opts)) (not (contains? property-types type'))) + (str "invalid type: " (:type opts)) + + (and (seq (:cardinality opts)) (not (contains? property-cardinalities cardinality'))) + (str "invalid cardinality: " (:cardinality opts)) + + :else + nil)) + + nil)) + +(defn build-tag-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for upsert"}} + (let [name (normalize-tag-name (:name options))] + (if (seq name) + {:ok? true + :action {:type :upsert-tag + :repo repo + :graph (core/repo->graph repo) + :name name}} + {:ok? false + :error {:code :missing-tag-name + :message "tag name is required"}})))) + +(defn- cardinality->db + [value] + (when-let [v (normalize-property-cardinality value)] + (case v + "many" :db.cardinality/many + "one" :db.cardinality/one + nil))) + +(defn- property-schema + [options] + (cond-> {} + (seq (:type options)) + (assoc :logseq.property/type (keyword (normalize-property-type (:type options)))) + + (seq (:cardinality options)) + (assoc :db/cardinality (cardinality->db (:cardinality options))) + + (contains? options :hide) + (assoc :logseq.property/hide? (boolean (:hide options))) + + (contains? options :public) + (assoc :logseq.property/public? (boolean (:public options))))) + +(defn build-property-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for upsert"}} + (let [name (normalize-property-name (:name options)) + invalid-message (invalid-options? :upsert-property options)] + (cond + (not (seq name)) + {:ok? false + :error {:code :missing-property-name + :message "property name is required"}} + + (seq invalid-message) + {:ok? false + :error {:code :invalid-options + :message invalid-message}} + + :else + {:ok? true + :action {:type :upsert-property + :repo repo + :graph (core/repo->graph repo) + :name name + :schema (property-schema options)}})))) + +(defn- pull-page-by-name + [config repo page-name selector] + (transport/invoke config :thread-api/pull false + [repo selector [:block/name (common-util/page-name-sanity-lc page-name)]])) + +(defn- tag-entity? + [entity] + (some #(= :logseq.class/Tag (:db/ident %)) + (:block/tags entity))) + +(defn execute-upsert-tag + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + existing (pull-page-by-name cfg (:repo action) (:name action) + [:db/id :block/name :block/title + {:block/tags [:db/ident]}]) + existing-id (:db/id existing)] + (cond + (and existing-id (not (tag-entity? existing))) + {:status :error + :error {:code :tag-name-conflict + :message "tag already exists as a page and is not a tag"}} + + :else + (p/let [_ (when-not existing-id + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:create-page [(:name action) {:class? true}]]] + {}])) + page (or (when existing-id existing) + (pull-page-by-name cfg (:repo action) (:name action) + [:db/id :block/name :block/title + {:block/tags [:db/ident]}])) + page-id (:db/id page)] + (cond + (not page-id) + {:status :error + :error {:code :tag-not-found + :message "tag not found after upsert"}} + + (not (tag-entity? page)) + {:status :error + :error {:code :tag-create-not-tag + :message "created entity is not tagged as :logseq.class/Tag"}} + + :else + {:status :ok + :data {:result [page-id]}})))))) + +(def ^:private property-selector + [:db/id :db/ident :block/name :block/title :logseq.property/type]) + +(defn- property-entity? + [entity] + (some? (:logseq.property/type entity))) + +(defn execute-upsert-property + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + existing (pull-page-by-name cfg (:repo action) (:name action) property-selector) + existing-id (:db/id existing)] + (cond + (and existing-id (not (property-entity? existing))) + {:status :error + :error {:code :property-name-conflict + :message "property already exists as a page and is not a property"}} + + :else + (p/let [property-ident (when (property-entity? existing) + (:db/ident existing)) + property-opts (cond-> {} + (nil? property-ident) + (assoc :property-name (:name action))) + _ (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:upsert-property [property-ident + (:schema action) + property-opts]]] + {}]) + property (pull-page-by-name cfg (:repo action) (:name action) property-selector) + property-id (:db/id property)] + (if property-id + {:status :ok + :data {:result [property-id]}} + {:status :error + :error {:code :property-not-found + :message "property not found after upsert"}})))))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index b5668c728f..2a3bde01a1 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -11,6 +11,7 @@ [logseq.cli.command.remove :as remove-command] [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] + [logseq.cli.command.upsert :as upsert-command] [logseq.cli.command.update :as update-command] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -69,6 +70,13 @@ :message "tag name is required"} :summary summary}) +(defn- missing-property-name-result + [summary] + {:ok? false + :error {:code :missing-property-name + :message "property name is required"} + :summary summary}) + (defn- missing-type-result [summary] {:ok? false @@ -106,6 +114,7 @@ server-command/entries list-command/entries add-command/entries + upsert-command/entries remove-command/entries update-command/entries query-command/entries @@ -153,9 +162,6 @@ (seq (:blocks-file opts)) has-args?) show-targets (filter some? [(:id opts) (:uuid opts) (:page opts)]) - remove-targets (filter some? [(:id opts) - (some-> (:uuid opts) string/trim) - (some-> (:page opts) string/trim)]) update-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)])] (cond (:help opts) @@ -174,17 +180,24 @@ (and (= command :add-page) (not (seq (:page opts)))) (missing-page-name-result summary) - (and (= command :add-tag) (not (seq (some-> (:name opts) string/trim)))) + (and (= command :upsert-tag) (not (seq (some-> (:name opts) string/trim)))) (missing-tag-name-result summary) - (and (= command :remove) (seq args)) - (command-core/invalid-options-result summary "remove does not accept subcommands") + (and (= command :upsert-property) (not (seq (some-> (:name opts) string/trim)))) + (missing-property-name-result summary) - (and (= command :remove) (empty? remove-targets)) + (and (= command :upsert-property) (upsert-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) + + (and (= command :remove-block) (empty? (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]))) (missing-target-result summary) - (and (= command :remove) (> (count remove-targets) 1)) - (command-core/invalid-options-result summary "only one of --id, --uuid, or --page is allowed") + (and (= command :remove-page) (not (seq (some-> (:name opts) string/trim)))) + (missing-page-name-result summary) + + (and (#{:remove-tag :remove-property} command) + (empty? (filter some? [(:id opts) (some-> (:name opts) string/trim)]))) + (missing-target-result summary) (and (= command :update-block) (update-command/invalid-options? opts)) (command-core/invalid-options-result summary (update-command/invalid-options? opts)) @@ -207,8 +220,9 @@ (list-command/invalid-options? command opts)) (command-core/invalid-options-result summary (list-command/invalid-options? command opts)) - (and (= command :remove) (remove-command/invalid-options? opts)) - (command-core/invalid-options-result summary (remove-command/invalid-options? opts)) + (and (#{:remove-block :remove-page :remove-tag :remove-property} command) + (remove-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (remove-command/invalid-options? command opts)) (and (= command :show) (show-command/invalid-options? opts)) (command-core/invalid-options-result summary (show-command/invalid-options? opts)) @@ -268,7 +282,7 @@ :message "missing command"} :summary summary}) - (and (= 1 (count args)) (#{"graph" "server" "list" "add" "query"} (first args))) + (and (= 1 (count args)) (#{"graph" "server" "list" "add" "upsert" "remove" "query"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) :else @@ -372,14 +386,17 @@ :add-page (add-command/build-add-page-action options repo) - :add-tag - (add-command/build-add-tag-action options repo) + :upsert-tag + (upsert-command/build-tag-action options repo) + + :upsert-property + (upsert-command/build-property-action options repo) :update-block (update-command/build-action options repo) - :remove - (remove-command/build-action options repo) + (:remove-block :remove-page :remove-tag :remove-property) + (remove-command/build-action command options repo) :query (query-command/build-action options repo config) @@ -423,9 +440,13 @@ :list-property (list-command/execute-list-property action config) :add-block (add-command/execute-add-block action config) :add-page (add-command/execute-add-page action config) - :add-tag (add-command/execute-add-tag action config) + :upsert-tag (upsert-command/execute-upsert-tag action config) + :upsert-property (upsert-command/execute-upsert-property action config) :update-block (update-command/execute-update action config) - :remove (remove-command/execute-remove action config) + :remove-block (remove-command/execute-remove-block action config) + :remove-page (remove-command/execute-remove-page action config) + :remove-tag (remove-command/execute-remove-tag action config) + :remove-property (remove-command/execute-remove-property action config) :query (query-command/execute-query action config) :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) @@ -440,8 +461,8 @@ :message "unknown action"}}))] (assoc result :command (or (:command action) (:type action)) - :context (select-keys action [:repo :graph :page :id :ids :uuid :block :blocks - :name + :context (select-keys action [:repo :graph :page :name :id :ids :uuid :block :blocks + :schema :source :target :update-tags :update-properties :remove-tags :remove-properties :export-type :output :import-type :input]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 3ff9292e45..6052212040 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -93,17 +93,30 @@ :missing-tag-name "Use --name " :missing-query "Use --query " :unknown-query "Use `logseq query list` to see available queries" + :ambiguous-tag-name "Retry with --id " + :ambiguous-property-name "Retry with --id " :data-dir-permission "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR" :server-owned-by-other "Retry from the process owner that started the server" :server-start-timeout-orphan "Check and stop lingering db-worker-node processes, then retry" nil)) +(defn- format-candidates + [candidates] + (when (seq candidates) + (str "\nCandidates:\n" + (string/join "\n" + (map (fn [{:keys [id name]}] + (str " " id " " (or name "-"))) + candidates))))) + (defn- format-error [error] - (let [{:keys [code message]} error + (let [{:keys [code message candidates]} error hint (error-hint error) - message* (style/bold-keywords message ["option" "command" "argument"])] + message* (style/bold-keywords message ["option" "command" "argument"]) + candidates* (format-candidates candidates)] (cond-> (str "Error (" (name (or code :error)) "): " message*) + candidates* (str candidates*) hint (str "\nHint: " hint)))) (defn- maybe-ident-header @@ -248,14 +261,37 @@ [_context ids] (str "Added tag:\n" (pr-str (vec (or ids []))))) -(defn- format-remove - [{:keys [repo page uuid id ids]}] +(defn- format-upsert-tag + [_context ids] + (str "Upserted tag:\n" (pr-str (vec (or ids []))))) + +(defn- format-upsert-property + [_context ids] + (str "Upserted property:\n" (pr-str (vec (or ids []))))) + +(defn- format-remove-block + [{:keys [repo uuid id ids]}] (cond - (seq page) (str "Removed page: " page " (repo: " repo ")") (seq uuid) (str "Removed block: " uuid " (repo: " repo ")") (seq ids) (str "Removed blocks: " (count ids) " (repo: " repo ")") (some? id) (str "Removed block: " id " (repo: " repo ")") - :else (str "Removed item (repo: " repo ")"))) + :else (str "Removed block (repo: " repo ")"))) + +(defn- format-remove-page + [{:keys [repo name]}] + (str "Removed page: " name " (repo: " repo ")")) + +(defn- format-remove-tag + [{:keys [repo name id]}] + (if (seq name) + (str "Removed tag: " name " (repo: " repo ")") + (str "Removed tag: " id " (repo: " repo ")"))) + +(defn- format-remove-property + [{:keys [repo name id]}] + (if (seq name) + (str "Removed property: " name " (repo: " repo ")") + (str "Removed property: " id " (repo: " repo ")"))) (defn- format-update-block [{:keys [repo source target update-tags update-properties remove-tags remove-properties]}] @@ -319,7 +355,12 @@ :add-block (format-add-block context (:result data)) :add-page (format-add-page context (:result data)) :add-tag (format-add-tag context (:result data)) - :remove (format-remove context) + :upsert-tag (format-upsert-tag context (:result data)) + :upsert-property (format-upsert-property context (:result data)) + :remove-block (format-remove-block context) + :remove-page (format-remove-page context) + :remove-tag (format-remove-tag context) + :remove-property (format-remove-property context) :update-block (format-update-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 252997f9e9..bbbafd346f 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -60,6 +60,7 @@ (is (string/includes? plain-summary "Graph Management")) (is (string/includes? plain-summary "list")) (is (string/includes? plain-summary "add")) + (is (string/includes? plain-summary "upsert")) (is (string/includes? plain-summary "remove")) (is (string/includes? plain-summary "update")) (is (string/includes? plain-summary "query")) @@ -73,8 +74,12 @@ (is (contains-bold? summary "list property")) (is (contains-bold? summary "add block")) (is (contains-bold? summary "add page")) - (is (contains-bold? summary "add tag")) - (is (contains-bold? summary "remove")) + (is (contains-bold? summary "upsert tag")) + (is (contains-bold? summary "upsert property")) + (is (contains-bold? summary "remove block")) + (is (contains-bold? summary "remove page")) + (is (contains-bold? summary "remove tag")) + (is (contains-bold? summary "remove property")) (is (contains-bold? summary "update")) (is (contains-bold? summary "query")) (is (contains-bold? summary "query list")) @@ -134,17 +139,27 @@ (is (string/includes? plain-summary "Global options:")) (is (string/includes? plain-summary "Command options:")))) - (testing "remove command shows help" + (testing "upsert group shows subcommands" (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["remove" "--help"])) + (commands/parse-args ["upsert"])) summary (:summary result) plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? plain-summary "Usage: logseq remove")) + (is (string/includes? plain-summary "upsert tag")) + (is (string/includes? plain-summary "upsert property")) + (is (contains-bold? summary "upsert tag")) + (is (contains-bold? summary "upsert property")))) + + (testing "remove block command shows help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["remove" "block" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq remove block")) (is (string/includes? plain-summary "Command options:")) (is (contains-bold? summary "--id")) - (is (contains-bold? summary "--uuid")) - (is (contains-bold? summary "--page")))) + (is (contains-bold? summary "--uuid")))) (testing "update command shows help" (let [result (binding [style/*color-enabled?* true] @@ -192,7 +207,7 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) -(deftest test-parse-args-help-add-group +(deftest test-parse-args-help-add-upsert-group (testing "add group shows subcommands" (let [result (binding [style/*color-enabled?* true] (commands/parse-args ["add"])) @@ -201,10 +216,19 @@ (is (true? (:help? result))) (is (string/includes? plain-summary "add block")) (is (string/includes? plain-summary "add page")) - (is (string/includes? plain-summary "add tag")) (is (contains-bold? summary "add block")) - (is (contains-bold? summary "add page")) - (is (contains-bold? summary "add tag"))))) + (is (contains-bold? summary "add page")))) + + (testing "upsert group shows subcommands" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["upsert"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "upsert tag")) + (is (string/includes? plain-summary "upsert property")) + (is (contains-bold? summary "upsert tag")) + (is (contains-bold? summary "upsert property"))))) (deftest test-parse-args-help-alignment (testing "graph group aligns subcommand columns" @@ -268,13 +292,26 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) -(deftest test-parse-args-rejects-legacy-remove-subcommands - (testing "rejects legacy remove subcommands" - (doseq [args [["remove" "block"] - ["remove" "page"]]] - (let [result (commands/parse-args args)] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))))) +(deftest test-parse-args-remove-help-and-rejects-add-tag + (testing "bare remove shows remove subcommand help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["remove"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "remove block")) + (is (string/includes? plain-summary "remove page")) + (is (string/includes? plain-summary "remove tag")) + (is (string/includes? plain-summary "remove property")) + (is (contains-bold? summary "remove block")) + (is (contains-bold? summary "remove page")) + (is (contains-bold? summary "remove tag")) + (is (contains-bold? summary "remove property")))) + + (testing "rejects removed add tag command" + (let [result (commands/parse-args ["add" "tag" "--name" "Quote"])] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code])))))) (deftest test-parse-args-rejects-graph-option (testing "rejects legacy --graph option" @@ -826,42 +863,69 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) -(deftest test-verb-subcommand-parse-add-remove - (testing "remove requires target" - (let [result (commands/parse-args ["remove"])] - (is (false? (:ok? result))) - (is (= :missing-target (get-in result [:error :code]))))) - - (testing "remove parses with id" - (let [result (commands/parse-args ["remove" "--id" "10"])] +(deftest test-verb-subcommand-parse-upsert-remove + (testing "remove block parses with id" + (let [result (commands/parse-args ["remove" "block" "--id" "10"])] (is (true? (:ok? result))) - (is (= :remove (:command result))) + (is (= :remove-block (:command result))) (is (= 10 (get-in result [:options :id]))))) - (testing "remove parses with uuid" - (let [result (commands/parse-args ["remove" "--uuid" "abc"])] + (testing "remove page parses with name" + (let [result (commands/parse-args ["remove" "page" "--name" "Home"])] (is (true? (:ok? result))) - (is (= :remove (:command result))) - (is (= "abc" (get-in result [:options :uuid]))))) + (is (= :remove-page (:command result))) + (is (= "Home" (get-in result [:options :name]))))) - (testing "remove parses with page" - (let [result (commands/parse-args ["remove" "--page" "Home"])] + (testing "remove tag parses with name" + (let [result (commands/parse-args ["remove" "tag" "--name" "Quote"])] (is (true? (:ok? result))) - (is (= :remove (:command result))) - (is (= "Home" (get-in result [:options :page]))))) + (is (= :remove-tag (:command result))) + (is (= "Quote" (get-in result [:options :name]))))) - (testing "remove rejects multiple selectors" - (let [result (commands/parse-args ["remove" "--id" "1" "--page" "Home"])] + (testing "remove property parses with id" + (let [result (commands/parse-args ["remove" "property" "--id" "123"])] + (is (true? (:ok? result))) + (is (= :remove-property (:command result))) + (is (= 123 (get-in result [:options :id]))))) + + (testing "remove block rejects empty id vector" + (let [result (commands/parse-args ["remove" "block" "--id" "[]"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "remove rejects empty id vector" - (let [result (commands/parse-args ["remove" "--id" "[]"])] + (testing "remove block rejects invalid id vector" + (let [result (commands/parse-args ["remove" "block" "--id" "[1 \"no\"]"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "remove rejects invalid id vector" - (let [result (commands/parse-args ["remove" "--id" "[1 \"no\"]"])] + (testing "upsert tag parses with name" + (let [result (commands/parse-args ["upsert" "tag" "--name" "Quote"])] + (is (true? (:ok? result))) + (is (= :upsert-tag (:command result))) + (is (= "Quote" (get-in result [:options :name]))))) + + (testing "upsert property parses with type and cardinality" + (let [result (commands/parse-args ["upsert" "property" + "--name" "owner" + "--type" "node" + "--cardinality" "many"])] + (is (true? (:ok? result))) + (is (= :upsert-property (:command result))) + (is (= "owner" (get-in result [:options :name]))) + (is (= "node" (get-in result [:options :type]))) + (is (= "many" (get-in result [:options :cardinality]))))) + + (testing "upsert property rejects invalid type" + (let [result (commands/parse-args ["upsert" "property" + "--name" "owner" + "--type" "wat"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "upsert property rejects invalid cardinality" + (let [result (commands/parse-args ["upsert" "property" + "--name" "owner" + "--cardinality" "triple"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -973,21 +1037,10 @@ (is (= "[\"TagA\"]" (get-in result [:options :tags]))) (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) - (testing "add tag requires name" - (let [result (commands/parse-args ["add" "tag"])] - (is (false? (:ok? result))) - (is (= :missing-tag-name (get-in result [:error :code]))))) - - (testing "add tag parses with name" + (testing "add tag is no longer supported" (let [result (commands/parse-args ["add" "tag" "--name" "Quote"])] - (is (true? (:ok? result))) - (is (= :add-tag (:command result))) - (is (= "Quote" (get-in result [:options :name]))))) - - (testing "add tag rejects blank name" - (let [result (commands/parse-args ["add" "tag" "--name" " "])] (is (false? (:ok? result))) - (is (= :missing-tag-name (get-in result [:error :code])))))) + (is (= :unknown-command (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-update-target-page (testing "update parses with target page" @@ -1113,7 +1166,8 @@ (testing "verb subcommands reject unknown flags" (doseq [args [["list" "page" "--wat"] ["add" "block" "--wat"] - ["remove" "--wat"] + ["remove" "block" "--wat"] + ["upsert" "tag" "--wat"] ["update" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] @@ -1207,7 +1261,7 @@ (is (= (cli-server/db-worker-dev-script-path) (get-in result [:action :script-path])))))) -(deftest test-build-action-inspect-edit +(deftest test-build-action-inspect-edit-add-upsert (testing "list page requires repo" (let [parsed {:ok? true :command :list-page :options {}} result (commands/build-action parsed {})] @@ -1232,35 +1286,79 @@ (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) - (testing "add tag requires name" - (let [parsed {:ok? true :command :add-tag :options {}} + (testing "upsert tag requires name" + (let [parsed {:ok? true :command :upsert-tag :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-tag-name (get-in result [:error :code]))))) - (testing "add tag builds normalized action" - (let [parsed {:ok? true :command :add-tag :options {:name " #Quote "}} + (testing "upsert tag builds normalized action" + (let [parsed {:ok? true :command :upsert-tag :options {:name " #Quote "}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= {:type :add-tag + (is (= {:type :upsert-tag :repo "logseq_db_demo" :graph "demo" :name "Quote"} (:action result))))) - (testing "remove requires target" - (let [parsed {:ok? true :command :remove :options {}} + (testing "upsert property coerces schema options" + (let [parsed {:ok? true + :command :upsert-property + :options {:name "owner" + :type "node" + :cardinality "many" + :hide true + :public false}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= {:type :upsert-property + :repo "logseq_db_demo" + :graph "demo" + :name "owner" + :schema {:logseq.property/type :node + :db/cardinality :db.cardinality/many + :logseq.property/hide? true + :logseq.property/public? false}} + (:action result))))) + + ) + +(deftest test-build-action-inspect-edit-remove-show + + (testing "remove block requires target" + (let [parsed {:ok? true :command :remove-block :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) - (testing "remove normalizes id vector in build action" - (let [parsed {:ok? true :command :remove :options {:id "[1 2]"}} + (testing "remove block normalizes id vector in build action" + (let [parsed {:ok? true :command :remove-block :options {:id "[1 2]"}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= :remove (get-in result [:action :type]))) + (is (= :remove-block (get-in result [:action :type]))) (is (= [1 2] (get-in result [:action :ids]))))) + (testing "remove page requires name" + (let [parsed {:ok? true :command :remove-page :options {}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :missing-page-name (get-in result [:error :code]))))) + + (testing "remove tag parses by id" + (let [parsed {:ok? true :command :remove-tag :options {:id 42}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :remove-tag (get-in result [:action :type]))) + (is (= 42 (get-in result [:action :id]))))) + + (testing "remove property parses by name" + (let [parsed {:ok? true :command :remove-property :options {:name "owner"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :remove-property (get-in result [:action :type]))) + (is (= "owner" (get-in result [:action :name]))))) + (testing "show requires target" (let [parsed {:ok? true :command :show :options {}} result (commands/build-action parsed {:repo "demo"})] @@ -1386,15 +1484,17 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) -(deftest test-execute-add-tag-builds-create-page-op +(deftest test-execute-upsert-tag-builds-create-page-op (async done (let [ops* (atom nil) created?* (atom false) + orig-list-graphs cli-server/list-graphs orig-ensure-server! cli-server/ensure-server! orig-invoke transport/invoke - action {:type :add-tag + action {:type :upsert-tag :repo "demo" :name "Quote"}] + (set! cli-server/list-graphs (fn [_] ["demo"])) (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) (set! transport/invoke (fn [_ method _ args] (case method @@ -1412,7 +1512,7 @@ (reset! ops* ops) {:result :ok}) (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (add-command/execute-add-tag action {})] + (-> (p/let [result (commands/execute action {})] (is (= :ok (:status result))) (is (= [4242] (get-in result [:data :result]))) (is (= [[:create-page ["Quote" {:class? true}]]] @@ -1420,17 +1520,20 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke) (done))))))) -(deftest test-execute-add-tag-rejects-existing-non-tag-page +(deftest test-execute-upsert-tag-rejects-existing-non-tag-page (async done - (let [action {:type :add-tag + (let [action {:type :upsert-tag :repo "demo" :name "Home"} + orig-list-graphs cli-server/list-graphs orig-ensure-server! cli-server/ensure-server! orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] ["demo"])) (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) (set! transport/invoke (fn [_ method _ args] (case method @@ -1444,24 +1547,27 @@ :thread-api/apply-outliner-ops (throw (ex-info "should not create tag" {:args args})) (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (add-command/execute-add-tag action {}) - (p/then (fn [_] - (is false "expected add tag conflict error"))) + (-> (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-name-conflict (get-in result [:error :code])))) (p/catch (fn [e] - (is (= :tag-name-conflict (:code (ex-data e)))))) + (is false (str "unexpected error: " e)))) (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke) (done))))))) -(deftest test-execute-add-tag-idempotent-when-tag-exists +(deftest test-execute-upsert-tag-idempotent-when-tag-exists (async done (let [apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs orig-ensure-server! cli-server/ensure-server! orig-invoke transport/invoke - action {:type :add-tag + action {:type :upsert-tag :repo "demo" :name "Quote"}] + (set! cli-server/list-graphs (fn [_] ["demo"])) (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) (set! transport/invoke (fn [_ method _ args] (case method @@ -1476,13 +1582,159 @@ (swap! apply-calls* inc) {:result :ok}) (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (add-command/execute-add-tag action {})] + (-> (p/let [result (commands/execute action {})] (is (= :ok (:status result))) (is (= [4242] (get-in result [:data :result]))) (is (= 0 @apply-calls*))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-property-emits-upsert-op + (async done + (let [ops* (atom nil) + created?* (atom false) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-property + :repo "demo" + :name "owner" + :schema {:logseq.property/type :node + :db/cardinality :db.cardinality/many}}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (if @created?* + {:db/id 654 + :db/ident :user.property/owner + :block/name "owner" + :block/title "owner" + :logseq.property/type :node} + {}) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! created?* true) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [[:upsert-property [nil + {:logseq.property/type :node + :db/cardinality :db.cardinality/many} + {:property-name "owner"}]]] + @ops*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-remove-tag-property + (async done + (let [ops* (atom []) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke + (fn [_ method _ args] + (case method + :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"}] + :thread-api/api-list-properties [{:db/id 2 :block/title "owner"}] + :thread-api/pull (let [[_ selector lookup] args] + (cond + (= lookup 1) + {:db/id 1 + :block/title "Quote" + :block/uuid (uuid "00000000-0000-0000-0000-000000000011") + :block/tags [{:db/ident :logseq.class/Tag}] + :logseq.property/public? true} + + (= lookup 2) + {:db/id 2 + :db/ident :user.property/owner + :block/title "owner" + :block/uuid (uuid "00000000-0000-0000-0000-000000000022") + :logseq.property/type :node + :logseq.property/public? true} + + (= lookup [:block/name "quote"]) + {:db/id 1 + :block/title "Quote" + :block/uuid (uuid "00000000-0000-0000-0000-000000000011") + :block/tags [{:db/ident :logseq.class/Tag}] + :logseq.property/public? true} + + (= lookup [:block/name "owner"]) + {:db/id 2 + :db/ident :user.property/owner + :block/title "owner" + :block/uuid (uuid "00000000-0000-0000-0000-000000000022") + :logseq.property/type :node + :logseq.property/public? true} + + :else + (throw (ex-info "unexpected pull lookup" + {:lookup lookup :selector selector})))) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (swap! ops* conj ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [tag-result (commands/execute {:type :remove-tag + :repo "demo" + :name "Quote"} + {}) + property-result (commands/execute {:type :remove-property + :repo "demo" + :id 2} + {})] + (is (= :ok (:status tag-result))) + (is (= :ok (:status property-result))) + (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000011")]]] + (first @ops*))) + (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000022")]]] + (second @ops*)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-remove-tag-ambiguous-name + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke + (fn [_ method _ _] + (case method + :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"} + {:db/id 2 :block/title "QUOTE"}] + (throw (ex-info "unexpected invoke" {:method method}))))) + (-> (p/let [result (commands/execute {:type :remove-tag + :repo "demo" + :name "Quote"} + {})] + (is (= :error (:status result))) + (is (= :ambiguous-tag-name (get-in result [:error :code]))) + (is (= 2 (count (get-in result [:error :candidates]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke) (done))))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index d6b0601446..ff794149f7 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -90,7 +90,7 @@ "Count: 1") result))))) -(deftest test-human-output-add-remove +(deftest test-human-output-add-upsert-remove (testing "add block renders ids in two lines" (let [result (format/format-result {:status :ok :command :add-block @@ -109,24 +109,60 @@ {:output-format nil})] (is (= "Added page:\n[123]" result)))) - (testing "add tag renders ids in two lines" + (testing "upsert tag renders ids in two lines" (let [result (format/format-result {:status :ok - :command :add-tag + :command :upsert-tag :context {:repo "demo-repo" :name "Quote"} :data {:result [321]}} {:output-format nil})] - (is (= "Added tag:\n[321]" result)))) + (is (= "Upserted tag:\n[321]" result)))) + + (testing "upsert property renders ids in two lines" + (let [result (format/format-result {:status :ok + :command :upsert-property + :context {:repo "demo-repo" + :name "owner"} + :data {:result [654]}} + {:output-format nil})] + (is (= "Upserted property:\n[654]" result)))) (testing "remove page renders a succinct success line" (let [result (format/format-result {:status :ok - :command :remove + :command :remove-page :context {:repo "demo-repo" - :page "Home"} + :name "Home"} :data {:result {:ok true}}} {:output-format nil})] (is (= "Removed page: Home (repo: demo-repo)" result)))) + (testing "remove block with id list renders block count" + (let [result (format/format-result {:status :ok + :command :remove-block + :context {:repo "demo-repo" + :ids [1 2 3]} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Removed blocks: 3 (repo: demo-repo)" result)))) + + (testing "remove tag renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :remove-tag + :context {:repo "demo-repo" + :name "Quote"} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Removed tag: Quote (repo: demo-repo)" result)))) + + (testing "remove property renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :remove-property + :context {:repo "demo-repo" + :name "owner"} + :data {:result {:ok true}}} + {:output-format nil})] + (is (= "Removed property: owner (repo: demo-repo)" result)))) + (testing "update block renders a succinct success line" (let [result (format/format-result {:status :ok :command :update-block @@ -430,7 +466,22 @@ {:output-format nil})] (is (= (str "Error (server-start-timeout-orphan): db-worker-node failed to create lock\n" "Hint: Check and stop lingering db-worker-node processes, then retry") - result))))) + result)))) + + (testing "remove tag ambiguity includes candidate list" + (let [result (format/format-result {:status :error + :command :remove-tag + :error {:code :ambiguous-tag-name + :message "multiple tags match name: Quote" + :candidates [{:id 1 :name "Quote"} + {:id 2 :name "QUOTE"}]}} + {:output-format nil})] + (is (string/includes? result "Error (ambiguous-tag-name):")) + (is (string/includes? result "multiple tags match name: Quote")) + (is (string/includes? result "1")) + (is (string/includes? result "2")) + (is (string/includes? result "Quote")) + (is (string/includes? result "QUOTE"))))) (deftest test-human-output-doctor (testing "doctor renders concise check summary" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 36f9aa0ed9..dec868d709 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -282,7 +282,7 @@ _ (p/delay 100) show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) show-payload (parse-json-output show-result) - remove-page-result (run-cli ["--repo" "content-graph" "remove" "--page" "TestPage"] data-dir cfg-path) + remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--name" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] @@ -437,7 +437,7 @@ "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) update-payload (parse-json-output update-result) - remove-result (run-cli ["--repo" repo "remove" "--id" (str block-id)] data-dir cfg-path) + remove-result (run-cli ["--repo" repo "remove" "block" "--id" (str block-id)] data-dir cfg-path) remove-payload (parse-json-output remove-result) query-after-remove (run-query data-dir cfg-path repo "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" @@ -857,19 +857,19 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-add-tag-create-and-use +(deftest ^:long test-cli-upsert-tag-create-and-use (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-create")] + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-tag-create")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - repo "add-tag-create-graph" + repo "upsert-tag-create-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) _ (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) - add-tag-result (run-cli ["--repo" repo - "add" "tag" - "--name" "CliQuote"] - data-dir cfg-path) - add-tag-payload (parse-json-output add-tag-result) + upsert-tag-result (run-cli ["--repo" repo + "upsert" "tag" + "--name" "CliQuote"] + data-dir cfg-path) + upsert-tag-payload (parse-json-output upsert-tag-result) list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) @@ -878,17 +878,17 @@ add-block-result (run-cli ["--repo" repo "add" "block" "--target-page-name" "Home" - "--content" "Tagged by add tag" + "--content" "Tagged by upsert tag" "--tags" "[\"CliQuote\"]"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - block-tag-names (query-tags data-dir cfg-path repo "Tagged by add tag") + block-tag-names (query-tags data-dir cfg-path repo "Tagged by upsert tag") stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code add-tag-result)) - (pr-str (:error add-tag-payload))) - (is (= "ok" (:status add-tag-payload))) + (is (= 0 (:exit-code upsert-tag-result)) + (pr-str (:error upsert-tag-payload))) + (is (= "ok" (:status upsert-tag-payload))) (is (= "ok" (:status list-tag-payload))) (is (contains? tag-names "CliQuote")) (is (= 0 (:exit-code add-block-result)) @@ -901,19 +901,19 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-add-tag-rejects-existing-non-tag-page +(deftest ^:long test-cli-upsert-tag-rejects-existing-non-tag-page (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-conflict")] + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-tag-conflict")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - repo "add-tag-conflict-graph" + repo "upsert-tag-conflict-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) _ (run-cli ["--repo" repo "add" "page" "--page" "ConflictPage"] data-dir cfg-path) - add-tag-result (run-cli ["--repo" repo - "add" "tag" - "--name" "ConflictPage"] - data-dir cfg-path) - add-tag-payload (parse-json-output add-tag-result) + upsert-tag-result (run-cli ["--repo" repo + "upsert" "tag" + "--name" "ConflictPage"] + data-dir cfg-path) + upsert-tag-payload (parse-json-output upsert-tag-result) list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) @@ -921,9 +921,9 @@ set) stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 1 (:exit-code add-tag-result))) - (is (= "error" (:status add-tag-payload))) - (is (string/includes? (get-in add-tag-payload [:error :message]) + (is (= 0 (:exit-code upsert-tag-result))) + (is (= "error" (:status upsert-tag-payload))) + (is (string/includes? (get-in upsert-tag-payload [:error :message]) "already exists as a page and is not a tag")) (is (not (contains? tag-names "ConflictPage"))) (is (= "ok" (:status stop-payload))) @@ -932,29 +932,29 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-add-tag-idempotent-existing-tag +(deftest ^:long test-cli-upsert-tag-idempotent-existing-tag (async done - (let [data-dir (node-helper/create-tmp-dir "db-worker-add-tag-idempotent")] + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-tag-idempotent")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - repo "add-tag-idempotent-graph" + repo "upsert-tag-idempotent-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - first-add-result (run-cli ["--repo" repo "add" "tag" "--name" "StableTag"] - data-dir cfg-path) - first-add-payload (parse-json-output first-add-result) - second-add-result (run-cli ["--repo" repo "add" "tag" "--name" "StableTag"] - data-dir cfg-path) - second-add-payload (parse-json-output second-add-result) + first-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] + data-dir cfg-path) + first-upsert-payload (parse-json-output first-upsert-result) + second-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] + data-dir cfg-path) + second-upsert-payload (parse-json-output second-upsert-result) list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) stable-tags (->> (get-in list-tag-payload [:data :items]) (filter #(= "StableTag" (or (:block/title %) (:title %) (:name %))))) stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code first-add-result))) - (is (= "ok" (:status first-add-payload))) - (is (= 0 (:exit-code second-add-result))) - (is (= "ok" (:status second-add-payload))) + (is (= 0 (:exit-code first-upsert-result))) + (is (= "ok" (:status first-upsert-payload))) + (is (= 0 (:exit-code second-upsert-result))) + (is (= "ok" (:status second-upsert-payload))) (is (= 1 (count stable-tags))) (is (= "ok" (:status stop-payload))) (done)) @@ -962,6 +962,73 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-and-remove-tag-property + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-remove-tag-property")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "upsert-remove-tag-property-graph" + tag-name "CliQuoteTagX" + property-name "CliOwnerPropX" + property-name-lc (common-util/page-name-sanity-lc property-name) + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + upsert-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" tag-name] data-dir cfg-path) + upsert-tag-payload (parse-json-output upsert-tag-result) + upsert-property-result (run-cli ["--repo" repo + "upsert" "property" + "--name" property-name + "--type" "node" + "--cardinality" "many"] + data-dir cfg-path) + upsert-property-payload (parse-json-output upsert-property-result) + update-property-result (run-cli ["--repo" repo + "upsert" "property" + "--name" property-name + "--type" "node" + "--cardinality" "one"] + data-dir cfg-path) + update-property-payload (parse-json-output update-property-result) + property-schema-before-remove (run-query data-dir cfg-path repo + "[:find ?type ?cardinality :in $ ?name :where [?p :block/name ?name] [?p :logseq.property/type ?type] [?p :db/cardinality ?cardinality]]" + (pr-str [property-name-lc])) + remove-tag-result (run-cli ["--repo" repo "remove" "tag" "--name" tag-name] data-dir cfg-path) + remove-tag-payload (parse-json-output remove-tag-result) + remove-property-result (run-cli ["--repo" repo "remove" "property" "--name" property-name] data-dir cfg-path) + remove-property-payload (parse-json-output remove-property-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + list-property-result (run-cli ["--repo" repo "list" "property"] data-dir cfg-path) + list-property-payload (parse-json-output list-property-result) + tag-names (->> (get-in list-tag-payload [:data :items]) + (map #(or (:block/title %) (:title %) (:name %))) + set) + property-names (->> (get-in list-property-payload [:data :items]) + (map #(or (:block/title %) (:title %) (:name %))) + set) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code upsert-tag-result))) + (is (= "ok" (:status upsert-tag-payload))) + (is (= 0 (:exit-code upsert-property-result))) + (is (= "ok" (:status upsert-property-payload))) + (is (= 0 (:exit-code update-property-result))) + (is (= "ok" (:status update-property-payload))) + (is (= [["node" "one"]] + (get-in property-schema-before-remove [:data :result]))) + (is (= 0 (:exit-code remove-tag-result))) + (is (= "ok" (:status remove-tag-payload)) + (pr-str remove-tag-payload)) + (is (= 0 (:exit-code remove-property-result))) + (is (= "ok" (:status remove-property-payload)) + (pr-str remove-property-payload)) + (is (not (contains? tag-names tag-name))) + (is (not (contains? property-names property-name))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-query (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-query") From 67645700cccccfc07e67c02853c80dec06508b31 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 2 Mar 2026 18:59:48 +0800 Subject: [PATCH 093/375] 044-logseq-cli-upsert-block-page.md --- .../044-logseq-cli-upsert-block-page.md | 191 ++++++ docs/cli/logseq-cli.md | 23 +- src/main/logseq/cli/command/add.cljs | 547 +++++++++--------- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/update.cljs | 34 +- src/main/logseq/cli/command/upsert.cljs | 283 ++++++++- src/main/logseq/cli/commands.cljs | 45 +- src/main/logseq/cli/format.cljs | 44 +- src/test/logseq/cli/commands_test.cljs | 414 ++++++++++--- src/test/logseq/cli/format_test.cljs | 24 +- src/test/logseq/cli/integration_test.cljs | 360 ++++++++---- 11 files changed, 1385 insertions(+), 582 deletions(-) create mode 100644 docs/agent-guide/044-logseq-cli-upsert-block-page.md diff --git a/docs/agent-guide/044-logseq-cli-upsert-block-page.md b/docs/agent-guide/044-logseq-cli-upsert-block-page.md new file mode 100644 index 0000000000..43bf316684 --- /dev/null +++ b/docs/agent-guide/044-logseq-cli-upsert-block-page.md @@ -0,0 +1,191 @@ +# Logseq CLI Upsert Block and Upsert Page Implementation Plan + +Goal: Consolidate block and page write commands by replacing `add block`, `add page`, and `update` with `upsert block` and `upsert page` while preserving current db-worker-node write behavior. + +Architecture: Keep db-worker-node RPC and outliner operation contracts unchanged, and implement command consolidation in CLI parsing, action building, execution, and formatting layers. +Architecture: Reuse existing helper logic from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` first, then fold shared behavior into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +Architecture: Route all mutations through existing `:thread-api/apply-outliner-ops` and `:thread-api/pull` calls so `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` require no new thread APIs. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Logseq CLI transport, db-worker-node, and outliner ops. + +Related: Relates to `docs/agent-guide/027-logseq-cli-update-command.md` and builds on `docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md`. +Document naming follows @planning-documents with sequence `044`. + +## Problem statement + +The current CLI splits block mutations across `add block` and `update`, while page writes are exposed as `add page`. +This creates an inconsistent user model and duplicates validation and formatting paths in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +The current block update property options only support built-in properties, which prevents consistent upsert behavior for custom properties. +The current `add page` flow applies tags after create, but page property behavior for existing pages is not fully upsert-like because `create-page` may no-op when the page exists. +The db-worker-node layer already exposes stable generic APIs, so this feature should be implemented as a CLI surface refactor without protocol changes. + +## Testing Plan + +I will use @test-driven-development for all implementation batches. +I will write all RED tests for parser, action builder, formatter, and integration flows before changing implementation behavior. +I will add parser and builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert block` and `upsert page` command forms and for hard-removal behavior of `add` and `update`. +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `:upsert-block` and `:upsert-page`. +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that verify block creation, block update, block move, page creation, and page update through only `upsert` commands. +I will add one integration test that verifies `upsert page` updates properties on an existing page, which closes the current `add page` gap. +I will verify RED failures come from missing behavior and not from broken test setup. +I will run focused GREEN tests after minimal implementation, then refactor, then rerun focused tests and full lint-test. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Area | Current implementation | Change target | +| --- | --- | --- | +| Command entries | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` defines `["add" "block"]` and `["add" "page"]`, and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` defines `["update"]`. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` defines `["upsert" "block"]` and `["upsert" "page"]` together with existing upsert subcommands. | +| Validation and dispatch | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` validates and dispatches `:add-block`, `:add-page`, and `:update-block` separately. | Replace with `:upsert-block` and `:upsert-page` parse and dispatch paths. | +| Help and group summaries | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` top-level summary and group handling still expose `add` and `update`. | Expose only `upsert` for these write cases. | +| Formatter | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` has `format-add-block`, `format-add-page`, and `format-update-block`. | Replace with upsert-focused formatter routes while preserving existing output contract style. | +| Worker APIs | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` already provide `:thread-api/apply-outliner-ops`. | No new worker endpoint or transport shape. | +| Page create behavior | `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/page.cljs` `create!` can return existing page with no transaction. | `upsert page` applies properties and tags explicitly on existing page to ensure real upsert behavior. | + +## Interface contract proposal + +`upsert block` supports two modes with deterministic priority. +If `--id` or `--uuid` is provided, `upsert block` always runs update mode. +If neither `--id` nor `--uuid` is provided, `upsert block` runs create mode. +For `upsert block` update mode, add, update, and remove property options must support all existing properties, not only built-in properties. +`upsert page` requires `--page` and always resolves an existing or newly created page, then applies add, update, and remove semantics for tags and properties. +For `upsert page`, all add, update, and remove tag or property operations require the target tag and property to already exist, otherwise the command returns an error. + +| Command | Input signal | Behavior | Existing code path to reuse | +| --- | --- | --- | --- | +| `upsert block` create mode | `--id` and `--uuid` are absent and a content source is present. | Insert blocks under target with existing add semantics and support add, update, and remove tag or property options in post-insert ops. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` add helpers plus update option helpers. | +| `upsert block` update mode | `--id` or `--uuid` is present. | Move and or add, update, and remove tags or properties with existing update semantics, and support both built-in and custom properties in property options. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` update helpers. | +| `upsert page` | `--page` is present. | Create page if missing, then apply add, update, and remove tags and properties for both new and existing pages, with strict existing-tag and existing-property validation. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` plus explicit remove op wiring. | + +## Architecture sketch + +```text +CLI args + -> parse-args in commands.cljs + -> build-action returns :upsert-block or :upsert-page + -> execute routes to upsert command executor + -> transport/invoke :thread-api/apply-outliner-ops and :thread-api/pull + -> db-worker-node /v1/invoke passthrough + -> db_core thread-api handlers and outliner-op/apply-ops! +``` + +## Plan + +1. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert block` create mode with `--content` and for update mode with `--id` plus `--update-tags`. +2. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that confirm `--id` or `--uuid` forces update mode even when create inputs are also present. +3. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert page --page ` with tags and properties. +4. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to define expected unknown-command behavior for legacy `add block`, `add page`, and `update`. +5. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-block` action shape in create mode. +6. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-block` action shape in update mode, including custom property option inputs. +7. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-page` action shape including resolved options. +8. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-block` delegates to insert-style ops for create mode. +9. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-block` delegates to move and property ops for update mode across built-in and custom properties. +10. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-page` applies add, update, and remove property or tag ops on an already existing page. +11. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-page` returns errors when any referenced tag or property does not exist. +12. Add RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output text of `:upsert-block` and `:upsert-page`. +13. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert block` create mode id outputs. +14. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert block` update mode move behavior and custom property updates. +15. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page` create and update-existing behaviors. +16. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page` erroring when referenced tags or properties do not exist. +17. Run focused RED commands and confirm failures are expectation failures rather than transport or fixture errors. +18. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` spec to include block and page options while keeping existing tag and property specs. +19. Implement `build-block-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to classify create mode versus update mode with `--id` or `--uuid` priority, and normalize property options for all property identifiers. +20. Implement `build-page-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` for `upsert page`. +21. Extract or reuse add helper functions from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` for reading blocks, parsing tags, parsing properties, and resolving ids. +22. Extract or reuse update helper functions from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` for source and target resolution and move option mapping. +23. Implement `execute-upsert-block` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` that branches by mode and calls reused logic without behavior drift, including custom property update support. +24. Implement `execute-upsert-page` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so add, update, and remove property or tag ops are applied after resolving the page entity in both create and existing-page paths. +25. Enforce strict `upsert page` validation and execution behavior where missing tags or properties fail fast instead of creating missing entities. +26. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` table entries and finalize-command validation for `:upsert-block` and `:upsert-page`. +27. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` build and execute case dispatch to remove `:add-block`, `:add-page`, and `:update-block` routing. +28. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` top-level summary and group-help triggers to reflect the new command family. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with `format-upsert-block` and `format-upsert-page` and command dispatch keys. +30. Keep `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` unchanged unless test evidence proves a missing worker behavior. +31. Update CLI documentation in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to replace add and update examples with upsert equivalents. +32. Run focused GREEN tests and confirm parser, builder, formatter, and integration cases pass. +33. Refactor duplicated helper wiring between add, update, and upsert modules while preserving test behavior. +34. Rerun focused test set after refactor to confirm no regressions. +35. Run `bb dev:lint-and-test` as final regression verification. + +## Edge cases + +`upsert block` with `--id` or `--uuid` plus create inputs must deterministically run update mode because source selectors have priority. +`upsert block` update mode must keep current `--pos` validation where `sibling` is invalid for page targets and `--pos` requires a target. +`upsert block` update mode property options must support all existing properties, including non built-in properties. +`upsert block` create mode must preserve current default target fallback to today journal when no target selector is provided. +`upsert block` create mode with `--blocks` or `--blocks-file` must keep the existing restriction that tags and properties cannot be combined if that restriction is still required by current insert behavior. +`upsert block` and `upsert page` must support remove options for tags and properties in addition to add and update options. +`upsert page` must return stable `data.result` id vectors for JSON, EDN, and human output just like current add command id outputs. +`upsert page` on an existing page must apply property updates and removals explicitly so upsert semantics are true for both create and existing states. +`upsert page` must error when any tag or property referenced by add, update, or remove options does not already exist. +Legacy command behavior must be hard removal with standard `unknown-command` errors for `add block`, `add page`, and `update`. +Help output must not regress command grouping or ANSI formatting alignment in `commands_test`. + +## Verification commands and expected output + +Run parser and builder tests during RED. + +```bash +bb dev:test -v logseq.cli.commands-test +``` + +Expected RED behavior is failing assertions for missing `upsert block` and `upsert page` paths before implementation. + +Run formatter tests during RED. + +```bash +bb dev:test -v logseq.cli.format-test +``` + +Expected RED behavior is failing assertions for unknown command formatter branches for upsert block and upsert page. + +Run focused integration tests after implementation. + +```bash +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-block-create-json-output-returns-ids +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-block-update-move +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-page-create-and-update-existing +``` + +Expected GREEN behavior is zero failures and zero errors for these tests. + +Run full verification. + +```bash +bb dev:lint-and-test +``` + +Expected GREEN behavior is full suite pass with exit code `0`. + +## Testing Details + +Behavior tests will assert command-level outcomes through real CLI execution and Datascript queries instead of only checking mock invocation counts. +Unit-level command tests will assert parse, validation, and action-shape behavior at module boundaries. +Integration tests will verify persisted graph state changes for both create and update paths of upsert block and upsert page. + +## Implementation Details + +- Keep db-worker-node API contracts unchanged and implement all command-surface changes in CLI modules only. +- Add `upsert block` and `upsert page` entries to `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Reuse add and update helper functions to minimize behavior drift and reduce migration risk. +- Ensure `upsert block` and `upsert page` support add, update, and remove options for tags and properties. +- Ensure `upsert block` property update options accept all existing properties, including custom properties, not only built-in properties. +- Ensure `upsert page` applies tags and properties after resolving page entity so existing pages are updated too. +- Ensure `upsert page` fails fast when referenced tags or properties do not exist, and never auto-creates them through upsert-page mutation options. +- Remove old command routes in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` for `add block`, `add page`, and `update`, returning standard `unknown-command`. +- Update formatter dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` for new command ids. +- Update command summaries in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` to keep help output accurate. +- Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` examples and command reference text. +- Keep implementation batches aligned to @test-driven-development RED, GREEN, and refactor phases. + +## Question + +No open questions. +Decided: remove `add block`, `add page`, and `update` immediately. +Decided: in `upsert block`, `--id` or `--uuid` means update mode and absence of both means create mode. +Decided: support add, update, and remove semantics for tags and properties. +Decided: in `upsert block` update mode, property mutation options support all existing properties, including custom properties. +Decided: for `upsert page`, add, update, and remove tag or property options require existing tags and properties, otherwise return error. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 41cfe0ba11..17dd89fa96 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -94,10 +94,11 @@ Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags - `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties -- `add block --content [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - add blocks; defaults to today’s journal page if no target is given -- `add block --blocks [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector -- `add block --blocks-file [--target-page-name |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file -- `add page --page ` - create a page +- `upsert block --content [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - create blocks; defaults to today’s journal page if no target is given +- `upsert block --blocks [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector +- `upsert block --blocks-file [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file +- `upsert block --id |--uuid [--target-id |--target-uuid |--target-page ] [--pos first-child|last-child|sibling] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update and/or move a block +- `upsert page --page [--tags ] [--properties ] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - create or update a page - `move --id |--uuid --target-id |--target-uuid |--target-page [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) - `remove --id |--uuid |--page ` - remove blocks (by db/id or UUID) or pages - `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) @@ -115,8 +116,10 @@ Subcommands: list page [options] List pages list tag [options] List tags list property [options] List properties - add block [options] Add blocks - add page [options] Create page + upsert block [options] Upsert block + upsert page [options] Upsert page + upsert tag [options] Upsert tag + upsert property [options] Upsert property move [options] Move block remove [options] Remove block or page search [options] Search graph @@ -138,15 +141,15 @@ Output formats: - Global `--output ` applies to all commands - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. -- `add page` and `add block` return created entity ids in `data.result` for JSON/EDN output, and include ids in human output. +- `upsert page` and `upsert block` return entity ids in `data.result` for JSON/EDN output, and include ids in human output. - Human example: ```text - Added page: + Upserted page: [123] ``` - Human example: ```text - Added blocks: + Upserted blocks: [201 202] ``` - JSON example: `{"status":"ok","data":{"result":[123]}}` @@ -180,7 +183,7 @@ Examples: node ./dist/logseq.js graph create --repo demo node ./dist/logseq.js graph export --type edn --output /tmp/demo.edn --repo demo node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --repo demo-import -node ./dist/logseq.js add block --target-page-name TestPage --content "hello world" +node ./dist/logseq.js upsert block --target-page TestPage --content "hello world" node ./dist/logseq.js move --uuid --target-page TargetPage node ./dist/logseq.js search "hello" node ./dist/logseq.js show --page TestPage --output json diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 90c858923b..1bc62a057b 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -16,28 +16,6 @@ [logseq.db.frontend.property.type :as db-property-type] [promesa.core :as p])) -(def ^:private content-add-spec - {:content {:desc "Block content for add"} - :blocks {:desc "EDN vector of blocks for add"} - :blocks-file {:desc "EDN file of blocks for add"} - :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} - :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."} - :target-id {:desc "Target block db/id" - :coerce :long} - :target-uuid {:desc "Target block UUID"} - :target-page-name {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling). Default: last-child"} - :status {:desc "Task status (todo, doing, done, etc.)"}}) - -(def ^:private add-page-spec - {:page {:desc "Page name"} - :tags {:desc "EDN vector of tags. Identifiers can be id, :db/ident, or :block/title."} - :properties {:desc "EDN map of built-in properties. Identifiers can be id, :db/ident, or :block/title."}}) - -(def entries - [(core/command-entry ["add" "block"] :add-block "Add blocks" content-add-spec) - (core/command-entry ["add" "page"] :add-page "Create page" add-page-spec)]) - (defn- today-page-title [config repo] (p/let [journal (transport/invoke config :thread-api/pull false @@ -194,24 +172,6 @@ ordered-uuids))] (created-ids-in-order ordered-uuids entities :block))))) -(defn- resolve-created-page-ids - [config repo page create-result] - (let [page-uuid (some-> create-result second normalized-uuid)] - (if page-uuid - (p/let [page-entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid] [:block/uuid page-uuid]])] - (created-ids-in-order [page-uuid] [page-entity] :page)) - (p/let [page-entity (transport/invoke config :thread-api/pull false - [repo [:db/id :block/uuid] - [:block/name (common-util/page-name-sanity-lc page)]]) - page-id (:db/id page-entity)] - (if (some? page-id) - [page-id] - (throw (ex-info "unable to resolve created page id" - {:code :add-id-resolution-failed - :entity-kind :page - :page page}))))))) - (defn- extract-page-refs [title] (when (string? title) @@ -531,85 +491,95 @@ (true? (get-in property [:schema :public?]))) (defn parse-properties-option - [value] - (if-not (seq value) - {:ok? true :value nil} - (let [parsed (parse-edn-option value)] - (cond - (nil? parsed) - (invalid-options-result "properties must be valid EDN map") + ([value] + (parse-properties-option value {:allow-non-built-in? false})) + ([value {:keys [allow-non-built-in?] + :or {allow-non-built-in? false}}] + (if-not (seq value) + {:ok? true :value nil} + (let [parsed (parse-edn-option value)] + (cond + (nil? parsed) + (invalid-options-result "properties must be valid EDN map") - (not (map? parsed)) - (invalid-options-result "properties must be a map") + (not (map? parsed)) + (invalid-options-result "properties must be a map") - (empty? parsed) - (invalid-options-result "properties must be a non-empty map") + (empty? parsed) + (invalid-options-result "properties must be a non-empty map") - :else - (loop [prop-entries (seq parsed) - acc {}] - (if (empty? prop-entries) - {:ok? true :value acc} - (let [[k v] (first prop-entries) - key-result (normalize-property-key-input k)] - (if-not key-result - (invalid-options-result (str "invalid property key: " k)) - (let [{:keys [type value]} key-result - key-ident value] - (if (= type :id) - (recur (rest prop-entries) (assoc acc key-ident v)) - (let [property (get db-property/built-in-properties key-ident)] - (cond - (nil? property) - (invalid-options-result (str "unknown built-in property: " key-ident)) + :else + (loop [prop-entries (seq parsed) + acc {}] + (if (empty? prop-entries) + {:ok? true :value acc} + (let [[k v] (first prop-entries) + key-result (normalize-property-key-input k)] + (if-not key-result + (invalid-options-result (str "invalid property key: " k)) + (let [{:keys [type value]} key-result + key-ident value] + (if (= type :id) + (recur (rest prop-entries) (assoc acc key-ident v)) + (let [property (get db-property/built-in-properties key-ident)] + (cond + (nil? property) + (if allow-non-built-in? + (recur (rest prop-entries) (assoc acc key-ident v)) + (invalid-options-result (str "unknown built-in property: " key-ident))) - (not (property-public? property)) - (invalid-options-result (str "property is not public: " key-ident)) + (not (property-public? property)) + (invalid-options-result (str "property is not public: " key-ident)) - :else - (let [{:keys [ok? value message]} (normalize-property-values property v) - normalized-value value] - (if-not ok? - (invalid-options-result (str "invalid value for " key-ident ": " message)) - (recur (rest prop-entries) (assoc acc key-ident normalized-value)))))))))))))))) + :else + (let [{:keys [ok? value message]} (normalize-property-values property v) + normalized-value value] + (if-not ok? + (invalid-options-result (str "invalid value for " key-ident ": " message)) + (recur (rest prop-entries) (assoc acc key-ident normalized-value))))))))))))))))) (defn parse-properties-vector-option - [value] - (if-not (seq value) - {:ok? true :value nil} - (let [parsed (parse-edn-option value)] - (cond - (nil? parsed) - (invalid-options-result "properties must be valid EDN vector") + ([value] + (parse-properties-vector-option value {:allow-non-built-in? false})) + ([value {:keys [allow-non-built-in?] + :or {allow-non-built-in? false}}] + (if-not (seq value) + {:ok? true :value nil} + (let [parsed (parse-edn-option value)] + (cond + (nil? parsed) + (invalid-options-result "properties must be valid EDN vector") - (not (vector? parsed)) - (invalid-options-result "properties must be a vector") + (not (vector? parsed)) + (invalid-options-result "properties must be a vector") - (empty? parsed) - (invalid-options-result "properties must be a non-empty vector") + (empty? parsed) + (invalid-options-result "properties must be a non-empty vector") - :else - (loop [prop-entries (seq parsed) - acc []] - (if (empty? prop-entries) - {:ok? true :value acc} - (let [entry (first prop-entries) - key-result (normalize-property-key-input entry)] - (if-not key-result - (invalid-options-result (str "invalid property key: " entry)) - (let [{:keys [type value]} key-result] - (if (= type :id) - (recur (rest prop-entries) (conj acc value)) - (let [property (get db-property/built-in-properties value)] - (cond - (nil? property) - (invalid-options-result (str "unknown built-in property: " value)) + :else + (loop [prop-entries (seq parsed) + acc []] + (if (empty? prop-entries) + {:ok? true :value acc} + (let [entry (first prop-entries) + key-result (normalize-property-key-input entry)] + (if-not key-result + (invalid-options-result (str "invalid property key: " entry)) + (let [{:keys [type value]} key-result] + (if (= type :id) + (recur (rest prop-entries) (conj acc value)) + (let [property (get db-property/built-in-properties value)] + (cond + (nil? property) + (if allow-non-built-in? + (recur (rest prop-entries) (conj acc value)) + (invalid-options-result (str "unknown built-in property: " value))) - (not (property-public? property)) - (invalid-options-result (str "property is not public: " value)) + (not (property-public? property)) + (invalid-options-result (str "property is not public: " value)) - :else - (recur (rest prop-entries) (conj acc value)))))))))))))) + :else + (recur (rest prop-entries) (conj acc value))))))))))))))) (defn invalid-options? [opts] @@ -755,129 +725,219 @@ :date (resolve-date-page-id config repo value) (p/resolved value)))) +(def ^:private property-entity-selector + [:db/id :db/ident :block/name :block/title + :logseq.property/type :db/cardinality :logseq.property/public?]) + +(defn- property-entity? + [entity] + (some? (:logseq.property/type entity))) + +(defn- property-entity-public? + [entity] + (not (false? (:logseq.property/public? entity)))) + +(defn- property-entity->property + [entity] + {:schema {:type (or (:logseq.property/type entity) :default) + :cardinality (if (= :db.cardinality/many (:db/cardinality entity)) + :many + :one) + :public? (property-entity-public? entity)}}) + +(defn- lookup-property-entity + [config repo property-key] + (let [lookup-by-title (fn [title] + (pull-entity config repo property-entity-selector + [:block/name (common-util/page-name-sanity-lc title)]))] + (cond + (number? property-key) + (pull-entity config repo property-entity-selector property-key) + + (keyword? property-key) + (p/let [entity (pull-entity config repo property-entity-selector [:db/ident property-key])] + (if (or (:db/id entity) (qualified-keyword? property-key)) + entity + (lookup-by-title (name property-key)))) + + (string? property-key) + (let [text (string/trim property-key) + ident (normalize-property-key text)] + (if-not (seq text) + (p/resolved nil) + (p/let [entity (lookup-by-title text)] + (if (:db/id entity) + entity + (if ident + (pull-entity config repo property-entity-selector [:db/ident ident]) + (p/resolved nil)))))) + + :else + (p/resolved nil)))) + +(defn- resolve-property-entry-allow-non-built-in + [config repo property-key] + (p/let [entity (lookup-property-entity config repo property-key) + ident (:db/ident entity)] + (cond + (nil? (:db/id entity)) + (throw (ex-info "property not found" + {:code :property-not-found + :property property-key})) + + (not (property-entity? entity)) + (throw (ex-info "target is not a property" + {:code :invalid-property-target + :property property-key})) + + (nil? ident) + (throw (ex-info "property not found" + {:code :property-not-found + :property property-key})) + + (not (property-entity-public? entity)) + (throw (ex-info "property is not public" + {:code :property-not-public + :property ident})) + + :else + {:ident ident + :property (property-entity->property entity)}))) + (defn resolve-properties - [config repo properties] - (if-not (seq properties) - (p/resolved nil) - (p/let [resolved-entries (p/all - (map (fn [[k v]] - (p/let [{:keys [ident property]} - (cond - (keyword? k) - (let [property (get db-property/built-in-properties k)] - (when-not property - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property k}))) - (when-not (property-public? property) - (throw (ex-info "property is not public" - {:code :property-not-public :property k}))) - (p/resolved {:ident k :property property})) + ([config repo properties] + (resolve-properties config repo properties {:allow-non-built-in? false})) + ([config repo properties {:keys [allow-non-built-in?] + :or {allow-non-built-in? false}}] + (if-not (seq properties) + (p/resolved nil) + (p/let [resolved-entries (p/all + (map (fn [[k v]] + (p/let [{:keys [ident property]} + (if allow-non-built-in? + (resolve-property-entry-allow-non-built-in config repo k) + (cond + (keyword? k) + (let [property (get db-property/built-in-properties k)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property k}))) + (p/resolved {:ident k :property property})) - (number? k) - (p/let [entity (pull-entity config repo [:db/ident] k) - ident (:db/ident entity) - property (get db-property/built-in-properties ident)] - (cond - (nil? ident) - (throw (ex-info "property not found" - {:code :property-not-found :property k})) + (number? k) + (p/let [entity (pull-entity config repo [:db/ident] k) + ident (:db/ident entity) + property (get db-property/built-in-properties ident)] + (cond + (nil? ident) + (throw (ex-info "property not found" + {:code :property-not-found :property k})) - (nil? property) - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property ident})) + (nil? property) + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property ident})) - (not (property-public? property)) - (throw (ex-info "property is not public" - {:code :property-not-public :property ident})) + (not (property-public? property)) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident})) - :else - {:ident ident :property property})) + :else + {:ident ident :property property})) - (string? k) - (let [ident (or (property-title->ident k) - (normalize-property-key k)) - property (get db-property/built-in-properties ident)] - (when-not property - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property k}))) - (when-not (property-public? property) - (throw (ex-info "property is not public" - {:code :property-not-public :property ident}))) - (p/resolved {:ident ident :property property})) + (string? k) + (let [ident (or (property-title->ident k) + (normalize-property-key k)) + property (get db-property/built-in-properties ident)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident}))) + (p/resolved {:ident ident :property property})) - :else - (p/rejected (ex-info "invalid property key" - {:code :invalid-property :property k}))) - {:keys [ok? value message]} (normalize-property-values property v)] - (when-not ok? - (throw (ex-info "invalid property value" - {:code :invalid-property-value - :property ident - :message message}))) - (let [many? (= :many (get-in property [:schema :cardinality])) - values (if many? - (if (and (coll? value) (not (string? value))) value [value]) - [value])] - (p/let [resolved (p/all (map #(resolve-property-value config repo property %) values)) - final-value (if many? (vec resolved) (first resolved))] - [ident final-value])))) - properties))] - (into {} resolved-entries)))) + :else + (p/rejected (ex-info "invalid property key" + {:code :invalid-property :property k})))) + {:keys [ok? value message]} (normalize-property-values property v)] + (when-not ok? + (throw (ex-info "invalid property value" + {:code :invalid-property-value + :property ident + :message message}))) + (let [many? (= :many (get-in property [:schema :cardinality])) + values (if many? + (if (and (coll? value) (not (string? value))) value [value]) + [value])] + (p/let [resolved (p/all (map #(resolve-property-value config repo property %) values)) + final-value (if many? (vec resolved) (first resolved))] + [ident final-value])))) + properties))] + (into {} resolved-entries))))) (defn resolve-property-identifiers - [config repo properties] - (if-not (seq properties) - (p/resolved nil) - (p/let [resolved-entries (p/all - (map (fn [k] - (cond - (keyword? k) - (let [property (get db-property/built-in-properties k)] - (when-not property - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property k}))) - (when-not (property-public? property) - (throw (ex-info "property is not public" - {:code :property-not-public :property k}))) - (p/resolved k)) + ([config repo properties] + (resolve-property-identifiers config repo properties {:allow-non-built-in? false})) + ([config repo properties {:keys [allow-non-built-in?] + :or {allow-non-built-in? false}}] + (if-not (seq properties) + (p/resolved nil) + (p/let [resolved-entries (p/all + (map (fn [k] + (if allow-non-built-in? + (p/let [{:keys [ident]} (resolve-property-entry-allow-non-built-in config repo k)] + ident) + (cond + (keyword? k) + (let [property (get db-property/built-in-properties k)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property k}))) + (p/resolved k)) - (number? k) - (p/let [entity (pull-entity config repo [:db/ident] k) - ident (:db/ident entity) - property (get db-property/built-in-properties ident)] - (cond - (nil? ident) - (throw (ex-info "property not found" - {:code :property-not-found :property k})) + (number? k) + (p/let [entity (pull-entity config repo [:db/ident] k) + ident (:db/ident entity) + property (get db-property/built-in-properties ident)] + (cond + (nil? ident) + (throw (ex-info "property not found" + {:code :property-not-found :property k})) - (nil? property) - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property ident})) + (nil? property) + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property ident})) - (not (property-public? property)) - (throw (ex-info "property is not public" - {:code :property-not-public :property ident})) + (not (property-public? property)) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident})) - :else - ident)) + :else + ident)) - (string? k) - (let [ident (or (property-title->ident k) - (normalize-property-key k)) - property (get db-property/built-in-properties ident)] - (when-not property - (throw (ex-info "unknown built-in property" - {:code :unknown-property :property k}))) - (when-not (property-public? property) - (throw (ex-info "property is not public" - {:code :property-not-public :property ident}))) - (p/resolved ident)) + (string? k) + (let [ident (or (property-title->ident k) + (normalize-property-key k)) + property (get db-property/built-in-properties ident)] + (when-not property + (throw (ex-info "unknown built-in property" + {:code :unknown-property :property k}))) + (when-not (property-public? property) + (throw (ex-info "property is not public" + {:code :property-not-public :property ident}))) + (p/resolved ident)) - :else - (p/rejected (ex-info "invalid property key" - {:code :invalid-property :property k})))) - properties))] - (vec resolved-entries)))) + :else + (p/rejected (ex-info "invalid property key" + {:code :invalid-property :property k}))))) + properties))] + (vec resolved-entries))))) (defn- resolve-add-target [config {:keys [repo target-id target-uuid target-page-name]}] @@ -978,35 +1038,6 @@ :properties properties :blocks blocks}})))))))) -(defn build-add-page-action - [options repo] - (if-not (seq repo) - {:ok? false - :error {:code :missing-repo - :message "repo is required for add"}} - (let [page (some-> (:page options) string/trim)] - (if (seq page) - (let [tags-result (parse-tags-option (:tags options)) - properties-result (parse-properties-option (:properties options))] - (cond - (not (:ok? tags-result)) - tags-result - - (not (:ok? properties-result)) - properties-result - - :else - {:ok? true - :action {:type :add-page - :repo repo - :graph (core/repo->graph repo) - :page page - :tags (:value tags-result) - :properties (:value properties-result)}})) - {:ok? false - :error {:code :missing-page-name - :message "page name is required"}})))) - (defn execute-add-block [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) @@ -1065,31 +1096,3 @@ created-ids (resolve-created-block-ids cfg (:repo action) blocks-for-insert insert-result)] {:status :ok :data {:result created-ids}}))) - -(defn execute-add-page - [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - tags (resolve-tags cfg (:repo action) (:tags action)) - tag-ids (when (seq tags) - (->> tags (map :db/id) (remove nil?) vec)) - properties (resolve-properties cfg (:repo action) (:properties action)) - options (cond-> {} - (seq properties) (assoc :properties properties)) - ops [[:create-page [(:page action) options]]] - create-result (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}]) - _ (when (seq tag-ids) - (p/let [page-name (common-util/page-name-sanity-lc (:page action)) - page (pull-entity cfg (:repo action) [:db/id :block/uuid] [:block/name page-name]) - page-uuid (:block/uuid page)] - (when-not page-uuid - (throw (ex-info "page not found" {:code :page-not-found :page (:page action)}))) - (p/all - (map (fn [tag-id] - (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) - [[:batch-set-property [[page-uuid] :block/tags tag-id {}]]] - {}])) - tag-ids)))) - created-ids (resolve-created-page-ids cfg (:repo action) (:page action) create-result)] - {:status :ok - :data {:result created-ids}}))) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index cb9ac273d3..1a99ba6ddf 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -93,7 +93,7 @@ (defn top-level-summary [table] (let [groups [{:title "Graph Inspect and Edit" - :commands #{"list" "add" "upsert" "remove" "update" "query" "show"}} + :commands #{"list" "upsert" "remove" "query" "show"}} {:title "Graph Management" :commands #{"graph" "server" "doctor"}}] render-group (fn [{:keys [title commands]}] diff --git a/src/main/logseq/cli/command/update.cljs b/src/main/logseq/cli/command/update.cljs index 4f6fa24d0c..8e3f5e8fa4 100644 --- a/src/main/logseq/cli/command/update.cljs +++ b/src/main/logseq/cli/command/update.cljs @@ -8,23 +8,6 @@ [logseq.common.util :as common-util] [promesa.core :as p])) -(def ^:private update-spec - {:id {:desc "Source block db/id" - :coerce :long} - :uuid {:desc "Source block UUID"} - :target-id {:desc "Target block db/id" - :coerce :long} - :target-uuid {:desc "Target block UUID"} - :target-page {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling). Default: first-child"} - :update-tags {:desc "Tags to add/update (EDN vector)"} - :update-properties {:desc "Properties to add/update (EDN map)"} - :remove-tags {:desc "Tags to remove (EDN vector)"} - :remove-properties {:desc "Properties to remove (EDN vector)"}}) - -(def entries - [(core/command-entry ["update"] :update-block "Update block" update-spec)]) - (def ^:private update-positions #{"first-child" "last-child" "sibling"}) @@ -152,9 +135,13 @@ page-name (some-> (:target-page options) string/trim) pos (some-> (:pos options) string/trim string/lower-case) update-tags-result (add-command/parse-tags-option (:update-tags options)) - update-properties-result (add-command/parse-properties-option (:update-properties options)) + update-properties-result (add-command/parse-properties-option + (:update-properties options) + {:allow-non-built-in? true}) remove-tags-result (add-command/parse-tags-vector-option (:remove-tags options)) - remove-properties-result (add-command/parse-properties-vector-option (:remove-properties options)) + remove-properties-result (add-command/parse-properties-vector-option + (:remove-properties options) + {:allow-non-built-in? true}) update-tags (:value update-tags-result) update-properties (:value update-properties-result) remove-tags (:value remove-tags-result) @@ -222,9 +209,12 @@ opts (when target (pos->opts (:pos action))) update-tags (add-command/resolve-tags cfg (:repo action) (:update-tags action)) remove-tags (add-command/resolve-tags cfg (:repo action) (:remove-tags action)) - update-properties (add-command/resolve-properties cfg (:repo action) (:update-properties action)) - remove-properties (add-command/resolve-property-identifiers cfg (:repo action) - (:remove-properties action)) + update-properties (add-command/resolve-properties + cfg (:repo action) (:update-properties action) + {:allow-non-built-in? true}) + remove-properties (add-command/resolve-property-identifiers + cfg (:repo action) (:remove-properties action) + {:allow-non-built-in? true}) block-id (:db/id source) block-ids [block-id] update-tag-ids (when (seq update-tags) diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index 94b91d5a6b..211a56dcf3 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -1,12 +1,43 @@ (ns logseq.cli.command.upsert "Upsert-related CLI commands." (:require [clojure.string :as string] + [logseq.cli.command.add :as add-command] [logseq.cli.command.core :as core] + [logseq.cli.command.update :as update-command] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [promesa.core :as p])) +(def ^:private upsert-block-spec + {:id {:desc "Source block db/id (forces update mode)" + :coerce :long} + :uuid {:desc "Source block UUID (forces update mode)"} + :target-id {:desc "Target block db/id" + :coerce :long} + :target-uuid {:desc "Target block UUID"} + :target-page {:desc "Target page name"} + :pos {:desc "Position (first-child, last-child, sibling). Default: create=last-child, update=first-child"} + :content {:desc "Block content for create mode"} + :blocks {:desc "EDN vector of blocks for create mode"} + :blocks-file {:desc "EDN file of blocks for create mode"} + :status {:desc "Task status (todo, doing, done, etc.)"} + :tags {:desc "Tags to add in create mode (EDN vector). Identifiers can be id, :db/ident, or :block/title."} + :properties {:desc "Properties to add in create mode (EDN map). Identifiers can be id, :db/ident, or :block/title."} + :update-tags {:desc "Tags to add/update (EDN vector)"} + :update-properties {:desc "Properties to add/update (EDN map)"} + :remove-tags {:desc "Tags to remove (EDN vector)"} + :remove-properties {:desc "Properties to remove (EDN vector)"}}) + +(def ^:private upsert-page-spec + {:page {:desc "Page name"} + :tags {:desc "Tags to add (EDN vector). Identifiers can be id, :db/ident, or :block/title."} + :properties {:desc "Properties to add (EDN map). Identifiers can be id, :db/ident, or :block/title."} + :update-tags {:desc "Tags to add/update (EDN vector)"} + :update-properties {:desc "Properties to add/update (EDN map)"} + :remove-tags {:desc "Tags to remove (EDN vector)"} + :remove-properties {:desc "Properties to remove (EDN vector)"}}) + (def ^:private upsert-tag-spec {:name {:desc "Tag name"}}) @@ -20,7 +51,9 @@ :coerce :boolean}}) (def entries - [(core/command-entry ["upsert" "tag"] :upsert-tag "Upsert tag" upsert-tag-spec) + [(core/command-entry ["upsert" "block"] :upsert-block "Upsert block" upsert-block-spec) + (core/command-entry ["upsert" "page"] :upsert-page "Upsert page" upsert-page-spec) + (core/command-entry ["upsert" "tag"] :upsert-tag "Upsert tag" upsert-tag-spec) (core/command-entry ["upsert" "property"] :upsert-property "Upsert property" upsert-property-spec)]) (def ^:private property-types @@ -56,6 +89,16 @@ (defn invalid-options? [command opts] (case command + :upsert-block + (let [opts (cond-> opts + (seq (:target-page opts)) + (assoc :target-page-name (:target-page opts))) + update-mode? (or (some? (:id opts)) + (seq (some-> (:uuid opts) string/trim)))] + (if update-mode? + (update-command/invalid-options? opts) + (add-command/invalid-options? opts))) + :upsert-property (let [type' (normalize-property-type (:type opts)) cardinality' (normalize-property-cardinality (:cardinality opts))] @@ -71,6 +114,115 @@ nil)) +(defn update-mode? + [opts] + (or (some? (:id opts)) + (seq (some-> (:uuid opts) string/trim)))) + +(defn build-block-action + [options args repo] + (let [update-mode* (update-mode? options)] + (if update-mode* + (let [options (cond-> options + (seq (:target-page options)) + (assoc :target-page (:target-page options)))] + (-> (update-command/build-action options repo) + (update :action + (fn [action] + (when action + (assoc action :type :upsert-block :mode :update)))))) + (let [options (cond-> options + (seq (:target-page options)) + (assoc :target-page-name (:target-page options)) + true + (dissoc :target-page)) + create-result (add-command/build-add-block-action options args repo) + update-tags-result (add-command/parse-tags-option (:update-tags options)) + update-properties-result (add-command/parse-properties-option + (:update-properties options) + {:allow-non-built-in? true}) + remove-tags-result (add-command/parse-tags-vector-option (:remove-tags options)) + remove-properties-result (add-command/parse-properties-vector-option + (:remove-properties options) + {:allow-non-built-in? true})] + (cond + (not (:ok? create-result)) + create-result + + (not (:ok? update-tags-result)) + update-tags-result + + (not (:ok? update-properties-result)) + update-properties-result + + (not (:ok? remove-tags-result)) + remove-tags-result + + (not (:ok? remove-properties-result)) + remove-properties-result + + :else + (-> create-result + (update :action + (fn [action] + (-> action + (assoc :type :upsert-block + :mode :create + :update-tags (:value update-tags-result) + :update-properties (:value update-properties-result) + :remove-tags (:value remove-tags-result) + :remove-properties (:value remove-properties-result))))))))))) + +(defn build-page-action + [options repo] + (if-not (seq repo) + {:ok? false + :error {:code :missing-repo + :message "repo is required for upsert"}} + (let [page (some-> (:page options) string/trim) + tags-result (add-command/parse-tags-option (:tags options)) + properties-result (add-command/parse-properties-option (:properties options)) + update-tags-result (add-command/parse-tags-option (:update-tags options)) + update-properties-result (add-command/parse-properties-option (:update-properties options)) + remove-tags-result (add-command/parse-tags-vector-option (:remove-tags options)) + remove-properties-result (add-command/parse-properties-vector-option (:remove-properties options))] + (cond + (not (seq page)) + {:ok? false + :error {:code :missing-page-name + :message "page name is required"}} + + (not (:ok? tags-result)) + tags-result + + (not (:ok? properties-result)) + properties-result + + (not (:ok? update-tags-result)) + update-tags-result + + (not (:ok? update-properties-result)) + update-properties-result + + (not (:ok? remove-tags-result)) + remove-tags-result + + (not (:ok? remove-properties-result)) + remove-properties-result + + :else + {:ok? true + :action {:type :upsert-page + :repo repo + :graph (core/repo->graph repo) + :page page + :tags (:value tags-result) + :properties (:value properties-result) + :update-tags (:value update-tags-result) + :update-properties (:value update-properties-result) + :remove-tags (:value remove-tags-result) + :remove-properties (:value remove-properties-result)}})))) + (defn build-tag-action [options repo] (if-not (seq repo) @@ -143,6 +295,135 @@ (transport/invoke config :thread-api/pull false [repo selector [:block/name (common-util/page-name-sanity-lc page-name)]])) +(defn- ensure-property-identifiers-exist! + [config repo property-idents] + (if (seq property-idents) + (p/all + (map (fn [property-ident] + (p/let [entity (transport/invoke config :thread-api/pull false + [repo [:db/id] [:db/ident property-ident]])] + (when-not (:db/id entity) + (throw (ex-info "property not found" + {:code :property-not-found + :property property-ident}))))) + (distinct property-idents))) + (p/resolved nil))) + +(defn- ensure-page-entity! + [config repo page-name] + (p/let [existing (pull-page-by-name config repo page-name [:db/id :block/uuid])] + (if (:db/id existing) + existing + (p/let [_ (transport/invoke config :thread-api/apply-outliner-ops false + [repo [[:create-page [page-name {}]]] {}]) + created (pull-page-by-name config repo page-name [:db/id :block/uuid])] + (if (:db/id created) + created + (throw (ex-info "page not found after upsert" + {:code :page-not-found + :page page-name}))))))) + +(defn- append-tag-and-property-ops + [ops block-ids {:keys [update-tag-ids remove-tag-ids update-properties remove-properties]}] + (cond-> ops + (seq remove-tag-ids) + (into (map (fn [tag-id] + [:batch-delete-property-value [block-ids :block/tags tag-id]]) + remove-tag-ids)) + + (seq remove-properties) + (into (map (fn [property-id] + [:batch-remove-property [block-ids property-id]]) + remove-properties)) + + (seq update-tag-ids) + (into (map (fn [tag-id] + [:batch-set-property [block-ids :block/tags tag-id {}]]) + update-tag-ids)) + + (seq update-properties) + (into (map (fn [[k v]] + [:batch-set-property [block-ids k v {}]]) + update-properties)))) + +(defn- execute-extra-upsert-block-ops! + [action config block-ids] + (if (seq block-ids) + (p/let [cfg (cli-server/ensure-server! config (:repo action)) + update-tags (add-command/resolve-tags cfg (:repo action) (:update-tags action)) + remove-tags (add-command/resolve-tags cfg (:repo action) (:remove-tags action)) + update-properties (add-command/resolve-properties + cfg (:repo action) (:update-properties action) + {:allow-non-built-in? true}) + remove-properties (add-command/resolve-property-identifiers + cfg (:repo action) (:remove-properties action) + {:allow-non-built-in? true}) + update-property-idents (keys (or update-properties {})) + _ (ensure-property-identifiers-exist! cfg (:repo action) update-property-idents) + _ (ensure-property-identifiers-exist! cfg (:repo action) remove-properties) + ops (append-tag-and-property-ops [] + block-ids + {:update-tag-ids (->> update-tags (map :db/id) (remove nil?) distinct vec) + :remove-tag-ids (->> remove-tags (map :db/id) (remove nil?) distinct vec) + :update-properties update-properties + :remove-properties remove-properties})] + (when (seq ops) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) ops {}]))) + (p/resolved nil))) + +(defn execute-upsert-block + [action config] + (-> (if (= :update (:mode action)) + (update-command/execute-update (assoc action :type :update-block) config) + (p/let [result (add-command/execute-add-block (assoc action :type :add-block) config) + created-ids (vec (or (get-in result [:data :result]) [])) + _ (execute-extra-upsert-block-ops! action config created-ids)] + {:status :ok + :data {:result created-ids}})) + (p/catch (fn [e] + {:status :error + :error {:code (or (get-in (ex-data e) [:code]) :exception) + :message (or (ex-message e) (str e))}})))) + +(defn execute-upsert-page + [action config] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + page (ensure-page-entity! cfg (:repo action) (:page action)) + page-id (:db/id page) + block-ids [page-id] + add-tags (add-command/resolve-tags cfg (:repo action) (:tags action)) + update-tags (add-command/resolve-tags cfg (:repo action) (:update-tags action)) + remove-tags (add-command/resolve-tags cfg (:repo action) (:remove-tags action)) + add-properties (add-command/resolve-properties cfg (:repo action) (:properties action)) + update-properties (add-command/resolve-properties cfg (:repo action) (:update-properties action)) + remove-properties (add-command/resolve-property-identifiers cfg (:repo action) + (:remove-properties action)) + merged-properties (merge (or add-properties {}) (or update-properties {})) + _ (ensure-property-identifiers-exist! cfg (:repo action) (keys merged-properties)) + _ (ensure-property-identifiers-exist! cfg (:repo action) remove-properties) + update-tag-ids (->> (concat (or add-tags []) (or update-tags [])) + (map :db/id) + (remove nil?) + distinct + vec) + remove-tag-ids (->> remove-tags (map :db/id) (remove nil?) distinct vec) + ops (append-tag-and-property-ops [] + block-ids + {:update-tag-ids update-tag-ids + :remove-tag-ids remove-tag-ids + :update-properties merged-properties + :remove-properties remove-properties}) + _ (when (seq ops) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) ops {}]))] + {:status :ok + :data {:result [page-id]}}) + (p/catch (fn [e] + {:status :error + :error {:code (or (get-in (ex-data e) [:code]) :exception) + :message (or (ex-message e) (str e))}})))) + (defn- tag-entity? [entity] (some #(= :logseq.class/Tag (:db/ident %)) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 2a3bde01a1..d7f40c2615 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -2,7 +2,6 @@ "Command parsing and action building for the Logseq CLI." (:require [babashka.cli :as cli] [clojure.string :as string] - [logseq.cli.command.add :as add-command] [logseq.cli.command.core :as command-core] [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] @@ -12,7 +11,6 @@ [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] [logseq.cli.command.upsert :as upsert-command] - [logseq.cli.command.update :as update-command] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -49,13 +47,6 @@ :message "block or page is required"} :summary summary}) -(defn- missing-source-result - [summary] - {:ok? false - :error {:code :missing-source - :message "source block is required"} - :summary summary}) - (defn- missing-page-name-result [summary] {:ok? false @@ -113,10 +104,8 @@ (vec (concat graph-command/entries server-command/entries list-command/entries - add-command/entries upsert-command/entries remove-command/entries - update-command/entries query-command/entries show-command/entries doctor-command/entries))) @@ -162,7 +151,7 @@ (seq (:blocks-file opts)) has-args?) show-targets (filter some? [(:id opts) (:uuid opts) (:page opts)]) - update-sources (filter some? [(:id opts) (some-> (:uuid opts) string/trim)])] + upsert-update-mode? (upsert-command/update-mode? opts)] (cond (:help opts) (command-core/help-result cmd-summary) @@ -171,13 +160,13 @@ (not (seq graph))) (missing-graph-result summary) - (and (= command :add-block) (not has-content?)) + (and (= command :upsert-block) (not upsert-update-mode?) (not has-content?)) (missing-content-result summary) - (and (= command :add-block) (add-command/invalid-options? opts)) - (command-core/invalid-options-result summary (add-command/invalid-options? opts)) + (and (= command :upsert-block) (upsert-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) - (and (= command :add-page) (not (seq (:page opts)))) + (and (= command :upsert-page) (not (seq (:page opts)))) (missing-page-name-result summary) (and (= command :upsert-tag) (not (seq (some-> (:name opts) string/trim)))) @@ -199,12 +188,6 @@ (empty? (filter some? [(:id opts) (some-> (:name opts) string/trim)]))) (missing-target-result summary) - (and (= command :update-block) (update-command/invalid-options? opts)) - (command-core/invalid-options-result summary (update-command/invalid-options? opts)) - - (and (= command :update-block) (empty? update-sources)) - (missing-source-result summary) - (and (= command :show) (empty? show-targets)) (missing-target-result summary) @@ -282,7 +265,7 @@ :message "missing command"} :summary summary}) - (and (= 1 (count args)) (#{"graph" "server" "list" "add" "upsert" "remove" "query"} (first args))) + (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) :else @@ -380,11 +363,11 @@ (:list-page :list-tag :list-property) (list-command/build-action command options repo) - :add-block - (add-command/build-add-block-action options args repo) + :upsert-block + (upsert-command/build-block-action options args repo) - :add-page - (add-command/build-add-page-action options repo) + :upsert-page + (upsert-command/build-page-action options repo) :upsert-tag (upsert-command/build-tag-action options repo) @@ -392,9 +375,6 @@ :upsert-property (upsert-command/build-property-action options repo) - :update-block - (update-command/build-action options repo) - (:remove-block :remove-page :remove-tag :remove-property) (remove-command/build-action command options repo) @@ -438,11 +418,10 @@ :list-page (list-command/execute-list-page action config) :list-tag (list-command/execute-list-tag action config) :list-property (list-command/execute-list-property action config) - :add-block (add-command/execute-add-block action config) - :add-page (add-command/execute-add-page action config) + :upsert-block (upsert-command/execute-upsert-block action config) + :upsert-page (upsert-command/execute-upsert-page action config) :upsert-tag (upsert-command/execute-upsert-tag action config) :upsert-property (upsert-command/execute-upsert-property action config) - :update-block (update-command/execute-update action config) :remove-block (remove-command/execute-remove-block action config) :remove-page (remove-command/execute-remove-page action config) :remove-tag (remove-command/execute-remove-tag action config) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 6052212040..e6d1efc675 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -249,17 +249,24 @@ (cond-> [(str "Server " (name status) ": " repo)] (and host port) (conj (str "Host: " host " Port: " port)))))) -(defn- format-add-block - [_context ids] - (str "Added blocks:\n" (pr-str (vec (or ids []))))) +(defn- format-upsert-block + [{:keys [repo source target update-tags update-properties remove-tags remove-properties]} result] + (if (vector? result) + (str "Upserted blocks:\n" (pr-str (vec (or result [])))) + (let [change-parts (cond-> [] + (seq update-tags) (conj (str "tags:+" (count update-tags))) + (seq update-properties) (conj (str "properties:+" (count update-properties))) + (seq remove-tags) (conj (str "remove-tags:+" (count remove-tags))) + (seq remove-properties) (conj (str "remove-properties:+" (count remove-properties)))) + changes (when (seq change-parts) + (str ", " (string/join ", " change-parts))) + move-fragment (when (seq target) + (str " -> " target))] + (str "Upserted block: " source (or move-fragment "") " (repo: " repo (or changes "") ")")))) -(defn- format-add-page +(defn- format-upsert-page [_context ids] - (str "Added page:\n" (pr-str (vec (or ids []))))) - -(defn- format-add-tag - [_context ids] - (str "Added tag:\n" (pr-str (vec (or ids []))))) + (str "Upserted page:\n" (pr-str (vec (or ids []))))) (defn- format-upsert-tag [_context ids] @@ -293,19 +300,6 @@ (str "Removed property: " name " (repo: " repo ")") (str "Removed property: " id " (repo: " repo ")"))) -(defn- format-update-block - [{:keys [repo source target update-tags update-properties remove-tags remove-properties]}] - (let [change-parts (cond-> [] - (seq update-tags) (conj (str "tags:+" (count update-tags))) - (seq update-properties) (conj (str "properties:+" (count update-properties))) - (seq remove-tags) (conj (str "remove-tags:+" (count remove-tags))) - (seq remove-properties) (conj (str "remove-properties:+" (count remove-properties)))) - changes (when (seq change-parts) - (str ", " (string/join ", " change-parts))) - move-fragment (when (seq target) - (str " -> " target))] - (str "Updated block: " source (or move-fragment "") " (repo: " repo (or changes "") ")"))) - (defn- format-graph-export [{:keys [export-type output]}] (str "Exported " export-type " to " output)) @@ -352,16 +346,14 @@ (format-server-action command data) :list-page (format-list-page (:items data) now-ms) (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) - :add-block (format-add-block context (:result data)) - :add-page (format-add-page context (:result data)) - :add-tag (format-add-tag context (:result data)) + :upsert-block (format-upsert-block context (:result data)) + :upsert-page (format-upsert-page context (:result data)) :upsert-tag (format-upsert-tag context (:result data)) :upsert-property (format-upsert-property context (:result data)) :remove-block (format-remove-block context) :remove-page (format-remove-page context) :remove-tag (format-remove-tag context) :remove-property (format-remove-property context) - :update-block (format-update-block context) :graph-export (format-graph-export context) :graph-import (format-graph-import context) :query (format-query-results (:result data)) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index bbbafd346f..969eb7afbf 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -59,10 +59,8 @@ (is (string/includes? plain-summary "Graph Inspect and Edit")) (is (string/includes? plain-summary "Graph Management")) (is (string/includes? plain-summary "list")) - (is (string/includes? plain-summary "add")) (is (string/includes? plain-summary "upsert")) (is (string/includes? plain-summary "remove")) - (is (string/includes? plain-summary "update")) (is (string/includes? plain-summary "query")) (is (string/includes? plain-summary "show")) (is (string/includes? plain-summary "doctor")) @@ -72,15 +70,14 @@ (is (contains-bold? summary "list page")) (is (contains-bold? summary "list tag")) (is (contains-bold? summary "list property")) - (is (contains-bold? summary "add block")) - (is (contains-bold? summary "add page")) + (is (contains-bold? summary "upsert block")) + (is (contains-bold? summary "upsert page")) (is (contains-bold? summary "upsert tag")) (is (contains-bold? summary "upsert property")) (is (contains-bold? summary "remove block")) (is (contains-bold? summary "remove page")) (is (contains-bold? summary "remove tag")) (is (contains-bold? summary "remove property")) - (is (contains-bold? summary "update")) (is (contains-bold? summary "query")) (is (contains-bold? summary "query list")) (is (contains-bold? summary "show")) @@ -161,16 +158,17 @@ (is (contains-bold? summary "--id")) (is (contains-bold? summary "--uuid")))) - (testing "update command shows help" + (testing "upsert block command shows help" (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["update" "--help"])) + (commands/parse-args ["upsert" "block" "--help"])) summary (:summary result) plain-summary (strip-ansi summary)] (is (true? (:help? result))) - (is (string/includes? plain-summary "Usage: logseq update")) + (is (string/includes? plain-summary "Usage: logseq upsert block")) (is (string/includes? plain-summary "Command options:")) (is (contains-bold? summary "--id")) (is (contains-bold? summary "--uuid")) + (is (contains-bold? summary "--content")) (is (contains-bold? summary "--target-id")) (is (contains-bold? summary "--target-uuid")) (is (contains-bold? summary "--update-tags")) @@ -207,17 +205,11 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) -(deftest test-parse-args-help-add-upsert-group - (testing "add group shows subcommands" - (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["add"])) - summary (:summary result) - plain-summary (strip-ansi summary)] - (is (true? (:help? result))) - (is (string/includes? plain-summary "add block")) - (is (string/includes? plain-summary "add page")) - (is (contains-bold? summary "add block")) - (is (contains-bold? summary "add page")))) +(deftest test-parse-args-help-upsert-group + (testing "add group is removed" + (let [result (commands/parse-args ["add"])] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code]))))) (testing "upsert group shows subcommands" (let [result (binding [style/*color-enabled?* true] @@ -225,8 +217,12 @@ summary (:summary result) plain-summary (strip-ansi summary)] (is (true? (:help? result))) + (is (string/includes? plain-summary "upsert block")) + (is (string/includes? plain-summary "upsert page")) (is (string/includes? plain-summary "upsert tag")) (is (string/includes? plain-summary "upsert property")) + (is (contains-bold? summary "upsert block")) + (is (contains-bold? summary "upsert page")) (is (contains-bold? summary "upsert tag")) (is (contains-bold? summary "upsert property"))))) @@ -282,6 +278,14 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) + (testing "rejects removed write commands" + (doseq [args [["add" "block" "--content" "x"] + ["add" "page" "--page" "Home"] + ["update" "--id" "1" "--update-tags" "[\"TagA\"]"]]] + (let [result (commands/parse-args args)] + (is (false? (:ok? result))) + (is (= :unknown-command (get-in result [:error :code])))))) + (testing "errors on missing command" (let [result (commands/parse-args [])] (is (false? (:ok? result))) @@ -634,11 +638,11 @@ (deftest test-help-tags-properties-identifiers (testing "add help mentions tag and property identifiers" (let [summary (:summary (binding [style/*color-enabled?* true] - (commands/parse-args ["add" "block" "--help"])))] + (commands/parse-args ["upsert" "block" "--help"])))] (is (string/includes? (strip-ansi summary) "Identifiers can be id, :db/ident, or :block/title."))) (let [summary (:summary (binding [style/*color-enabled?* true] - (commands/parse-args ["add" "page" "--help"])))] + (commands/parse-args ["upsert" "page" "--help"])))] (is (string/includes? (strip-ansi summary) "Identifiers can be id, :db/ident, or :block/title."))))) @@ -927,126 +931,149 @@ "--name" "owner" "--cardinality" "triple"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code])))))) - (testing "update requires source selector" - (let [result (commands/parse-args ["update" "--target-id" "10"])] +(deftest test-verb-subcommand-parse-upsert-block-mode + + (testing "upsert block create mode requires content when source selectors are absent" + (let [result (commands/parse-args ["upsert" "block" "--target-id" "10"])] (is (false? (:ok? result))) - (is (= :missing-source (get-in result [:error :code]))))) + (is (= :missing-content (get-in result [:error :code]))))) - (testing "update requires target or update/remove options" - (let [result (commands/parse-args ["update" "--id" "1"])] + (testing "upsert block update mode requires target or update/remove options" + (let [result (commands/parse-args ["upsert" "block" "--id" "1"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "update parses with source and target" - (let [result (commands/parse-args ["update" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] + (testing "upsert block parses with source and target" + (let [result (commands/parse-args ["upsert" "block" "--uuid" "abc" "--target-uuid" "def" "--pos" "last-child"])] (is (true? (:ok? result))) - (is (= :update-block (:command result))) + (is (= :upsert-block (:command result))) (is (= "abc" (get-in result [:options :uuid]))) (is (= "def" (get-in result [:options :target-uuid]))) (is (= "last-child" (get-in result [:options :pos]))))) - (testing "update parses with tags and properties" - (let [result (commands/parse-args ["update" "--id" "1" + (testing "upsert block parses with update tags and properties" + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--update-tags" "[\"TagA\"]" "--update-properties" "{:logseq.property/publishing-public? true}"])] (is (true? (:ok? result))) - (is (= :update-block (:command result))) + (is (= :upsert-block (:command result))) (is (= "[\"TagA\"]" (get-in result [:options :update-tags]))) (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :update-properties]))))) - (testing "update allows update without move target" - (let [result (commands/parse-args ["update" "--uuid" "abc" + (testing "upsert block allows updates without move target" + (let [result (commands/parse-args ["upsert" "block" "--uuid" "abc" "--update-tags" "[\"TagA\"]"])] (is (true? (:ok? result))) - (is (= :update-block (:command result))) - (is (= "abc" (get-in result [:options :uuid])))))) + (is (= :upsert-block (:command result))) + (is (= "abc" (get-in result [:options :uuid]))))) + + (testing "upsert block forces update mode when id and content are both provided" + (let [result (commands/parse-args ["upsert" "block" + "--id" "1" + "--content" "hello" + "--update-tags" "[\"TagA\"]"])] + (is (true? (:ok? result))) + (is (= :upsert-block (:command result))) + (is (= 1 (get-in result [:options :id]))) + (is (= "hello" (get-in result [:options :content]))) + (is (= "[\"TagA\"]" (get-in result [:options :update-tags])))))) (deftest test-verb-subcommand-parse-add - (testing "add block requires content source" - (let [result (commands/parse-args ["add" "block"])] + (testing "upsert block create mode requires content source" + (let [result (commands/parse-args ["upsert" "block"])] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) - (testing "add block parses with content" - (let [result (commands/parse-args ["add" "block" "--content" "hello"])] + (testing "upsert block create mode parses with content" + (let [result (commands/parse-args ["upsert" "block" "--content" "hello"])] (is (true? (:ok? result))) - (is (= :add-block (:command result))) + (is (= :upsert-block (:command result))) (is (= "hello" (get-in result [:options :content]))))) - (testing "add block parses with target selectors and pos" - (let [result (commands/parse-args ["add" "block" + (testing "upsert block create mode parses with target selectors and pos" + (let [result (commands/parse-args ["upsert" "block" "--content" "hello" "--target-uuid" "abc" "--pos" "first-child"])] (is (true? (:ok? result))) - (is (= :add-block (:command result))) + (is (= :upsert-block (:command result))) (is (= "abc" (get-in result [:options :target-uuid]))) (is (= "first-child" (get-in result [:options :pos]))))) - (testing "add block parses with tags and properties" - (let [result (commands/parse-args ["add" "block" + (testing "upsert block create mode parses with tags and properties" + (let [result (commands/parse-args ["upsert" "block" "--content" "hello" "--tags" "[\"TagA\" \"TagB\"]" "--properties" "{:logseq.property/publishing-public? true}"])] (is (true? (:ok? result))) - (is (= :add-block (:command result))) + (is (= :upsert-block (:command result))) (is (= "[\"TagA\" \"TagB\"]" (get-in result [:options :tags]))) (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) - (testing "add block rejects invalid pos" - (let [result (commands/parse-args ["add" "block" + (testing "upsert block rejects invalid pos" + (let [result (commands/parse-args ["upsert" "block" "--content" "hello" "--pos" "middle"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "add block rejects tags with blocks payload" - (let [result (commands/parse-args ["add" "block" + (testing "upsert block rejects tags with blocks payload" + (let [result (commands/parse-args ["upsert" "block" "--blocks" "[]" "--tags" "[\"TagA\"]"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "add block rejects properties with blocks-file payload" - (let [result (commands/parse-args ["add" "block" + (testing "upsert block rejects properties with blocks-file payload" + (let [result (commands/parse-args ["upsert" "block" "--blocks-file" "/tmp/blocks.edn" "--properties" "{:logseq.property/publishing-public? true}"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "add page requires page name" - (let [result (commands/parse-args ["add" "page"])] + (testing "upsert page requires page name" + (let [result (commands/parse-args ["upsert" "page"])] (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) - (testing "add page parses with name" - (let [result (commands/parse-args ["add" "page" "--page" "Home"])] + (testing "upsert page parses with name" + (let [result (commands/parse-args ["upsert" "page" "--page" "Home"])] (is (true? (:ok? result))) - (is (= :add-page (:command result))) + (is (= :upsert-page (:command result))) (is (= "Home" (get-in result [:options :page]))))) - (testing "add page parses with tags and properties" - (let [result (commands/parse-args ["add" "page" + (testing "upsert page parses with tags and properties" + (let [result (commands/parse-args ["upsert" "page" "--page" "Home" "--tags" "[\"TagA\"]" "--properties" "{:logseq.property/publishing-public? true}"])] (is (true? (:ok? result))) - (is (= :add-page (:command result))) + (is (= :upsert-page (:command result))) (is (= "[\"TagA\"]" (get-in result [:options :tags]))) (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) - (testing "add tag is no longer supported" + (testing "upsert page parses update and remove options" + (let [result (commands/parse-args ["upsert" "page" + "--page" "Home" + "--update-tags" "[\"TagB\"]" + "--remove-properties" "[:logseq.property/deadline]"])] + (is (true? (:ok? result))) + (is (= :upsert-page (:command result))) + (is (= "[\"TagB\"]" (get-in result [:options :update-tags]))) + (is (= "[:logseq.property/deadline]" (get-in result [:options :remove-properties]))))) + + (testing "legacy add tag is no longer supported" (let [result (commands/parse-args ["add" "tag" "--name" "Quote"])] (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) (deftest test-verb-subcommand-parse-update-target-page - (testing "update parses with target page" - (let [result (commands/parse-args ["update" "--id" "1" "--target-page" "Home"])] + (testing "upsert block update mode parses with target page" + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--target-page" "Home"])] (is (true? (:ok? result))) - (is (= :update-block (:command result))) + (is (= :upsert-block (:command result))) (is (= 1 (get-in result [:options :id]))) (is (= "Home" (get-in result [:options :target-page])))))) @@ -1165,10 +1192,10 @@ (deftest test-verb-subcommand-parse-flags (testing "verb subcommands reject unknown flags" (doseq [args [["list" "page" "--wat"] - ["add" "block" "--wat"] + ["upsert" "block" "--wat"] + ["upsert" "page" "--wat"] ["remove" "block" "--wat"] ["upsert" "tag" "--wat"] - ["update" "--wat"] ["show" "--wat"]]] (let [result (commands/parse-args args)] (is (false? (:ok? result))) @@ -1269,19 +1296,19 @@ (is (= :missing-repo (get-in result [:error :code]))))) (testing "add block requires content" - (let [parsed {:ok? true :command :add-block :options {}} + (let [parsed {:ok? true :command :upsert-block :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) (testing "add block builds insert-blocks op" - (let [parsed {:ok? true :command :add-block :options {:content "hello"}} + (let [parsed {:ok? true :command :upsert-block :options {:content "hello"}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= :add-block (get-in result [:action :type]))))) + (is (= :upsert-block (get-in result [:action :type]))))) (testing "add page requires name" - (let [parsed {:ok? true :command :add-page :options {}} + (let [parsed {:ok? true :command :upsert-page :options {}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) @@ -1374,7 +1401,7 @@ (deftest test-build-action-add-validates-properties (testing "add block rejects unknown property" - (let [parsed (commands/parse-args ["add" "block" + (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--properties" "{:not/a 1}"]) result (commands/build-action parsed {:repo "demo"})] @@ -1382,7 +1409,7 @@ (is (= :invalid-options (get-in result [:error :code]))))) (testing "add block accepts property title key" - (let [parsed (commands/parse-args ["add" "block" + (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--properties" "{\"Publishing Public?\" true}"]) result (commands/build-action parsed {:repo "demo"})] @@ -1391,7 +1418,7 @@ (-> result :action :properties keys first))))) (testing "add block rejects non-public built-in property" - (let [parsed (commands/parse-args ["add" "block" + (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--properties" "{:logseq.property/heading 1}"]) result (commands/build-action parsed {:repo "demo"})] @@ -1399,7 +1426,7 @@ (is (= :invalid-options (get-in result [:error :code]))))) (testing "add block rejects invalid checkbox value" - (let [parsed (commands/parse-args ["add" "block" + (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--properties" "{:logseq.property/publishing-public? \"nope\"}"]) result (commands/build-action parsed {:repo "demo"})] @@ -1408,7 +1435,7 @@ (deftest test-build-action-add-accepts-tag-ids (testing "add block accepts numeric tag ids" - (let [parsed (commands/parse-args ["add" "block" + (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--tags" "[42]"]) result (commands/build-action parsed {:repo "demo"})] @@ -1424,63 +1451,87 @@ (is (= {:type :id :value 42} (normalize-property-key-input 42))))) (deftest test-build-action-update - (testing "update requires source selector" - (let [parsed {:ok? true :command :update-block :options {:target-id 2}} + (testing "upsert block create mode requires content when source selector is absent" + (let [parsed {:ok? true :command :upsert-block :options {:target-id 2}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) - (is (= :missing-source (get-in result [:error :code]))))) + (is (= :missing-content (get-in result [:error :code]))))) - (testing "update requires target or update/remove options" - (let [parsed {:ok? true :command :update-block :options {:id 1}} + (testing "upsert block update mode requires target or update/remove options" + (let [parsed {:ok? true :command :upsert-block :options {:id 1}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update accepts update tags without target" (let [parsed {:ok? true - :command :update-block + :command :upsert-block :options {:id 1 :update-tags "[\"TagA\"]"}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= :update-block (get-in result [:action :type]))) + (is (= :upsert-block (get-in result [:action :type]))) (is (= ["TagA"] (get-in result [:action :update-tags]))))) (testing "update rejects invalid update tags" (let [parsed {:ok? true - :command :update-block + :command :upsert-block :options {:id 1 :update-tags "{:tag \"no\"}"}} result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code])))))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "upsert block forces update mode when id and content are both provided" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 :content "hello" :update-tags "[\"TagA\"]"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :upsert-block (get-in result [:action :type]))) + (is (= 1 (get-in result [:action :id]))) + (is (= ["TagA"] (get-in result [:action :update-tags]))))) + + (testing "update accepts custom property identifiers" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 + :update-properties "{:user.property/owner \"alice\"}" + :remove-properties "[:user.property/owner]"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :upsert-block (get-in result [:action :type]))) + (is (= {:user.property/owner "alice"} + (get-in result [:action :update-properties]))) + (is (= [:user.property/owner] + (get-in result [:action :remove-properties])))))) (deftest test-update-parse-validation (testing "update rejects multiple source selectors" - (let [result (commands/parse-args ["update" "--id" "1" "--uuid" "abc" "--target-id" "2"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--uuid" "abc" "--target-id" "2"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update rejects multiple target selectors" - (let [result (commands/parse-args ["update" "--id" "1" "--target-id" "2" "--target-uuid" "def"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--target-id" "2" "--target-uuid" "def"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update rejects invalid position" - (let [result (commands/parse-args ["update" "--id" "1" "--target-id" "2" "--pos" "middle"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--target-id" "2" "--pos" "middle"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update rejects sibling pos for page target" - (let [result (commands/parse-args ["update" "--id" "1" "--target-page" "Home" "--pos" "sibling"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--target-page" "Home" "--pos" "sibling"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update rejects legacy target-page-name option" - (let [result (commands/parse-args ["update" "--id" "1" "--target-page-name" "Home"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--target-page-name" "Home"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "update rejects pos without target" - (let [result (commands/parse-args ["update" "--id" "1" "--pos" "last-child" "--update-tags" "[\"TagA\"]"])] + (let [result (commands/parse-args ["upsert" "block" "--id" "1" "--pos" "last-child" "--update-tags" "[\"TagA\"]"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) @@ -1637,6 +1688,178 @@ (set! transport/invoke orig-invoke) (done))))))) +(deftest test-execute-upsert-block-create-applies-extra-tag-property-ops + (async done + (let [ops* (atom nil) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-execute-add-block add-command/execute-add-block + orig-resolve-tags add-command/resolve-tags + orig-resolve-properties add-command/resolve-properties + orig-resolve-property-identifiers add-command/resolve-property-identifiers + orig-invoke transport/invoke + action {:type :upsert-block + :mode :create + :repo "demo" + :update-tags [:tag/new] + :remove-tags [:tag/old] + :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} + :remove-properties [:logseq.property/publishing-public?]}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! add-command/execute-add-block (fn [_ _] + (p/resolved {:status :ok + :data {:result [11 12]}}))) + (set! add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond + (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil)))) + (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) + (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (and (vector? lookup) + (= :db/ident (first lookup))) + {:db/id 99} + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [11 12] (get-in result [:data :result]))) + (is (= [[:batch-delete-property-value [[11 12] :block/tags 202]] + [:batch-remove-property [[11 12] :logseq.property/publishing-public?]] + [:batch-set-property [[11 12] :block/tags 101 {}]] + [:batch-set-property [[11 12] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + @ops*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! add-command/execute-add-block orig-execute-add-block) + (set! add-command/resolve-tags orig-resolve-tags) + (set! add-command/resolve-properties orig-resolve-properties) + (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-page-applies-ops-on-existing-page + (async done + (let [ops* (atom nil) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-resolve-tags add-command/resolve-tags + orig-resolve-properties add-command/resolve-properties + orig-resolve-property-identifiers add-command/resolve-property-identifiers + orig-invoke transport/invoke + action {:type :upsert-page + :repo "demo" + :page "Home" + :tags [:tag/new] + :update-tags [:tag/next] + :remove-tags [:tag/old] + :properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} + :update-properties {:logseq.property/publishing-public? true} + :remove-properties [:logseq.property/deadline]}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond + (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/next]) [{:db/id 303}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil)))) + (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) + (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup [:block/name "home"]) + {:db/id 50 + :block/uuid (uuid "00000000-0000-0000-0000-000000000050")} + + (and (vector? lookup) (= :db/ident (first lookup))) + {:db/id 888} + + :else {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {}) + ops @ops*] + (is (= :ok (:status result))) + (is (= [50] (get-in result [:data :result]))) + (is (= 6 (count ops))) + (is (some #(= [:batch-delete-property-value [[50] :block/tags 202]] %) ops)) + (is (some #(= [:batch-remove-property [[50] :logseq.property/deadline]] %) ops)) + (is (some #(= [:batch-set-property [[50] :block/tags 101 {}]] %) ops)) + (is (some #(= [:batch-set-property [[50] :block/tags 303 {}]] %) ops)) + (is (some #(= [:batch-set-property [[50] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]] %) ops)) + (is (some #(= [:batch-set-property [[50] :logseq.property/publishing-public? true {}]] %) ops))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! add-command/resolve-tags orig-resolve-tags) + (set! add-command/resolve-properties orig-resolve-properties) + (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-page-errors-when-property-does-not-exist + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-resolve-tags add-command/resolve-tags + orig-resolve-properties add-command/resolve-properties + orig-resolve-property-identifiers add-command/resolve-property-identifiers + orig-invoke transport/invoke + action {:type :upsert-page + :repo "demo" + :page "Home" + :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"}}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! add-command/resolve-tags (fn [_ _ _] (p/resolved nil))) + (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) + (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup [:block/name "home"]) + {:db/id 50} + + (and (vector? lookup) (= :db/ident (first lookup))) + {} + + :else {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not apply ops when property lookup fails" + {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :property-not-found (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! add-command/resolve-tags orig-resolve-tags) + (set! add-command/resolve-properties orig-resolve-properties) + (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-remove-tag-property (async done (let [ops* (atom []) @@ -1749,7 +1972,8 @@ orig-resolve-properties add-command/resolve-properties orig-resolve-property-identifiers add-command/resolve-property-identifiers orig-invoke transport/invoke - action {:type :update-block + action {:type :upsert-block + :mode :update :repo "demo" :id 1 :target-id 2 @@ -1765,8 +1989,8 @@ (= tags [:tag/new]) [{:db/id 101}] (= tags [:tag/old]) [{:db/id 202}] :else nil)))) - (set! add-command/resolve-properties (fn [_ _ properties] (p/resolved properties))) - (set! add-command/resolve-property-identifiers (fn [_ _ properties] (p/resolved properties))) + (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) + (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) (set! transport/invoke (fn [_ method _ args] (swap! calls* conj {:method method :args args}) (case method diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index ff794149f7..91c8528894 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -91,23 +91,23 @@ result))))) (deftest test-human-output-add-upsert-remove - (testing "add block renders ids in two lines" + (testing "upsert block renders ids in two lines" (let [result (format/format-result {:status :ok - :command :add-block + :command :upsert-block :context {:repo "demo-repo" :blocks ["a" "b"]} :data {:result [201 202]}} {:output-format nil})] - (is (= "Added blocks:\n[201 202]" result)))) + (is (= "Upserted blocks:\n[201 202]" result)))) - (testing "add page renders ids in two lines" + (testing "upsert page renders ids in two lines" (let [result (format/format-result {:status :ok - :command :add-page + :command :upsert-page :context {:repo "demo-repo" :page "Home"} :data {:result [123]}} {:output-format nil})] - (is (= "Added page:\n[123]" result)))) + (is (= "Upserted page:\n[123]" result)))) (testing "upsert tag renders ids in two lines" (let [result (format/format-result {:status :ok @@ -163,9 +163,9 @@ {:output-format nil})] (is (= "Removed property: owner (repo: demo-repo)" result)))) - (testing "update block renders a succinct success line" + (testing "upsert block update mode renders a succinct success line" (let [result (format/format-result {:status :ok - :command :update-block + :command :upsert-block :context {:repo "demo-repo" :source "source-uuid" :target "target-uuid" @@ -175,17 +175,17 @@ :remove-properties [:logseq.property/deadline]} :data {:result {:ok true}}} {:output-format nil})] - (is (= "Updated block: source-uuid -> target-uuid (repo: demo-repo, tags:+1, properties:+1, remove-tags:+1, remove-properties:+1)" result)))) + (is (= "Upserted block: source-uuid -> target-uuid (repo: demo-repo, tags:+1, properties:+1, remove-tags:+1, remove-properties:+1)" result)))) - (testing "update without move target renders a succinct success line" + (testing "upsert block update without move target renders a succinct success line" (let [result (format/format-result {:status :ok - :command :update-block + :command :upsert-block :context {:repo "demo-repo" :source "source-uuid" :update-tags ["TagA"]} :data {:result {:ok true}}} {:output-format nil})] - (is (= "Updated block: source-uuid (repo: demo-repo, tags:+1)" result))))) + (is (= "Upserted block: source-uuid (repo: demo-repo, tags:+1)" result))))) (deftest test-human-output-graph-import-export (testing "graph export renders a succinct success line" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index dec868d709..2ea2ac4f47 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -129,7 +129,7 @@ (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "tags-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "tags-graph" "add" "page" "--page" "Home"] data-dir cfg-path)] + _ (run-cli ["--repo" "tags-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path)] {:cfg-path cfg-path :repo "tags-graph"})) (defn- stop-repo! @@ -269,7 +269,7 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) - add-page-result (run-cli ["--repo" "content-graph" "add" "page" "--page" "TestPage"] data-dir cfg-path) + add-page-result (run-cli ["--repo" "content-graph" "upsert" "page" "--page" "TestPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) list-page-result (run-cli ["--repo" "content-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) @@ -277,7 +277,7 @@ list-tag-payload (parse-json-output list-tag-result) list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["--repo" "content-graph" "add" "block" "--target-page-name" "TestPage" "--content" "Test block"] data-dir cfg-path) + add-block-result (run-cli ["--repo" "content-graph" "upsert" "block" "--target-page" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) @@ -288,7 +288,8 @@ stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) - (is (= "ok" (:status add-block-payload))) + (is (= "ok" (:status add-block-payload)) + (pr-str (:error add-block-payload))) (is (= "ok" (:status list-page-payload))) (is (vector? (get-in list-page-payload [:data :items]))) (is (= "ok" (:status list-tag-payload))) @@ -306,14 +307,14 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-add-page-json-output-returns-id +(deftest ^:long test-cli-upsert-page-json-output-returns-id (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-add-page-json-id") repo "add-page-json-id-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - add-page-result (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + add-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) page-ids (get-in add-page-payload [:data :result]) page-id (first page-ids) @@ -335,7 +336,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-add-block-json-output-returns-ids +(deftest ^:long test-cli-upsert-block-create-json-output-returns-ids (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-add-block-json-ids") repo "add-block-json-ids-graph"] @@ -345,10 +346,10 @@ {:block/title "Sibling"}]) _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) add-block-result (run-cli ["--repo" repo - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--blocks" blocks-edn] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) @@ -361,8 +362,10 @@ set) stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code add-block-result))) - (is (= "ok" (:status add-block-payload))) + (is (= 0 (:exit-code add-block-result)) + (pr-str (:error add-block-payload))) + (is (= "ok" (:status add-block-payload)) + (pr-str (:error add-block-payload))) (is (vector? block-ids)) (is (= 3 (count block-ids))) (is (= 3 (count (distinct block-ids)))) @@ -383,15 +386,15 @@ _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) add-page-result (run-cli ["--repo" repo "--output" "edn" - "add" "page" + "upsert" "page" "--page" "Home"] data-dir cfg-path) add-page-payload (parse-edn-output add-page-result) page-ids (get-in add-page-payload [:data :result]) add-block-result (run-cli ["--repo" repo "--output" "edn" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "EDN block"] data-dir cfg-path) add-block-payload (parse-edn-output add-block-result) @@ -421,18 +424,18 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - add-page-result (run-cli ["--repo" repo "add" "page" "--page" "ChainPage"] data-dir cfg-path) + add-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "ChainPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) page-id (first-result-id add-page-payload) add-block-result (run-cli ["--repo" repo - "add" "block" + "upsert" "block" "--target-id" (str page-id) "--content" "Chain block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) block-id (first-result-id add-block-payload) update-result (run-cli ["--repo" repo - "update" + "upsert" "block" "--id" (str block-id) "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) @@ -462,16 +465,96 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-page-create-and-update-existing + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-page-existing") + repo "upsert-page-existing-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + create-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + create-payload (parse-json-output create-result) + page-id (first-result-id create-payload) + update-result (run-cli ["--repo" repo + "upsert" "page" + "--page" "Home" + "--update-properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + update-payload (parse-json-output update-result) + update-id (first-result-id update-payload) + property-after-update (query-property data-dir cfg-path repo "Home" + ":logseq.property/publishing-public?") + remove-result (run-cli ["--repo" repo + "upsert" "page" + "--page" "Home" + "--remove-properties" "[:logseq.property/publishing-public?]"] + data-dir cfg-path) + remove-payload (parse-json-output remove-result) + remove-id (first-result-id remove-payload) + property-after-remove (query-property data-dir cfg-path repo "Home" + ":logseq.property/publishing-public?") + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code create-result))) + (is (= "ok" (:status create-payload))) + (is (number? page-id)) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status update-payload))) + (is (= page-id update-id)) + (is (= true property-after-update)) + (is (= 0 (:exit-code remove-result))) + (is (= "ok" (:status remove-payload))) + (is (= page-id remove-id)) + (is (nil? property-after-remove)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-upsert-page-errors-on-missing-tags-properties + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-page-missing") + repo "upsert-page-missing-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + missing-tag-result (run-cli ["--repo" repo + "upsert" "page" + "--page" "Home" + "--update-tags" "[\"MissingTag\"]"] + data-dir cfg-path) + missing-tag-payload (parse-json-output missing-tag-result) + missing-property-result (run-cli ["--repo" repo + "upsert" "page" + "--page" "Home" + "--update-properties" "{:not/a 1}"] + data-dir cfg-path) + missing-property-payload (parse-json-output missing-property-result) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 1 (:exit-code missing-tag-result))) + (is (= "error" (:status missing-tag-payload))) + (is (= :tag-not-found (keyword (get-in missing-tag-payload [:error :code])))) + (is (= 1 (:exit-code missing-property-result))) + (is (= "error" (:status missing-property-payload))) + (is (= :invalid-options (keyword (get-in missing-property-payload [:error :code])))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-add-block-rewrites-page-ref (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "ref-rewrite-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "ref-rewrite-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" "ref-rewrite-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) add-block-result (run-cli ["--repo" "ref-rewrite-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "See [[New Page]]"] data-dir cfg-path) add-block-payload (parse-json-output-safe add-block-result "add-block") @@ -513,10 +596,10 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "uuid-ref-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" "uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) _ (run-cli ["--repo" "uuid-ref-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Target block"] data-dir cfg-path) _ (p/delay 100) @@ -525,8 +608,8 @@ (pr-str ["Target block"])) target-uuid (first (first (get-in target-query-payload [:data :result]))) add-block-result (run-cli ["--repo" "uuid-ref-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" (str "See [[" target-uuid "]]")] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) @@ -564,11 +647,11 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "missing-uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" "missing-uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) missing-uuid (str (random-uuid)) add-block-result (run-cli ["--repo" "missing-uuid-ref-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" (str "See [[" missing-uuid "]]")] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) @@ -588,23 +671,23 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-tags")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) add-page-result (run-cli ["--repo" "tags-graph" - "add" "page" + "upsert" "page" "--page" "TaggedPage" "--tags" "[\"Quote\"]" "--properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) add-block-result (run-cli ["--repo" "tags-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Tagged block" "--tags" "[\"Quote\"]" "--properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) add-block-ident-result (run-cli ["--repo" "tags-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Tagged block ident" "--tags" "[:logseq.class/Quote-block]"] data-dir cfg-path) @@ -612,14 +695,14 @@ deadline-prop-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) publishing-prop-title (get-in db-property/built-in-properties [:logseq.property/publishing-public? :title]) add-page-title-result (run-cli ["--repo" "tags-graph" - "add" "page" + "upsert" "page" "--page" "TaggedPageTitle" "--properties" (str "{\"" publishing-prop-title "\" true}")] data-dir cfg-path) add-page-title-payload (parse-json-output add-page-title-result) add-block-title-result (run-cli ["--repo" "tags-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Tagged block title" "--properties" (str "{\"" deadline-prop-title "\" \"2026-01-25T12:00:00Z\"}")] data-dir cfg-path) @@ -670,15 +753,15 @@ deadline-id (find-item-id (get-in list-property-payload [:data :items]) deadline-title) publishing-id (find-item-id (get-in list-property-payload [:data :items]) publishing-title) add-page-id-result (run-cli ["--repo" repo - "add" "page" + "upsert" "page" "--page" "TaggedPageId" "--tags" (pr-str [quote-tag-id]) "--properties" (pr-str {publishing-id true})] data-dir cfg-path) add-page-id-payload (parse-json-output add-page-id-result) add-block-id-result (run-cli ["--repo" repo - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Tagged block id" "--tags" (pr-str [quote-tag-id]) "--properties" (pr-str {deadline-id "2026-01-25T12:00:00Z"})] @@ -793,8 +876,8 @@ tag-a-name "Quote" tag-b-name "Math" add-block-result (run-cli ["--repo" repo - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Update block" "--tags" "[:logseq.class/Quote-block]" "--properties" "{:logseq.property/publishing-public? true}"] @@ -806,7 +889,7 @@ block-node (find-block-by-title (get-in show-home-payload [:data :root]) "Update block") block-id (node-id block-node) update-result (run-cli ["--repo" repo - "update" + "upsert" "block" "--id" (str block-id) "--update-tags" "[:logseq.class/Math-block]" "--remove-tags" "[:logseq.class/Quote-block]" @@ -828,6 +911,63 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-block-update-custom-property + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-block-custom-property")] + (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) + upsert-property-result (run-cli ["--repo" repo + "upsert" "property" + "--name" "owner" + "--type" "default"] + data-dir cfg-path) + upsert-property-payload (parse-json-output upsert-property-result) + add-block-result (run-cli ["--repo" repo + "upsert" "block" + "--target-page" "Home" + "--content" "Block with custom property"] + data-dir cfg-path) + add-block-payload (parse-json-output add-block-result) + _ (p/delay 100) + show-home (run-cli ["--repo" repo "show" "--page" "Home"] data-dir cfg-path) + show-home-payload (parse-json-output show-home) + block-node (find-block-by-title (get-in show-home-payload [:data :root]) "Block with custom property") + block-id (node-id block-node) + update-result (run-cli ["--repo" repo + "upsert" "block" + "--id" (str block-id) + "--update-properties" "{:user.property/owner \"alice\"}"] + data-dir cfg-path) + update-payload (parse-json-output update-result) + _ (p/delay 100) + property-after-update (query-property data-dir cfg-path repo "Block with custom property" + ":user.property/owner") + remove-result (run-cli ["--repo" repo + "upsert" "block" + "--id" (str block-id) + "--remove-properties" "[:user.property/owner]"] + data-dir cfg-path) + remove-payload (parse-json-output remove-result) + _ (p/delay 100) + property-after-remove (query-property data-dir cfg-path repo "Block with custom property" + ":user.property/owner") + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code upsert-property-result))) + (is (= "ok" (:status upsert-property-payload))) + (is (= 0 (:exit-code add-block-result))) + (is (= "ok" (:status add-block-payload))) + (is (some? block-id)) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status update-payload))) + (is (some? property-after-update)) + (is (= 0 (:exit-code remove-result))) + (is (= "ok" (:status remove-payload))) + (is (nil? property-after-remove)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-add-tags-rejects-missing-tag (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-missing")] @@ -835,8 +975,8 @@ _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "tags-missing-graph"] data-dir cfg-path) add-block-result (run-cli ["--repo" "tags-missing-graph" - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Block with missing tag" "--tags" "[\"MissingTag\"]"] data-dir cfg-path) @@ -864,7 +1004,7 @@ repo "upsert-tag-create-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "add" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) upsert-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" "CliQuote"] @@ -876,8 +1016,8 @@ (map #(or (:block/title %) (:title %) (:name %))) set) add-block-result (run-cli ["--repo" repo - "add" "block" - "--target-page-name" "Home" + "upsert" "block" + "--target-page" "Home" "--content" "Tagged by upsert tag" "--tags" "[\"CliQuote\"]"] data-dir cfg-path) @@ -908,7 +1048,7 @@ repo "upsert-tag-conflict-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "add" "page" "--page" "ConflictPage"] data-dir cfg-path) + _ (run-cli ["--repo" repo "upsert" "page" "--page" "ConflictPage"] data-dir cfg-path) upsert-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" "ConflictPage"] @@ -1037,13 +1177,13 @@ _ (fs/writeFileSync cfg-path "{:output-format :json}") create-result (run-cli ["graph" "create" "--repo" "query-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) - _ (run-cli ["--repo" "query-graph" "add" "page" "--page" "QueryPage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "add" "block" - "--target-page-name" "QueryPage" + _ (run-cli ["--repo" "query-graph" "upsert" "page" "--page" "QueryPage"] data-dir cfg-path) + _ (run-cli ["--repo" "query-graph" "upsert" "block" + "--target-page" "QueryPage" "--content" "Query block"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "add" "block" - "--target-page-name" "QueryPage" + _ (run-cli ["--repo" "query-graph" "upsert" "block" + "--target-page" "QueryPage" "--content" "Query block"] data-dir cfg-path) _ (p/delay 100) @@ -1074,22 +1214,22 @@ _ (fs/writeFileSync cfg-path "{:output-format :json}") create-result (run-cli ["graph" "create" "--repo" "task-query-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) - _ (run-cli ["--repo" "task-query-graph" "add" "page" "--page" "Tasks"] data-dir cfg-path) + _ (run-cli ["--repo" "task-query-graph" "upsert" "page" "--page" "Tasks"] data-dir cfg-path) _ (run-cli ["--repo" "task-query-graph" - "add" "block" - "--target-page-name" "Tasks" + "upsert" "block" + "--target-page" "Tasks" "--content" "Task one" "--status" "doing"] data-dir cfg-path) _ (run-cli ["--repo" "task-query-graph" - "add" "block" - "--target-page-name" "Tasks" + "upsert" "block" + "--target-page" "Tasks" "--content" "Task two" "--status" "doing"] data-dir cfg-path) _ (run-cli ["--repo" "task-query-graph" - "add" "block" - "--target-page-name" "Tasks" + "upsert" "block" + "--target-page" "Tasks" "--content" "Task three" "--status" "todo"] data-dir cfg-path) @@ -1194,9 +1334,9 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "recent-updated-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "recent-updated-graph" "add" "page" "--page" "RecentPage"] data-dir cfg-path) - _ (run-cli ["--repo" "recent-updated-graph" "add" "block" - "--target-page-name" "RecentPage" + _ (run-cli ["--repo" "recent-updated-graph" "upsert" "page" "--page" "RecentPage"] data-dir cfg-path) + _ (run-cli ["--repo" "recent-updated-graph" "upsert" "block" + "--target-page" "RecentPage" "--content" "Recent block"] data-dir cfg-path) _ (p/delay 100) @@ -1291,20 +1431,20 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "add" "page" "--page" "NestedPage"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" "--content" "Inner"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "upsert" "page" "--page" "NestedPage"] data-dir cfg-path) + _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" "Inner"] data-dir cfg-path) show-nested (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-nested-payload (parse-json-output show-nested) _inner-node (find-block-by-title (get-in show-nested-payload [:data :root]) "Inner") inner-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" "Inner") middle-content (str "See [[" inner-uuid "]]") - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" middle-content] data-dir cfg-path) show-middle (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-middle-payload (parse-json-output show-middle) _middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) middle-content) middle-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" middle-content) - _ (run-cli ["--repo" "nested-refs" "add" "block" "--target-page-name" "NestedPage" + _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) show-outer (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-outer-payload (parse-json-output show-outer) @@ -1327,15 +1467,15 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) _target-show-payload (parse-json-output target-show) target-uuid (query-block-uuid-by-title data-dir cfg-path "linked-refs-graph" "TargetPage") target-title "TargetPage" ref-content (str "See [[" target-uuid "]]") ref-title (str "See [[" target-title "]]") - _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" "--content" ref-content] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" "--content" ref-content] data-dir cfg-path) _ (p/delay 100) source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "SourcePage"] data-dir cfg-path) source-payload (parse-json-output source-show) @@ -1365,22 +1505,22 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest ^:long test-cli-update-block-move +(deftest ^:long test-cli-upsert-block-update-move (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "add" "block" "--target-page-name" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "upsert" "block" "--target-page" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) _ (p/delay 100) source-show (run-cli ["--repo" "move-graph" "show" "--page" "SourcePage"] data-dir cfg-path) source-payload (parse-json-output source-show) parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") parent-id (node-id parent-node) - _ (run-cli ["--repo" "move-graph" "add" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) - update-result (run-cli ["--repo" "move-graph" "update" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "move-graph" "upsert" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) + update-result (run-cli ["--repo" "move-graph" "upsert" "block" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) update-payload (parse-json-output update-result) target-show (run-cli ["--repo" "move-graph" "show" "--page" "TargetPage"] data-dir cfg-path) target-payload (parse-json-output target-show) @@ -1404,15 +1544,15 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "page" "--page" "PosPage"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-page-name" "PosPage" "--content" "Parent"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "upsert" "page" "--page" "PosPage"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-page" "PosPage" "--content" "Parent"] data-dir cfg-path) _ (p/delay 100) parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) parent-payload (parse-json-output parent-show) parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") parent-id (node-id parent-node) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "add" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) + _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) final-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) final-payload (parse-json-output final-show) final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") @@ -1452,7 +1592,7 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "list-id-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) list-tag-result (run-cli ["list" "tag"] data-dir cfg-path) @@ -1479,7 +1619,7 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "human-list-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page" "--output" "human"] data-dir cfg-path) output (:output list-page-result)] (is (= 0 (:exit-code list-page-result))) @@ -1497,7 +1637,7 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "show-page-block-graph"] data-dir cfg-path) - _ (run-cli ["add" "page" "--page" "TestPage"] data-dir cfg-path) + _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page" "--expand"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) page-item (some (fn [item] @@ -1536,14 +1676,14 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "show-multi-id-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "add" "page" "--page" "MultiPage"] + _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "page" "--page" "MultiPage"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "add" "block" - "--target-page-name" "MultiPage" + _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "block" + "--target-page" "MultiPage" "--content" "Multi show one"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "add" "block" - "--target-page-name" "MultiPage" + _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "block" + "--target-page" "MultiPage" "--content" "Multi show two"] data-dir cfg-path) _ (p/delay 100) @@ -1608,10 +1748,10 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "show-multi-id-contained-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "page" "--page" "ParentPage"] + _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "page" "--page" "ParentPage"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" - "--target-page-name" "ParentPage" + _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "block" + "--target-page" "ParentPage" "--content" "Parent Block"] data-dir cfg-path) parent-query (run-cli ["--repo" "show-multi-id-contained-graph" "query" @@ -1620,7 +1760,7 @@ data-dir cfg-path) parent-payload (parse-json-output parent-query) parent-id (get-in parent-payload [:data :result]) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "add" "block" + _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) @@ -1662,14 +1802,14 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "query-pipe-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "add" "page" "--page" "PipePage"] + _ (run-cli ["--repo" "query-pipe-graph" "upsert" "page" "--page" "PipePage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "add" "block" - "--target-page-name" "PipePage" + _ (run-cli ["--repo" "query-pipe-graph" "upsert" "block" + "--target-page" "PipePage" "--content" "Pipe One"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "add" "block" - "--target-page-name" "PipePage" + _ (run-cli ["--repo" "query-pipe-graph" "upsert" "block" + "--target-page" "PipePage" "--content" "Pipe Two"] data-dir cfg-path) _ (p/delay 100) @@ -1736,14 +1876,14 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "query-stdin-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "add" "page" "--page" "PipePage"] + _ (run-cli ["--repo" "query-stdin-graph" "upsert" "page" "--page" "PipePage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "add" "block" - "--target-page-name" "PipePage" + _ (run-cli ["--repo" "query-stdin-graph" "upsert" "block" + "--target-page" "PipePage" "--content" "Pipe One"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "add" "block" - "--target-page-name" "PipePage" + _ (run-cli ["--repo" "query-stdin-graph" "upsert" "block" + "--target-page" "PipePage" "--content" "Pipe Two"] data-dir cfg-path) _ (p/delay 100) @@ -1798,8 +1938,8 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "add" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) list-page-result (run-cli ["--repo" "linked-refs-graph" "list" "page" "--expand"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) @@ -1809,7 +1949,7 @@ (get-in list-page-payload [:data :items])) page-id (or (:db/id page-item) (:id page-item)) blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") - _ (run-cli ["--repo" "linked-refs-graph" "add" "block" "--target-page-name" "SourcePage" + _ (run-cli ["--repo" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" "--blocks" blocks-edn] data-dir cfg-path) show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) @@ -1842,8 +1982,8 @@ import-graph "import-edn-graph" export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "page" "--page" "ExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "ExportPage" "--content" "Export content"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "upsert" "page" "--page" "ExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "upsert" "block" "--target-page" "ExportPage" "--content" "Export content"] data-dir cfg-path) export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "edn" @@ -1881,8 +2021,8 @@ import-graph "import-sqlite-graph" export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "add" "block" "--target-page-name" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "upsert" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) + _ (run-cli ["--repo" export-graph "upsert" "block" "--target-page" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "sqlite" From fa3fda584920b9783eb39e278acb75ac69617003 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 2 Mar 2026 20:57:45 +0800 Subject: [PATCH 094/375] 045-logseq-cli-property-type-and-upsert-option-unification.md --- ...erty-type-and-upsert-option-unification.md | 180 +++++++++ docs/cli/logseq-cli.md | 12 +- src/main/logseq/cli/command/upsert.cljs | 365 ++++++++++++------ src/main/logseq/cli/commands.cljs | 49 ++- src/main/logseq/cli/common/mcp/tools.cljs | 7 +- src/main/logseq/cli/format.cljs | 34 +- src/test/logseq/cli/commands_test.cljs | 262 +++++++++++-- src/test/logseq/cli/format_test.cljs | 19 +- src/test/logseq/cli/integration_test.cljs | 147 ++++++- .../logseq/cli/mcp_tools_contract_test.cljs | 3 +- 10 files changed, 895 insertions(+), 183 deletions(-) create mode 100644 docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md diff --git a/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md b/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md new file mode 100644 index 0000000000..c15b2def60 --- /dev/null +++ b/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md @@ -0,0 +1,180 @@ +# Logseq CLI Property Type and Upsert Option Unification Implementation Plan + +Goal: Add a property type column to `list property`, add `--id` update-mode semantics to `upsert block/page/tag/property`, and remove duplicated `--tags` or `--properties` options from `upsert block/page` in favor of `--update-tags` or `--update-properties`. + +Architecture: Keep the existing `logseq-cli -> transport/invoke -> db-worker-node :thread-api/*` contract unchanged and implement behavior changes in CLI parsing, action building, execution, and formatting. +Architecture: Extend property list payload shaping so non-expanded property items include `:logseq.property/type`, then render a dedicated property table in human output with a `TYPE` column. +Architecture: Treat `--id` as an explicit update signal for all upsert entity commands, and keep create paths only when `--id` is absent. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, logseq-cli command modules, db-worker-node thread APIs. + +Related: Builds on `docs/agent-guide/044-logseq-cli-upsert-block-page.md` and relates to `docs/agent-guide/043-logseq-cli-tag-property-management.md`. + +## Problem statement + +Current `list property` human output in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` uses the same formatter as `list tag`, so no property-type column is rendered. + +Current non-expanded property list items from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` are built by `minimal-list-item`, which does not include `:logseq.property/type`. + +Current `upsert block` already supports `--id` and treats it as update mode via `update-mode?` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, but `upsert page`, `upsert tag`, and `upsert property` do not accept `--id`. + +Current `upsert block/page` specs include both `--tags` or `--properties` and `--update-tags` or `--update-properties` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, which duplicates semantics and increases parser and action complexity. + +Current parser validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` requires `--page` for `upsert page` and requires `--name` for `upsert property`, so there is no update-by-id mode for those commands. + +## Testing Plan + +I will use `@test-driven-development` for all implementation batches. + +I will add parser and action RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for new `--id` contracts on `upsert page`, `upsert tag`, and `upsert property`. + +I will add formatter RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for the `list property` `TYPE` column and its value normalization. + +I will add contract RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/mcp_tools_contract_test.cljs` to ensure non-expanded property list items carry `:logseq.property/type`. + +I will add integration RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for update-by-id flows and for rejecting removed `--tags` or `--properties` flags on `upsert block/page`. + +I will use `@clojure-debug` only when failures indicate fixture or async harness issues rather than missing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Requirement | Current behavior | Gap | +| --- | --- | --- | +| `list property` shows type column. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` renders tag and property with the same columns (`ID`, `TITLE`, optional `IDENT`, timestamps). | No `TYPE` column in human output, and non-expanded payload currently omits type. | +| `upsert block/page/tag/property` supports `--id` and `--id` forces update mode. | `upsert block` supports this already, but `upsert page/tag/property` specs do not expose `--id` and still depend on page/name creation-first contracts. | Missing update-by-id mode for three upsert commands. | +| `upsert block/page` removes `--tags` or `--properties` and uses `--update-tags` or `--update-properties` only. | Both old and new options are accepted and merged in action building and execution. | Duplicate option surface and duplicate parsing paths remain. | + +## Target contract + +`upsert block` keeps current `--id` update-mode behavior and remove legacy create-only `--tags` or `--properties` options. + +`upsert page` accepts `--id` as update mode, and accepts `--page` only for create mode. + +`upsert tag` accepts `--id` as update mode, and keeps `--name` for create mode. + +`upsert property` accepts `--id` as update mode, and keeps `--name` for create mode. + +`upsert tag --id ` with no additional mutation options is a successful no-op after id lookup and tag-class validation. + +`upsert page --id --page ` is invalid and must fail as conflicting selectors. + +When `--id` is provided for any upsert command, create-specific resolution paths must be skipped and the command must fail if the target id does not exist or has the wrong entity class. + +`upsert block/page` should reject `--tags` and `--properties` as unknown options after spec cleanup, with guidance to use `--update-tags` and `--update-properties`. + +Update-by-id failures should use new id-mode specific error codes so scripts can distinguish id lookup and id class mismatch from create-mode validation failures. + +## Architecture sketch + +```text +list property + -> /src/main/logseq/cli/command/list.cljs execute-list-property + -> /src/main/frontend/worker/db_core.cljs :thread-api/api-list-properties + -> /src/main/logseq/cli/common/mcp/tools.cljs list-properties + -> /src/main/logseq/cli/format.cljs format-list-property (new dedicated formatter) +``` + +```text +upsert page/tag/property --id + -> /src/main/logseq/cli/commands.cljs parse + finalize-command + -> /src/main/logseq/cli/command/upsert.cljs update-mode detection and action build + -> transport/invoke :thread-api/pull for id/entity validation + -> transport/invoke :thread-api/apply-outliner-ops for update ops only +``` + +## Detailed implementation plan + +1. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert page --id 10 --update-properties ...` parses as `:upsert-page` and no longer requires `--page`. +2. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert tag --id 10` parses and `upsert property --id 10 --type node` parses. +3. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert block --tags ...` and `upsert page --properties ...` fail with `:invalid-options` due to removed flags. +4. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert page --id --page ` fails with a selector conflict error. +5. Add RED build-action tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert page/tag/property` actions that include `:mode :update` when `--id` is present. +6. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` verifying `upsert tag --id ` with no update fields returns `:ok` and no mutation ops. +7. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` verifying update-by-id rejects missing ids and wrong entity classes with new id-mode specific error codes. +8. Add RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting `list property` human output includes `TYPE` header and per-row values. +9. Add RED contract tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/mcp_tools_contract_test.cljs` asserting non-expanded `list-properties` items include `:logseq.property/type`. +10. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page --id`, `upsert tag --id`, and `upsert property --id` update mode behavior, including tag no-op behavior. +11. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` verifying `upsert page --id --page` fails with selector conflict and `upsert block/page` reject removed `--tags` and `--properties` options. +12. Run focused RED commands and verify failures are behavior assertions, not fixture failures. +13. Update specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to remove `:tags` and `:properties` from block/page specs and add `:id` to page/tag/property specs. +14. Update option validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` so `upsert page` and `upsert property` required-field checks are mode-aware instead of unconditional, and add explicit selector-conflict validation for `upsert page --id --page`. +15. Refactor `build-page-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to support create mode by `--page` and update mode by `--id`. +16. Refactor `build-tag-action` and `build-property-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to support update mode by `--id` with mode-specific required options. +17. Add shared helper(s) in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to pull entities by id and validate class/type constraints before updates. +18. Update `execute-upsert-page`, `execute-upsert-tag`, and `execute-upsert-property` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so update mode uses id lookup, skips creation paths, and applies only update semantics, with `upsert tag --id` no-op when no mutation fields are provided. +19. Introduce explicit id-mode error codes in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` for id-not-found and id-type-mismatch failures. +20. Remove all `:tags` and `:properties` action wiring from page/block upsert flows in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, and keep only `:update-tags` and `:update-properties`. +21. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` to include `:logseq.property/type` in non-expanded property list items. +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to split property formatting from tag formatting and render a dedicated `TYPE` column for `:list-property`. +23. Update docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to remove `--tags` or `--properties` from `upsert block/page` docs, document update-by-id behavior across all upsert commands, and document selector conflict behavior. +24. Run focused GREEN tests for commands, format, and integration, then run `bb dev:lint-and-test`. +25. Refactor only after GREEN to reduce duplication in upsert mode branching, then rerun focused tests and full suite. + +## Edge cases + +`upsert page --id ` must fail when id points to a block that is not a page entity. + +`upsert tag --id ` must fail when id points to a page not tagged with `:logseq.class/Tag`. + +`upsert property --id ` must fail when id points to an entity without `:logseq.property/type`. + +`upsert tag --id ` with no mutation options must return success without issuing mutation ops. + +`upsert page --id --page ` must fail with a dedicated selector conflict error. + +Update-by-id missing target and class mismatch failures must return new id-mode specific error codes. + +`upsert block` create mode with `--blocks` or `--blocks-file` must preserve existing validation behavior after removing `--tags` and `--properties` options. + +Property type display should remain stable for built-in and custom properties, and missing type values should render as `-` instead of throwing. + +JSON and EDN list outputs should remain backward compatible except for the additive `type` field on property items. + +## Verification commands and expected output + +| Command | Expected output | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test` | Parser, action, and execute tests for mode switching and option removal pass. | +| `bb dev:test -v logseq.cli.format-test` | Human formatter tests pass with `TYPE` column coverage for `list property`. | +| `bb dev:test -v logseq.cli.mcp-tools-contract-test` | Contract tests pass with `:logseq.property/type` present in non-expanded property items. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-page-create-and-update-existing` | Existing page upsert flow still passes after mode refactor. | +| `bb dev:test -v logseq.cli.integration-test` | New `--id` update-mode and removed-option behavior pass end to end. | +| `bb dev:lint-and-test` | Full suite passes with exit code `0`. | + +## Testing Details + +Tests cover CLI behavior at parser, action, executor, formatter, and end-to-end levels, and they assert entity outcomes instead of internal helper wiring. + +Tests verify that update-by-id mode never creates entities and that legacy duplicated options are no longer accepted for block/page upsert. + +Tests verify that `list property` human and structured output both include property-type information in their respective contracts. + +## Implementation Details + +- Keep db-worker-node thread API names unchanged and avoid adding new transport methods. +- Add `--id` to `upsert page/tag/property` specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Remove `--tags` and `--properties` from `upsert block/page` specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Make finalize validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` mode-aware for page/property required options. +- Rework `build-page-action`, `build-tag-action`, and `build-property-action` to branch on create vs update mode. +- Add id-based entity validation helpers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Make `upsert tag --id` with no mutation fields a successful no-op after id and class validation. +- Reject `upsert page --id --page` as explicit selector conflict. +- Add new id-mode specific error codes for id-not-found and id-type-mismatch paths. +- Include `:logseq.property/type` in non-expanded property list payload from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs`. +- Split property-specific table rendering in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to add `TYPE` column. +- Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` command reference and examples. +- Keep implementation and debugging workflow aligned with `@test-driven-development` and `@clojure-debug`. + +## Question + +No open questions. + +Decided: `upsert tag --id ` with no additional mutation options is a successful no-op. + +Decided: `upsert page --id --page ` is rejected as conflicting selectors. + +Decided: update-by-id failures use new id-mode specific error codes. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 17dd89fa96..2f0bfdfd55 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -93,12 +93,17 @@ Server ownership behavior: Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags -- `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties +- `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties (`TYPE` is included by default even without `--expand`) - `upsert block --content [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - create blocks; defaults to today’s journal page if no target is given - `upsert block --blocks [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector - `upsert block --blocks-file [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file - `upsert block --id |--uuid [--target-id |--target-uuid |--target-page ] [--pos first-child|last-child|sibling] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update and/or move a block -- `upsert page --page [--tags ] [--properties ] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - create or update a page +- `upsert page --page [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - create (or update by page name) a page +- `upsert page --id [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update a page by id (cannot be combined with `--page`) +- `upsert tag --name ` - create or upsert a tag by name +- `upsert tag --id ` - validate and upsert a tag by id (no-op when no other mutation options are provided) +- `upsert property --name [--type ] [--cardinality one|many] [--hide true|false] [--public true|false]` - create or update a property by name +- `upsert property --id [--type ] [--cardinality one|many] [--hide true|false] [--public true|false]` - update a property by id - `move --id |--uuid --target-id |--target-uuid |--target-page [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) - `remove --id |--uuid |--page ` - remove blocks (by db/id or UUID) or pages - `search [--type page|block|tag|property|all] [--tag ] [--case-sensitive] [--sort updated-at|created-at] [--order asc|desc]` - search across pages, blocks, tags, and properties (query is positional) @@ -140,7 +145,8 @@ Revision: Output formats: - Global `--output ` applies to all commands - For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. -- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. `list property` includes a dedicated `TYPE` column. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- For `list property`, `TYPE` is returned in default output (without `--expand`) for human and structured (`json`/`edn`) formats. - `upsert page` and `upsert block` return entity ids in `data.result` for JSON/EDN output, and include ids in human output. - Human example: ```text diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index 211a56dcf3..357d1026c5 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -22,27 +22,29 @@ :blocks {:desc "EDN vector of blocks for create mode"} :blocks-file {:desc "EDN file of blocks for create mode"} :status {:desc "Task status (todo, doing, done, etc.)"} - :tags {:desc "Tags to add in create mode (EDN vector). Identifiers can be id, :db/ident, or :block/title."} - :properties {:desc "Properties to add in create mode (EDN map). Identifiers can be id, :db/ident, or :block/title."} :update-tags {:desc "Tags to add/update (EDN vector)"} :update-properties {:desc "Properties to add/update (EDN map)"} :remove-tags {:desc "Tags to remove (EDN vector)"} :remove-properties {:desc "Properties to remove (EDN vector)"}}) (def ^:private upsert-page-spec - {:page {:desc "Page name"} - :tags {:desc "Tags to add (EDN vector). Identifiers can be id, :db/ident, or :block/title."} - :properties {:desc "Properties to add (EDN map). Identifiers can be id, :db/ident, or :block/title."} + {:id {:desc "Target page db/id (forces update mode)" + :coerce :long} + :page {:desc "Page name"} :update-tags {:desc "Tags to add/update (EDN vector)"} :update-properties {:desc "Properties to add/update (EDN map)"} :remove-tags {:desc "Tags to remove (EDN vector)"} :remove-properties {:desc "Properties to remove (EDN vector)"}}) (def ^:private upsert-tag-spec - {:name {:desc "Tag name"}}) + {:id {:desc "Target tag db/id (forces update mode)" + :coerce :long} + :name {:desc "Tag name"}}) (def ^:private upsert-property-spec - {:name {:desc "Property name"} + {:id {:desc "Target property db/id (forces update mode)" + :coerce :long} + :name {:desc "Property name"} :type {:desc "Property type (default, number, date, datetime, checkbox, url, node, json, string)"} :cardinality {:desc "Property cardinality (one, many)"} :hide {:desc "Hide property" @@ -101,8 +103,13 @@ :upsert-property (let [type' (normalize-property-type (:type opts)) - cardinality' (normalize-property-cardinality (:cardinality opts))] + cardinality' (normalize-property-cardinality (:cardinality opts)) + name (normalize-property-name (:name opts)) + selectors (filter some? [(:id opts) name])] (cond + (> (count selectors) 1) + "only one of --id or --name is allowed" + (and (seq (:type opts)) (not (contains? property-types type'))) (str "invalid type: " (:type opts)) @@ -112,6 +119,18 @@ :else nil)) + :upsert-page + (let [page (some-> (:page opts) string/trim) + selectors (filter some? [(:id opts) page])] + (when (> (count selectors) 1) + "only one of --id or --page is allowed")) + + :upsert-tag + (let [name (normalize-tag-name (:name opts)) + selectors (filter some? [(:id opts) name])] + (when (> (count selectors) 1) + "only one of --id or --name is allowed")) + nil)) (defn update-mode? @@ -179,25 +198,24 @@ {:ok? false :error {:code :missing-repo :message "repo is required for upsert"}} - (let [page (some-> (:page options) string/trim) - tags-result (add-command/parse-tags-option (:tags options)) - properties-result (add-command/parse-properties-option (:properties options)) + (let [id (:id options) + page (some-> (:page options) string/trim) update-tags-result (add-command/parse-tags-option (:update-tags options)) update-properties-result (add-command/parse-properties-option (:update-properties options)) remove-tags-result (add-command/parse-tags-vector-option (:remove-tags options)) - remove-properties-result (add-command/parse-properties-vector-option (:remove-properties options))] + remove-properties-result (add-command/parse-properties-vector-option (:remove-properties options)) + invalid-message (invalid-options? :upsert-page options)] (cond - (not (seq page)) + (seq invalid-message) + {:ok? false + :error {:code :invalid-options + :message invalid-message}} + + (and (not (some? id)) (not (seq page))) {:ok? false :error {:code :missing-page-name :message "page name is required"}} - (not (:ok? tags-result)) - tags-result - - (not (:ok? properties-result)) - properties-result - (not (:ok? update-tags-result)) update-tags-result @@ -212,16 +230,16 @@ :else {:ok? true - :action {:type :upsert-page - :repo repo - :graph (core/repo->graph repo) - :page page - :tags (:value tags-result) - :properties (:value properties-result) - :update-tags (:value update-tags-result) - :update-properties (:value update-properties-result) - :remove-tags (:value remove-tags-result) - :remove-properties (:value remove-properties-result)}})))) + :action (cond-> {:type :upsert-page + :repo repo + :graph (core/repo->graph repo) + :mode (if (some? id) :update :create) + :update-tags (:value update-tags-result) + :update-properties (:value update-properties-result) + :remove-tags (:value remove-tags-result) + :remove-properties (:value remove-properties-result)} + (some? id) (assoc :id id) + (seq page) (assoc :page page))})))) (defn build-tag-action [options repo] @@ -229,13 +247,32 @@ {:ok? false :error {:code :missing-repo :message "repo is required for upsert"}} - (let [name (normalize-tag-name (:name options))] - (if (seq name) + (let [id (:id options) + name (normalize-tag-name (:name options)) + invalid-message (invalid-options? :upsert-tag options)] + (cond + (seq invalid-message) + {:ok? false + :error {:code :invalid-options + :message invalid-message}} + + (some? id) {:ok? true :action {:type :upsert-tag + :mode :update + :id id + :repo repo + :graph (core/repo->graph repo)}} + + (seq name) + {:ok? true + :action {:type :upsert-tag + :mode :create :repo repo :graph (core/repo->graph repo) :name name}} + + :else {:ok? false :error {:code :missing-tag-name :message "tag name is required"}})))) @@ -269,10 +306,11 @@ {:ok? false :error {:code :missing-repo :message "repo is required for upsert"}} - (let [name (normalize-property-name (:name options)) + (let [id (:id options) + name (normalize-property-name (:name options)) invalid-message (invalid-options? :upsert-property options)] (cond - (not (seq name)) + (and (not (some? id)) (not (seq name))) {:ok? false :error {:code :missing-property-name :message "property name is required"}} @@ -284,11 +322,13 @@ :else {:ok? true - :action {:type :upsert-property - :repo repo - :graph (core/repo->graph repo) - :name name - :schema (property-schema options)}})))) + :action (cond-> {:type :upsert-property + :mode (if (some? id) :update :create) + :repo repo + :graph (core/repo->graph repo) + :schema (property-schema options)} + (some? id) (assoc :id id) + (seq name) (assoc :name name))})))) (defn- pull-page-by-name [config repo page-name selector] @@ -323,6 +363,93 @@ {:code :page-not-found :page page-name}))))))) +(def ^:private upsert-id-not-found-code + :upsert-id-not-found) + +(def ^:private upsert-id-type-mismatch-code + :upsert-id-type-mismatch) + +(def ^:private page-selector + [:db/id :block/uuid :block/name :block/title]) + +(def ^:private tag-selector + [:db/id :block/uuid :block/name :block/title + {:block/tags [:db/ident]}]) + +(def ^:private property-selector + [:db/id :db/ident :block/uuid :block/name :block/title :logseq.property/type]) + +(defn- page-entity? + [entity] + (seq (:block/name entity))) + +(defn- tag-entity? + [entity] + (some #(= :logseq.class/Tag (:db/ident %)) + (:block/tags entity))) + +(defn- property-entity? + [entity] + (some? (:logseq.property/type entity))) + +(defn- pull-entity-by-id + [config repo selector id] + (transport/invoke config :thread-api/pull false + [repo selector id])) + +(defn- throw-upsert-id-not-found! + [entity-type id] + (throw (ex-info (str entity-type " not found for id") + {:code upsert-id-not-found-code + :entity-type entity-type + :id id}))) + +(defn- throw-upsert-id-type-mismatch! + [entity-type id] + (throw (ex-info (str "id does not reference expected " entity-type) + {:code upsert-id-type-mismatch-code + :entity-type entity-type + :id id}))) + +(defn- ensure-page-by-id! + [config repo id] + (p/let [entity (pull-entity-by-id config repo page-selector id)] + (cond + (not (:db/id entity)) + (throw-upsert-id-not-found! "page" id) + + (not (page-entity? entity)) + (throw-upsert-id-type-mismatch! "page" id) + + :else + entity))) + +(defn- ensure-tag-by-id! + [config repo id] + (p/let [entity (pull-entity-by-id config repo tag-selector id)] + (cond + (not (:db/id entity)) + (throw-upsert-id-not-found! "tag" id) + + (not (tag-entity? entity)) + (throw-upsert-id-type-mismatch! "tag" id) + + :else + entity))) + +(defn- ensure-property-by-id! + [config repo id] + (p/let [entity (pull-entity-by-id config repo property-selector id)] + (cond + (not (:db/id entity)) + (throw-upsert-id-not-found! "property" id) + + (not (property-entity? entity)) + (throw-upsert-id-type-mismatch! "property" id) + + :else + entity))) + (defn- append-tag-and-property-ops [ops block-ids {:keys [update-tag-ids remove-tag-ids update-properties remove-properties]}] (cond-> ops @@ -389,20 +516,20 @@ (defn execute-upsert-page [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - page (ensure-page-entity! cfg (:repo action) (:page action)) + update-by-id? (= :update (:mode action)) + page (if update-by-id? + (ensure-page-by-id! cfg (:repo action) (:id action)) + (ensure-page-entity! cfg (:repo action) (:page action))) page-id (:db/id page) block-ids [page-id] - add-tags (add-command/resolve-tags cfg (:repo action) (:tags action)) update-tags (add-command/resolve-tags cfg (:repo action) (:update-tags action)) remove-tags (add-command/resolve-tags cfg (:repo action) (:remove-tags action)) - add-properties (add-command/resolve-properties cfg (:repo action) (:properties action)) update-properties (add-command/resolve-properties cfg (:repo action) (:update-properties action)) remove-properties (add-command/resolve-property-identifiers cfg (:repo action) (:remove-properties action)) - merged-properties (merge (or add-properties {}) (or update-properties {})) - _ (ensure-property-identifiers-exist! cfg (:repo action) (keys merged-properties)) + _ (ensure-property-identifiers-exist! cfg (:repo action) (keys (or update-properties {}))) _ (ensure-property-identifiers-exist! cfg (:repo action) remove-properties) - update-tag-ids (->> (concat (or add-tags []) (or update-tags [])) + update-tag-ids (->> (or update-tags []) (map :db/id) (remove nil?) distinct @@ -412,7 +539,7 @@ block-ids {:update-tag-ids update-tag-ids :remove-tag-ids remove-tag-ids - :update-properties merged-properties + :update-properties update-properties :remove-properties remove-properties}) _ (when (seq ops) (transport/invoke cfg :thread-api/apply-outliner-ops false @@ -424,85 +551,99 @@ :error {:code (or (get-in (ex-data e) [:code]) :exception) :message (or (ex-message e) (str e))}})))) -(defn- tag-entity? - [entity] - (some #(= :logseq.class/Tag (:db/ident %)) - (:block/tags entity))) - (defn execute-upsert-tag [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - existing (pull-page-by-name cfg (:repo action) (:name action) - [:db/id :block/name :block/title - {:block/tags [:db/ident]}]) - existing-id (:db/id existing)] - (cond - (and existing-id (not (tag-entity? existing))) - {:status :error - :error {:code :tag-name-conflict - :message "tag already exists as a page and is not a tag"}} - - :else - (p/let [_ (when-not existing-id - (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) - [[:create-page [(:name action) {:class? true}]]] - {}])) - page (or (when existing-id existing) - (pull-page-by-name cfg (:repo action) (:name action) + update-by-id? (= :update (:mode action))] + (if update-by-id? + (p/let [entity (ensure-tag-by-id! cfg (:repo action) (:id action))] + {:status :ok + :data {:result [(:db/id entity)]}}) + (p/let [existing (pull-page-by-name cfg (:repo action) (:name action) [:db/id :block/name :block/title - {:block/tags [:db/ident]}])) - page-id (:db/id page)] + {:block/tags [:db/ident]}]) + existing-id (:db/id existing)] (cond - (not page-id) + (and existing-id (not (tag-entity? existing))) {:status :error - :error {:code :tag-not-found - :message "tag not found after upsert"}} - - (not (tag-entity? page)) - {:status :error - :error {:code :tag-create-not-tag - :message "created entity is not tagged as :logseq.class/Tag"}} + :error {:code :tag-name-conflict + :message "tag already exists as a page and is not a tag"}} :else - {:status :ok - :data {:result [page-id]}})))))) + (p/let [_ (when-not existing-id + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:create-page [(:name action) {:class? true}]]] + {}])) + page (or (when existing-id existing) + (pull-page-by-name cfg (:repo action) (:name action) + [:db/id :block/name :block/title + {:block/tags [:db/ident]}])) + page-id (:db/id page)] + (cond + (not page-id) + {:status :error + :error {:code :tag-not-found + :message "tag not found after upsert"}} -(def ^:private property-selector - [:db/id :db/ident :block/name :block/title :logseq.property/type]) + (not (tag-entity? page)) + {:status :error + :error {:code :tag-create-not-tag + :message "created entity is not tagged as :logseq.class/Tag"}} -(defn- property-entity? - [entity] - (some? (:logseq.property/type entity))) + :else + {:status :ok + :data {:result [page-id]}})))))) + (p/catch (fn [e] + {:status :error + :error {:code (or (get-in (ex-data e) [:code]) :exception) + :message (or (ex-message e) (str e))}})))) (defn execute-upsert-property [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - existing (pull-page-by-name cfg (:repo action) (:name action) property-selector) - existing-id (:db/id existing)] - (cond - (and existing-id (not (property-entity? existing))) - {:status :error - :error {:code :property-name-conflict - :message "property already exists as a page and is not a property"}} - - :else - (p/let [property-ident (when (property-entity? existing) - (:db/ident existing)) - property-opts (cond-> {} - (nil? property-ident) - (assoc :property-name (:name action))) - _ (transport/invoke cfg :thread-api/apply-outliner-ops false - [(:repo action) - [[:upsert-property [property-ident - (:schema action) - property-opts]]] - {}]) - property (pull-page-by-name cfg (:repo action) (:name action) property-selector) - property-id (:db/id property)] - (if property-id - {:status :ok - :data {:result [property-id]}} + update-by-id? (= :update (:mode action))] + (if update-by-id? + (p/let [existing (ensure-property-by-id! cfg (:repo action) (:id action)) + property-ident (:db/ident existing) + _ (when (seq (:schema action)) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:upsert-property [property-ident + (:schema action) + {}]]] + {}]))] + {:status :ok + :data {:result [(:db/id existing)]}}) + (p/let [existing (pull-page-by-name cfg (:repo action) (:name action) property-selector) + existing-id (:db/id existing)] + (cond + (and existing-id (not (property-entity? existing))) {:status :error - :error {:code :property-not-found - :message "property not found after upsert"}})))))) + :error {:code :property-name-conflict + :message "property already exists as a page and is not a property"}} + + :else + (p/let [property-ident (when (property-entity? existing) + (:db/ident existing)) + property-opts (cond-> {} + (nil? property-ident) + (assoc :property-name (:name action))) + _ (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:upsert-property [property-ident + (:schema action) + property-opts]]] + {}]) + property (pull-page-by-name cfg (:repo action) (:name action) property-selector) + property-id (:db/id property)] + (if property-id + {:status :ok + :data {:result [property-id]}} + {:status :error + :error {:code :property-not-found + :message "property not found after upsert"}})))))) + (p/catch (fn [e] + {:status :error + :error {:code (or (get-in (ex-data e) [:code]) :exception) + :message (or (ex-message e) (str e))}})))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index d7f40c2615..560f89a94e 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -139,6 +139,29 @@ (string/join " " (cond-> (vec dispatch) wrong-input (conj wrong-input)))) +(defn- legacy-upsert-option-guidance + [args message] + (let [subcommand (vec (take 2 args))] + (cond + (and (= ["upsert" "block"] subcommand) + (re-find #"Unknown option:\s*:tags" (or message ""))) + "unknown option: --tags; use --update-tags" + + (and (= ["upsert" "block"] subcommand) + (re-find #"Unknown option:\s*:properties" (or message ""))) + "unknown option: --properties; use --update-properties" + + (and (= ["upsert" "page"] subcommand) + (re-find #"Unknown option:\s*:tags" (or message ""))) + "unknown option: --tags; use --update-tags" + + (and (= ["upsert" "page"] subcommand) + (re-find #"Unknown option:\s*:properties" (or message ""))) + "unknown option: --properties; use --update-properties" + + :else + nil))) + (defn- ^:large-vars/cleanup-todo finalize-command [summary {:keys [command opts args cmds spec]}] (let [opts (command-core/normalize-opts opts) @@ -166,18 +189,30 @@ (and (= command :upsert-block) (upsert-command/invalid-options? command opts)) (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) - (and (= command :upsert-page) (not (seq (:page opts)))) + (and (= command :upsert-page) (upsert-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) + + (and (= command :upsert-page) + (not (some? (:id opts))) + (not (seq (:page opts)))) (missing-page-name-result summary) - (and (= command :upsert-tag) (not (seq (some-> (:name opts) string/trim)))) - (missing-tag-name-result summary) + (and (= command :upsert-tag) (upsert-command/invalid-options? command opts)) + (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) - (and (= command :upsert-property) (not (seq (some-> (:name opts) string/trim)))) - (missing-property-name-result summary) + (and (= command :upsert-tag) + (not (some? (:id opts))) + (not (seq (some-> (:name opts) string/trim)))) + (missing-tag-name-result summary) (and (= command :upsert-property) (upsert-command/invalid-options? command opts)) (command-core/invalid-options-result summary (upsert-command/invalid-options? command opts)) + (and (= command :upsert-property) + (not (some? (:id opts))) + (not (seq (some-> (:name opts) string/trim)))) + (missing-property-name-result summary) + (and (= command :remove-block) (empty? (filter some? [(:id opts) (some-> (:uuid opts) string/trim)]))) (missing-target-result summary) @@ -291,7 +326,9 @@ (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) (some? data) - (command-core/cli-error->result summary data) + (if-let [guided-message (legacy-upsert-option-guidance args (:msg data))] + (command-core/invalid-options-result summary guided-message) + (command-core/cli-error->result summary data)) :else (command-core/unknown-command-result summary (str "unknown command: " (string/join " " args)))))))))) diff --git a/src/main/logseq/cli/common/mcp/tools.cljs b/src/main/logseq/cli/common/mcp/tools.cljs index 3f2cbc528a..7a1ab2b635 100644 --- a/src/main/logseq/cli/common/mcp/tools.cljs +++ b/src/main/logseq/cli/common/mcp/tools.cljs @@ -22,7 +22,8 @@ :block/title (:block/title e) :block/created-at (:block/created-at e) :block/updated-at (:block/updated-at e)} - (:db/ident e) (assoc :db/ident (:db/ident e)))) + (:db/ident e) (assoc :db/ident (:db/ident e)) + (:logseq.property/type e) (assoc :logseq.property/type (:logseq.property/type e)))) (defn list-properties "Main fn for ListProperties tool" @@ -46,7 +47,9 @@ (update :logseq.property/classes #(mapv :db/ident %)) (:logseq.property/description e) (update :logseq.property/description db-property/property-value-content)) - (minimal-list-item e))))))) + ;; Keep property type in default list output (without --expand). + (assoc (minimal-list-item e) + :logseq.property/type (:logseq.property/type e)))))))) (defn list-tags "Main fn for ListTags tool" diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index e6d1efc675..3658007189 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -172,7 +172,7 @@ headers (mapv #(format-list-row % include-ident? now-ms) items)))) -(defn- format-list-tag-or-property +(defn- format-list-tag [items now-ms] (let [items (or items []) include-ident? (boolean (some :db/ident items)) @@ -183,6 +183,35 @@ headers (mapv #(format-list-row % include-ident? now-ms) items)))) +(defn- normalize-property-type + [value] + (cond + (keyword? value) (name value) + (nil? value) "-" + :else (str value))) + +(defn- format-list-property-row + [item include-ident? now-ms] + (let [base [(or (:db/id item) (:id item)) + (or (:title item) (:block/title item) (:name item)) + (normalize-property-type (:logseq.property/type item))] + with-ident (cond-> base + include-ident? (conj (:db/ident item))) + updated (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms) + created (human-ago (or (:created-at item) (:block/created-at item)) now-ms)] + (conj with-ident updated created))) + +(defn- format-list-property + [items now-ms] + (let [items (or items []) + include-ident? (boolean (some :db/ident items)) + headers (into ["ID" "TITLE" "TYPE"] + (concat (or (maybe-ident-header items) []) + ["UPDATED-AT" "CREATED-AT"]))] + (format-counted-table + headers + (mapv #(format-list-property-row % include-ident? now-ms) items)))) + (defn- format-graph-list [graphs] (format-counted-table @@ -345,7 +374,8 @@ (:server-start :server-stop :server-restart) (format-server-action command data) :list-page (format-list-page (:items data) now-ms) - (:list-tag :list-property) (format-list-tag-or-property (:items data) now-ms) + :list-tag (format-list-tag (:items data) now-ms) + :list-property (format-list-property (:items data) now-ms) :upsert-block (format-upsert-block context (:result data)) :upsert-page (format-upsert-page context (:result data)) :upsert-tag (format-upsert-tag context (:result data)) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 969eb7afbf..320e3463e8 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -635,16 +635,23 @@ (is (= (str "1 See [[Target [[Inner]]]]") (strip-ansi output)))))) -(deftest test-help-tags-properties-identifiers - (testing "add help mentions tag and property identifiers" +(deftest test-help-upsert-update-options + (testing "upsert block help includes update options and removes legacy flags" (let [summary (:summary (binding [style/*color-enabled?* true] (commands/parse-args ["upsert" "block" "--help"])))] - (is (string/includes? (strip-ansi summary) - "Identifiers can be id, :db/ident, or :block/title."))) + (is (string/includes? (strip-ansi summary) "--update-tags")) + (is (string/includes? (strip-ansi summary) "--update-properties")) + (is (not (string/includes? (strip-ansi summary) "--tags"))) + (is (not (string/includes? (strip-ansi summary) "--properties"))))) + + (testing "upsert page help includes update options and removes legacy flags" (let [summary (:summary (binding [style/*color-enabled?* true] (commands/parse-args ["upsert" "page" "--help"])))] - (is (string/includes? (strip-ansi summary) - "Identifiers can be id, :db/ident, or :block/title."))))) + (is (string/includes? (strip-ansi summary) "--id")) + (is (string/includes? (strip-ansi summary) "--update-tags")) + (is (string/includes? (strip-ansi summary) "--update-properties")) + (is (not (string/includes? (strip-ansi summary) "--tags"))) + (is (not (string/includes? (strip-ansi summary) "--properties")))))) (deftest test-show-json-edn-strips-block-uuid (testing "show json/edn removes :block/uuid recursively while keeping :db/id" @@ -908,6 +915,12 @@ (is (= :upsert-tag (:command result))) (is (= "Quote" (get-in result [:options :name]))))) + (testing "upsert tag parses with id" + (let [result (commands/parse-args ["upsert" "tag" "--id" "10"])] + (is (true? (:ok? result))) + (is (= :upsert-tag (:command result))) + (is (= 10 (get-in result [:options :id]))))) + (testing "upsert property parses with type and cardinality" (let [result (commands/parse-args ["upsert" "property" "--name" "owner" @@ -919,6 +932,15 @@ (is (= "node" (get-in result [:options :type]))) (is (= "many" (get-in result [:options :cardinality]))))) + (testing "upsert property parses with id and type" + (let [result (commands/parse-args ["upsert" "property" + "--id" "11" + "--type" "node"])] + (is (true? (:ok? result))) + (is (= :upsert-property (:command result))) + (is (= 11 (get-in result [:options :id]))) + (is (= "node" (get-in result [:options :type]))))) + (testing "upsert property rejects invalid type" (let [result (commands/parse-args ["upsert" "property" "--name" "owner" @@ -1002,15 +1024,15 @@ (is (= "abc" (get-in result [:options :target-uuid]))) (is (= "first-child" (get-in result [:options :pos]))))) - (testing "upsert block create mode parses with tags and properties" + (testing "upsert block create mode parses with update tags and update properties" (let [result (commands/parse-args ["upsert" "block" "--content" "hello" - "--tags" "[\"TagA\" \"TagB\"]" - "--properties" "{:logseq.property/publishing-public? true}"])] + "--update-tags" "[\"TagA\" \"TagB\"]" + "--update-properties" "{:logseq.property/publishing-public? true}"])] (is (true? (:ok? result))) (is (= :upsert-block (:command result))) - (is (= "[\"TagA\" \"TagB\"]" (get-in result [:options :tags]))) - (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) + (is (= "[\"TagA\" \"TagB\"]" (get-in result [:options :update-tags]))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :update-properties]))))) (testing "upsert block rejects invalid pos" (let [result (commands/parse-args ["upsert" "block" @@ -1019,20 +1041,21 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "upsert block rejects tags with blocks payload" + (testing "upsert block rejects removed --tags option" (let [result (commands/parse-args ["upsert" "block" - "--blocks" "[]" + "--content" "hello" "--tags" "[\"TagA\"]"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) - (testing "upsert block rejects properties with blocks-file payload" + (testing "upsert block rejects removed --properties option" (let [result (commands/parse-args ["upsert" "block" - "--blocks-file" "/tmp/blocks.edn" + "--content" "hello" "--properties" "{:logseq.property/publishing-public? true}"])] (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-verb-subcommand-parse-upsert-page-mode (testing "upsert page requires page name" (let [result (commands/parse-args ["upsert" "page"])] (is (false? (:ok? result))) @@ -1044,15 +1067,22 @@ (is (= :upsert-page (:command result))) (is (= "Home" (get-in result [:options :page]))))) - (testing "upsert page parses with tags and properties" + (testing "upsert page parses with id update mode" + (let [result (commands/parse-args ["upsert" "page" + "--id" "42" + "--update-properties" "{:logseq.property/publishing-public? true}"])] + (is (true? (:ok? result))) + (is (= :upsert-page (:command result))) + (is (= 42 (get-in result [:options :id]))) + (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :update-properties]))))) + + (testing "upsert page rejects removed --tags and --properties options" (let [result (commands/parse-args ["upsert" "page" "--page" "Home" "--tags" "[\"TagA\"]" "--properties" "{:logseq.property/publishing-public? true}"])] - (is (true? (:ok? result))) - (is (= :upsert-page (:command result))) - (is (= "[\"TagA\"]" (get-in result [:options :tags]))) - (is (= "{:logseq.property/publishing-public? true}" (get-in result [:options :properties]))))) + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) (testing "upsert page parses update and remove options" (let [result (commands/parse-args ["upsert" "page" @@ -1064,6 +1094,13 @@ (is (= "[\"TagB\"]" (get-in result [:options :update-tags]))) (is (= "[:logseq.property/deadline]" (get-in result [:options :remove-properties]))))) + (testing "upsert page rejects selector conflict for --id and --page" + (let [result (commands/parse-args ["upsert" "page" + "--id" "10" + "--page" "Home"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "legacy add tag is no longer supported" (let [result (commands/parse-args ["add" "tag" "--name" "Quote"])] (is (false? (:ok? result))) @@ -1313,6 +1350,18 @@ (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) + (testing "upsert page by id builds update action" + (let [parsed {:ok? true + :command :upsert-page + :options {:id 42 + :update-properties "{:logseq.property/publishing-public? true}"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= :update (get-in result [:action :mode]))) + (is (= 42 (get-in result [:action :id])))))) + +(deftest test-build-action-upsert-tag-property + (testing "upsert tag requires name" (let [parsed {:ok? true :command :upsert-tag :options {}} result (commands/build-action parsed {:repo "demo"})] @@ -1324,11 +1373,23 @@ result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-tag + :mode :create :repo "logseq_db_demo" :graph "demo" :name "Quote"} (:action result))))) + (testing "upsert tag by id builds update action" + (let [parsed {:ok? true :command :upsert-tag :options {:id 123}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= {:type :upsert-tag + :mode :update + :repo "logseq_db_demo" + :graph "demo" + :id 123} + (:action result))))) + (testing "upsert property coerces schema options" (let [parsed {:ok? true :command :upsert-property @@ -1340,6 +1401,7 @@ result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-property + :mode :create :repo "logseq_db_demo" :graph "demo" :name "owner" @@ -1349,6 +1411,23 @@ :logseq.property/public? false}} (:action result))))) + (testing "upsert property by id builds update action" + (let [parsed {:ok? true + :command :upsert-property + :options {:id 654 + :type "node" + :cardinality "many"}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= {:type :upsert-property + :mode :update + :repo "logseq_db_demo" + :graph "demo" + :id 654 + :schema {:logseq.property/type :node + :db/cardinality :db.cardinality/many}} + (:action result))))) + ) (deftest test-build-action-inspect-edit-remove-show @@ -1400,27 +1479,27 @@ (is (= [1 2] (get-in result [:action :ids])))))) (deftest test-build-action-add-validates-properties - (testing "add block rejects unknown property" + (testing "add block accepts custom property key in update-properties" (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" - "--properties" "{:not/a 1}"]) + "--update-properties" "{:not/a 1}"]) result (commands/build-action parsed {:repo "demo"})] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))))) + (is (true? (:ok? result))) + (is (= {:not/a 1} (get-in result [:action :update-properties]))))) (testing "add block accepts property title key" (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" - "--properties" "{\"Publishing Public?\" true}"]) + "--update-properties" "{\"Publishing Public?\" true}"]) result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) (is (= :logseq.property/publishing-public? - (-> result :action :properties keys first))))) + (-> result :action :update-properties keys first))))) (testing "add block rejects non-public built-in property" (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" - "--properties" "{:logseq.property/heading 1}"]) + "--update-properties" "{:logseq.property/heading 1}"]) result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -1428,7 +1507,7 @@ (testing "add block rejects invalid checkbox value" (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" - "--properties" "{:logseq.property/publishing-public? \"nope\"}"]) + "--update-properties" "{:logseq.property/publishing-public? \"nope\"}"]) result (commands/build-action parsed {:repo "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) @@ -1437,10 +1516,10 @@ (testing "add block accepts numeric tag ids" (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" - "--tags" "[42]"]) + "--update-tags" "[42]"]) result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) - (is (= [42] (get-in result [:action :tags])))))) + (is (= [42] (get-in result [:action :update-tags])))))) (deftest test-tag-lookup-ref-accepts-id (let [tag-lookup-ref #'add-command/tag-lookup-ref] @@ -1688,6 +1767,120 @@ (set! transport/invoke orig-invoke) (done))))))) +(deftest test-execute-upsert-tag-by-id-no-op + (async done + (let [apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-tag + :mode :update + :repo "demo" + :id 4242}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup 4242) + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-id-mode-validates-target-entity + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (case lookup + 100 {} + 101 {:db/id 101 + :block/uuid (uuid "00000000-0000-0000-0000-000000000101")} + 200 {} + 201 {:db/id 201 + :block/name "not-a-tag" + :block/title "Not a tag" + :block/tags [{:db/ident :logseq.class/Page}]} + 300 {} + 301 {:db/id 301 + :block/name "not-a-property" + :block/title "Not a property"} + {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not mutate on invalid id update mode" {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [page-missing (commands/execute {:type :upsert-page + :mode :update + :repo "demo" + :id 100} + {}) + page-mismatch (commands/execute {:type :upsert-page + :mode :update + :repo "demo" + :id 101} + {}) + tag-missing (commands/execute {:type :upsert-tag + :mode :update + :repo "demo" + :id 200} + {}) + tag-mismatch (commands/execute {:type :upsert-tag + :mode :update + :repo "demo" + :id 201} + {}) + property-missing (commands/execute {:type :upsert-property + :mode :update + :repo "demo" + :id 300} + {}) + property-mismatch (commands/execute {:type :upsert-property + :mode :update + :repo "demo" + :id 301} + {})] + (is (= :error (:status page-missing))) + (is (= :upsert-id-not-found (get-in page-missing [:error :code]))) + (is (= :error (:status page-mismatch))) + (is (= :upsert-id-type-mismatch (get-in page-mismatch [:error :code]))) + (is (= :error (:status tag-missing))) + (is (= :upsert-id-not-found (get-in tag-missing [:error :code]))) + (is (= :error (:status tag-mismatch))) + (is (= :upsert-id-type-mismatch (get-in tag-mismatch [:error :code]))) + (is (= :error (:status property-missing))) + (is (= :upsert-id-not-found (get-in property-missing [:error :code]))) + (is (= :error (:status property-mismatch))) + (is (= :upsert-id-type-mismatch (get-in property-mismatch [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-upsert-block-create-applies-extra-tag-property-ops (async done (let [ops* (atom nil) @@ -1760,17 +1953,14 @@ action {:type :upsert-page :repo "demo" :page "Home" - :tags [:tag/new] :update-tags [:tag/next] :remove-tags [:tag/old] - :properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} :update-properties {:logseq.property/publishing-public? true} :remove-properties [:logseq.property/deadline]}] (set! cli-server/list-graphs (fn [_] ["demo"])) (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) (set! add-command/resolve-tags (fn [_ _ tags] (p/resolved (cond - (= tags [:tag/new]) [{:db/id 101}] (= tags [:tag/next]) [{:db/id 303}] (= tags [:tag/old]) [{:db/id 202}] :else nil)))) @@ -1796,12 +1986,10 @@ ops @ops*] (is (= :ok (:status result))) (is (= [50] (get-in result [:data :result]))) - (is (= 6 (count ops))) + (is (= 4 (count ops))) (is (some #(= [:batch-delete-property-value [[50] :block/tags 202]] %) ops)) (is (some #(= [:batch-remove-property [[50] :logseq.property/deadline]] %) ops)) - (is (some #(= [:batch-set-property [[50] :block/tags 101 {}]] %) ops)) (is (some #(= [:batch-set-property [[50] :block/tags 303 {}]] %) ops)) - (is (some #(= [:batch-set-property [[50] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]] %) ops)) (is (some #(= [:batch-set-property [[50] :logseq.property/publishing-public? true {}]] %) ops))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 91c8528894..e7b0fa96a6 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -81,12 +81,27 @@ :command :list-property :data {:items [{:block/title "Prop" :db/id 99 + :logseq.property/type :node :block/created-at 40000 :block/updated-at 90000}]}} {:output-format nil :now-ms 100000})] - (is (= (str "ID TITLE UPDATED-AT CREATED-AT\n" - "99 Prop 10s ago 1m ago\n" + (is (= (str "ID TITLE TYPE UPDATED-AT CREATED-AT\n" + "99 Prop node 10s ago 1m ago\n" + "Count: 1") + result)))) + + (testing "list property renders missing type as -" + (let [result (format/format-result {:status :ok + :command :list-property + :data {:items [{:block/title "Untyped" + :db/id 100 + :block/created-at 40000 + :block/updated-at 90000}]}} + {:output-format nil + :now-ms 100000})] + (is (= (str "ID TITLE TYPE UPDATED-AT CREATED-AT\n" + "100 Untyped - 10s ago 1m ago\n" "Count: 1") result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 2ea2ac4f47..96cc9bca0e 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -533,10 +533,8 @@ missing-property-payload (parse-json-output missing-property-result) stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 1 (:exit-code missing-tag-result))) (is (= "error" (:status missing-tag-payload))) (is (= :tag-not-found (keyword (get-in missing-tag-payload [:error :code])))) - (is (= 1 (:exit-code missing-property-result))) (is (= "error" (:status missing-property-payload))) (is (= :invalid-options (keyword (get-in missing-property-payload [:error :code])))) (is (= "ok" (:status stop-payload))) @@ -545,6 +543,119 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-id-mode-for-page-tag-property + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-id-mode") + repo "upsert-id-mode-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + create-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + create-page-payload (parse-json-output create-page-result) + page-id (first-result-id create-page-payload) + update-page-result (run-cli ["--repo" repo + "upsert" "page" + "--id" (str page-id) + "--update-properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + update-page-payload (parse-json-output update-page-result) + page-value (query-property data-dir cfg-path repo "Home" ":logseq.property/publishing-public?") + create-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] data-dir cfg-path) + create-tag-payload (parse-json-output create-tag-result) + tag-id (first-result-id create-tag-payload) + noop-tag-result (run-cli ["--repo" repo "upsert" "tag" "--id" (str tag-id)] data-dir cfg-path) + noop-tag-payload (parse-json-output noop-tag-result) + create-property-result (run-cli ["--repo" repo + "upsert" "property" + "--name" "OwnerProp" + "--type" "default"] + data-dir cfg-path) + create-property-payload (parse-json-output create-property-result) + property-id (first-result-id create-property-payload) + property-name (common-util/page-name-sanity-lc "OwnerProp") + update-property-result (run-cli ["--repo" repo + "upsert" "property" + "--id" (str property-id) + "--type" "node" + "--cardinality" "many"] + data-dir cfg-path) + update-property-payload (parse-json-output update-property-result) + property-schema (run-query data-dir cfg-path repo + "[:find ?type . :in $ ?name :where [?p :block/name ?name] [?p :logseq.property/type ?type]]" + (pr-str [property-name])) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code create-page-result))) + (is (= "ok" (:status create-page-payload))) + (is (number? page-id)) + (is (= 0 (:exit-code update-page-result))) + (is (= "ok" (:status update-page-payload))) + (is (= page-id (first-result-id update-page-payload))) + (is (true? page-value)) + (is (= 0 (:exit-code create-tag-result))) + (is (= "ok" (:status create-tag-payload))) + (is (number? tag-id)) + (is (= 0 (:exit-code noop-tag-result))) + (is (= "ok" (:status noop-tag-payload))) + (is (= tag-id (first-result-id noop-tag-payload))) + (is (= 0 (:exit-code create-property-result))) + (is (= "ok" (:status create-property-payload))) + (is (number? property-id)) + (is (= 0 (:exit-code update-property-result))) + (is (= "ok" (:status update-property-payload))) + (is (= property-id (first-result-id update-property-payload))) + (is (= "node" (get-in property-schema [:data :result]))) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-upsert-rejects-legacy-flags-and-selector-conflict + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-legacy-options") + repo "upsert-legacy-options-graph"] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + create-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + create-page-payload (parse-json-output create-page-result) + page-id (first-result-id create-page-payload) + legacy-block-result (run-cli ["--repo" repo + "upsert" "block" + "--target-page" "Home" + "--content" "Legacy block" + "--tags" "[\"Quote\"]"] + data-dir cfg-path) + legacy-page-result (run-cli ["--repo" repo + "upsert" "page" + "--page" "Home" + "--properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + conflict-result (run-cli ["--repo" repo + "upsert" "page" + "--id" (str page-id) + "--page" "Home"] + data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code create-page-result))) + (is (= "ok" (:status create-page-payload))) + (is (= 1 (:exit-code legacy-block-result))) + (is (string/includes? (:output legacy-block-result) "invalid-options")) + (is (string/includes? (:output legacy-block-result) "--update-tags")) + (is (= 1 (:exit-code legacy-page-result))) + (is (string/includes? (:output legacy-page-result) "invalid-options")) + (is (string/includes? (:output legacy-page-result) "--update-properties")) + (is (= 1 (:exit-code conflict-result))) + (is (string/includes? (:output conflict-result) "invalid-options")) + (is (string/includes? (:output conflict-result) "only one of --id or --page")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-add-block-rewrites-page-ref (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] @@ -673,23 +784,23 @@ add-page-result (run-cli ["--repo" "tags-graph" "upsert" "page" "--page" "TaggedPage" - "--tags" "[\"Quote\"]" - "--properties" "{:logseq.property/publishing-public? true}"] + "--update-tags" "[\"Quote\"]" + "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) add-block-result (run-cli ["--repo" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block" - "--tags" "[\"Quote\"]" - "--properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] + "--update-tags" "[\"Quote\"]" + "--update-properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) add-block-ident-result (run-cli ["--repo" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block ident" - "--tags" "[:logseq.class/Quote-block]"] + "--update-tags" "[:logseq.class/Quote-block]"] data-dir cfg-path) add-block-ident-payload (parse-json-output add-block-ident-result) deadline-prop-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) @@ -697,14 +808,14 @@ add-page-title-result (run-cli ["--repo" "tags-graph" "upsert" "page" "--page" "TaggedPageTitle" - "--properties" (str "{\"" publishing-prop-title "\" true}")] + "--update-properties" (str "{\"" publishing-prop-title "\" true}")] data-dir cfg-path) add-page-title-payload (parse-json-output add-page-title-result) add-block-title-result (run-cli ["--repo" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block title" - "--properties" (str "{\"" deadline-prop-title "\" \"2026-01-25T12:00:00Z\"}")] + "--update-properties" (str "{\"" deadline-prop-title "\" \"2026-01-25T12:00:00Z\"}")] data-dir cfg-path) add-block-title-payload (parse-json-output add-block-title-result) _ (p/delay 100) @@ -755,16 +866,16 @@ add-page-id-result (run-cli ["--repo" repo "upsert" "page" "--page" "TaggedPageId" - "--tags" (pr-str [quote-tag-id]) - "--properties" (pr-str {publishing-id true})] + "--update-tags" (pr-str [quote-tag-id]) + "--update-properties" (pr-str {publishing-id true})] data-dir cfg-path) add-page-id-payload (parse-json-output add-page-id-result) add-block-id-result (run-cli ["--repo" repo "upsert" "block" "--target-page" "Home" "--content" "Tagged block id" - "--tags" (pr-str [quote-tag-id]) - "--properties" (pr-str {deadline-id "2026-01-25T12:00:00Z"})] + "--update-tags" (pr-str [quote-tag-id]) + "--update-properties" (pr-str {deadline-id "2026-01-25T12:00:00Z"})] data-dir cfg-path) add-block-id-payload (parse-json-output add-block-id-result) _ (p/delay 100) @@ -879,8 +990,8 @@ "upsert" "block" "--target-page" "Home" "--content" "Update block" - "--tags" "[:logseq.class/Quote-block]" - "--properties" "{:logseq.property/publishing-public? true}"] + "--update-tags" "[:logseq.class/Quote-block]" + "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) @@ -978,7 +1089,7 @@ "upsert" "block" "--target-page" "Home" "--content" "Block with missing tag" - "--tags" "[\"MissingTag\"]"] + "--update-tags" "[\"MissingTag\"]"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) list-tag-result (run-cli ["--repo" "tags-missing-graph" "list" "tag"] data-dir cfg-path) @@ -988,8 +1099,8 @@ set) stop-result (run-cli ["server" "stop" "--repo" "tags-missing-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 1 (:exit-code add-block-result))) (is (= "error" (:status add-block-payload))) + (is (= :tag-not-found (keyword (get-in add-block-payload [:error :code])))) (is (not (contains? tag-names "MissingTag"))) (is (= "ok" (:status stop-payload))) (done)) @@ -1019,7 +1130,7 @@ "upsert" "block" "--target-page" "Home" "--content" "Tagged by upsert tag" - "--tags" "[\"CliQuote\"]"] + "--update-tags" "[\"CliQuote\"]"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) diff --git a/src/test/logseq/cli/mcp_tools_contract_test.cljs b/src/test/logseq/cli/mcp_tools_contract_test.cljs index 1ec1ef77b7..0f52049fba 100644 --- a/src/test/logseq/cli/mcp_tools_contract_test.cljs +++ b/src/test/logseq/cli/mcp_tools_contract_test.cljs @@ -48,6 +48,7 @@ (deftest test-list-non-expanded-contract (let [db (create-test-db) required-keys #{:db/id :block/title :block/created-at :block/updated-at} + required-property-keys #{:db/id :block/title :block/created-at :block/updated-at :logseq.property/type} visible-page (some #(when (= "Visible Page" (:block/title %)) %) (cli-common-mcp-tools/list-pages db {})) custom-tag-entity (first-user-tag-entity db) @@ -68,7 +69,7 @@ (testing "list-properties non-expanded includes stable id and timestamps" (is (some? custom-property)) - (is (set/subset? required-keys (set (keys custom-property))))))) + (is (set/subset? required-property-keys (set (keys custom-property))))))) (deftest test-list-tags-and-properties-include-built-in-default (let [db (create-test-db) From 0331c598dffa5dbea8fe55552634979381755662 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 2 Mar 2026 22:40:14 +0800 Subject: [PATCH 095/375] 046-logseq-cli-upsert-tag-rename-by-id.md --- .../046-logseq-cli-upsert-tag-rename-by-id.md | 147 ++++++++++++ docs/cli/logseq-cli.md | 3 +- src/main/logseq/cli/command/upsert.cljs | 74 +++++- src/test/logseq/cli/commands_test.cljs | 217 +++++++++++++++++- src/test/logseq/cli/integration_test.cljs | 92 ++++++++ 5 files changed, 517 insertions(+), 16 deletions(-) create mode 100644 docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md diff --git a/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md new file mode 100644 index 0000000000..b40caa6ad5 --- /dev/null +++ b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md @@ -0,0 +1,147 @@ +# Logseq CLI Upsert Tag Rename by ID Implementation Plan + +Goal: Allow `logseq upsert tag --id --name ` to rename an existing tag while preserving the current create-by-name and validate-by-id behavior. + +Architecture: Keep the current `logseq-cli -> transport/invoke -> :thread-api/apply-outliner-ops` integration and implement rename-by-id entirely in the CLI command layer. +Architecture: Reuse the existing db-worker-node `:rename-page` outliner op instead of introducing a new thread API. +Architecture: Keep `upsert tag --id ` with no `--name` as an id-validation no-op for backward compatibility. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript pull queries, db-worker-node outliner ops. + +Related: Builds on `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/042-logseq-cli-add-tag-command.md`, `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/043-logseq-cli-tag-property-management.md`, and `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md`. + +## Problem statement + +Current `upsert tag` validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` rejects `--id` and `--name` together with the error `only one of --id or --name is allowed`. + +Current update mode in `execute-upsert-tag` only calls `ensure-tag-by-id!` and returns success without mutation when `:mode` is `:update`. + +Current db-worker-node path already supports page rename through `:thread-api/apply-outliner-ops` with `[:rename-page [page-uuid new-title]]` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`, so rename behavior can be reused without adding new APIs. + +The user-visible gap is that a command like `logseq upsert tag --repo --id 180 --name "Project Renamed"` should rename tag `180`, but currently fails at option validation. + +## Testing Plan + +I will use `@test-driven-development` for all implementation. + +I will write all RED tests first in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` and `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` before any production edits. + +I will verify RED failures are behavior failures by asserting error codes or missing mutation ops rather than fixture or async setup problems. + +I will use `@clojure-debug` only if async transport stubs or db-worker test harness behavior is unclear. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Requirement | Current behavior | Gap | +| --- | --- | --- | +| `upsert tag --id --name ` renames an existing tag. | `invalid-options?` for `:upsert-tag` rejects mixed selectors in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. | Rename-by-id cannot be triggered. | +| `upsert tag --id ` stays supported. | `execute-upsert-tag` update mode validates id and returns `[id]` without mutation. | Must remain unchanged for compatibility. | +| Rename execution uses existing db-worker-node contracts. | db-worker-node already handles `:rename-page` via `:thread-api/apply-outliner-ops` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`. | CLI does not currently call rename op in tag update path. | + +## Target contract + +`upsert tag --name ` keeps create or idempotent-create semantics. + +`upsert tag --id ` keeps id-validation no-op semantics and returns the same id. + +`upsert tag --id --name ` renames the target tag to `` using the existing outliner rename op. + +`upsert tag --id --name ` must fail with `:upsert-id-not-found` or `:upsert-id-type-mismatch` when id is invalid or not a tag. + +`upsert tag --id --name ` must no-op when `` normalizes to the same `:block/name` as the target. + +`upsert tag --id --name ` must fail if `` belongs to another non-tag page with `:tag-name-conflict`. + +`upsert tag --id --name ` must fail if `` belongs to another existing tag to avoid ambiguous cross-tag merges. + +## Architecture sketch + +```text +CLI + logseq upsert tag --id 180 --name "Project Renamed" + -> /src/main/logseq/cli/commands.cljs finalize-command + -> /src/main/logseq/cli/command/upsert.cljs build-tag-action + -> /src/main/logseq/cli/command/upsert.cljs execute-upsert-tag + 1) ensure-tag-by-id! + 2) conflict lookup by target name + 3) transport/invoke :thread-api/apply-outliner-ops + [repo [[:rename-page [tag-uuid new-name]]] {}] + 4) pull by id and return same id +DB Worker + /src/main/frontend/worker/db_core.cljs + :rename-page handler -> outliner-core/save-block! with new title +``` + +## Detailed implementation plan + +1. Add a RED parse/build test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert tag --id 180 --name "Project Renamed"` is accepted and routed to `:upsert-tag`. +2. Add a RED action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode tag action keeps `:id` and includes normalized `:name`. +3. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode with both id and name emits exactly one `:rename-page` op through `:thread-api/apply-outliner-ops`. +4. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode with id-only remains no-op and does not call `:thread-api/apply-outliner-ops`. +5. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting rename target conflict with non-tag page returns `:tag-name-conflict`. +6. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting rename target conflict with another tag returns a dedicated conflict error code. +7. Add a RED integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a tag, fetches its id, runs `upsert tag --id --name `, and verifies the new title appears in `list tag`. +8. Add a RED integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` asserting the old name no longer appears in `list tag` after rename. +9. Run focused RED commands and confirm failures are expected contract failures. +10. Update `invalid-options?` for `:upsert-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to permit `--id` plus `--name` and keep invalid checks for empty or malformed names. +11. Update `build-tag-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so update mode accepts optional `:name` and keeps create mode semantics unchanged. +12. Add helper logic in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to detect whether normalized rename target equals current tag name and skip mutation in that case. +13. Add helper logic in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to pull by target name and detect conflicts with other entities before rename. +14. Update `execute-upsert-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` update branch to call `:rename-page` when id and name are both provided. +15. Keep error handling in `execute-upsert-tag` aligned with existing `:upsert-id-not-found` and `:upsert-id-type-mismatch` contracts, and add one dedicated rename-conflict code for tag-to-tag collisions. +16. Update CLI reference docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document `upsert tag --id --name ` rename semantics and conflict behavior. +17. Run focused GREEN tests for `commands_test` and targeted integration tests. +18. Run `bb dev:test -v logseq.cli.commands-test` and `bb dev:test -v logseq.cli.integration-test` to confirm no regressions. +19. Run `bb dev:lint-and-test` as final verification. +20. Refactor duplicated tag-name normalization or conflict checks only after GREEN, then rerun focused tests. + +## Edge cases + +- `--name` with leading `#` in update mode should normalize exactly like create mode. +- `--name` that trims to blank should return `:invalid-options` and not hit db-worker. +- Rename to current name with different casing should follow normalized-name no-op behavior. +- Rename target that already exists as another tag should fail deterministically and not mutate either tag. +- Rename target that exists as a non-tag page should fail with `:tag-name-conflict`. +- Rename by id must preserve the original `:db/id` in command output. +- Id-mode not-found and type-mismatch errors must remain stable for scripts. + +## Verification commands and expected output + +| Command | Expected output | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test` | Upsert tag parse, build, and execute tests pass including rename-by-id and no-op id-only paths. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-tag-id-rename` | End-to-end rename-by-id test passes with renamed tag visible in list output. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-tag-id-rename-conflict` | Conflict behavior test passes and returns expected error code/message. | +| `bb dev:lint-and-test` | Full suite passes with exit code `0`. | + +## Testing Details + +The new tests verify user-observable behavior at parser, action, executor, and CLI integration levels. + +The tests assert command outputs, mutation calls, and list/query observable state instead of helper internals. + +The tests keep existing id-only no-op behavior covered so rename support does not regress current automation scripts. + +## Implementation Details + +- Modify tag option validation only in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Keep db-worker-node thread API signatures unchanged in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`. +- Reuse existing `:rename-page` outliner op instead of introducing a new op. +- Preserve create-by-name idempotency path for `upsert tag --name`. +- Preserve id-only validate path for `upsert tag --id`. +- Add deterministic rename conflict handling before invoking rename op. +- Keep error code stability for existing id lookup and type mismatch failures. +- Update CLI docs to reflect rename-by-id and no-op-by-id contracts. +- Follow `@test-driven-development` sequence strictly and use `@clojure-debug` only for harness issues. + +## Question + +Resolved: choose option 1. + +Rename-to-existing-tag returns a dedicated conflict error and must not be treated as success by returning the existing tag id. + +This prevents implicit merges and accidental retargeting. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 2f0bfdfd55..894e34583a 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -101,7 +101,8 @@ Inspect and edit commands: - `upsert page --page [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - create (or update by page name) a page - `upsert page --id [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update a page by id (cannot be combined with `--page`) - `upsert tag --name ` - create or upsert a tag by name -- `upsert tag --id ` - validate and upsert a tag by id (no-op when no other mutation options are provided) +- `upsert tag --id [--name ]` - validate a tag by id; when `--name` is provided, rename that tag id (no-op if normalized name is unchanged) +- `upsert tag --id --name ` conflicts: returns `tag-name-conflict` when target name is a non-tag page, and `tag-rename-conflict` when target name is another existing tag - `upsert property --name [--type ] [--cardinality one|many] [--hide true|false] [--public true|false]` - create or update a property by name - `upsert property --id [--type ] [--cardinality one|many] [--hide true|false] [--public true|false]` - update a property by id - `move --id |--uuid --target-id |--target-uuid |--target-page [--pos first-child|last-child|sibling]` - move a block and its children (defaults to first-child) diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index 357d1026c5..08e8f1909f 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -126,10 +126,14 @@ "only one of --id or --page is allowed")) :upsert-tag - (let [name (normalize-tag-name (:name opts)) - selectors (filter some? [(:id opts) name])] - (when (> (count selectors) 1) - "only one of --id or --name is allowed")) + (let [name-provided? (contains? opts :name) + name (normalize-tag-name (:name opts))] + (cond + (and name-provided? (not (seq name))) + "tag name must not be blank" + + :else + nil)) nil)) @@ -258,11 +262,12 @@ (some? id) {:ok? true - :action {:type :upsert-tag - :mode :update - :id id - :repo repo - :graph (core/repo->graph repo)}} + :action (cond-> {:type :upsert-tag + :mode :update + :id id + :repo repo + :graph (core/repo->graph repo)} + (seq name) (assoc :name name))} (seq name) {:ok? true @@ -437,6 +442,35 @@ :else entity))) +(def ^:private tag-rename-conflict-code + :tag-rename-conflict) + +(defn- normalized-tag-lookup-name + [value] + (some-> value normalize-tag-name common-util/page-name-sanity-lc)) + +(defn- rename-target-same-as-current? + [entity target-name] + (= (:block/name entity) + (normalized-tag-lookup-name target-name))) + +(defn- rename-target-conflict + [entity target] + (let [target-id (:db/id target) + current-id (:db/id entity)] + (cond + (or (nil? target-id) + (= target-id current-id)) + nil + + (not (tag-entity? target)) + {:code :tag-name-conflict + :message "tag already exists as a page and is not a tag"} + + :else + {:code tag-rename-conflict-code + :message "rename target already exists as a tag"}))) + (defn- ensure-property-by-id! [config repo id] (p/let [entity (pull-entity-by-id config repo property-selector id)] @@ -556,9 +590,25 @@ (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) update-by-id? (= :update (:mode action))] (if update-by-id? - (p/let [entity (ensure-tag-by-id! cfg (:repo action) (:id action))] - {:status :ok - :data {:result [(:db/id entity)]}}) + (p/let [entity (ensure-tag-by-id! cfg (:repo action) (:id action)) + target-name (:name action) + target (when (seq target-name) + (pull-page-by-name cfg (:repo action) target-name tag-selector)) + conflict (when (and (seq target-name) + (not (rename-target-same-as-current? entity target-name))) + (rename-target-conflict entity target)) + _ (when (and (seq target-name) + (not conflict) + (not (rename-target-same-as-current? entity target-name))) + (transport/invoke cfg :thread-api/apply-outliner-ops false + [(:repo action) + [[:rename-page [(:block/uuid entity) target-name]]] + {}]))] + (if conflict + {:status :error + :error conflict} + {:status :ok + :data {:result [(:db/id entity)]}})) (p/let [existing (pull-page-by-name cfg (:repo action) (:name action) [:db/id :block/name :block/title {:block/tags [:db/ident]}]) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 320e3463e8..2a9fd13e7f 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -921,6 +921,15 @@ (is (= :upsert-tag (:command result))) (is (= 10 (get-in result [:options :id]))))) + (testing "upsert tag parses with id and name" + (let [result (commands/parse-args ["upsert" "tag" + "--id" "10" + "--name" "Project Renamed"])] + (is (true? (:ok? result))) + (is (= :upsert-tag (:command result))) + (is (= 10 (get-in result [:options :id]))) + (is (= "Project Renamed" (get-in result [:options :name]))))) + (testing "upsert property parses with type and cardinality" (let [result (commands/parse-args ["upsert" "property" "--name" "owner" @@ -1390,6 +1399,24 @@ :id 123} (:action result))))) + (testing "upsert tag by id with name builds update action" + (let [parsed {:ok? true :command :upsert-tag :options {:id 123 :name " #Project Renamed "}} + result (commands/build-action parsed {:repo "demo"})] + (is (true? (:ok? result))) + (is (= {:type :upsert-tag + :mode :update + :repo "logseq_db_demo" + :graph "demo" + :id 123 + :name "Project Renamed"} + (:action result))))) + + (testing "upsert tag by id rejects blank rename name" + (let [parsed {:ok? true :command :upsert-tag :options {:id 123 :name " "}} + result (commands/build-action parsed {:repo "demo"})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "upsert property coerces schema options" (let [parsed {:ok? true :command :upsert-property @@ -1426,9 +1453,7 @@ :id 654 :schema {:logseq.property/type :node :db/cardinality :db.cardinality/many}} - (:action result))))) - - ) + (:action result)))))) (deftest test-build-action-inspect-edit-remove-show @@ -1804,6 +1829,192 @@ (set! transport/invoke orig-invoke) (done))))))) +(deftest test-execute-upsert-tag-by-id-with-name-emits-rename-op + (async done + (let [ops* (atom nil) + apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-tag + :mode :update + :repo "demo" + :id 4242 + :name "Project Renamed"} + tag-uuid (uuid "00000000-0000-0000-0000-000000004242")] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid tag-uuid + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} + + (= lookup [:block/name "project renamed"]) + {} + + :else + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (swap! apply-calls* inc) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 1 @apply-calls*)) + (is (= [[:rename-page [tag-uuid "Project Renamed"]]] + @ops*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-tag-by-id-with-name-no-op-when-normalized-name-matches + (async done + (let [apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-tag + :mode :update + :repo "demo" + :id 4242 + :name " #QUOTE "}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-tag-by-id-with-name-rejects-existing-non-tag-page + (async done + (let [apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-tag + :mode :update + :repo "demo" + :id 4242 + :name "Project Renamed"}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} + + (= lookup [:block/name "project renamed"]) + {:db/id 5000 + :block/name "project renamed" + :block/title "Project Renamed" + :block/tags [{:db/ident :logseq.class/Page}]} + + :else + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-name-conflict (get-in result [:error :code]))) + (is (= 0 @apply-calls*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-upsert-tag-by-id-with-name-rejects-existing-tag + (async done + (let [apply-calls* (atom 0) + orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:type :upsert-tag + :mode :update + :repo "demo" + :id 4242 + :name "Project Renamed"}] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) + (set! transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} + + (= lookup [:block/name "project renamed"]) + {:db/id 9001 + :block/uuid (uuid "00000000-0000-0000-0000-000000009001") + :block/name "project renamed" + :block/title "Project Renamed" + :block/tags [{:db/ident :logseq.class/Tag}]} + + :else + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))) + (-> (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-rename-conflict (get-in result [:error :code]))) + (is (= 0 @apply-calls*))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-upsert-id-mode-validates-target-entity (async done (let [orig-list-graphs cli-server/list-graphs diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 96cc9bca0e..9407093818 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1213,6 +1213,98 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-tag-id-rename + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-tag-id-rename")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "upsert-tag-id-rename-graph" + source-name "CliRenameSource" + target-name "CliRenameTarget" + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + create-result (run-cli ["--repo" repo "upsert" "tag" "--name" source-name] + data-dir cfg-path) + create-payload (parse-json-output create-result) + source-id (first-result-id create-payload) + rename-result (run-cli ["--repo" repo + "upsert" "tag" + "--id" (str source-id) + "--name" target-name] + data-dir cfg-path) + rename-payload (parse-json-output rename-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + tags (get-in list-tag-payload [:data :items]) + tag-names (->> tags + (map #(or (:block/title %) (:title %) (:name %))) + set) + target-id (find-item-id tags target-name) + source-id-after (find-item-id tags source-name) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code create-result))) + (is (= "ok" (:status create-payload))) + (is (number? source-id)) + (is (= 0 (:exit-code rename-result)) + (pr-str rename-payload)) + (is (= "ok" (:status rename-payload)) + (pr-str rename-payload)) + (is (= [source-id] (get-in rename-payload [:data :result]))) + (is (contains? tag-names target-name)) + (is (not (contains? tag-names source-name))) + (is (= source-id target-id)) + (is (nil? source-id-after)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-upsert-tag-id-rename-conflict + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-tag-id-rename-conflict")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + repo "upsert-tag-id-rename-conflict-graph" + source-name "CliRenameConflictSource" + existing-name "CliRenameConflictExisting" + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + source-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" source-name] + data-dir cfg-path) + source-upsert-payload (parse-json-output source-upsert-result) + source-id (first-result-id source-upsert-payload) + existing-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" existing-name] + data-dir cfg-path) + existing-upsert-payload (parse-json-output existing-upsert-result) + rename-result (run-cli ["--repo" repo + "upsert" "tag" + "--id" (str source-id) + "--name" existing-name] + data-dir cfg-path) + rename-payload (parse-json-output rename-result) + list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-payload (parse-json-output list-tag-result) + tag-names (->> (get-in list-tag-payload [:data :items]) + (map #(or (:block/title %) (:title %) (:name %))) + set) + stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= 0 (:exit-code source-upsert-result))) + (is (= "ok" (:status source-upsert-payload))) + (is (number? source-id)) + (is (= 0 (:exit-code existing-upsert-result))) + (is (= "ok" (:status existing-upsert-payload))) + (is (= 0 (:exit-code rename-result))) + (is (= "error" (:status rename-payload))) + (is (= :tag-rename-conflict (keyword (get-in rename-payload [:error :code])))) + (is (contains? tag-names source-name)) + (is (contains? tag-names existing-name)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-upsert-and-remove-tag-property (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-remove-tag-property")] From 859c4cee41e953ede9d6d4f6933aa1c5e418320b Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 2 Mar 2026 12:12:51 -0500 Subject: [PATCH 096/375] fix: graph export's command option conflicts with global option --output was being used as a global and command option. This resulted in buggy, coupled behavior e.g. unable to set a global output unless the export file was named human, edn or json. `--ouput` was also not listed as a command option which was confusing. Renamed the command --ouput to --file for explicit and uncoupled behavior --- .../006-logseq-cli-import-export.md | 4 +-- docs/cli/logseq-cli.md | 6 ++--- src/main/logseq/cli/command/graph.cljs | 10 +++---- src/main/logseq/cli/commands.cljs | 14 +++++----- src/main/logseq/cli/format.cljs | 4 +-- src/test/logseq/cli/commands_test.cljs | 27 +++++++++++-------- src/test/logseq/cli/format_test.cljs | 2 +- src/test/logseq/cli/integration_test.cljs | 4 +-- 8 files changed, 38 insertions(+), 33 deletions(-) diff --git a/docs/agent-guide/006-logseq-cli-import-export.md b/docs/agent-guide/006-logseq-cli-import-export.md index f84ab5b0d7..42df290964 100644 --- a/docs/agent-guide/006-logseq-cli-import-export.md +++ b/docs/agent-guide/006-logseq-cli-import-export.md @@ -19,8 +19,8 @@ Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/ Prefer graph-scoped subcommands to keep import/export with graph management: -- `logseq graph export --type edn --output [--repo ]` -- `logseq graph export --type sqlite --output [--repo ]` +- `logseq graph export --type edn --file [--repo ]` +- `logseq graph export --type sqlite --file [--repo ]` - `logseq graph import --type edn --input --repo ` - `logseq graph import --type sqlite --input --repo ` diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 894e34583a..6254450d39 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -72,7 +72,7 @@ Graph commands: - `graph remove --repo ` - remove a graph - `graph validate --repo ` - validate graph data - `graph info [--repo ]` - show graph metadata (defaults to current graph) -- `graph export --type edn|sqlite --output [--repo ]` - export a graph to EDN or SQLite +- `graph export --type edn|sqlite --file [--repo ]` - export a graph to EDN or SQLite - `graph import --type edn|sqlite --input --repo ` - import a graph from EDN or SQLite (new graph only) For any command that requires `--repo`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). `graph import` fails if the target graph already exists. @@ -145,7 +145,7 @@ Revision: Output formats: - Global `--output ` applies to all commands -- For `graph export`, `--output` refers to the destination file path. Output formatting is controlled via `:output-format` in config or `LOGSEQ_CLI_OUTPUT`. +- Output formatting is controlled via global `--output`, `:output-format` in config, or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. `list property` includes a dedicated `TYPE` column. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. - For `list property`, `TYPE` is returned in default output (without `--expand`) for human and structured (`json`/`edn`) formats. - `upsert page` and `upsert block` return entity ids in `data.result` for JSON/EDN output, and include ids in human output. @@ -188,7 +188,7 @@ Examples: ```bash node ./dist/logseq.js graph create --repo demo -node ./dist/logseq.js graph export --type edn --output /tmp/demo.edn --repo demo +node ./dist/logseq.js graph export --type edn --file /tmp/demo.edn --repo demo node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --repo demo-import node ./dist/logseq.js upsert block --target-page TestPage --content "hello world" node ./dist/logseq.js move --uuid --target-page TargetPage diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index ccec6d6e9c..d5a646274a 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -9,7 +9,7 @@ (def ^:private graph-export-spec {:type {:desc "Export type (edn, sqlite)"} - :output {:desc "Output path"}}) + :file {:desc "Export file path"}}) (def ^:private graph-import-spec {:type {:desc "Import type (edn, sqlite)"} @@ -107,7 +107,7 @@ :graph (core/repo->graph repo)}}))) (defn build-export-action - [repo export-type output] + [repo export-type file] (if-not (seq repo) {:ok? false :error {:code :missing-repo @@ -117,7 +117,7 @@ :repo repo :graph (core/repo->graph repo) :export-type export-type - :output output}})) + :file file}})) (defn build-import-action [repo import-type input] @@ -201,9 +201,9 @@ (js/Buffer.from export-result "base64") export-result) format (if (= export-type "sqlite") :sqlite :edn)] - (transport/write-output {:format format :path (:output action) :data data}) + (transport/write-output {:format format :path (:file action) :data data}) {:status :ok - :data {:message (str "wrote " (:output action))}}))) + :data {:message (str "wrote " (:file action))}}))) (defn execute-graph-import [action config] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 560f89a94e..685fd98393 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -82,11 +82,11 @@ :message "input is required"} :summary summary}) -(defn- missing-output-result +(defn- missing-file-result [summary] {:ok? false - :error {:code :missing-output - :message "output is required"} + :error {:code :missing-file + :message "file is required"} :summary summary}) (defn- missing-query-result @@ -248,8 +248,8 @@ (and (= command :graph-export) (not (seq (graph-command/normalize-import-export-type (:type opts))))) (missing-type-result summary) - (and (= command :graph-export) (not (seq (:output opts)))) - (missing-output-result summary) + (and (= command :graph-export) (not (seq (:file opts)))) + (missing-file-result summary) (and (= command :graph-export) (not (contains? (graph-command/import-export-types) @@ -387,7 +387,7 @@ :graph-export (let [export-type (graph-command/normalize-import-export-type (:type options))] - (graph-command/build-export-action repo export-type (:output options))) + (graph-command/build-export-action repo export-type (:file options))) :graph-import (let [import-repo (command-core/resolve-repo (:repo options)) @@ -481,4 +481,4 @@ :schema :source :target :update-tags :update-properties :remove-tags :remove-properties - :export-type :output :import-type :input]))))) + :export-type :file :import-type :input]))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 3658007189..ffa017fcb1 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -330,8 +330,8 @@ (str "Removed property: " id " (repo: " repo ")"))) (defn- format-graph-export - [{:keys [export-type output]}] - (str "Exported " export-type " to " output)) + [{:keys [export-type file]}] + (str "Exported " export-type " to " file)) (defn- format-graph-import [{:keys [import-type input]}] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 2a9fd13e7f..97aa3a879a 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1183,14 +1183,14 @@ (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) - (testing "graph export parses with type and output" + (testing "graph export parses with type and file" (let [result (commands/parse-args ["graph" "export" "--type" "edn" - "--output" "export.edn"])] + "--file" "export.edn"])] (is (true? (:ok? result))) (is (= :graph-export (:command result))) (is (= "edn" (get-in result [:options :type]))) - (is (= "export.edn" (get-in result [:options :output]))))) + (is (= "export.edn" (get-in result [:options :file]))))) (testing "graph import parses with type, input, and repo" (let [result (commands/parse-args ["graph" "import" @@ -1204,14 +1204,19 @@ (is (= "demo" (get-in result [:options :repo]))))) (testing "graph export requires type" - (let [result (commands/parse-args ["graph" "export" "--output" "export.edn"])] + (let [result (commands/parse-args ["graph" "export" "--file" "export.edn"])] (is (false? (:ok? result))) (is (= :missing-type (get-in result [:error :code]))))) - (testing "graph export requires output" + (testing "graph export requires file" (let [result (commands/parse-args ["graph" "export" "--type" "edn"])] (is (false? (:ok? result))) - (is (= :missing-output (get-in result [:error :code]))))) + (is (= :missing-file (get-in result [:error :code]))))) + + (testing "graph export accepts global output format and still requires file" + (let [result (commands/parse-args ["graph" "export" "--type" "edn" "--output" "json"])] + (is (false? (:ok? result))) + (is (= :missing-file (get-in result [:error :code]))))) (testing "graph import requires repo" (let [result (commands/parse-args ["graph" "import" @@ -1280,7 +1285,7 @@ (testing "graph export uses config repo" (let [parsed {:ok? true :command :graph-export - :options {:type "edn" :output "export.edn"}} + :options {:type "edn" :file "export.edn"}} result (commands/build-action parsed {:repo "demo"})] (is (true? (:ok? result))) (is (= :graph-export (get-in result [:action :type]))))) @@ -2485,22 +2490,22 @@ :repo "logseq_db_demo" :graph "demo" :export-type "edn" - :output "/tmp/export.edn" + :file "/tmp/export.edn" :allow-missing-graph true} {}) sqlite-result (commands/execute {:type :graph-export :repo "logseq_db_demo" :graph "demo" :export-type "sqlite" - :output "/tmp/export.sqlite" + :file "/tmp/export.sqlite" :allow-missing-graph true} {})] (is (= :ok (:status edn-result))) (is (= :ok (:status sqlite-result))) (is (= "edn" (get-in edn-result [:context :export-type]))) - (is (= "/tmp/export.edn" (get-in edn-result [:context :output]))) + (is (= "/tmp/export.edn" (get-in edn-result [:context :file]))) (is (= "sqlite" (get-in sqlite-result [:context :export-type]))) - (is (= "/tmp/export.sqlite" (get-in sqlite-result [:context :output]))) + (is (= "/tmp/export.sqlite" (get-in sqlite-result [:context :file]))) (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] [:thread-api/export-db-base64 true ["logseq_db_demo"]]] @invoke-calls)) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index e7b0fa96a6..11c4695525 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -207,7 +207,7 @@ (let [result (format/format-result {:status :ok :command :graph-export :context {:export-type "edn" - :output "/tmp/export.edn"}} + :file "/tmp/export.edn"}} {:output-format nil})] (is (= "Exported edn to /tmp/export.edn" result)))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9407093818..31fb08d9f8 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -2190,7 +2190,7 @@ export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "edn" - "--output" export-path] data-dir cfg-path) + "--file" export-path] data-dir cfg-path) export-payload (parse-json-output export-result) _ (run-cli ["--repo" import-graph "graph" "import" @@ -2229,7 +2229,7 @@ export-result (run-cli ["--repo" export-graph "graph" "export" "--type" "sqlite" - "--output" export-path] data-dir cfg-path) + "--file" export-path] data-dir cfg-path) export-payload (parse-json-output export-result) _ (run-cli ["--repo" import-graph "graph" "import" From 4073d47fe41e42ad3df9402017c4f97c0c95ef53 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 2 Mar 2026 12:32:51 -0500 Subject: [PATCH 097/375] fix: frontend lint --- src/main/frontend/worker/sync/crypt.cljs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/main/frontend/worker/sync/crypt.cljs b/src/main/frontend/worker/sync/crypt.cljs index 5f6b41aa23..2b1d65b31b 100644 --- a/src/main/frontend/worker/sync/crypt.cljs +++ b/src/main/frontend/worker/sync/crypt.cljs @@ -27,11 +27,11 @@ (def ^:private invalid-coerce ::invalid-coerce) (def ^:private invalid-transit ::invalid-transit) -(defn- native-worker? +(defn native-worker? [] native-env?) -(defn- Date: Mon, 2 Mar 2026 18:17:56 -0500 Subject: [PATCH 098/375] fix: confusing and incorrect naming for --repo global option Repo refers to internal identifiers that start with 'logseq_db_' and used to start with 'logseq_local_'. The --repo option was not being used in that way and was just a reference to a graph name. While it's reasonable for internal CLIs like db-worker-node.js to use --repo, it is needlessly confusing to introduce repo to users. Almost all of our apps and docs label graphs as 'graph' and not 'repo' --- .../agent-guide/002-logseq-cli-subcommands.md | 12 +- .../003-db-worker-node-cli-orchestration.md | 16 +- .../006-logseq-cli-import-export.md | 12 +- ...logseq-cli-thread-api-and-command-split.md | 15 +- ...logseq-cli-db-graph-default-dir-locking.md | 2 +- .../031-logseq-cli-doctor-command.md | 2 +- ...e-db-prefix-in-user-visible-graph-names.md | 22 +- .../046-logseq-cli-upsert-tag-rename-by-id.md | 2 +- docs/cli/logseq-cli.md | 32 +- src/main/logseq/cli/command/core.cljs | 13 +- src/main/logseq/cli/command/graph.cljs | 4 +- src/main/logseq/cli/command/server.cljs | 2 +- src/main/logseq/cli/commands.cljs | 25 +- src/main/logseq/cli/config.cljs | 6 +- src/main/logseq/cli/format.cljs | 6 +- src/test/logseq/cli/commands_test.cljs | 94 ++-- src/test/logseq/cli/config_test.cljs | 26 +- src/test/logseq/cli/format_test.cljs | 6 +- src/test/logseq/cli/integration_test.cljs | 512 +++++++++--------- 19 files changed, 391 insertions(+), 418 deletions(-) diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md index 63f31ef6b9..8e14f38fd0 100644 --- a/docs/agent-guide/002-logseq-cli-subcommands.md +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -57,7 +57,7 @@ Global options apply to all subcommands and are parsed before subcommand options | --help | Show help | Available at top level and per subcommand. | | --version | Show version | Prints build time and revision. | | --config PATH | Config file path | Defaults to ~/logseq/cli.edn. | -| --repo REPO | Graph name | Used as current repo. | +| --graph GRAPH | Graph name | Used as current graph. | | --timeout-ms MS | Request timeout | Integer milliseconds. | | --output FORMAT | Output format | One of human, json, edn. | @@ -66,11 +66,11 @@ Each subcommand uses a nested path and its own options. | Subcommand path | Required args | Options | Notes | | --- | --- | --- | --- | | graph list | none | --output | Lists all graphs. | -| graph create | none | --repo GRAPH, --output | Creates and switches graph. | -| graph switch | none | --repo GRAPH, --output | Switches current graph. | -| graph remove | none | --repo GRAPH, --output | Removes graph. | -| graph validate | none | --repo GRAPH, --output | Validates graph. | -| graph info | none | --repo GRAPH, --output | Shows metadata, defaults to config repo if omitted. | +| graph create | none | --graph GRAPH, --output | Creates and switches graph. | +| graph switch | none | --graph GRAPH, --output | Switches current graph. | +| graph remove | none | --graph GRAPH, --output | Removes graph. | +| graph validate | none | --graph GRAPH, --output | Validates graph. | +| graph info | none | --graph GRAPH, --output | Shows metadata, defaults to config repo if omitted. | | block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. | | block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. | | search | QUERY | --type page|block|tag|property|all, --tag NAME, --case-sensitive, --sort updated-at|created-at, --order asc|desc, --output | Search text is positional and required. Human output columns: ID (db/id), TITLE. Block reference UUIDs in text are resolved recursively up to 10 levels. | diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md index eb88386fb9..53fe6268d9 100644 --- a/docs/agent-guide/003-db-worker-node-cli-orchestration.md +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -11,7 +11,7 @@ Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, re ## Requirements 1. Refactor `db-worker-node`: startup must require `--repo`; on startup it must open or create that graph; it must not switch graphs at runtime; it must create a lock file so a graph can be served by only one db-worker-node instance; it only needs to bind to localhost. -2. In `logseq-cli`, all commands requiring `--repo` or any graph operations must connect to or create the corresponding db-worker-node server. +2. In `logseq-cli`, all commands requiring `--graph` or any graph operations must connect to or create the corresponding db-worker-node server. 3. db-worker-node server must not be started manually; logseq-cli is fully responsible. 4. Add `server` subcommand(s) to logseq-cli for managing db-worker-node servers. @@ -59,7 +59,7 @@ Files: - New: `src/main/logseq/cli/server.cljs` (process management + lock handling) Key changes: -- **Repo resolution**: for all graph/content commands, require `--repo` or resolved repo from config; otherwise error. +- **Repo resolution**: for all graph/content commands, require `--graph` or resolved repo from config; otherwise error. - **Ensure server** (new helper `ensure-server!`): 1. Derive data-dir, repo dir, and lock file path from repo. 2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`. @@ -73,12 +73,12 @@ Key changes: Suggested command group: - `server list`: list servers from lock files (repo, pid, port, status). -- `server start --repo `: start server for repo. -- `server stop --repo `: stop server (SIGTERM or `/v1/shutdown`). -- `server restart --repo `: stop + start. +- `server start --graph `: start server for repo. +- `server stop --graph `: stop server (SIGTERM or `/v1/shutdown`). +- `server restart --graph `: stop + start. Implementation notes: -- `start|stop|restart` require `--repo`. +- `start|stop|restart` require `--graph`. - `list` scans data-dir for repo directories, reads lock files, and verifies status. - Consider adding `/v1/shutdown` in db-worker-node for graceful stop. @@ -104,8 +104,8 @@ Implementation notes: ## Open Questions -1. Should `graph list` require `--repo`? If not, define a “global” server or out-of-band access to data-dir. - - Answer: No --repo needed, using 'out-of-band access to data-dir' way +1. Should `graph list` require `--graph`? If not, define a “global” server or out-of-band access to data-dir. + - Answer: No --graph needed, using 'out-of-band access to data-dir' way 2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). - lockfile name:`db-worker.lock`, - Location: inside repo dir (e.g. `~/logseq/cli-graphs//db-worker.lock`). diff --git a/docs/agent-guide/006-logseq-cli-import-export.md b/docs/agent-guide/006-logseq-cli-import-export.md index 42df290964..461edee5aa 100644 --- a/docs/agent-guide/006-logseq-cli-import-export.md +++ b/docs/agent-guide/006-logseq-cli-import-export.md @@ -19,14 +19,14 @@ Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/ Prefer graph-scoped subcommands to keep import/export with graph management: -- `logseq graph export --type edn --file [--repo ]` -- `logseq graph export --type sqlite --file [--repo ]` -- `logseq graph import --type edn --input --repo ` -- `logseq graph import --type sqlite --input --repo ` +- `logseq graph export --type edn --file [--graph ]` +- `logseq graph export --type sqlite --file [--graph ]` +- `logseq graph import --type edn --input --graph ` +- `logseq graph import --type sqlite --input --graph ` Notes: - `graph import` only supports importing into a new graph name; it must not overwrite an existing graph. -- `--repo` is required for import, and required unless the current graph is set in config for export. +- `--graph` is required for import, and required unless the current graph is set in config for export. ## Current Capabilities (Baseline) @@ -70,7 +70,7 @@ Notes: ## Edge Cases - Large SQLite exports may exceed JSON limits if not base64/transit encoded; ensure streaming-safe or chunked base64 handling. -- Import should fail fast if the repo is missing and `--repo` is not provided, or if input file does not exist. +- Import should fail fast if the repo is missing and `--graph` is not provided, or if input file does not exist. - SQLite import while the repo is open must close/reopen connections to avoid stale datascript state. - EDN import should validate the export shape and surface readable errors when EDN is invalid or incompatible. - Overwrite behavior should be explicit for SQLite imports to prevent accidental data loss. diff --git a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md index e3b67af23e..0b3a8d77db 100644 --- a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md +++ b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md @@ -1,6 +1,6 @@ # Logseq CLI Thread API Keywords And Command Split Implementation Plan -Goal: Replace thread-api string usage with keywords, standardize CLI repo option to --repo, and split logseq.cli.commands into per-subcommand namespaces. +Goal: Replace thread-api string usage with keywords, standardize CLI repo/graph option to --graph, and split logseq.cli.commands into per-subcommand namespaces. Architecture: Update transport and db-worker-node boundaries to accept keyword methods while still serializing over HTTP. Refactor CLI command parsing into a shared dispatcher plus per-subcommand namespaces under a new command directory. Keep existing CLI behavior and output stable while updating option naming and error hints. @@ -10,11 +10,11 @@ Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md, docs/age ## Problem statement -The current CLI and db-worker-node codebase mixes thread-api method strings with keyword-based APIs, which makes it easy to introduce mismatches and reduces consistency with the thread-api macro design. The CLI option naming also has legacy --graph hints and expectations that conflict with the newer --repo naming, which creates confusion for users and tests. The logseq.cli.commands namespace has grown large and mixes parsing, validation, and execution for multiple command groups, which makes maintenance and ownership difficult. +The current CLI and db-worker-node codebase mixes thread-api method strings with keyword-based APIs, which makes it easy to introduce mismatches and reduces consistency with the thread-api macro design. The logseq.cli.commands namespace has grown large and mixes parsing, validation, and execution for multiple command groups, which makes maintenance and ownership difficult. ## Testing Plan -I will add unit tests to ensure all CLI thread-api method invocations use keywords and still serialize correctly through transport when invoking db-worker-node. I will add unit tests to ensure any error hint or help text for missing graph/repo uses --repo and no longer mentions --graph. I will add unit tests for command parsing and action building to cover the new per-subcommand namespaces and ensure summaries and help still match expected output. I will update db-worker-node tests to assert keyword method handling for repo validation and allowed non-repo methods. NOTE: I will write *all* tests before I add any implementation behavior. +I will add unit tests to ensure all CLI thread-api method invocations use keywords and still serialize correctly through transport when invoking db-worker-node. I will add unit tests for command parsing and action building to cover the new per-subcommand namespaces and ensure summaries and help still match expected output. I will update db-worker-node tests to assert keyword method handling for repo validation and allowed non-repo methods. NOTE: I will write *all* tests before I add any implementation behavior. ## Plan @@ -36,7 +36,7 @@ I will add unit tests to ensure all CLI thread-api method invocations use keywor 9. Update any db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` that assert method strings to use keyword expectations and verify non-repo method handling. -10. Replace any --graph option references in CLI formatting and tests by updating `src/main/logseq/cli/format.cljs` and `src/test/logseq/cli/format_test.cljs` to use --repo. +10. ~~Replace any --graph option references in CLI formatting and tests by updating `src/main/logseq/cli/format.cljs` and `src/test/logseq/cli/format_test.cljs` to use --repo.~~ 11. Search for any `:graph` option wiring in `src/main/logseq/cli/commands.cljs` and remove CLI option parsing for --graph, including any help or usage text, while preserving graph-specific subcommands like `graph create`. @@ -62,15 +62,14 @@ The tests will focus on behavior by asserting that CLI invocations still produce - Normalize thread-api method values at transport and db-worker-node boundaries to accept keywords and serialize as strings over HTTP. - Replace all explicit "thread-api/..." literals in CLI and db-worker-node call sites with :thread-api/... keywords. -- Remove --graph option handling and update error hints to use --repo. -- Preserve graph subcommands and graph naming semantics while standardizing on :repo options. +- Preserve graph subcommands and graph naming semantics while standardizing on :graph options. - Move per-subcommand parsing and execution helpers into `src/main/logseq/cli/command/` namespaces and keep `logseq.cli.commands` as a facade. - Keep action map shapes stable to avoid downstream changes in format or execution. - Update tests to match keyword method expectations and new module layout. -- Ensure public CLI output and behavior remain unchanged aside from --repo messaging. +- Ensure public CLI output and behavior remain unchanged ## Question -Resolved: Remove --graph entirely and fail fast on any --graph usage. +~~Resolved: Remove --graph entirely and fail fast on any --graph usage.~~ --- diff --git a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md index 1f139e372d..1384d26901 100644 --- a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md +++ b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md @@ -53,7 +53,7 @@ NOTE: I will write *all* tests before I add any implementation behavior. ## Integration sketch ```text -CLI --repo demo +CLI --graph demo -> command-core resolves internal repo: logseq_db_demo -> graph-dir resolver maps repo to graph key: demo -> fs paths use ~/logseq/graphs/demo diff --git a/docs/agent-guide/031-logseq-cli-doctor-command.md b/docs/agent-guide/031-logseq-cli-doctor-command.md index 13cc97b068..4a2872cd3b 100644 --- a/docs/agent-guide/031-logseq-cli-doctor-command.md +++ b/docs/agent-guide/031-logseq-cli-doctor-command.md @@ -178,6 +178,6 @@ Resolved: `doctor` will fail fast on the first failed check. Resolved: `doctor` will treat `:starting` servers as warnings when script and data-dir checks pass. -Resolved: `doctor` will support a future `--repo` scoped deep check that verifies per-graph lock path and repo directory access without starting the daemon. +Resolved: `doctor` will support a future `--graph` scoped deep check that verifies per-graph lock path and repo directory access without starting the daemon. --- diff --git a/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md index 1b338148c3..e26f5a5c23 100644 --- a/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md +++ b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md @@ -59,7 +59,7 @@ I will add RTC tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handle I will extend CLI formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to assert user-facing fields strip exactly one prefix and never introduce additional prefixes. -I will extend CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `graph list` and server output paths with unprefixed and prefix-like graph names, and assert prefix-like `--repo` values are treated as graph-name content instead of invalid input. +I will extend CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `graph list` and server output paths with unprefixed and prefix-like graph names, and assert prefix-like `--graph` values are treated as graph-name content instead of invalid input. I will add legacy graph discovery tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` to verify old directory names do not produce multi-prefix repo ids. @@ -77,7 +77,7 @@ NOTE: I will write all tests before I add any implementation behavior. 2. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/db/persist_test.cljs` to verify worker and Electron graph sources are canonicalized to one internal prefix. 3. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handler/db_based/rtc_test.cljs` for remote graph mapping and download paths with prefixed and double-prefixed payload names. 4. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` with failing cases where `:repo` or `:graph` includes one or two prefixes and output uses one-layer stripping only. -5. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with failing cases for graph list and server output using unprefixed and prefix-like `--repo` values, and assert prefix-like values are not rejected by argument validation. +5. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with failing cases for graph list and server output using unprefixed and prefix-like `--graph` values, and assert prefix-like values are not rejected by argument validation. 6. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` for legacy directory names that already contain `logseq_db_`. 7. Run focused tests and confirm failures reflect behavior gaps rather than setup errors. @@ -116,7 +116,7 @@ NOTE: I will write all tests before I add any implementation behavior. 26. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` so `repo->graph` strips exactly one leading prefix for user-visible output. 27. Verify `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` uses one-layer display normalization consistently for human, JSON, and EDN user-facing graph fields. 28. Update `/Users/rcmerci/gh-repos/logseq/deps/cli/src/logseq/cli/commands/graph.cljs` to canonicalize internal repo identifiers before list rendering and server-target resolution. -29. Ensure CLI parsing and command execution treat `--repo` as a graph-name string, so leading `logseq_db_` is interpreted as part of graph name when present and is not rejected by input validation. +29. Ensure CLI parsing and command execution treat `--graph` as a graph-name string, so leading `logseq_db_` is interpreted as part of graph name when present and is not rejected by input validation. ### Phase 7: Verification and release gate. @@ -124,11 +124,11 @@ NOTE: I will write all tests before I add any implementation behavior. 31. Run `bb dev:test -v 'frontend.db.persist-test'` and confirm merged-source canonicalization behavior is stable. 32. Run `bb dev:test -v 'frontend.handler.db-based.rtc-test'` and confirm remote ingestion cannot produce multi-prefix repos. 33. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm CLI display fields apply one-layer strip behavior. -34. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm graph command behavior remains stable with unprefixed and prefix-like `--repo` input. +34. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm graph command behavior remains stable with unprefixed and prefix-like `--graph` input. 35. Run `bb dev:test -v 'logseq.cli.common.graph-test'` and confirm legacy graph discovery does not emit double-prefixed repo names. 36. Run `bb dev:lint-and-test` and confirm `0 failures, 0 errors`. 37. Perform a manual graph-list smoke check on web and Electron to confirm normal graphs display without prefix and legacy doubles display with one remaining prefix. -38. Perform a manual CLI smoke check with `logseq graph list`, `logseq server status --repo demo`, and `logseq server status --repo logseq_db_demo` to confirm both inputs are accepted as graph names. +38. Perform a manual CLI smoke check with `logseq graph list`, `logseq server status --graph demo`, and `logseq server status --graph logseq_db_demo` to confirm both inputs are accepted as graph names. ## Edge cases @@ -141,9 +141,9 @@ NOTE: I will write all tests before I add any implementation behavior. | Legacy disk directory is named `logseq_db_logseq_db_demo`. | Discovery canonicalization collapses to internal repo `logseq_db_demo`. | | Remote graph payload returns `graph-name` as `logseq_db_demo`. | RTC mapping keeps internal repo `logseq_db_demo` and user-visible name `demo`. | | Remote graph payload returns `graph-name` as `logseq_db_logseq_db_demo`. | RTC mapping canonicalizes to internal repo `logseq_db_demo`, and user-visible name is `demo` after one-layer display strip. | -| CLI receives `--repo demo`. | Command works and output graph name is `demo`. | -| CLI receives `--repo logseq_db_demo`. | Command treats `logseq_db_` as part of graph name and does not fail argument validation. | -| CLI receives `--repo logseq_db_logseq_db_demo`. | Command treats the full value as graph name content and does not fail argument validation. | +| CLI receives `--graph demo`. | Command works and output graph name is `demo`. | +| CLI receives `--graph logseq_db_demo`. | Command treats `logseq_db_` as part of graph name and does not fail argument validation. | +| CLI receives `--graph logseq_db_logseq_db_demo`. | Command treats the full value as graph name content and does not fail argument validation. | | Non-user-visible fields like `data-testid` include repo id. | Existing selectors remain unchanged unless canonicalization is required to prevent duplicate graph entries. | ## Verification commands and expected outputs @@ -164,7 +164,7 @@ Web and Electron manual checks should show no new multi-prefix repo entries. Web and Electron display should strip one prefix only at render time. -CLI human output should match one-layer strip semantics, and CLI `--repo` should treat prefix-like values as normal graph names. +CLI human output should match one-layer strip semantics, and CLI `--graph` should treat prefix-like values as normal graph names. ## Testing Details @@ -184,7 +184,7 @@ CLI tests assert command behavior and output formatting remain stable for unpref - Reuse shared helpers across frontend, Electron, and CLI. - Preserve `data-testid` compatibility unless canonicalization makes key updates unavoidable. - Avoid one-time metadata migration for existing persisted graphs. -- Treat CLI `--repo` input as raw graph name where `logseq_db_` may be part of the name. +- Treat CLI `--graph` input as raw graph name where `logseq_db_` may be part of the name. - Keep internal thread-api contracts based on prefixed repo ids. - Follow `@test-driven-development` for RED, GREEN, and REFACTOR order. - Validate final patch with `@prompts/review.md` checklist. @@ -197,7 +197,7 @@ Decision: The implementation focus is to identify and fix all paths that can cre Decision: This change does not include a one-time metadata migration for existing persisted legacy values. -Decision: CLI `--repo` option treats leading `logseq_db_` as graph-name content, not as forbidden prefix. +Decision: CLI `--graph` option treats leading `logseq_db_` as graph-name content, not as forbidden prefix. Decision: Treat `data-testid` stability as a strict compatibility requirement for `clj-e2e`. diff --git a/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md index b40caa6ad5..3c8824ad74 100644 --- a/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md +++ b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md @@ -18,7 +18,7 @@ Current update mode in `execute-upsert-tag` only calls `ensure-tag-by-id!` and r Current db-worker-node path already supports page rename through `:thread-api/apply-outliner-ops` with `[:rename-page [page-uuid new-title]]` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`, so rename behavior can be reused without adding new APIs. -The user-visible gap is that a command like `logseq upsert tag --repo --id 180 --name "Project Renamed"` should rename tag `180`, but currently fails at option validation. +The user-visible gap is that a command like `logseq upsert tag --graph --id 180 --name "Project Renamed"` should rename tag `180`, but currently fails at option validation. ## Testing Plan diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 6254450d39..725b4b22b1 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -52,7 +52,7 @@ Graph directories on disk are stored as user-facing graph names (for example, `d Migration note: If you previously used `~/.logseq/cli-graphs` or `~/.logseq/cli.edn`, pass `--data-dir` or `--config` to continue using those locations. Supported keys include: -- `:repo` +- `:graph` - `:data-dir` - `:timeout-ms` - `:output-format` (use `:json` or `:edn` for scripting) @@ -67,22 +67,22 @@ Verbose logging: Graph commands: - `graph list` - list all db graphs -- `graph create --repo ` - create a new db graph and switch to it -- `graph switch --repo ` - switch current graph -- `graph remove --repo ` - remove a graph -- `graph validate --repo ` - validate graph data -- `graph info [--repo ]` - show graph metadata (defaults to current graph) -- `graph export --type edn|sqlite --file [--repo ]` - export a graph to EDN or SQLite -- `graph import --type edn|sqlite --input --repo ` - import a graph from EDN or SQLite (new graph only) +- `graph create --graph ` - create a new db graph and switch to it +- `graph switch --graph ` - switch current graph +- `graph remove --graph ` - remove a graph +- `graph validate --graph ` - validate graph data +- `graph info [--graph ]` - show graph metadata (defaults to current graph) +- `graph export --type edn|sqlite --file [--graph ]` - export a graph to EDN or SQLite +- `graph import --type edn|sqlite --input --graph ` - import a graph from EDN or SQLite (new graph only) -For any command that requires `--repo`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). `graph import` fails if the target graph already exists. +For any command that requires `--graph`, if the target graph does not exist, the CLI returns `graph not exists` (except for `graph create`). `graph import` fails if the target graph already exists. Server commands: - `server list` - list running db-worker-node servers -- `server status --repo ` - show server status for a graph -- `server start --repo ` - start db-worker-node for a graph -- `server stop --repo ` - stop db-worker-node for a graph -- `server restart --repo ` - restart db-worker-node for a graph +- `server status --graph ` - show server status for a graph +- `server start --graph ` - start db-worker-node for a graph +- `server stop --graph ` - stop db-worker-node for a graph +- `server restart --graph ` - restart db-worker-node for a graph - `doctor [--dev-script]` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness (`--dev-script` checks `static/db-worker-node.js` explicitly) Server ownership behavior: @@ -187,9 +187,9 @@ id8 └── b8 Examples: ```bash -node ./dist/logseq.js graph create --repo demo -node ./dist/logseq.js graph export --type edn --file /tmp/demo.edn --repo demo -node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --repo demo-import +node ./dist/logseq.js graph create --graph demo +node ./dist/logseq.js graph export --type edn --file /tmp/demo.edn --graph demo +node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --graph demo-import node ./dist/logseq.js upsert block --target-page TestPage --content "hello world" node ./dist/logseq.js move --uuid --target-page TargetPage node ./dist/logseq.js search "hello" diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 1a99ba6ddf..4c28609237 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -12,7 +12,7 @@ :version {:desc "Show version" :coerce :boolean} :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)"} - :repo {:desc "Graph name"} + :graph {:desc "Graph name"} :data-dir {:desc "Path to db-worker data dir (default ~/logseq/graphs)"} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} @@ -197,13 +197,6 @@ {:opts opts :args (rest remaining)})) {:opts opts :args remaining}))))) -(defn legacy-graph-opt? - [raw-args] - (some (fn [token] - (or (= token "--graph") - (string/starts-with? token "--graph="))) - raw-args)) - (defn cli-error->result [summary {:keys [msg]}] (invalid-options-result summary (or msg "invalid options"))) @@ -227,5 +220,5 @@ (defn pick-graph [options _command-args config] - (or (:repo options) - (:repo config))) + (or (:graph options) + (:graph config))) diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index d5a646274a..6c2a948001 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -150,7 +150,7 @@ (:direct-pass? action) (:args action))] (when-let [repo (:persist-repo action)] - (cli-config/update-config! config {:repo repo})) + (cli-config/update-config! config {:graph repo})) (if-let [write (:write action)] (let [{:keys [format path]} write] (transport/write-output {:format format :path path :data result}) @@ -167,7 +167,7 @@ :error {:code :graph-not-found :message (str "graph not found: " graph)}} (p/let [_ (cli-server/ensure-server! config (:repo action))] - (cli-config/update-config! config {:repo graph}) + (cli-config/update-config! config {:graph graph}) {:status :ok :data {:message (str "switched to " graph)}}))))) diff --git a/src/main/logseq/cli/command/server.cljs b/src/main/logseq/cli/command/server.cljs index 2d37477988..0cad4f5065 100644 --- a/src/main/logseq/cli/command/server.cljs +++ b/src/main/logseq/cli/command/server.cljs @@ -5,7 +5,7 @@ [promesa.core :as p])) (def ^:private server-spec - {:repo {:desc "Graph name"}}) + {:graph {:desc "Graph name"}}) (def entries [(core/command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 685fd98393..894daf4939 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -26,13 +26,6 @@ :message "graph name is required"} :summary summary}) -(defn- missing-repo-result - [summary] - {:ok? false - :error {:code :missing-repo - :message "repo is required"} - :summary summary}) - (defn- missing-content-result [summary] {:ok? false @@ -167,7 +160,7 @@ (let [opts (command-core/normalize-opts opts) args (vec args) cmd-summary (command-core/command-summary {:cmds cmds :spec spec}) - graph (:repo opts) + graph (:graph opts) has-args? (seq args) has-content? (or (seq (:content opts)) (seq (:blocks opts)) @@ -262,8 +255,8 @@ (and (= command :graph-import) (not (seq (:input opts)))) (missing-input-result summary) - (and (= command :graph-import) (not (seq (:repo opts)))) - (missing-repo-result summary) + (and (= command :graph-import) (not (seq (:graph opts)))) + (missing-graph-result summary) (and (= command :graph-import) (not (contains? (graph-command/import-export-types) @@ -271,8 +264,8 @@ (command-core/invalid-options-result summary (str "invalid type: " (:type opts))) (and (#{:server-status :server-start :server-stop :server-restart} command) - (not (seq (:repo opts)))) - (missing-repo-result summary) + (not (seq (:graph opts)))) + (missing-graph-result summary) :else (command-core/ok-result command opts args summary)))) @@ -282,13 +275,9 @@ (defn parse-args [raw-args] (let [summary (command-core/top-level-summary table) - legacy-graph-opt? (command-core/legacy-graph-opt? raw-args) {:keys [opts args]} (command-core/parse-leading-global-opts raw-args) {:keys [args id-from-stdin?]} (inject-stdin-id-arg (vec args))] (cond - legacy-graph-opt? - (command-core/invalid-options-result summary "unknown option: --graph") - (:version opts) (command-core/ok-result :version opts [] summary) @@ -380,7 +369,7 @@ (let [{:keys [command options args]} parsed graph (command-core/pick-graph options args config) repo (command-core/resolve-repo graph) - server-repo (command-core/resolve-repo (:repo options))] + server-repo (command-core/resolve-repo (:graph options))] (case command (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) (graph-command/build-graph-action command graph repo) @@ -390,7 +379,7 @@ (graph-command/build-export-action repo export-type (:file options))) :graph-import - (let [import-repo (command-core/resolve-repo (:repo options)) + (let [import-repo (command-core/resolve-repo (:graph options)) import-type (graph-command/normalize-import-export-type (:type options))] (graph-command/build-import-action import-repo import-type (:input options))) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 11dc5c987f..f27152b557 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -46,7 +46,7 @@ (let [path (or config-path (default-config-path)) current (or (read-config-file path) {}) filtered-current (dissoc current :auth-token :retries) - filtered-updates (dissoc updates :auth-token :retries) + filtered-updates (dissoc (or updates {}) :auth-token :retries) next (merge filtered-current filtered-updates)] (ensure-config-dir! path) (.writeFileSync fs path (pr-str next)) @@ -56,8 +56,8 @@ [] (let [env (.-env js/process)] (cond-> {} - (seq (gobj/get env "LOGSEQ_CLI_REPO")) - (assoc :repo (gobj/get env "LOGSEQ_CLI_REPO")) + (seq (gobj/get env "LOGSEQ_CLI_GRAPH")) + (assoc :graph (gobj/get env "LOGSEQ_CLI_GRAPH")) (seq (gobj/get env "LOGSEQ_CLI_DATA_DIR")) (assoc :data-dir (gobj/get env "LOGSEQ_CLI_DATA_DIR")) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index ffa017fcb1..cb053a7b73 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -87,8 +87,8 @@ (defn- error-hint [{:keys [code]}] (case code - :missing-graph "Use --repo " - :missing-repo "Use --repo " + :missing-graph "Use --graph " + :missing-repo "Use --graph " :missing-content "Use --content or pass content as args" :missing-tag-name "Use --name " :missing-query "Use --query " @@ -221,7 +221,7 @@ (defn- format-server-list [servers] (format-counted-table - ["REPO" "STATUS" "HOST" "PORT" "PID" "OWNER"] + ["GRAPH" "STATUS" "HOST" "PORT" "PID" "OWNER"] (mapv (fn [server] [(:repo server) (:status server) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 97aa3a879a..6d7f0384c9 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -87,7 +87,7 @@ (is (contains-bold? summary "server list")) (is (contains-bold? summary "server start")) (is (contains-bold? summary "--help")) - (is (contains-bold? summary "--repo")) + (is (contains-bold? summary "--graph")) (is (re-find #"\u001b\[[0-9;]*mCommands\u001b\[[0-9;]*m:" summary)) (is (re-find #"\u001b\[[0-9;]*moptions\u001b\[[0-9;]*m:" summary)))) @@ -317,14 +317,6 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) -(deftest test-parse-args-rejects-graph-option - (testing "rejects legacy --graph option" - (let [result (commands/parse-args ["--graph" "demo" "graph" "list"])] - (is (false? (:ok? result))) - (is (= :invalid-options (get-in result [:error :code]))) - (is (= "unknown option: --graph" - (strip-ansi (get-in result [:error :message]))))))) - (deftest test-parse-args-global-options (testing "global output option is accepted" (let [result (commands/parse-args ["--output" "json" "graph" "list"])] @@ -1178,7 +1170,7 @@ (is (= "[\"Hello\"]" (get-in result [:options :inputs])))))) (deftest test-verb-subcommand-parse-graph-import-export - (testing "graph create requires --repo even with positional args" + (testing "graph create requires --graph even with positional args" (let [result (commands/parse-args ["graph" "create" "demo"])] (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) @@ -1196,12 +1188,12 @@ (let [result (commands/parse-args ["graph" "import" "--type" "sqlite" "--input" "import.sqlite" - "--repo" "demo"])] + "--graph" "demo"])] (is (true? (:ok? result))) (is (= :graph-import (:command result))) (is (= "sqlite" (get-in result [:options :type]))) (is (= "import.sqlite" (get-in result [:options :input]))) - (is (= "demo" (get-in result [:options :repo]))))) + (is (= "demo" (get-in result [:options :graph]))))) (testing "graph export requires type" (let [result (commands/parse-args ["graph" "export" "--file" "export.edn"])] @@ -1218,27 +1210,27 @@ (is (false? (:ok? result))) (is (= :missing-file (get-in result [:error :code]))))) - (testing "graph import requires repo" + (testing "graph import requires graph" (let [result (commands/parse-args ["graph" "import" "--type" "edn" "--input" "import.edn"])] (is (false? (:ok? result))) - (is (= :missing-repo (get-in result [:error :code]))))) + (is (= :missing-graph (get-in result [:error :code]))))) (testing "graph import rejects unknown type" (let [result (commands/parse-args ["graph" "import" "--type" "zip" "--input" "import.zip" - "--repo" "demo"])] + "--graph" "demo"])] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) (testing "server status accepts prefix-like repo option values" (let [result (commands/parse-args ["server" "status" - "--repo" "logseq_db_logseq_db_demo"])] + "--graph" "logseq_db_logseq_db_demo"])] (is (true? (:ok? result))) (is (= :server-status (:command result))) - (is (= "logseq_db_logseq_db_demo" (get-in result [:options :repo])))))) + (is (= "logseq_db_logseq_db_demo" (get-in result [:options :graph])))))) (deftest test-verb-subcommand-parse-flags (testing "verb subcommands reject unknown flags" @@ -1271,14 +1263,14 @@ (is (= :missing-graph (get-in result [:error :code]))))) (testing "graph-switch uses graph name" - (let [parsed {:ok? true :command :graph-switch :options {:repo "demo"}} + (let [parsed {:ok? true :command :graph-switch :options {:graph "demo"}} result (commands/build-action parsed {})] (is (true? (:ok? result))) (is (= :graph-switch (get-in result [:action :type]))))) (testing "graph-info defaults to config repo" (let [parsed {:ok? true :command :graph-info :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :graph-info (get-in result [:action :type]))))) @@ -1286,7 +1278,7 @@ (let [parsed {:ok? true :command :graph-export :options {:type "edn" :file "export.edn"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :graph-export (get-in result [:action :type]))))) @@ -1312,13 +1304,13 @@ (is (= :missing-repo (get-in result [:error :code]))))) (testing "server stop builds action" - (let [parsed {:ok? true :command :server-stop :options {:repo "demo"}} + (let [parsed {:ok? true :command :server-stop :options {:graph "demo"}} result (commands/build-action parsed {})] (is (true? (:ok? result))) (is (= :server-stop (get-in result [:action :type]))))) (testing "server status canonicalizes multi-prefixed repo option" - (let [parsed {:ok? true :command :server-status :options {:repo "logseq_db_logseq_db_demo"}} + (let [parsed {:ok? true :command :server-status :options {:graph "logseq_db_logseq_db_demo"}} result (commands/build-action parsed {})] (is (true? (:ok? result))) (is (= :server-status (get-in result [:action :type]))) @@ -1348,19 +1340,19 @@ (testing "add block requires content" (let [parsed {:ok? true :command :upsert-block :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) (testing "add block builds insert-blocks op" (let [parsed {:ok? true :command :upsert-block :options {:content "hello"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :upsert-block (get-in result [:action :type]))))) (testing "add page requires name" (let [parsed {:ok? true :command :upsert-page :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) @@ -1369,7 +1361,7 @@ :command :upsert-page :options {:id 42 :update-properties "{:logseq.property/publishing-public? true}"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :update (get-in result [:action :mode]))) (is (= 42 (get-in result [:action :id])))))) @@ -1378,13 +1370,13 @@ (testing "upsert tag requires name" (let [parsed {:ok? true :command :upsert-tag :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-tag-name (get-in result [:error :code]))))) (testing "upsert tag builds normalized action" (let [parsed {:ok? true :command :upsert-tag :options {:name " #Quote "}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-tag :mode :create @@ -1395,7 +1387,7 @@ (testing "upsert tag by id builds update action" (let [parsed {:ok? true :command :upsert-tag :options {:id 123}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-tag :mode :update @@ -1406,7 +1398,7 @@ (testing "upsert tag by id with name builds update action" (let [parsed {:ok? true :command :upsert-tag :options {:id 123 :name " #Project Renamed "}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-tag :mode :update @@ -1418,7 +1410,7 @@ (testing "upsert tag by id rejects blank rename name" (let [parsed {:ok? true :command :upsert-tag :options {:id 123 :name " "}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -1430,7 +1422,7 @@ :cardinality "many" :hide true :public false}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-property :mode :create @@ -1449,7 +1441,7 @@ :options {:id 654 :type "node" :cardinality "many"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:type :upsert-property :mode :update @@ -1464,46 +1456,46 @@ (testing "remove block requires target" (let [parsed {:ok? true :command :remove-block :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) (testing "remove block normalizes id vector in build action" (let [parsed {:ok? true :command :remove-block :options {:id "[1 2]"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :remove-block (get-in result [:action :type]))) (is (= [1 2] (get-in result [:action :ids]))))) (testing "remove page requires name" (let [parsed {:ok? true :command :remove-page :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-page-name (get-in result [:error :code]))))) (testing "remove tag parses by id" (let [parsed {:ok? true :command :remove-tag :options {:id 42}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :remove-tag (get-in result [:action :type]))) (is (= 42 (get-in result [:action :id]))))) (testing "remove property parses by name" (let [parsed {:ok? true :command :remove-property :options {:name "owner"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :remove-property (get-in result [:action :type]))) (is (= "owner" (get-in result [:action :name]))))) (testing "show requires target" (let [parsed {:ok? true :command :show :options {}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-target (get-in result [:error :code]))))) (testing "show normalizes id vector in build action" (let [parsed {:ok? true :command :show :options {:id "[1 2]"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :show (get-in result [:action :type]))) (is (= [1 2] (get-in result [:action :ids])))))) @@ -1513,7 +1505,7 @@ (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--update-properties" "{:not/a 1}"]) - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= {:not/a 1} (get-in result [:action :update-properties]))))) @@ -1521,7 +1513,7 @@ (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--update-properties" "{\"Publishing Public?\" true}"]) - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :logseq.property/publishing-public? (-> result :action :update-properties keys first))))) @@ -1530,7 +1522,7 @@ (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--update-properties" "{:logseq.property/heading 1}"]) - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -1538,7 +1530,7 @@ (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--update-properties" "{:logseq.property/publishing-public? \"nope\"}"]) - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) @@ -1547,7 +1539,7 @@ (let [parsed (commands/parse-args ["upsert" "block" "--content" "hello" "--update-tags" "[42]"]) - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= [42] (get-in result [:action :update-tags])))))) @@ -1562,13 +1554,13 @@ (deftest test-build-action-update (testing "upsert block create mode requires content when source selector is absent" (let [parsed {:ok? true :command :upsert-block :options {:target-id 2}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :missing-content (get-in result [:error :code]))))) (testing "upsert block update mode requires target or update/remove options" (let [parsed {:ok? true :command :upsert-block :options {:id 1}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -1576,7 +1568,7 @@ (let [parsed {:ok? true :command :upsert-block :options {:id 1 :update-tags "[\"TagA\"]"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :upsert-block (get-in result [:action :type]))) (is (= ["TagA"] (get-in result [:action :update-tags]))))) @@ -1585,7 +1577,7 @@ (let [parsed {:ok? true :command :upsert-block :options {:id 1 :update-tags "{:tag \"no\"}"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code]))))) @@ -1593,7 +1585,7 @@ (let [parsed {:ok? true :command :upsert-block :options {:id 1 :content "hello" :update-tags "[\"TagA\"]"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :upsert-block (get-in result [:action :type]))) (is (= 1 (get-in result [:action :id]))) @@ -1605,7 +1597,7 @@ :options {:id 1 :update-properties "{:user.property/owner \"alice\"}" :remove-properties "[:user.property/owner]"}} - result (commands/build-action parsed {:repo "demo"})] + result (commands/build-action parsed {:graph "demo"})] (is (true? (:ok? result))) (is (= :upsert-block (get-in result [:action :type]))) (is (= {:user.property/owner "alice"} diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index a20df6fdb9..6c0e91623b 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -24,22 +24,22 @@ (let [dir (node-helper/create-tmp-dir) cfg-path (node-path/join dir "cli.edn") _ (fs/writeFileSync cfg-path - (str "{:repo \"file-repo\" " + (str "{:graph \"file-repo\" " ":data-dir \"file-data\" " ":timeout-ms 111 " ":output-format :edn}")) - env {"LOGSEQ_CLI_REPO" "env-repo" + env {"LOGSEQ_CLI_GRAPH" "env-repo" "LOGSEQ_CLI_DATA_DIR" "env-data" "LOGSEQ_CLI_TIMEOUT_MS" "222" "LOGSEQ_CLI_OUTPUT" "json"} opts {:config-path cfg-path - :repo "cli-repo" + :graph "cli-repo" :data-dir "cli-data" :timeout-ms 333 :output-format :human} result (with-env env #(config/resolve-config opts))] (is (= cfg-path (:config-path result))) - (is (= "cli-repo" (:repo result))) + (is (= "cli-repo" (:graph result))) (is (= "cli-data" (:data-dir result))) (is (= 333 (:timeout-ms result))) (is (nil? (:auth-token result))) @@ -49,11 +49,11 @@ (deftest test-env-overrides-file (let [dir (node-helper/create-tmp-dir) cfg-path (node-path/join dir "cli.edn") - _ (fs/writeFileSync cfg-path "{:repo \"file-repo\" :data-dir \"file-data\"}") - env {"LOGSEQ_CLI_REPO" "env-repo" + _ (fs/writeFileSync cfg-path "{:graph \"file-repo\" :data-dir \"file-data\"}") + env {"LOGSEQ_CLI_GRAPH" "env-repo" "LOGSEQ_CLI_DATA_DIR" "env-data"} result (with-env env #(config/resolve-config {:config-path cfg-path}))] - (is (= "env-repo" (:repo result))) + (is (= "env-repo" (:graph result))) (is (= "env-data" (:data-dir result))))) (deftest test-output-format-env-overrides-file @@ -87,22 +87,22 @@ (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") - _ (fs/writeFileSync cfg-path "{:repo \"old\"}") - _ (config/update-config! {:config-path cfg-path} {:repo "new"}) + _ (fs/writeFileSync cfg-path "{:graph \"old\"}") + _ (config/update-config! {:config-path cfg-path} {:graph "new"}) contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] - (is (= "new" (:repo parsed))))) + (is (= "new" (:graph parsed))))) (deftest test-update-config-strips-removed-options (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") - _ (fs/writeFileSync cfg-path "{:repo \"old\"}") + _ (fs/writeFileSync cfg-path "{:graph \"old\"}") _ (config/update-config! {:config-path cfg-path} - {:repo "new" + {:graph "new" :auth-token "secret" :retries 2}) contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] - (is (= "new" (:repo parsed))) + (is (= "new" (:graph parsed))) (is (not (contains? parsed :auth-token))) (is (not (contains? parsed :retries))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 11c4695525..bb6e6987ea 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -284,7 +284,7 @@ :pid 9876 :owner-source :cli}]}} {:output-format nil})] - (is (= (str "REPO STATUS HOST PORT PID OWNER\n" + (is (= (str "GRAPH STATUS HOST PORT PID OWNER\n" "demo-repo :ready 127.0.0.1 1234 9876 :cli\n" "Count: 1") result)))) @@ -298,7 +298,7 @@ :port 1234 :pid 9876}]}} {:output-format nil})] - (is (= (str "REPO STATUS HOST PORT PID OWNER\n" + (is (= (str "GRAPH STATUS HOST PORT PID OWNER\n" "demo-repo :ready 127.0.0.1 1234 9876 -\n" "Count: 1") result))))) @@ -460,7 +460,7 @@ :message "graph name is required"}} {:output-format nil})] (is (= (str "Error (missing-graph): graph name is required\n" - "Hint: Use --repo ") + "Hint: Use --graph ") result)))) (testing "owner mismatch includes ownership hint" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 31fb08d9f8..0fca0d4422 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -128,18 +128,18 @@ [data-dir] (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "tags-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "tags-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path)] + _ (run-cli ["graph" "create" "--graph" "tags-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "tags-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path)] {:cfg-path cfg-path :repo "tags-graph"})) (defn- stop-repo! [data-dir cfg-path repo] - (p/let [result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path)] + (p/let [result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path)] (parse-json-output result))) (defn- run-query [data-dir cfg-path repo query inputs] - (p/let [result (run-cli ["--repo" repo "query" "--query" query "--inputs" inputs] + (p/let [result (run-cli ["--graph" repo "query" "--query" query "--inputs" inputs] data-dir cfg-path)] (parse-json-output result))) @@ -174,7 +174,7 @@ (defn- list-items [data-dir cfg-path repo list-type] - (p/let [result (run-cli ["--repo" repo "list" list-type] data-dir cfg-path)] + (p/let [result (run-cli ["--graph" repo "list" list-type] data-dir cfg-path)] (parse-json-output result))) (defn- find-item-id @@ -229,7 +229,7 @@ (fs/chmodSync repo-dir 365) (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - result (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + result (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) payload (parse-json-output result)] (is (= 1 (:exit-code result))) (is (= "error" (:status payload))) @@ -245,11 +245,11 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "demo-graph"] data-dir cfg-path) + create-result (run-cli ["graph" "create" "--graph" "demo-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) info-result (run-cli ["graph" "info"] data-dir cfg-path) info-payload (parse-json-output info-result) - stop-result (run-cli ["server" "stop" "--repo" "demo-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "demo-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-result))) (is (= "ok" (:status create-payload))) @@ -268,23 +268,23 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "content-graph"] data-dir cfg-path) - add-page-result (run-cli ["--repo" "content-graph" "upsert" "page" "--page" "TestPage"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "content-graph"] data-dir cfg-path) + add-page-result (run-cli ["--graph" "content-graph" "upsert" "page" "--page" "TestPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) - list-page-result (run-cli ["--repo" "content-graph" "list" "page"] data-dir cfg-path) + list-page-result (run-cli ["--graph" "content-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) - list-tag-result (run-cli ["--repo" "content-graph" "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" "content-graph" "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) - list-property-result (run-cli ["--repo" "content-graph" "list" "property"] data-dir cfg-path) + list-property-result (run-cli ["--graph" "content-graph" "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - add-block-result (run-cli ["--repo" "content-graph" "upsert" "block" "--target-page" "TestPage" "--content" "Test block"] data-dir cfg-path) + add-block-result (run-cli ["--graph" "content-graph" "upsert" "block" "--target-page" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - show-result (run-cli ["--repo" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) + show-result (run-cli ["--graph" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) show-payload (parse-json-output show-result) - remove-page-result (run-cli ["--repo" "content-graph" "remove" "page" "--name" "TestPage"] data-dir cfg-path) + remove-page-result (run-cli ["--graph" "content-graph" "remove" "page" "--name" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) - stop-result (run-cli ["server" "stop" "--repo" "content-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) @@ -313,8 +313,8 @@ repo "add-page-json-id-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - add-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + add-page-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) page-ids (get-in add-page-payload [:data :result]) page-id (first page-ids) @@ -322,7 +322,7 @@ "[:find ?id . :in $ ?page-name :where [?id :block/name ?page-name]]" (pr-str [(common-util/page-name-sanity-lc "Home")])) queried-page-id (get-in query-payload [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) @@ -345,9 +345,9 @@ :block/children [{:block/title "Child"}]} {:block/title "Sibling"}]) _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) - add-block-result (run-cli ["--repo" repo + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + add-block-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--blocks" blocks-edn] @@ -360,7 +360,7 @@ block-titles (->> (get-in title-query-payload [:data :result]) (map first) set) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-block-result)) (pr-str (:error add-block-payload))) @@ -383,15 +383,15 @@ repo "add-edn-id-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - add-page-result (run-cli ["--repo" repo + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + add-page-result (run-cli ["--graph" repo "--output" "edn" "upsert" "page" "--page" "Home"] data-dir cfg-path) add-page-payload (parse-edn-output add-page-result) page-ids (get-in add-page-payload [:data :result]) - add-block-result (run-cli ["--repo" repo + add-block-result (run-cli ["--graph" repo "--output" "edn" "upsert" "block" "--target-page" "Home" @@ -399,7 +399,7 @@ data-dir cfg-path) add-block-payload (parse-edn-output add-block-result) block-ids (get-in add-block-payload [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= :ok (:status add-page-payload))) @@ -423,30 +423,30 @@ repo "add-id-chain-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - add-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "ChainPage"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + add-page-result (run-cli ["--graph" repo "upsert" "page" "--page" "ChainPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) page-id (first-result-id add-page-payload) - add-block-result (run-cli ["--repo" repo + add-block-result (run-cli ["--graph" repo "upsert" "block" "--target-id" (str page-id) "--content" "Chain block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) block-id (first-result-id add-block-payload) - update-result (run-cli ["--repo" repo + update-result (run-cli ["--graph" repo "upsert" "block" "--id" (str block-id) "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) update-payload (parse-json-output update-result) - remove-result (run-cli ["--repo" repo "remove" "block" "--id" (str block-id)] data-dir cfg-path) + remove-result (run-cli ["--graph" repo "remove" "block" "--id" (str block-id)] data-dir cfg-path) remove-payload (parse-json-output remove-result) query-after-remove (run-query data-dir cfg-path repo "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" (pr-str ["Chain block"])) removed-id (get-in query-after-remove [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) @@ -471,11 +471,11 @@ repo "upsert-page-existing-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - create-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + create-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) create-payload (parse-json-output create-result) page-id (first-result-id create-payload) - update-result (run-cli ["--repo" repo + update-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home" "--update-properties" "{:logseq.property/publishing-public? true}"] @@ -484,7 +484,7 @@ update-id (first-result-id update-payload) property-after-update (query-property data-dir cfg-path repo "Home" ":logseq.property/publishing-public?") - remove-result (run-cli ["--repo" repo + remove-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home" "--remove-properties" "[:logseq.property/publishing-public?]"] @@ -493,7 +493,7 @@ remove-id (first-result-id remove-payload) property-after-remove (query-property data-dir cfg-path repo "Home" ":logseq.property/publishing-public?") - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-result))) (is (= "ok" (:status create-payload))) @@ -518,20 +518,20 @@ repo "upsert-page-missing-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - missing-tag-result (run-cli ["--repo" repo + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + missing-tag-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home" "--update-tags" "[\"MissingTag\"]"] data-dir cfg-path) missing-tag-payload (parse-json-output missing-tag-result) - missing-property-result (run-cli ["--repo" repo + missing-property-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home" "--update-properties" "{:not/a 1}"] data-dir cfg-path) missing-property-payload (parse-json-output missing-property-result) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "error" (:status missing-tag-payload))) (is (= :tag-not-found (keyword (get-in missing-tag-payload [:error :code])))) @@ -549,23 +549,23 @@ repo "upsert-id-mode-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - create-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + create-page-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) create-page-payload (parse-json-output create-page-result) page-id (first-result-id create-page-payload) - update-page-result (run-cli ["--repo" repo + update-page-result (run-cli ["--graph" repo "upsert" "page" "--id" (str page-id) "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) update-page-payload (parse-json-output update-page-result) page-value (query-property data-dir cfg-path repo "Home" ":logseq.property/publishing-public?") - create-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] data-dir cfg-path) + create-tag-result (run-cli ["--graph" repo "upsert" "tag" "--name" "StableTag"] data-dir cfg-path) create-tag-payload (parse-json-output create-tag-result) tag-id (first-result-id create-tag-payload) - noop-tag-result (run-cli ["--repo" repo "upsert" "tag" "--id" (str tag-id)] data-dir cfg-path) + noop-tag-result (run-cli ["--graph" repo "upsert" "tag" "--id" (str tag-id)] data-dir cfg-path) noop-tag-payload (parse-json-output noop-tag-result) - create-property-result (run-cli ["--repo" repo + create-property-result (run-cli ["--graph" repo "upsert" "property" "--name" "OwnerProp" "--type" "default"] @@ -573,7 +573,7 @@ create-property-payload (parse-json-output create-property-result) property-id (first-result-id create-property-payload) property-name (common-util/page-name-sanity-lc "OwnerProp") - update-property-result (run-cli ["--repo" repo + update-property-result (run-cli ["--graph" repo "upsert" "property" "--id" (str property-id) "--type" "node" @@ -583,7 +583,7 @@ property-schema (run-query data-dir cfg-path repo "[:find ?type . :in $ ?name :where [?p :block/name ?name] [?p :logseq.property/type ?type]]" (pr-str [property-name])) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-page-result))) (is (= "ok" (:status create-page-payload))) @@ -617,27 +617,27 @@ repo "upsert-legacy-options-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - create-page-result (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + create-page-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) create-page-payload (parse-json-output create-page-result) page-id (first-result-id create-page-payload) - legacy-block-result (run-cli ["--repo" repo + legacy-block-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--content" "Legacy block" "--tags" "[\"Quote\"]"] data-dir cfg-path) - legacy-page-result (run-cli ["--repo" repo + legacy-page-result (run-cli ["--graph" repo "upsert" "page" "--page" "Home" "--properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) - conflict-result (run-cli ["--repo" repo + conflict-result (run-cli ["--graph" repo "upsert" "page" "--id" (str page-id) "--page" "Home"] data-dir cfg-path) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-page-result))) (is (= "ok" (:status create-page-payload))) @@ -661,16 +661,16 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "ref-rewrite-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "ref-rewrite-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) - add-block-result (run-cli ["--repo" "ref-rewrite-graph" + _ (run-cli ["graph" "create" "--graph" "ref-rewrite-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "ref-rewrite-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) + add-block-result (run-cli ["--graph" "ref-rewrite-graph" "upsert" "block" "--target-page" "Home" "--content" "See [[New Page]]"] data-dir cfg-path) add-block-payload (parse-json-output-safe add-block-result "add-block") _ (p/delay 100) - list-page-result (run-cli ["--repo" "ref-rewrite-graph" "list" "page"] data-dir cfg-path) + list-page-result (run-cli ["--graph" "ref-rewrite-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output-safe list-page-result "list-page") page-titles (->> (get-in list-page-payload [:data :items]) (map #(or (:block/title %) (:title %))) @@ -686,7 +686,7 @@ titles) ref-value (when ref-title (second (first (re-seq #"\[\[(.*?)\]\]" ref-title)))) - stop-result (run-cli ["server" "stop" "--repo" "ref-rewrite-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "ref-rewrite-graph"] data-dir cfg-path) stop-payload (parse-json-output-safe stop-result "server-stop")] (is (= 0 (:exit-code add-block-result))) (is (= "ok" (:status add-block-payload))) @@ -706,9 +706,9 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-uuid-ref")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "uuid-ref-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) - _ (run-cli ["--repo" "uuid-ref-graph" + _ (run-cli ["graph" "create" "--graph" "uuid-ref-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["--graph" "uuid-ref-graph" "upsert" "block" "--target-page" "Home" "--content" "Target block"] @@ -718,14 +718,14 @@ "[:find ?uuid :in $ ?title :where [?b :block/title ?title] [?b :block/uuid ?uuid]]" (pr-str ["Target block"])) target-uuid (first (first (get-in target-query-payload [:data :result]))) - add-block-result (run-cli ["--repo" "uuid-ref-graph" + add-block-result (run-cli ["--graph" "uuid-ref-graph" "upsert" "block" "--target-page" "Home" "--content" (str "See [[" target-uuid "]]")] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - list-page-result (run-cli ["--repo" "uuid-ref-graph" "list" "page"] data-dir cfg-path) + list-page-result (run-cli ["--graph" "uuid-ref-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) page-titles (->> (get-in list-page-payload [:data :items]) (map #(or (:block/title %) (:title %))) @@ -738,7 +738,7 @@ (string/includes? % (str "[[" target-uuid "]]"))) %) titles) - stop-result (run-cli ["server" "stop" "--repo" "uuid-ref-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "uuid-ref-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (string? target-uuid)) (is (= 0 (:exit-code add-block-result))) @@ -757,16 +757,16 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-missing-uuid-ref")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "missing-uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "missing-uuid-ref-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "missing-uuid-ref-graph" "upsert" "page" "--page" "Home"] data-dir cfg-path) missing-uuid (str (random-uuid)) - add-block-result (run-cli ["--repo" "missing-uuid-ref-graph" + add-block-result (run-cli ["--graph" "missing-uuid-ref-graph" "upsert" "block" "--target-page" "Home" "--content" (str "See [[" missing-uuid "]]")] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) - stop-result (run-cli ["server" "stop" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "missing-uuid-ref-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 1 (:exit-code add-block-result))) (is (= "error" (:status add-block-payload))) @@ -781,14 +781,14 @@ (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-tags")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) - add-page-result (run-cli ["--repo" "tags-graph" + add-page-result (run-cli ["--graph" "tags-graph" "upsert" "page" "--page" "TaggedPage" "--update-tags" "[\"Quote\"]" "--update-properties" "{:logseq.property/publishing-public? true}"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) - add-block-result (run-cli ["--repo" "tags-graph" + add-block-result (run-cli ["--graph" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block" @@ -796,7 +796,7 @@ "--update-properties" "{:logseq.property/deadline \"2026-01-25T12:00:00Z\"}"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) - add-block-ident-result (run-cli ["--repo" "tags-graph" + add-block-ident-result (run-cli ["--graph" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block ident" @@ -805,13 +805,13 @@ add-block-ident-payload (parse-json-output add-block-ident-result) deadline-prop-title (get-in db-property/built-in-properties [:logseq.property/deadline :title]) publishing-prop-title (get-in db-property/built-in-properties [:logseq.property/publishing-public? :title]) - add-page-title-result (run-cli ["--repo" "tags-graph" + add-page-title-result (run-cli ["--graph" "tags-graph" "upsert" "page" "--page" "TaggedPageTitle" "--update-properties" (str "{\"" publishing-prop-title "\" true}")] data-dir cfg-path) add-page-title-payload (parse-json-output add-page-title-result) - add-block-title-result (run-cli ["--repo" "tags-graph" + add-block-title-result (run-cli ["--graph" "tags-graph" "upsert" "block" "--target-page" "Home" "--content" "Tagged block title" @@ -863,14 +863,14 @@ publishing-title (get-in db-property/built-in-properties [:logseq.property/publishing-public? :title]) deadline-id (find-item-id (get-in list-property-payload [:data :items]) deadline-title) publishing-id (find-item-id (get-in list-property-payload [:data :items]) publishing-title) - add-page-id-result (run-cli ["--repo" repo + add-page-id-result (run-cli ["--graph" repo "upsert" "page" "--page" "TaggedPageId" "--update-tags" (pr-str [quote-tag-id]) "--update-properties" (pr-str {publishing-id true})] data-dir cfg-path) add-page-id-payload (parse-json-output add-page-id-result) - add-block-id-result (run-cli ["--repo" repo + add-block-id-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--content" "Tagged block id" @@ -908,7 +908,7 @@ repo-id (command-core/resolve-repo repo)] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) cfg (cli-config/resolve-config {:data-dir data-dir :config-path cfg-path :output-format :json}) @@ -965,13 +965,13 @@ repo "verbose-graph"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) {:keys [buffer restore!]} (capture-stderr!) - result (-> (run-cli ["--verbose" "--repo" repo "graph" "info"] data-dir cfg-path) + result (-> (run-cli ["--verbose" "--graph" repo "graph" "info"] data-dir cfg-path) (p/finally (fn [] (restore!)))) payload (parse-json-output-safe result "verbose graph info") stderr-text @buffer - _ (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path)] + _ (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path)] (is (= 0 (:exit-code result))) (is (= "ok" (:status payload))) (is (string/includes? stderr-text ":cli.transport/invoke")) @@ -986,7 +986,7 @@ (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) tag-a-name "Quote" tag-b-name "Math" - add-block-result (run-cli ["--repo" repo + add-block-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--content" "Update block" @@ -995,11 +995,11 @@ data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - show-home (run-cli ["--repo" repo "show" "--page" "Home"] data-dir cfg-path) + show-home (run-cli ["--graph" repo "show" "--page" "Home"] data-dir cfg-path) show-home-payload (parse-json-output show-home) block-node (find-block-by-title (get-in show-home-payload [:data :root]) "Update block") block-id (node-id block-node) - update-result (run-cli ["--repo" repo + update-result (run-cli ["--graph" repo "upsert" "block" "--id" (str block-id) "--update-tags" "[:logseq.class/Math-block]" @@ -1026,24 +1026,24 @@ (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-block-custom-property")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) - upsert-property-result (run-cli ["--repo" repo + upsert-property-result (run-cli ["--graph" repo "upsert" "property" "--name" "owner" "--type" "default"] data-dir cfg-path) upsert-property-payload (parse-json-output upsert-property-result) - add-block-result (run-cli ["--repo" repo + add-block-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--content" "Block with custom property"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) _ (p/delay 100) - show-home (run-cli ["--repo" repo "show" "--page" "Home"] data-dir cfg-path) + show-home (run-cli ["--graph" repo "show" "--page" "Home"] data-dir cfg-path) show-home-payload (parse-json-output show-home) block-node (find-block-by-title (get-in show-home-payload [:data :root]) "Block with custom property") block-id (node-id block-node) - update-result (run-cli ["--repo" repo + update-result (run-cli ["--graph" repo "upsert" "block" "--id" (str block-id) "--update-properties" "{:user.property/owner \"alice\"}"] @@ -1052,7 +1052,7 @@ _ (p/delay 100) property-after-update (query-property data-dir cfg-path repo "Block with custom property" ":user.property/owner") - remove-result (run-cli ["--repo" repo + remove-result (run-cli ["--graph" repo "upsert" "block" "--id" (str block-id) "--remove-properties" "[:user.property/owner]"] @@ -1084,20 +1084,20 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-tags-missing")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "tags-missing-graph"] data-dir cfg-path) - add-block-result (run-cli ["--repo" "tags-missing-graph" + _ (run-cli ["graph" "create" "--graph" "tags-missing-graph"] data-dir cfg-path) + add-block-result (run-cli ["--graph" "tags-missing-graph" "upsert" "block" "--target-page" "Home" "--content" "Block with missing tag" "--update-tags" "[\"MissingTag\"]"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) - list-tag-result (run-cli ["--repo" "tags-missing-graph" "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" "tags-missing-graph" "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) (map #(or (:block/title %) (:block/name %))) set) - stop-result (run-cli ["server" "stop" "--repo" "tags-missing-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "tags-missing-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "error" (:status add-block-payload))) (is (= :tag-not-found (keyword (get-in add-block-payload [:error :code])))) @@ -1114,19 +1114,19 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") repo "upsert-tag-create-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) - upsert-tag-result (run-cli ["--repo" repo + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "Home"] data-dir cfg-path) + upsert-tag-result (run-cli ["--graph" repo "upsert" "tag" "--name" "CliQuote"] data-dir cfg-path) upsert-tag-payload (parse-json-output upsert-tag-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) (map #(or (:block/title %) (:title %) (:name %))) set) - add-block-result (run-cli ["--repo" repo + add-block-result (run-cli ["--graph" repo "upsert" "block" "--target-page" "Home" "--content" "Tagged by upsert tag" @@ -1135,7 +1135,7 @@ add-block-payload (parse-json-output add-block-result) _ (p/delay 100) block-tag-names (query-tags data-dir cfg-path repo "Tagged by upsert tag") - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code upsert-tag-result)) (pr-str (:error upsert-tag-payload))) @@ -1158,19 +1158,19 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") repo "upsert-tag-conflict-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - _ (run-cli ["--repo" repo "upsert" "page" "--page" "ConflictPage"] data-dir cfg-path) - upsert-tag-result (run-cli ["--repo" repo + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "ConflictPage"] data-dir cfg-path) + upsert-tag-result (run-cli ["--graph" repo "upsert" "tag" "--name" "ConflictPage"] data-dir cfg-path) upsert-tag-payload (parse-json-output upsert-tag-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) (map #(or (:block/title %) (:title %) (:name %))) set) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code upsert-tag-result))) (is (= "error" (:status upsert-tag-payload))) @@ -1189,18 +1189,18 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") repo "upsert-tag-idempotent-graph" _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - first-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + first-upsert-result (run-cli ["--graph" repo "upsert" "tag" "--name" "StableTag"] data-dir cfg-path) first-upsert-payload (parse-json-output first-upsert-result) - second-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" "StableTag"] + second-upsert-result (run-cli ["--graph" repo "upsert" "tag" "--name" "StableTag"] data-dir cfg-path) second-upsert-payload (parse-json-output second-upsert-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) stable-tags (->> (get-in list-tag-payload [:data :items]) (filter #(= "StableTag" (or (:block/title %) (:title %) (:name %))))) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code first-upsert-result))) (is (= "ok" (:status first-upsert-payload))) @@ -1221,18 +1221,18 @@ source-name "CliRenameSource" target-name "CliRenameTarget" _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - create-result (run-cli ["--repo" repo "upsert" "tag" "--name" source-name] + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + create-result (run-cli ["--graph" repo "upsert" "tag" "--name" source-name] data-dir cfg-path) create-payload (parse-json-output create-result) source-id (first-result-id create-payload) - rename-result (run-cli ["--repo" repo + rename-result (run-cli ["--graph" repo "upsert" "tag" "--id" (str source-id) "--name" target-name] data-dir cfg-path) rename-payload (parse-json-output rename-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tags (get-in list-tag-payload [:data :items]) tag-names (->> tags @@ -1240,7 +1240,7 @@ set) target-id (find-item-id tags target-name) source-id-after (find-item-id tags source-name) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code create-result))) (is (= "ok" (:status create-payload))) @@ -1268,26 +1268,26 @@ source-name "CliRenameConflictSource" existing-name "CliRenameConflictExisting" _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - source-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" source-name] + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + source-upsert-result (run-cli ["--graph" repo "upsert" "tag" "--name" source-name] data-dir cfg-path) source-upsert-payload (parse-json-output source-upsert-result) source-id (first-result-id source-upsert-payload) - existing-upsert-result (run-cli ["--repo" repo "upsert" "tag" "--name" existing-name] + existing-upsert-result (run-cli ["--graph" repo "upsert" "tag" "--name" existing-name] data-dir cfg-path) existing-upsert-payload (parse-json-output existing-upsert-result) - rename-result (run-cli ["--repo" repo + rename-result (run-cli ["--graph" repo "upsert" "tag" "--id" (str source-id) "--name" existing-name] data-dir cfg-path) rename-payload (parse-json-output rename-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) tag-names (->> (get-in list-tag-payload [:data :items]) (map #(or (:block/title %) (:title %) (:name %))) set) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code source-upsert-result))) (is (= "ok" (:status source-upsert-payload))) @@ -1314,17 +1314,17 @@ property-name "CliOwnerPropX" property-name-lc (common-util/page-name-sanity-lc property-name) _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" repo] data-dir cfg-path) - upsert-tag-result (run-cli ["--repo" repo "upsert" "tag" "--name" tag-name] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + upsert-tag-result (run-cli ["--graph" repo "upsert" "tag" "--name" tag-name] data-dir cfg-path) upsert-tag-payload (parse-json-output upsert-tag-result) - upsert-property-result (run-cli ["--repo" repo + upsert-property-result (run-cli ["--graph" repo "upsert" "property" "--name" property-name "--type" "node" "--cardinality" "many"] data-dir cfg-path) upsert-property-payload (parse-json-output upsert-property-result) - update-property-result (run-cli ["--repo" repo + update-property-result (run-cli ["--graph" repo "upsert" "property" "--name" property-name "--type" "node" @@ -1334,13 +1334,13 @@ property-schema-before-remove (run-query data-dir cfg-path repo "[:find ?type ?cardinality :in $ ?name :where [?p :block/name ?name] [?p :logseq.property/type ?type] [?p :db/cardinality ?cardinality]]" (pr-str [property-name-lc])) - remove-tag-result (run-cli ["--repo" repo "remove" "tag" "--name" tag-name] data-dir cfg-path) + remove-tag-result (run-cli ["--graph" repo "remove" "tag" "--name" tag-name] data-dir cfg-path) remove-tag-payload (parse-json-output remove-tag-result) - remove-property-result (run-cli ["--repo" repo "remove" "property" "--name" property-name] data-dir cfg-path) + remove-property-result (run-cli ["--graph" repo "remove" "property" "--name" property-name] data-dir cfg-path) remove-property-payload (parse-json-output remove-property-result) - list-tag-result (run-cli ["--repo" repo "list" "tag"] data-dir cfg-path) + list-tag-result (run-cli ["--graph" repo "list" "tag"] data-dir cfg-path) list-tag-payload (parse-json-output list-tag-result) - list-property-result (run-cli ["--repo" repo "list" "property"] data-dir cfg-path) + list-property-result (run-cli ["--graph" repo "list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) tag-names (->> (get-in list-tag-payload [:data :items]) (map #(or (:block/title %) (:title %) (:name %))) @@ -1348,7 +1348,7 @@ property-names (->> (get-in list-property-payload [:data :items]) (map #(or (:block/title %) (:title %) (:name %))) set) - stop-result (run-cli ["server" "stop" "--repo" repo] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code upsert-tag-result))) (is (= "ok" (:status upsert-tag-payload))) @@ -1378,26 +1378,26 @@ query-text "[:find ?e :in $ ?title :where [?e :block/title ?title]]"] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "query-graph"] data-dir cfg-path) + create-result (run-cli ["graph" "create" "--graph" "query-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) - _ (run-cli ["--repo" "query-graph" "upsert" "page" "--page" "QueryPage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "upsert" "block" + _ (run-cli ["--graph" "query-graph" "upsert" "page" "--page" "QueryPage"] data-dir cfg-path) + _ (run-cli ["--graph" "query-graph" "upsert" "block" "--target-page" "QueryPage" "--content" "Query block"] data-dir cfg-path) - _ (run-cli ["--repo" "query-graph" "upsert" "block" + _ (run-cli ["--graph" "query-graph" "upsert" "block" "--target-page" "QueryPage" "--content" "Query block"] data-dir cfg-path) _ (p/delay 100) - query-result (run-cli ["--repo" "query-graph" + query-result (run-cli ["--graph" "query-graph" "query" "--query" query-text "--inputs" "[\"Query block\"]"] data-dir cfg-path) query-payload (parse-json-output query-result) result (get-in query-payload [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" "query-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "query-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status create-payload))) (is (= 0 (:exit-code query-result))) @@ -1415,22 +1415,22 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-task-query")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "task-query-graph"] data-dir cfg-path) + create-result (run-cli ["graph" "create" "--graph" "task-query-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) - _ (run-cli ["--repo" "task-query-graph" "upsert" "page" "--page" "Tasks"] data-dir cfg-path) - _ (run-cli ["--repo" "task-query-graph" + _ (run-cli ["--graph" "task-query-graph" "upsert" "page" "--page" "Tasks"] data-dir cfg-path) + _ (run-cli ["--graph" "task-query-graph" "upsert" "block" "--target-page" "Tasks" "--content" "Task one" "--status" "doing"] data-dir cfg-path) - _ (run-cli ["--repo" "task-query-graph" + _ (run-cli ["--graph" "task-query-graph" "upsert" "block" "--target-page" "Tasks" "--content" "Task two" "--status" "doing"] data-dir cfg-path) - _ (run-cli ["--repo" "task-query-graph" + _ (run-cli ["--graph" "task-query-graph" "upsert" "block" "--target-page" "Tasks" "--content" "Task three" @@ -1442,13 +1442,13 @@ task-entry (some (fn [entry] (when (= "task-search" (:name entry)) entry)) (get-in list-payload [:data :queries])) - query-result (run-cli ["--repo" "task-query-graph" + query-result (run-cli ["--graph" "task-query-graph" "query" "--name" "task-search" "--inputs" "[\"doing\"]"] data-dir cfg-path) query-payload (parse-json-output query-result) - query-nil-result (run-cli ["--repo" "task-query-graph" + query-nil-result (run-cli ["--graph" "task-query-graph" "query" "--name" "task-search" "--inputs" "[\"doing\" nil 1]"] @@ -1456,7 +1456,7 @@ query-nil-payload (parse-json-output query-nil-result) result (get-in query-payload [:data :result]) nil-result (get-in query-nil-payload [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" "task-query-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "task-query-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status create-payload))) (is (= "ok" (:status list-payload))) @@ -1483,25 +1483,25 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-status-query")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - create-result (run-cli ["graph" "create" "--repo" "status-query-graph"] data-dir cfg-path) + create-result (run-cli ["graph" "create" "--graph" "status-query-graph"] data-dir cfg-path) create-payload (parse-json-output create-result) _ (p/delay 100) list-result (run-cli ["query" "list"] data-dir cfg-path) list-payload (parse-json-output list-result) names (set (map :name (get-in list-payload [:data :queries]))) - status-result (run-cli ["--repo" "status-query-graph" + status-result (run-cli ["--graph" "status-query-graph" "query" "--name" "list-status"] data-dir cfg-path) status-payload (parse-json-output status-result) status-values (get-in status-payload [:data :result]) - priority-result (run-cli ["--repo" "status-query-graph" + priority-result (run-cli ["--graph" "status-query-graph" "query" "--name" "list-priority"] data-dir cfg-path) priority-payload (parse-json-output priority-result) priority-values (get-in priority-payload [:data :result]) - stop-result (run-cli ["server" "stop" "--repo" "status-query-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "status-query-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status create-payload))) (is (= "ok" (:status list-payload))) @@ -1536,21 +1536,21 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-recent-updated")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "recent-updated-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "recent-updated-graph" "upsert" "page" "--page" "RecentPage"] data-dir cfg-path) - _ (run-cli ["--repo" "recent-updated-graph" "upsert" "block" + _ (run-cli ["graph" "create" "--graph" "recent-updated-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "recent-updated-graph" "upsert" "page" "--page" "RecentPage"] data-dir cfg-path) + _ (run-cli ["--graph" "recent-updated-graph" "upsert" "block" "--target-page" "RecentPage" "--content" "Recent block"] data-dir cfg-path) _ (p/delay 100) - list-page-result (run-cli ["--repo" "recent-updated-graph" "list" "page" "--expand"] data-dir cfg-path) + list-page-result (run-cli ["--graph" "recent-updated-graph" "list" "page" "--expand"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) page-item (some (fn [item] (when (= "RecentPage" (or (:block/title item) (:title item))) item)) (get-in list-page-payload [:data :items])) page-id (or (:db/id page-item) (:id page-item)) - show-result (run-cli ["--repo" "recent-updated-graph" + show-result (run-cli ["--graph" "recent-updated-graph" "show" "--page" "RecentPage"] data-dir cfg-path) @@ -1564,7 +1564,7 @@ (when (= "recent-updated" (:name entry)) entry)) (get-in list-payload [:data :queries])) now-ms (js/Date.now) - query-result (run-cli ["--repo" "recent-updated-graph" + query-result (run-cli ["--graph" "recent-updated-graph" "query" "--name" "recent-updated" "--inputs" (pr-str [1 now-ms])] @@ -1572,32 +1572,32 @@ query-payload (parse-json-output query-result) result (get-in query-payload [:data :result]) future-now-ms (+ now-ms (* 10 86400000)) - future-query-result (run-cli ["--repo" "recent-updated-graph" + future-query-result (run-cli ["--graph" "recent-updated-graph" "query" "--name" "recent-updated" "--inputs" (pr-str [1 future-now-ms])] data-dir cfg-path) future-query-payload (parse-json-output future-query-result) future-result (get-in future-query-payload [:data :result]) - zero-result (run-cli ["--repo" "recent-updated-graph" + zero-result (run-cli ["--graph" "recent-updated-graph" "query" "--name" "recent-updated" "--inputs" "[0]"] data-dir cfg-path) zero-payload (parse-json-output zero-result) - nil-result (run-cli ["--repo" "recent-updated-graph" + nil-result (run-cli ["--graph" "recent-updated-graph" "query" "--name" "recent-updated" "--inputs" "[nil]"] data-dir cfg-path) nil-payload (parse-json-output nil-result) - neg-result (run-cli ["--repo" "recent-updated-graph" + neg-result (run-cli ["--graph" "recent-updated-graph" "query" "--name" "recent-updated" "--inputs" "[-1]"] data-dir cfg-path) neg-payload (parse-json-output neg-result) - stop-result (run-cli ["server" "stop" "--repo" "recent-updated-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "recent-updated-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status list-page-payload))) (is (some? page-id)) @@ -1633,26 +1633,26 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-nested-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "nested-refs"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "upsert" "page" "--page" "NestedPage"] data-dir cfg-path) - _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" "Inner"] data-dir cfg-path) - show-nested (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "nested-refs"] data-dir cfg-path) + _ (run-cli ["--graph" "nested-refs" "upsert" "page" "--page" "NestedPage"] data-dir cfg-path) + _ (run-cli ["--graph" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" "Inner"] data-dir cfg-path) + show-nested (run-cli ["--graph" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-nested-payload (parse-json-output show-nested) _inner-node (find-block-by-title (get-in show-nested-payload [:data :root]) "Inner") inner-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" "Inner") middle-content (str "See [[" inner-uuid "]]") - _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" + _ (run-cli ["--graph" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" middle-content] data-dir cfg-path) - show-middle (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) + show-middle (run-cli ["--graph" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-middle-payload (parse-json-output show-middle) _middle-node (find-block-by-title (get-in show-middle-payload [:data :root]) middle-content) middle-uuid (query-block-uuid-by-title data-dir cfg-path "nested-refs" middle-content) - _ (run-cli ["--repo" "nested-refs" "upsert" "block" "--target-page" "NestedPage" + _ (run-cli ["--graph" "nested-refs" "upsert" "block" "--target-page" "NestedPage" "--content" (str "Outer [[" middle-uuid "]]")] data-dir cfg-path) - show-outer (run-cli ["--repo" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) + show-outer (run-cli ["--graph" "nested-refs" "show" "--page" "NestedPage"] data-dir cfg-path) show-outer-payload (parse-json-output show-outer) outer-node (find-block-by-title (get-in show-outer-payload [:data :root]) "Outer [[See [[Inner]]]]") - stop-result (run-cli ["server" "stop" "--repo" "nested-refs"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "nested-refs"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (some? inner-uuid)) (is (some? middle-uuid)) @@ -1669,22 +1669,22 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) - target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) + target-show (run-cli ["--graph" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) _target-show-payload (parse-json-output target-show) target-uuid (query-block-uuid-by-title data-dir cfg-path "linked-refs-graph" "TargetPage") target-title "TargetPage" ref-content (str "See [[" target-uuid "]]") ref-title (str "See [[" target-title "]]") - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" "--content" ref-content] data-dir cfg-path) + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" "--content" ref-content] data-dir cfg-path) _ (p/delay 100) - source-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "SourcePage"] data-dir cfg-path) + source-show (run-cli ["--graph" "linked-refs-graph" "show" "--page" "SourcePage"] data-dir cfg-path) source-payload (parse-json-output source-show) ref-node (find-block-by-title (get-in source-payload [:data :root]) ref-title) ref-id (or (:db/id ref-node) (:id ref-node)) - target-show (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) + target-show (run-cli ["--graph" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) target-payload (parse-json-output target-show) linked-refs (get-in target-payload [:data :linked-references]) linked-blocks (:blocks linked-refs) @@ -1695,7 +1695,7 @@ (get-in block [:page :title]) (get-in block [:page :name]))) linked-blocks)) - stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "linked-refs-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (some? target-uuid)) (is (= "ok" (:status target-payload))) @@ -1713,23 +1713,23 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-move")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "move-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "move-graph" "upsert" "block" "--target-page" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "move-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "move-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) + _ (run-cli ["--graph" "move-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--graph" "move-graph" "upsert" "block" "--target-page" "SourcePage" "--content" "Parent Block"] data-dir cfg-path) _ (p/delay 100) - source-show (run-cli ["--repo" "move-graph" "show" "--page" "SourcePage"] data-dir cfg-path) + source-show (run-cli ["--graph" "move-graph" "show" "--page" "SourcePage"] data-dir cfg-path) source-payload (parse-json-output source-show) parent-node (find-block-by-title (get-in source-payload [:data :root]) "Parent Block") parent-id (node-id parent-node) - _ (run-cli ["--repo" "move-graph" "upsert" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) - update-result (run-cli ["--repo" "move-graph" "upsert" "block" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--graph" "move-graph" "upsert" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) + update-result (run-cli ["--graph" "move-graph" "upsert" "block" "--id" (str parent-id) "--target-page" "TargetPage"] data-dir cfg-path) update-payload (parse-json-output update-result) - target-show (run-cli ["--repo" "move-graph" "show" "--page" "TargetPage"] data-dir cfg-path) + target-show (run-cli ["--graph" "move-graph" "show" "--page" "TargetPage"] data-dir cfg-path) target-payload (parse-json-output target-show) moved-node (find-block-by-title (get-in target-payload [:data :root]) "Parent Block") child-node (find-block-by-title moved-node "Child Block") - stop-result (run-cli ["server" "stop" "--repo" "move-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "move-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status update-payload))) (is (some? parent-id)) @@ -1746,21 +1746,21 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "add-pos-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "upsert" "page" "--page" "PosPage"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-page" "PosPage" "--content" "Parent"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "add-pos-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "add-pos-graph" "upsert" "page" "--page" "PosPage"] data-dir cfg-path) + _ (run-cli ["--graph" "add-pos-graph" "upsert" "block" "--target-page" "PosPage" "--content" "Parent"] data-dir cfg-path) _ (p/delay 100) - parent-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) + parent-show (run-cli ["--graph" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) parent-payload (parse-json-output parent-show) parent-node (find-block-by-title (get-in parent-payload [:data :root]) "Parent") parent-id (node-id parent-node) - _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) - _ (run-cli ["--repo" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) - final-show (run-cli ["--repo" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) + _ (run-cli ["--graph" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "first-child" "--content" "First"] data-dir cfg-path) + _ (run-cli ["--graph" "add-pos-graph" "upsert" "block" "--target-id" (str parent-id) "--pos" "last-child" "--content" "Last"] data-dir cfg-path) + final-show (run-cli ["--graph" "add-pos-graph" "show" "--page" "PosPage"] data-dir cfg-path) final-payload (parse-json-output final-show) final-parent (find-block-by-title (get-in final-payload [:data :root]) "Parent") child-titles (map node-title (node-children final-parent)) - stop-result (run-cli ["server" "stop" "--repo" "add-pos-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "add-pos-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (some? parent-id)) (is (= ["First" "Last"] (vec child-titles))) @@ -1794,7 +1794,7 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "list-id-graph"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "list-id-graph"] data-dir cfg-path) _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) @@ -1802,7 +1802,7 @@ list-tag-payload (parse-json-output list-tag-result) list-property-result (run-cli ["list" "property"] data-dir cfg-path) list-property-payload (parse-json-output list-property-result) - stop-result (run-cli ["server" "stop" "--repo" "list-id-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "list-id-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status list-page-payload))) (is (every? #(contains? % :id) (get-in list-page-payload [:data :items]))) @@ -1821,7 +1821,7 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "human-list-graph"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "human-list-graph"] data-dir cfg-path) _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page" "--output" "human"] data-dir cfg-path) output (:output list-page-result)] @@ -1839,7 +1839,7 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "show-page-block-graph"] data-dir cfg-path) + _ (run-cli ["graph" "create" "--graph" "show-page-block-graph"] data-dir cfg-path) _ (run-cli ["upsert" "page" "--page" "TestPage"] data-dir cfg-path) list-page-result (run-cli ["list" "page" "--expand"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) @@ -1853,7 +1853,7 @@ show-by-id-payload (parse-json-output show-by-id-result) show-by-uuid-result (run-cli ["show" "--uuid" (str page-uuid)] data-dir cfg-path) show-by-uuid-payload (parse-json-output show-by-uuid-result) - stop-result (run-cli ["server" "stop" "--repo" "show-page-block-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "show-page-block-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status list-page-payload))) (is (some? page-item)) @@ -1878,33 +1878,33 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "show-multi-id-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "page" "--page" "MultiPage"] + _ (run-cli ["graph" "create" "--graph" "show-multi-id-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "show-multi-id-graph" "upsert" "page" "--page" "MultiPage"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "block" + _ (run-cli ["--graph" "show-multi-id-graph" "upsert" "block" "--target-page" "MultiPage" "--content" "Multi show one"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-graph" "upsert" "block" + _ (run-cli ["--graph" "show-multi-id-graph" "upsert" "block" "--target-page" "MultiPage" "--content" "Multi show two"] data-dir cfg-path) _ (p/delay 100) query-text "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" - query-one-result (run-cli ["--repo" "show-multi-id-graph" "query" + query-one-result (run-cli ["--graph" "show-multi-id-graph" "query" "--query" query-text "--inputs" (pr-str ["Multi show one"])] data-dir cfg-path) query-one-payload (parse-json-output query-one-result) block-one-id (get-in query-one-payload [:data :result]) - query-two-result (run-cli ["--repo" "show-multi-id-graph" "query" + query-two-result (run-cli ["--graph" "show-multi-id-graph" "query" "--query" query-text "--inputs" (pr-str ["Multi show two"])] data-dir cfg-path) query-two-payload (parse-json-output query-two-result) block-two-id (get-in query-two-payload [:data :result]) ids-edn (str "[" block-one-id " " block-two-id "]") - show-text-result (run-cli ["--repo" "show-multi-id-graph" "show" + show-text-result (run-cli ["--graph" "show-multi-id-graph" "show" "--id" ids-edn "--output" "human"] @@ -1913,13 +1913,13 @@ idx-one (string/index-of output "Multi show one") idx-two (string/index-of output "Multi show two") idx-delim (string/index-of output "================================================================") - show-json-result (run-cli ["--repo" "show-multi-id-graph" "show" + show-json-result (run-cli ["--graph" "show-multi-id-graph" "show" "--id" ids-edn] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) root-titles (set (map (comp node-title :root) show-data)) - stop-result (run-cli ["server" "stop" "--repo" "show-multi-id-graph"] + stop-result (run-cli ["server" "stop" "--graph" "show-multi-id-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code query-one-result))) @@ -1950,25 +1950,25 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-multi-id-contained")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "show-multi-id-contained-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "page" "--page" "ParentPage"] + _ (run-cli ["graph" "create" "--graph" "show-multi-id-contained-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "show-multi-id-contained-graph" "upsert" "page" "--page" "ParentPage"] data-dir cfg-path) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "block" + _ (run-cli ["--graph" "show-multi-id-contained-graph" "upsert" "block" "--target-page" "ParentPage" "--content" "Parent Block"] data-dir cfg-path) - parent-query (run-cli ["--repo" "show-multi-id-contained-graph" "query" + parent-query (run-cli ["--graph" "show-multi-id-contained-graph" "query" "--query" "[:find ?e . :in $ ?title :where [?e :block/title ?title]]" "--inputs" (pr-str ["Parent Block"])] data-dir cfg-path) parent-payload (parse-json-output parent-query) parent-id (get-in parent-payload [:data :result]) - _ (run-cli ["--repo" "show-multi-id-contained-graph" "upsert" "block" + _ (run-cli ["--graph" "show-multi-id-contained-graph" "upsert" "block" "--target-id" (str parent-id) "--content" "Child Block"] data-dir cfg-path) _ (p/delay 100) - show-children (run-cli ["--repo" "show-multi-id-contained-graph" + show-children (run-cli ["--graph" "show-multi-id-contained-graph" "show" "--page" "ParentPage"] data-dir cfg-path) @@ -1976,13 +1976,13 @@ child-node (find-block-by-title (get-in show-children-payload [:data :root]) "Child Block") child-id (or (:db/id child-node) (:id child-node)) ids-edn (str "[" parent-id " " child-id "]") - show-json-result (run-cli ["--repo" "show-multi-id-contained-graph" "show" + show-json-result (run-cli ["--graph" "show-multi-id-contained-graph" "show" "--id" ids-edn] data-dir cfg-path) show-json-payload (parse-json-output show-json-result) show-data (:data show-json-payload) root-titles (set (map (comp node-title :root) show-data)) - stop-result (run-cli ["server" "stop" "--repo" "show-multi-id-contained-graph"] + stop-result (run-cli ["server" "stop" "--graph" "show-multi-id-contained-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code parent-query))) @@ -2004,14 +2004,14 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-query-pipe")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "query-pipe-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "upsert" "page" "--page" "PipePage"] + _ (run-cli ["graph" "create" "--graph" "query-pipe-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "query-pipe-graph" "upsert" "page" "--page" "PipePage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "upsert" "block" + _ (run-cli ["--graph" "query-pipe-graph" "upsert" "block" "--target-page" "PipePage" "--content" "Pipe One"] data-dir cfg-path) - _ (run-cli ["--repo" "query-pipe-graph" "upsert" "block" + _ (run-cli ["--graph" "query-pipe-graph" "upsert" "block" "--target-page" "PipePage" "--content" "Pipe Two"] data-dir cfg-path) @@ -2021,14 +2021,14 @@ " :where" " [?e :block/title ?title]" " [(clojure.string/includes? ?title ?q)]]") - query-json-result (run-cli ["--repo" "query-pipe-graph" + query-json-result (run-cli ["--graph" "query-pipe-graph" "query" "--query" query-text "--inputs" (pr-str ["Pipe"])] data-dir cfg-path) query-json-payload (parse-json-output query-json-result) query-ids (get-in query-json-payload [:data :result]) - query-human-result (run-cli ["--repo" "query-pipe-graph" + query-human-result (run-cli ["--graph" "query-pipe-graph" "--output" "human" "query" "--query" query-text @@ -2046,7 +2046,7 @@ [node-bin cli-bin "--data-dir" data-arg "--config" cfg-arg - "--repo" repo-arg + "--graph" repo-arg "--output" "human" "query" "--query" query-arg @@ -2055,13 +2055,13 @@ [node-bin cli-bin "--data-dir" data-arg "--config" cfg-arg - "--repo" repo-arg + "--graph" repo-arg "--output" "human" "show" "--id"]) pipeline (str query-cmd " | xargs -I{} " show-cmd " {}") output (run-shell pipeline) - stop-result (run-cli ["server" "stop" "--repo" "query-pipe-graph"] + stop-result (run-cli ["server" "stop" "--graph" "query-pipe-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= (pr-str query-ids) query-human-output)) @@ -2078,14 +2078,14 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-query-stdin")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "query-stdin-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "upsert" "page" "--page" "PipePage"] + _ (run-cli ["graph" "create" "--graph" "query-stdin-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "query-stdin-graph" "upsert" "page" "--page" "PipePage"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "upsert" "block" + _ (run-cli ["--graph" "query-stdin-graph" "upsert" "block" "--target-page" "PipePage" "--content" "Pipe One"] data-dir cfg-path) - _ (run-cli ["--repo" "query-stdin-graph" "upsert" "block" + _ (run-cli ["--graph" "query-stdin-graph" "upsert" "block" "--target-page" "PipePage" "--content" "Pipe Two"] data-dir cfg-path) @@ -2095,14 +2095,14 @@ " :where" " [?e :block/title ?title]" " [(clojure.string/includes? ?title ?q)]]") - query-json-result (run-cli ["--repo" "query-stdin-graph" + query-json-result (run-cli ["--graph" "query-stdin-graph" "query" "--query" query-text "--inputs" (pr-str ["Pipe"])] data-dir cfg-path) query-json-payload (parse-json-output query-json-result) query-ids (get-in query-json-payload [:data :result]) - query-result (run-cli ["--repo" "query-stdin-graph" + query-result (run-cli ["--graph" "query-stdin-graph" "--output" "human" "query" "--query" query-text @@ -2110,7 +2110,7 @@ data-dir cfg-path) ids-text (string/trim (:output query-result)) show-result (with-redefs [show-command/read-stdin (fn [] ids-text)] - (run-cli ["--repo" "query-stdin-graph" + (run-cli ["--graph" "query-stdin-graph" "--output" "json" "show" "--id"] @@ -2121,7 +2121,7 @@ root-titles (set (map (comp node-title :root) show-data)) pipe-one (find-block-by-title root "Pipe One") pipe-two (find-block-by-title root "Pipe Two") - stop-result (run-cli ["server" "stop" "--repo" "query-stdin-graph"] + stop-result (run-cli ["server" "stop" "--graph" "query-stdin-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= (pr-str query-ids) ids-text)) @@ -2140,10 +2140,10 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-linked-refs")] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") - _ (run-cli ["graph" "create" "--repo" "linked-refs-graph"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) - list-page-result (run-cli ["--repo" "linked-refs-graph" "list" "page" "--expand"] + _ (run-cli ["graph" "create" "--graph" "linked-refs-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "page" "--page" "TargetPage"] data-dir cfg-path) + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "page" "--page" "SourcePage"] data-dir cfg-path) + list-page-result (run-cli ["--graph" "linked-refs-graph" "list" "page" "--expand"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) page-item (some (fn [item] @@ -2152,14 +2152,14 @@ (get-in list-page-payload [:data :items])) page-id (or (:db/id page-item) (:id page-item)) blocks-edn (str "[{:block/title \"Ref to TargetPage\" :block/refs [{:db/id " page-id "}]}]") - _ (run-cli ["--repo" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" + _ (run-cli ["--graph" "linked-refs-graph" "upsert" "block" "--target-page" "SourcePage" "--blocks" blocks-edn] data-dir cfg-path) - show-result (run-cli ["--repo" "linked-refs-graph" "show" "--page" "TargetPage"] + show-result (run-cli ["--graph" "linked-refs-graph" "show" "--page" "TargetPage"] data-dir cfg-path) show-payload (parse-json-output show-result) linked (get-in show-payload [:data :linked-references]) ref-block (first (:blocks linked)) - stop-result (run-cli ["server" "stop" "--repo" "linked-refs-graph"] data-dir cfg-path) + stop-result (run-cli ["server" "stop" "--graph" "linked-refs-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= "ok" (:status show-payload))) (is (some? page-id)) @@ -2184,22 +2184,22 @@ export-graph "export-edn-graph" import-graph "import-edn-graph" export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.edn") - _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "upsert" "page" "--page" "ExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "upsert" "block" "--target-page" "ExportPage" "--content" "Export content"] data-dir cfg-path) - export-result (run-cli ["--repo" export-graph + _ (run-cli ["graph" "create" "--graph" export-graph] data-dir cfg-path) + _ (run-cli ["--graph" export-graph "upsert" "page" "--page" "ExportPage"] data-dir cfg-path) + _ (run-cli ["--graph" export-graph "upsert" "block" "--target-page" "ExportPage" "--content" "Export content"] data-dir cfg-path) + export-result (run-cli ["--graph" export-graph "graph" "export" "--type" "edn" "--file" export-path] data-dir cfg-path) export-payload (parse-json-output export-result) - _ (run-cli ["--repo" import-graph + _ (run-cli ["--graph" import-graph "graph" "import" "--type" "edn" "--input" export-path] data-dir cfg-path) - list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-result (run-cli ["--graph" import-graph "list" "page"] data-dir cfg-path) list-payload (parse-json-output list-result) - stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) - stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + stop-export (run-cli ["server" "stop" "--graph" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--graph" import-graph] data-dir cfg-path)] (is (= 0 (:exit-code export-result))) (is (= "ok" (:status export-payload))) (is (fs/existsSync export-path)) @@ -2223,22 +2223,22 @@ export-graph "export-sqlite-graph" import-graph "import-sqlite-graph" export-path (node-path/join (node-helper/create-tmp-dir "exports") "graph.sqlite") - _ (run-cli ["graph" "create" "--repo" export-graph] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "upsert" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) - _ (run-cli ["--repo" export-graph "upsert" "block" "--target-page" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) - export-result (run-cli ["--repo" export-graph + _ (run-cli ["graph" "create" "--graph" export-graph] data-dir cfg-path) + _ (run-cli ["--graph" export-graph "upsert" "page" "--page" "SQLiteExportPage"] data-dir cfg-path) + _ (run-cli ["--graph" export-graph "upsert" "block" "--target-page" "SQLiteExportPage" "--content" "SQLite export content"] data-dir cfg-path) + export-result (run-cli ["--graph" export-graph "graph" "export" "--type" "sqlite" "--file" export-path] data-dir cfg-path) export-payload (parse-json-output export-result) - _ (run-cli ["--repo" import-graph + _ (run-cli ["--graph" import-graph "graph" "import" "--type" "sqlite" "--input" export-path] data-dir cfg-path) - list-result (run-cli ["--repo" import-graph "list" "page"] data-dir cfg-path) + list-result (run-cli ["--graph" import-graph "list" "page"] data-dir cfg-path) list-payload (parse-json-output list-result) - stop-export (run-cli ["server" "stop" "--repo" export-graph] data-dir cfg-path) - stop-import (run-cli ["server" "stop" "--repo" import-graph] data-dir cfg-path)] + stop-export (run-cli ["server" "stop" "--graph" export-graph] data-dir cfg-path) + stop-import (run-cli ["server" "stop" "--graph" import-graph] data-dir cfg-path)] (is (= 0 (:exit-code export-result))) (is (= "ok" (:status export-payload))) (is (fs/existsSync export-path)) From a1989f9c3d4b6e9757feb821b6a4bdb1277ab38e Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 3 Mar 2026 11:34:13 -0500 Subject: [PATCH 099/375] enhance: add aliases for more used global options --- src/main/logseq/cli/command/core.cljs | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 4c28609237..708b553ed1 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -11,13 +11,17 @@ :coerce :boolean} :version {:desc "Show version" :coerce :boolean} - :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)"} - :graph {:desc "Graph name"} + :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)" + :alias :c} + :graph {:desc "Graph name" + :alias :g} :data-dir {:desc "Path to db-worker data dir (default ~/logseq/graphs)"} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} - :output {:desc "Output format (human, json, edn). Default: human"} + :output {:desc "Output format (human, json, edn). Default: human" + :alias :o} :verbose {:desc "Enable verbose debug logging to stderr" + :alias :v :coerce :boolean}}) (defn global-spec From edf0853e0a541422da4ef9bd1609c1e034cb9880 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 3 Mar 2026 12:43:29 -0500 Subject: [PATCH 100/375] fix: crypt tests to handle new caching Also mark more ^:long tests so most unit tests can still be run locally in ~30s --- src/test/frontend/worker/sync/crypt_test.cljs | 16 +++++++++------- src/test/logseq/db_worker/ncc_bundle_test.cljs | 6 +++--- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/src/test/frontend/worker/sync/crypt_test.cljs b/src/test/frontend/worker/sync/crypt_test.cljs index 1e5f931301..611306fb04 100644 --- a/src/test/frontend/worker/sync/crypt_test.cljs +++ b/src/test/frontend/worker/sync/crypt_test.cljs @@ -150,13 +150,15 @@ (sync-crypt/ (p/let [_ (ensure-bundle-built!) @@ -156,7 +156,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest bundle-daemon-starts-with-empty-asset-manifest +(deftest ^:long bundle-daemon-starts-with-empty-asset-manifest (async done (let [child* (atom nil) original-manifest* (atom nil)] @@ -216,7 +216,7 @@ (write-asset-manifest! manifest)) (done))))))) -(deftest bundle-errors-are-actionable-when-manifest-or-entry-is-missing +(deftest ^:long bundle-errors-are-actionable-when-manifest-or-entry-is-missing (let [_ (ensure-bundle-built!) manifest-path (dist-path "db-worker-node-assets.json") manifest-backup-path (dist-path "db-worker-node-assets.json.bak") From 2b66fb0f1bdfb41f94126ec83f00b8a3fdaa411f Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 3 Mar 2026 13:04:25 -0500 Subject: [PATCH 101/375] fix: turn off noisy logging in db-worker-node-test So much noise makes it hard to see actual failure in test suite --- .../frontend/worker/db_worker_node_test.cljs | 123 +++++++++++------- 1 file changed, 73 insertions(+), 50 deletions(-) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 22961545dd..e4b8858ec2 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -2,7 +2,7 @@ (:require ["fs" :as fs] ["http" :as http] ["path" :as node-path] - [cljs.test :refer [async deftest is]] + [cljs.test :refer [async deftest is use-fixtures]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker.db-worker-node-lock :as db-lock] @@ -107,13 +107,43 @@ date-str (yyyymmdd (js/Date.))] (node-path/join repo-dir (str "db-worker-node-" date-str ".log")))) +(defn- start-daemon! + "Start daemon with quiet logging by default" + [opts] + (db-worker-node/start-daemon! (update opts :log-level #(or % "error")))) + +(defn- noisy-debug-line? + [line] + (or (string/includes? line ":listen-db-changes!") + (string/includes? line ":debug :db-gc"))) + +(defonce ^:private *orig-print-fn (atom nil)) + +(defn- quiet-debug-output-before + [] + (when-not @*orig-print-fn + (reset! *orig-print-fn *print-fn*)) + (set-print-fn! + (fn [line] + (when-not (and (string? line) (noisy-debug-line? line)) + (when-let [orig @*orig-print-fn] + (orig line)))))) + +(defn- quiet-debug-output-after + [] + (when-let [orig @*orig-print-fn] + (set-print-fn! orig))) + +(use-fixtures :each {:before quiet-debug-output-before + :after quiet-debug-output-after}) + (deftest db-worker-node-data-dir-permission-error (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly") repo (str "logseq_db_perm_" (subs (str (random-uuid)) 0 8))] (fs/chmodSync data-dir 365) - (-> (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (-> (start-daemon! {:data-dir data-dir + :repo repo}) (p/then (fn [_] (is false "expected data-dir permission error"))) (p/catch (fn [e] @@ -129,8 +159,8 @@ repo (str "logseq_db_log_" (subs (str (random-uuid)) 0 8)) log-file (log-path data-dir repo)] (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) _ (p/delay 50)] (is (fs/existsSync log-file))) @@ -148,10 +178,10 @@ repo (str "logseq_db_log_entries_" (subs (str (random-uuid)) 0 8)) log-file (log-path data-dir repo)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) - _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + _ (invoke-raw host port "thread-api/q" [repo nil]) _ (p/delay 50) contents (when (fs/existsSync log-file) (.toString (fs/readFileSync log-file) "utf8"))] @@ -226,9 +256,9 @@ repo (str "logseq_db_owner_cli_" (subs (str (random-uuid)) 0 8)) lock-file (lock-path data-dir repo)] (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo - :owner-source :cli}) + (start-daemon! {:data-dir data-dir + :repo repo + :owner-source :cli}) _ (reset! daemon {:stop! stop!}) lock-json (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8"))] (is (= "cli" (gobj/get lock-json "owner-source")))) @@ -246,9 +276,9 @@ repo (str "logseq_db_owner_electron_" (subs (str (random-uuid)) 0 8)) lock-file (lock-path data-dir repo)] (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo - :owner-source :electron}) + (start-daemon! {:data-dir data-dir + :repo repo + :owner-source :electron}) _ (reset! daemon {:stop! stop!}) lock-json (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8"))] (is (= "electron" (gobj/get lock-json "owner-source")))) @@ -320,8 +350,8 @@ data-dir (node-helper/create-tmp-dir "db-worker-set-context") repo (str "logseq_db_set_context_" (subs (str (random-uuid)) 0 8))] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:host host :port port :stop! stop!}) result (invoke host port "thread-api/set-context" [{:app "desktop"}])] (is (nil? result))) @@ -341,23 +371,17 @@ page-uuid (random-uuid) block-uuid (random-uuid)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! - {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) health (http-get host port "/healthz") ready (http-get host port "/readyz") _ (do (reset! daemon {:host host :port port :stop! stop!}) - (println "[db-worker-node-test] daemon started" {:host host :port port}) - (println "[db-worker-node-test] /healthz" health) (is (= 200 (:status health))) - (println "[db-worker-node-test] /readyz" ready) - (is (= 200 (:status ready))) - (println "[db-worker-node-test] repo" repo)) + (is (= 200 (:status ready)))) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) dbs (invoke host port "thread-api/list-db" []) _ (do - (println "[db-worker-node-test] list-db" dbs) (is (some #(= repo (:name %)) dbs))) lock-file (lock-path data-dir repo) _ (is (fs/existsSync lock-file)) @@ -387,7 +411,6 @@ :in $ ?uuid :where [?e :block/uuid ?uuid]] block-uuid]])] - (println "[db-worker-node-test] q result" result) (is (seq result))) (p/catch (fn [e] (println "[db-worker-node-test] e:" e) @@ -410,8 +433,8 @@ now (js/Date.now) page-uuid (random-uuid)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-a}) + (start-daemon! {:data-dir data-dir + :repo repo-a}) _ (reset! daemon-a {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) _ (invoke host port "thread-api/transact" @@ -428,8 +451,8 @@ (is (map? export-edn)) (p/let [_ ((:stop! @daemon-a)) {:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-b}) + (start-daemon! {:data-dir data-dir + :repo repo-b}) _ (reset! daemon-b {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) _ (invoke host port "thread-api/import-edn" [repo-b export-edn]) @@ -470,8 +493,8 @@ now (js/Date.now) page-uuid (random-uuid)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-a}) + (start-daemon! {:data-dir data-dir + :repo repo-a}) _ (reset! daemon-a {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo-a {}]) _ (invoke host port "thread-api/transact" @@ -489,8 +512,8 @@ (is (pos? (count export-base64))) (p/let [_ ((:stop! @daemon-a)) {:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo-b}) + (start-daemon! {:data-dir data-dir + :repo repo-b}) _ (reset! daemon-b {:stop! stop!}) _ (invoke host port "thread-api/import-db-base64" [repo-b export-base64]) _ (invoke host port "thread-api/create-or-open-db" [repo-b {}]) @@ -528,8 +551,8 @@ repo (str "logseq_db_mismatch_" (subs (str (random-uuid)) 0 8)) other-repo (str repo "_other")] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:host host :port port :stop! stop!}) {:keys [status body]} (invoke-raw host port "thread-api/create-or-open-db" [other-repo {}]) parsed (js->clj (js/JSON.parse body) :keywordize-keys true)] @@ -549,11 +572,11 @@ data-dir (node-helper/create-tmp-dir "db-worker-lock") repo (str "logseq_db_lock_" (subs (str (random-uuid)) 0 8))] (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!})] - (-> (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (-> (start-daemon! {:data-dir data-dir + :repo repo}) (p/then (fn [_] (is false "expected lock error"))) (p/catch (fn [e] @@ -572,8 +595,8 @@ repo (str "logseq_db_write_lease_pid_" (subs (str (random-uuid)) 0 8)) lock-file (lock-path data-dir repo)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) export-base64 (invoke host port "thread-api/export-db-base64" [repo]) @@ -602,8 +625,8 @@ repo (str "logseq_db_write_lease_owner_" (subs (str (random-uuid)) 0 8)) lock-file (lock-path data-dir repo)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) @@ -628,8 +651,8 @@ repo (str "logseq_db_write_lease_replaced_" (subs (str (random-uuid)) 0 8)) lock-file (lock-path data-dir repo)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) export-base64 (invoke host port "thread-api/export-db-base64" [repo]) @@ -663,8 +686,8 @@ (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) (fs/writeFileSync lock-file (js/JSON.stringify (clj->js stale-lock))) (-> (p/let [{:keys [stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) lock' (js->clj (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) :keywordize-keys true)] @@ -686,8 +709,8 @@ now (js/Date.now) page-uuid (random-uuid)] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) _ (invoke host port "thread-api/transact" From 703581d37b9756070469f70f23deb9676f53f4e1 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 3 Mar 2026 13:58:33 -0500 Subject: [PATCH 102/375] enhance: more user-friendly behavior when no args given Be more helpful when no args given like cp, zip and npm. Also fix frontend lint --- src/main/logseq/cli/commands.cljs | 14 ++------------ src/test/frontend/worker/db_worker_node_test.cljs | 3 +-- src/test/logseq/cli/commands_test.cljs | 4 ++-- 3 files changed, 5 insertions(+), 16 deletions(-) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 894daf4939..b1f09617d1 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -282,12 +282,7 @@ (command-core/ok-result :version opts [] summary) (empty? args) - (if (:help opts) - (command-core/help-result summary) - {:ok? false - :error {:code :missing-command - :message "missing command"} - :summary summary}) + (command-core/help-result summary) (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) @@ -304,12 +299,7 @@ (let [{:keys [cause] :as data} (ex-data e)] (cond (= cause :input-exhausted) - (if (:help opts) - (command-core/help-result summary) - {:ok? false - :error {:code :missing-command - :message "missing command"} - :summary summary}) + (command-core/help-result summary) (= cause :no-match) (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index e4b8858ec2..f747134f00 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -381,8 +381,7 @@ (is (= 200 (:status ready)))) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) dbs (invoke host port "thread-api/list-db" []) - _ (do - (is (some #(= repo (:name %)) dbs))) + _ (is (some #(= repo (:name %)) dbs)) lock-file (lock-path data-dir repo) _ (is (fs/existsSync lock-file)) lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8")) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 6d7f0384c9..aee1a906c0 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -286,10 +286,10 @@ (is (false? (:ok? result))) (is (= :unknown-command (get-in result [:error :code])))))) - (testing "errors on missing command" + (testing "no commands prints help" (let [result (commands/parse-args [])] (is (false? (:ok? result))) - (is (= :missing-command (get-in result [:error :code]))))) + (is (true? (:help? result))))) (testing "errors on unknown command" (let [result (commands/parse-args ["wat"])] From 76776c53a8809fbd6d64a7f408435ebfd256d4b2 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Wed, 4 Mar 2026 10:45:24 -0500 Subject: [PATCH 103/375] fix: old file graphs showing up in app list of graphs Both in sidebar but also in commands to select graphs like 'Select graph to open' --- src/main/frontend/db/persist.cljs | 10 +++++----- src/test/frontend/db/persist_test.cljs | 5 +++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/src/main/frontend/db/persist.cljs b/src/main/frontend/db/persist.cljs index 7e170a63a0..7767f5d2ca 100644 --- a/src/main/frontend/db/persist.cljs +++ b/src/main/frontend/db/persist.cljs @@ -8,10 +8,10 @@ [logseq.common.config :as common-config] [promesa.core :as p])) -(defn local-file-based-graph? +(defn- local-file-based-graph? [s] (and (string? s) - (string/starts-with? s common-config/file-version-prefix))) + (string/starts-with? s (str common-config/db-version-prefix common-config/file-version-prefix)))) (defn get-all-graphs [] @@ -27,9 +27,9 @@ (distinct (concat repos' - (map (fn [repo-name] - {:name (common-config/canonicalize-db-version-repo repo-name)}) - (some-> electron-disk-graphs bean/->clj)))))) + (->> (some-> electron-disk-graphs bean/->clj) + (map (fn [repo-name] + {:name (common-config/canonicalize-db-version-repo repo-name)}))))))) (defn delete-graph! [graph] diff --git a/src/test/frontend/db/persist_test.cljs b/src/test/frontend/db/persist_test.cljs index 6fce792a8a..557ce4edf1 100644 --- a/src/test/frontend/db/persist_test.cljs +++ b/src/test/frontend/db/persist_test.cljs @@ -12,7 +12,7 @@ (p/resolved [{:name "demo"} {:name "logseq_db_prefixed"} {:name "logseq_db_logseq_db_legacy"} - {:name "logseq_local_local-only"}])) + {:name "logseq_db_logseq_local_local-only"}])) util/electron? (constantly true) ipc/ipc (fn [_channel] (p/resolved #js ["logseq_db_remote" @@ -25,7 +25,8 @@ "logseq_db_remote" "logseq_db_remote-legacy"} (set names))) - (is (not-any? #(re-find #"^logseq_db_logseq_db_" %) names)))) + (is (not-any? #(re-find #"^logseq_db_logseq_db_" %) names)) + (is (not-any? #(re-find #"logseq_local_" %) names)))) (p/catch (fn [error] (is false (str error)))) (p/finally done)))) From 16ae5b2ef778b058265a824c450cbac27f4068ff Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Wed, 4 Mar 2026 14:32:17 -0500 Subject: [PATCH 104/375] fix: validate graph does not handle invalid graphs Invalid graphs were pretending to be valid. Instead validation errors should be displayed and command should exit 1. This also fixes other commands that were exiting 0 even though they returned an :error --- src/main/logseq/cli/command/graph.cljs | 56 +++++++++++++++------ src/main/logseq/cli/format.cljs | 20 ++++---- src/main/logseq/cli/main.cljs | 7 ++- src/test/logseq/cli/command/graph_test.cljs | 15 ++++++ src/test/logseq/cli/format_test.cljs | 16 ++++++ src/test/logseq/cli/integration_test.cljs | 4 +- src/test/logseq/cli/main_test.cljs | 6 +++ 7 files changed, 96 insertions(+), 28 deletions(-) create mode 100644 src/test/logseq/cli/command/graph_test.cljs diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 6c2a948001..2afd88a6ec 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.command.graph "Graph-related CLI commands." - (:require [clojure.string :as string] + (:require [cljs.pprint :as pprint] + [clojure.string :as string] [logseq.cli.command.core :as core] [logseq.cli.config :as cli-config] [logseq.cli.server :as cli-server] @@ -140,23 +141,46 @@ {:status :ok :data {:graphs graphs}})) +(defn- format-validation-errors + [errors] + (str "Graph invalid. Found " (count errors) + (if (= 1 (count errors)) " entity" " entities") + " with errors:\n" + (with-out-str (pprint/pprint errors)))) + +(defn- graph-validate-result + [result] + (if (seq (:errors result)) + {:status :error + :error {:code :graph-validation-failed + :message (format-validation-errors (:errors result))} + :data {:errors (:errors result)}} + {:status :ok :data {:result result}})) + (defn execute-invoke [action config] - (-> (p/let [cfg (if-let [repo (:repo action)] - (cli-server/ensure-server! config repo) - (p/resolved config)) - result (transport/invoke cfg - (:method action) - (:direct-pass? action) - (:args action))] - (when-let [repo (:persist-repo action)] - (cli-config/update-config! config {:graph repo})) - (if-let [write (:write action)] - (let [{:keys [format path]} write] - (transport/write-output {:format format :path path :data result}) - {:status :ok - :data {:message (str "wrote " path)}}) - {:status :ok :data {:result result}})))) + (p/let [cfg (if-let [repo (:repo action)] + (cli-server/ensure-server! config repo) + (p/resolved config)) + result (transport/invoke cfg + (:method action) + (:direct-pass? action) + (:args action))] + (when-let [repo (:persist-repo action)] + (cli-config/update-config! config {:graph repo})) + (let [write (:write action)] + (cond + (= :graph-validate (:command action)) + (graph-validate-result result) + + write + (let [{:keys [format path]} write] + (transport/write-output {:format format :path path :data result}) + {:status :ok + :data {:message (str "wrote " path)}}) + + :else + {:status :ok :data {:result result}})))) (defn execute-graph-switch [action config] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index cb053a7b73..fb8ffd0da4 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -115,9 +115,11 @@ hint (error-hint error) message* (style/bold-keywords message ["option" "command" "argument"]) candidates* (format-candidates candidates)] - (cond-> (str "Error (" (name (or code :error)) "): " message*) - candidates* (str candidates*) - hint (str "\nHint: " hint)))) + (if (= :graph-validation-failed code) + message* + (cond-> (str "Error (" (name (or code :error)) "): " message*) + candidates* (str candidates*) + hint (str "\nHint: " hint))))) (defn- maybe-ident-header [items] @@ -340,12 +342,12 @@ (defn- format-graph-action [command {:keys [graph]}] (let [verb (case command - :graph-create "created" - :graph-switch "switched" - :graph-remove "removed" - :graph-validate "validated" - "updated")] - (str "Graph " verb ": " graph))) + :graph-create "Created" + :graph-switch "Switched to" + :graph-remove "Removed" + :graph-validate "Validated" + "Updated")] + (str verb " graph " (pr-str graph)))) (defn- format-doctor [status checks] diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index 36c2feb87a..b069445e7a 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -19,6 +19,11 @@ "Options:" summary])) +(defn- result->exit-code + [result] + (or (:exit-code result) + (if (= :error (:status result)) 1 0))) + (defn run! ([args] (run! args {})) ([args _opts] @@ -69,7 +74,7 @@ (let [opts (cond-> cfg (:output-format result) (assoc :output-format (:output-format result)))] - {:exit-code 0 + {:exit-code (result->exit-code result) :output (format/format-result result opts)}))) (p/catch (fn [error] (let [data (ex-data error) diff --git a/src/test/logseq/cli/command/graph_test.cljs b/src/test/logseq/cli/command/graph_test.cljs new file mode 100644 index 0000000000..4560d2f804 --- /dev/null +++ b/src/test/logseq/cli/command/graph_test.cljs @@ -0,0 +1,15 @@ +(ns logseq.cli.command.graph-test + (:require [cljs.test :refer [deftest is]] + [clojure.string :as string] + [logseq.cli.command.graph :as graph-command])) + +(deftest test-graph-validate-result + (let [graph-validate-result #'graph-command/graph-validate-result + invalid-result (graph-validate-result {:errors [{:entity {:db/id 1} + :errors {:foo ["bad"]}}]}) + valid-result (graph-validate-result {:errors nil :datom-count 10})] + (is (= :error (:status invalid-result))) + (is (= :graph-validation-failed (get-in invalid-result [:error :code]))) + (is (string/includes? (get-in invalid-result [:error :message]) + "Found 1 entity with errors:")) + (is (= :ok (:status valid-result))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index bb6e6987ea..6189b2d7ef 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -45,6 +45,22 @@ {:output-format nil})] (is (= "Error (boom): nope" result))))) +(deftest test-format-graph-validation + (testing "graph validation success prints validated" + (let [result (format/format-result {:status :ok + :command :graph-validate + :context {:graph "foo"} + :data {:result {:errors nil}}} + {:output-format nil})] + (is (= "Validated graph \"foo\"" result)))) + (testing "graph validation error prints validation details without extra prefix" + (let [result (format/format-result {:status :error + :command :graph-validate + :error {:code :graph-validation-failed + :message "Found 1 entity with errors:\n({:entity {:db/id 1}})\n"}} + {:output-format nil})] + (is (= "Found 1 entity with errors:\n({:entity {:db/id 1}})\n" result))))) + (deftest test-human-output-list-page (testing "list page renders a table with count" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 0fca0d4422..540051ed36 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -1172,7 +1172,7 @@ set) stop-result (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path) stop-payload (parse-json-output stop-result)] - (is (= 0 (:exit-code upsert-tag-result))) + (is (= 1 (:exit-code upsert-tag-result))) (is (= "error" (:status upsert-tag-payload))) (is (string/includes? (get-in upsert-tag-payload [:error :message]) "already exists as a page and is not a tag")) @@ -1294,7 +1294,7 @@ (is (number? source-id)) (is (= 0 (:exit-code existing-upsert-result))) (is (= "ok" (:status existing-upsert-payload))) - (is (= 0 (:exit-code rename-result))) + (is (= 1 (:exit-code rename-result))) (is (= "error" (:status rename-payload))) (is (= :tag-rename-conflict (keyword (get-in rename-payload [:error :code])))) (is (contains? tag-names source-name)) diff --git a/src/test/logseq/cli/main_test.cljs b/src/test/logseq/cli/main_test.cljs index d211a998aa..9ea8aa82d3 100644 --- a/src/test/logseq/cli/main_test.cljs +++ b/src/test/logseq/cli/main_test.cljs @@ -25,3 +25,9 @@ (is false (str "unexpected error: " e)) (done))) (p/finally done)))) + +(deftest test-result->exit-code + (let [result->exit-code #'cli-main/result->exit-code] + (is (= 0 (result->exit-code {:status :ok}))) + (is (= 1 (result->exit-code {:status :error}))) + (is (= 7 (result->exit-code {:status :error :exit-code 7}))))) From aa2f05c1b6af87ad8c21a036b2fe2a14fcb9705b Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Wed, 4 Mar 2026 15:37:51 -0500 Subject: [PATCH 105/375] fix: integration test assertion Only assertion that fails locally --- src/test/logseq/cli/integration_test.cljs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 540051ed36..9623c87913 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -649,7 +649,7 @@ (is (string/includes? (:output legacy-page-result) "--update-properties")) (is (= 1 (:exit-code conflict-result))) (is (string/includes? (:output conflict-result) "invalid-options")) - (is (string/includes? (:output conflict-result) "only one of --id or --page")) + (is (string/includes? (style/strip-ansi (:output conflict-result)) "only one of --id or --page")) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From 426d4f10b3521823a3cae40478b22d9c68af386c Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Wed, 4 Mar 2026 18:00:06 -0500 Subject: [PATCH 106/375] enhance: validate cmd option to fix graph Give users the option to fix while validating. Validate implies read-only so give users the extra ability in cli to toggle this behavior --- src/main/frontend/worker/db/validate.cljs | 20 ++++++++++++-------- src/main/frontend/worker/db_core.cljs | 4 ++-- src/main/logseq/cli/command/graph.cljs | 11 ++++++++--- src/main/logseq/cli/commands.cljs | 2 +- 4 files changed, 23 insertions(+), 14 deletions(-) diff --git a/src/main/frontend/worker/db/validate.cljs b/src/main/frontend/worker/db/validate.cljs index 50094abfb7..38291ff504 100644 --- a/src/main/frontend/worker/db/validate.cljs +++ b/src/main/frontend/worker/db/validate.cljs @@ -220,12 +220,13 @@ {:fix-db? true}))) (defn validate-db - [conn] - (fix-extends-cardinality! conn) - (fix-icon-wrong-type! conn) - (db-migrate/ensure-built-in-data-exists! conn) - (fix-non-closed-values! conn) - (fix-num-prefix-db-idents! conn) + [conn {:keys [fix] :or {fix true}}] + (when fix + (fix-extends-cardinality! conn) + (fix-icon-wrong-type! conn) + (db-migrate/ensure-built-in-data-exists! conn) + (fix-non-closed-values! conn) + (fix-num-prefix-db-idents! conn)) (let [db @conn {:keys [errors datom-count entities]} (db-validate/validate-db! db) @@ -238,12 +239,15 @@ (if errors (do - (fix-invalid-blocks! conn errors) + (when fix + (fix-invalid-blocks! conn errors)) (shared-service/broadcast-to-clients! :log [:db-invalid :error {:msg "Validation errors" :errors errors}]) (shared-service/broadcast-to-clients! :notification - [(str "Validation detected " (count errors) " invalid block(s). These blocks may be buggy. Attempting to fix invalid blocks. Run validation again to see if they were fixed.") + [(str "Validation detected " (count errors) " invalid block(s). These blocks may be buggy." + (when fix + " Attempting to fix invalid blocks. Run validation again to see if they were fixed.")) :warning false])) (shared-service/broadcast-to-clients! :notification diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 194200d867..3241ac9a06 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -776,9 +776,9 @@ (worker-export/get-all-page->content @conn options))) (def-thread-api :thread-api/validate-db - [repo] + [repo & [opts]] (when-let [conn (worker-state/get-datascript-conn repo)] - (worker-db-validate/validate-db conn))) + (worker-db-validate/validate-db conn opts))) ;; Returns an export-edn map for given repo. When there's an unexpected error, a map ;; with key :export-edn-error is returned diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 2afd88a6ec..26c6e8d68e 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -16,12 +16,17 @@ {:type {:desc "Import type (edn, sqlite)"} :input {:desc "Input path"}}) +(def ^:private graph-validate-spec + {:fix {:desc "Attempt to fix validation errors" + :alias :f + :default false}}) + (def entries [(core/command-entry ["graph" "list"] :graph-list "List graphs" {}) (core/command-entry ["graph" "create"] :graph-create "Create graph" {}) (core/command-entry ["graph" "switch"] :graph-switch "Switch current graph" {}) (core/command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) - (core/command-entry ["graph" "validate"] :graph-validate "Validate graph" {}) + (core/command-entry ["graph" "validate"] :graph-validate "Validate graph" graph-validate-spec) (core/command-entry ["graph" "info"] :graph-info "Graph metadata" {}) (core/command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec) (core/command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec)]) @@ -44,7 +49,7 @@ :message "graph name is required"}}) (defn build-graph-action - [command graph repo] + [command graph repo options] (case command :graph-list {:ok? true @@ -94,7 +99,7 @@ :command :graph-validate :method :thread-api/validate-db :direct-pass? false - :args [repo] + :args [repo options] :repo repo :graph (core/repo->graph repo)}}) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index b1f09617d1..c427971775 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -362,7 +362,7 @@ server-repo (command-core/resolve-repo (:graph options))] (case command (:graph-list :graph-create :graph-switch :graph-remove :graph-validate :graph-info) - (graph-command/build-graph-action command graph repo) + (graph-command/build-graph-action command graph repo options) :graph-export (let [export-type (graph-command/normalize-import-export-type (:type options))] From 2449dc27e49fefb8fc48597bb742e218170703e9 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 4 Mar 2026 16:28:48 +0800 Subject: [PATCH 107/375] 047-logseq-cli-sync-command.md --- .../047-logseq-cli-sync-command.md | 225 +++++++++ docs/cli/logseq-cli.md | 24 + src/main/frontend/worker/db_core.cljs | 78 ++- src/main/frontend/worker/db_worker_node.cljs | 2 + src/main/frontend/worker/sync.cljs | 174 ++++++- src/main/frontend/worker/sync/crypt.cljs | 75 ++- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/command/graph.cljs | 3 +- src/main/logseq/cli/command/sync.cljs | 273 +++++++++++ src/main/logseq/cli/commands.cljs | 26 +- src/main/logseq/cli/config.cljs | 13 +- src/main/logseq/cli/format.cljs | 61 +++ src/test/frontend/worker/db_core_test.cljs | 32 +- .../frontend/worker/db_worker_node_test.cljs | 39 ++ src/test/frontend/worker/sync/crypt_test.cljs | 83 ++++ src/test/logseq/cli/command/sync_test.cljs | 450 ++++++++++++++++++ src/test/logseq/cli/commands_test.cljs | 52 ++ src/test/logseq/cli/config_test.cljs | 13 +- src/test/logseq/cli/format_test.cljs | 90 ++++ 19 files changed, 1675 insertions(+), 40 deletions(-) create mode 100644 docs/agent-guide/047-logseq-cli-sync-command.md create mode 100644 src/main/logseq/cli/command/sync.cljs create mode 100644 src/test/logseq/cli/command/sync_test.cljs diff --git a/docs/agent-guide/047-logseq-cli-sync-command.md b/docs/agent-guide/047-logseq-cli-sync-command.md new file mode 100644 index 0000000000..945d496e5e --- /dev/null +++ b/docs/agent-guide/047-logseq-cli-sync-command.md @@ -0,0 +1,225 @@ +# Logseq CLI Sync Command Implementation Plan + +Goal: Add `logseq sync` subcommands to inspect and operate db-sync through existing db-worker-node APIs. + +Architecture: The CLI parser and executor will gain a dedicated sync command module that maps subcommands to `:thread-api/db-sync-*` calls via `/v1/invoke`. +Architecture: A small worker API addition will expose runtime sync status, and sync config commands will support headless token setup through CLI-managed config values. +Architecture: The design will reuse existing graph lock and repo binding behavior in `logseq.cli.server/ensure-server!` and `frontend.worker.db-worker-node/repo-error`. + +Tech Stack: ClojureScript, babashka.cli, promesa, db-worker-node HTTP API, frontend.worker.sync. + +Related: Builds on `docs/agent-guide/031-logseq-cli-doctor-command.md`, `docs/agent-guide/033-desktop-db-worker-node-backend.md`, and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +The current CLI exposes graph, server, doctor, list, upsert, remove, query, and show commands, but it does not expose db-sync control or observability. + +`frontend.worker.db-core` already exposes operational db-sync thread APIs such as `:thread-api/db-sync-start`, `:thread-api/db-sync-stop`, `:thread-api/db-sync-upload-graph`, and `:thread-api/db-sync-grant-graph-access`. + +`frontend.worker.db-worker-node` already routes these methods over `/v1/invoke` with repo-lock safety checks, so the missing piece is a CLI command surface and one read API for status inspection. + +This plan keeps scope tight by reusing current transport and server lifecycle code, and only adds new worker behavior where inspection data is currently unavailable. + +I will use @planning-documents for naming, @writing-plans for task granularity, @logseq-cli for CLI integration expectations, and @test-driven-development for implementation sequence. + +## Testing Plan + +I will add parser and action unit tests that fail first for new `sync` command help, option validation, and action shaping. + +I will add command execution tests that fail first and verify `logseq.cli.transport/invoke` receives the exact method names and argument shapes for each sync subcommand. + +I will add format tests that fail first and verify human output for `sync status` and action commands, while keeping JSON and EDN behavior unchanged. + +I will add worker API tests that fail first for the new sync inspection API exposed through `/v1/invoke`. + +I will add one CLI integration test that fails first and verifies an end-to-end `sync status` flow on a temp graph and a started db-worker-node process. + +I will run targeted tests after each behavior slice and then run `bb dev:lint-and-test` before final review. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and CLI surface + +| CLI command | Purpose | Worker method | Repo required | +|---|---|---|---| +| `sync status [--graph ]` | View current db-sync runtime state and counters. | `:thread-api/db-sync-status` (new). | Yes. | +| `sync start [--graph ]` | Start db-sync websocket client for the graph. | `:thread-api/db-sync-start`. | Yes. | +| `sync stop [--graph ]` | Stop db-sync client for the running daemon. | `:thread-api/db-sync-stop`. | Yes, to target a graph daemon deterministically. | +| `sync upload [--graph ]` | Upload current graph snapshot and mark graph remote metadata. | `:thread-api/db-sync-upload-graph`. | Yes. | +| `sync download [--graph ]` | Download remote graph data and apply it to local graph storage. | `:thread-api/db-sync-download-graph` (new). | Yes. | +| `sync remote-graphs` | List remote graphs visible to current auth context. | `:thread-api/db-sync-list-remote-graphs` (new). | No. | +| `sync ensure-keys` | Ensure user RSA keys required by e2ee are present. | `:thread-api/db-sync-ensure-user-rsa-keys`. | No. | +| `sync grant-access --graph-id --email [--graph ]` | Grant encrypted graph key access to a target user email. | `:thread-api/db-sync-grant-graph-access`. | Yes. | +| `sync config set ` | Set one config value by key. | `:thread-api/set-db-sync-config`. | No. | +| `sync config get ` | Read one config value by key. | `:thread-api/get-db-sync-config` (new). | No. | +| `sync config unset ` | Remove one config value by key. | `:thread-api/set-db-sync-config`. | No. | + +The first release intentionally excludes asset download and raw kv import commands because they need more user-facing safety rails and payload tooling. + +`sync config set` supports `ws-url`, `http-base`, and `auth-token`, and `config set auth-token ` is the headless authentication entrypoint. + +`sync config get` and `sync config unset` reject unknown config keys. + +`sync status` will return normalized fields even when sync is not configured, so scripts can branch deterministically. + +`sync remote-graphs` and `sync download` require auth-token to be configured in headless mode. + +## Architecture and integration points + +```text +logseq sync + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs (parse/build/execute) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs (new) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs (ensure graph daemon) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs (POST /v1/invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs (repo checks + invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs thread APIs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs and sync/crypt.cljs +``` + +Worker additions will be minimal, with no protocol changes to cloud endpoints. + +CLI additions will follow existing `graph` and `server` command module patterns for spec, `entries`, `build-action`, and `execute-*` helpers. + +## Implementation plan + +### Phase 1. Add failing parser and help tests. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that top-level help includes `sync` and `sync status`. +2. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `logseq sync` shows subgroup help like `server` and `graph`. +3. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `sync config set|get|unset` and `sync grant-access` show in sync group help. +4. Run `bb dev:test -v logseq.cli.commands-test/test-help-output` and confirm failure references missing `sync` command rows. + +### Phase 2. Add failing action and execution tests for sync command module. + +5. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` with failing tests for `build-action` graph requirement on `sync status`. +6. Add a failing test for `build-action` rejection when `sync config set` is missing name or value. +7. Add a failing test for `build-action` rejection when `sync grant-access` misses `--graph-id` or `--email`. +8. Add a failing test for `build-action` graph requirement on `sync download`. +9. Add a failing execution test that stubs `logseq.cli.server/ensure-server!` and `logseq.cli.transport/invoke` and expects `:thread-api/db-sync-start` with `[repo]`. +10. Add a failing execution test that expects `:thread-api/db-sync-stop` with `[]` and still routes through `ensure-server!` using selected repo. +11. Add a failing execution test that expects `:thread-api/db-sync-upload-graph` with `[repo]`. +12. Add a failing execution test that expects `:thread-api/db-sync-download-graph` with `[repo]`. +13. Add a failing execution test that expects `:thread-api/db-sync-list-remote-graphs` for `sync remote-graphs`. +14. Add a failing execution test that expects `:thread-api/db-sync-ensure-user-rsa-keys` without repo. +15. Add a failing execution test that expects `:thread-api/db-sync-grant-graph-access` with `[repo graph-id email]`. +16. Add a failing execution test that expects `:thread-api/get-db-sync-config` for `sync config get `. +17. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config set ` and payload merge behavior. +18. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config unset ` and key removal behavior. +19. Add a failing execution test that verifies `sync config set auth-token ` updates worker-consumable token config for headless mode. +20. Run `bb dev:test -v logseq.cli.command.sync-test` and confirm failures are only from missing sync implementation. + +### Phase 3. Implement CLI sync command wiring. + +21. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` with sync option specs, `entries`, `build-action`, and `execute-*` functions. +22. Register `sync-command/entries` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` command table. +23. Extend `finalize-command` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` with sync-specific required-option checks. +24. Extend single-token group help routing in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to include `sync`. +25. Extend `build-action` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to call `sync-command/build-action`. +26. Extend `execute` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to route sync action types. +27. Add `sync` to top-level command grouping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs`. +28. Run `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` and `bb dev:test -v logseq.cli.command.sync-test` until green. + +### Phase 4. Add read-only worker APIs for sync inspection. + +29. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` accepts `thread-api/get-db-sync-config` without repo. +30. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` for `thread-api/db-sync-status` enforces repo and returns structured status. +31. Add `:thread-api/get-db-sync-config` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning current config map. +32. Add `:thread-api/db-sync-status` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning ws state, graph id, and sync counters for a repo. +33. Add or expose a small helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to compute status without requiring websocket side effects. +34. Add `:thread-api/db-sync-list-remote-graphs` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement cloud graph listing through worker sync HTTP helpers. +35. Add `:thread-api/db-sync-download-graph` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement remote snapshot download plus local import flow. +36. Update worker sync auth token resolution so `sync config set auth-token ` is used in headless mode when state token is missing. +37. Register `:thread-api/get-db-sync-config` and `:thread-api/db-sync-list-remote-graphs` in `non-repo-methods` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`. +38. Run `bb dev:test -v frontend.worker.db-worker-node-test` and fix only sync-related regressions. + +### Phase 5. Add output formatting tests and implementation. + +39. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync status`. +40. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync remote-graphs`. +41. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of sync action commands such as start, upload, and download. +42. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` verifying token redaction for `sync config get auth-token` in human output. +43. Implement sync human formatters in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with stable keys and token redaction. +44. Confirm JSON and EDN output behavior by running `bb dev:test -v logseq.cli.format-test`. + +### Phase 6. Add integration coverage and CLI docs. + +45. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a graph and runs `sync status` with `--output json`. +46. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config set auth-token` then `sync config get auth-token` behavior. +47. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config unset auth-token`. +48. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync remote-graphs --output json`. +49. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync download --graph ` flow with mocked remote snapshot response. +50. Implement any missing glue for integration stability in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs`. +51. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` command docs with sync command examples and error behaviors. +52. Run `bb dev:test -v logseq.cli.integration-test` to verify end-to-end behavior. + +### Phase 7. Final verification and cleanup. + +53. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and confirm exit code `0`. +54. Run manual smoke commands with a temp graph and confirm both `--output human` and `--output json` are stable. +55. Review help text alignment and command ordering to match existing CLI aesthetics. + +## Edge cases and error handling + +`sync status` must return a valid map even when `:ws-url` is missing, with an explicit inactive state rather than throwing. + +`sync start` must keep current behavior where missing `ws-url` or missing graph uuid results in no crash and a deterministic status response. + +`sync grant-access` must surface cloud errors with existing `http-error` path and preserve status code and body context. + +`sync config get auth-token` must redact token values in human output while keeping full value available in JSON and EDN output for scripting. + +`sync config set auth-token ` must write to the config file selected by `--config` (default `~/logseq/cli.edn`) so headless auth survives daemon restarts. + +`sync remote-graphs` must return a deterministic empty list when user has no remote graphs instead of returning nil. + +`sync download` must fail fast when the target local graph is missing required auth or remote graph metadata, and must report a clear sync-specific error code. + +Repo mismatch and lock ownership behavior must remain enforced by db-worker-node and must not be bypassed in CLI command code. + +All new options must keep kebab-case keyword naming and avoid introducing `_` forms. + +## Verification commands + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.command.sync-test` | Sync command unit tests pass with no failures. | +| `bb dev:test -v logseq.cli.commands-test/test-help-output` | Help output includes `sync` group and subcommands. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Worker invoke tests pass including new sync read APIs. | +| `bb dev:test -v logseq.cli.format-test` | Human and structured output tests pass including sync formatters. | +| `bb dev:test -v logseq.cli.integration-test` | CLI integration tests pass for sync status and config flow. | +| `bb dev:lint-and-test` | Full lint and unit suite passes with exit code `0`. | +| `node ./dist/logseq.js sync status --graph demo --output json` | Returns `{"status":"ok","data":...}` with sync status fields. | +| `node ./dist/logseq.js sync remote-graphs --output json` | Returns remote graph list in structured output. | +| `node ./dist/logseq.js sync download --graph demo` | Downloads remote graph snapshot and imports it into local graph data. | +| `node ./dist/logseq.js sync config set auth-token ` | Sets headless auth token for db-sync API calls. | +| `node ./dist/logseq.js sync config get auth-token --output json` | Returns configured token value in structured output. | +| `node ./dist/logseq.js sync config unset auth-token` | Removes configured token and returns success message. | + +## Testing Details + +The new tests verify behavior at parser level, action-building level, transport payload level, worker invoke contract level, output formatting level, and end-to-end CLI invocation level. + +The tests assert external behavior such as command availability, returned status payloads, and worker method invocations, instead of asserting internal helper implementation details. + +## Implementation Details + +- Add new file `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` for sync command ownership. +- Keep `sync` command wiring inside existing dispatch points in `commands.cljs` and do not introduce a second dispatcher. +- Add core worker sync inspection APIs, `get-db-sync-config` and `db-sync-status`, and reuse existing `set-db-sync-config` for config writes. +- Add worker sync APIs for remote graph listing and graph download to support `sync remote-graphs` and `sync download`. +- Reuse `transport/invoke` with existing `direct-pass?` handling and default to transit mode. +- Keep `sync status` output fields stable for scripting, including `repo`, `graph-id`, `ws-state`, and pending counters. +- Keep human output terse and redact auth-token values. +- Update `command.core/top-level-summary` and group-help routing so `sync` behaves like existing command groups. +- Keep all new keyword names kebab-case and avoid shadowed local names such as `bytes`. +- Update `docs/cli/logseq-cli.md` with command list, examples, and expected error hints. +- Run full lint and tests after targeted green passes. + +## Question + +No open question. + +This plan adopts option A and includes `sync config set|get|unset` with `config set auth-token ` as the token setup path. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 725b4b22b1..7f7c908c40 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -90,6 +90,30 @@ Server ownership behavior: - `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. - `server list` human output includes an `OWNER` column, and `server status` / `server list` include owner metadata in structured output (`--output json|edn`). +Sync commands: +- `sync status --graph ` - show db-sync runtime state for a graph daemon +- `sync start --graph ` - start db-sync websocket client for a graph +- `sync stop --graph ` - stop db-sync client on a graph daemon +- `sync upload --graph ` - upload local graph snapshot to remote +- `sync download --graph ` - download remote graph `` into a same-name local graph directory +- `sync remote-graphs [--graph ]` - list remote graphs visible to the current auth context +- `sync ensure-keys [--graph ]` - ensure user RSA keys for sync/e2ee +- `sync grant-access --graph --graph-id --email ` - grant encrypted graph access to a user +- `sync config set [--graph ] ws-url|http-base|auth-token|e2ee-password ` - set db-sync runtime config key +- `sync config get [--graph ] ws-url|http-base|auth-token|e2ee-password` - get db-sync runtime config key +- `sync config unset [--graph ] ws-url|http-base|auth-token|e2ee-password` - remove db-sync runtime config key + +Sync download behavior: +- `sync download` requires `--graph `. +- If a local graph with the same name already exists, the CLI returns `graph-exists`. +- If no remote graph with that name exists, the CLI returns `remote-graph-not-found`. +- For e2ee remote graphs in headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before download. + +Sync config persistence: +- `sync config set/unset` writes to the CLI config file selected by `--config`. +- If `--config` is not provided, the default config path is `~/logseq/cli.edn`. +- `sync config get` reads from that same config source. + Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages - `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 3241ac9a06..0f198be4eb 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -394,6 +394,14 @@ (reset! worker-state/*db-sync-config config) nil) +(def-thread-api :thread-api/get-db-sync-config + [] + @worker-state/*db-sync-config) + +(def-thread-api :thread-api/db-sync-status + [repo] + (db-sync/status repo)) + (def-thread-api :thread-api/db-sync-start [repo] (db-sync/start! repo)) @@ -418,10 +426,40 @@ [] (sync-crypt/ensure-user-rsa-keys!)) +(def-thread-api :thread-api/db-sync-list-remote-graphs + [] + (db-sync/list-remote-graphs!)) + (def-thread-api :thread-api/db-sync-upload-graph [repo] (db-sync/upload-graph! repo)) +(def-thread-api :thread-api/db-sync-download-graph + [repo] + (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph! repo) + row-count (count rows) + _ (when (seq rows) + ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) + repo rows true graph-id remote-tx graph-e2ee?))] + {:repo repo + :graph-id graph-id + :remote-tx remote-tx + :graph-e2ee? graph-e2ee? + :row-count row-count})) + +(def-thread-api :thread-api/db-sync-download-graph-by-id + [repo graph-id graph-e2ee?] + (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph-by-id! repo graph-id graph-e2ee?) + row-count (count rows) + _ (when (seq rows) + ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) + repo rows true graph-id remote-tx graph-e2ee?))] + {:repo repo + :graph-id graph-id + :remote-tx remote-tx + :graph-e2ee? graph-e2ee? + :row-count row-count})) + (def-thread-api :thread-api/set-infer-worker-proxy [infer-worker-proxy] (reset! worker-state/*infer-worker infer-worker-proxy) @@ -987,22 +1025,30 @@ (defn- prev-graph close-db!) - (when graph - (if (= graph prev-graph) - service - (do - (log/info :db-worker/init-service {:graph graph - :prev-graph prev-graph - :import-type (:import-type start-opts)}) - (p/let [service (shared-service/js fns) - #(on-become-master graph start-opts) - broadcast-data-types - {:import? (:import-type? start-opts)})] - (assert (p/promise? (get-in service [:status :ready]))) - (reset! *service [graph service]) - service)))))) + (cond + (nil? graph) + (do + (some-> prev-graph close-db!) + nil) + + (and (= graph prev-graph) service) + service + + :else + (do + (when (and prev-graph (not= graph prev-graph)) + (close-db! prev-graph)) + (log/info :db-worker/init-service {:graph graph + :prev-graph prev-graph + :import-type (:import-type start-opts)}) + (p/let [service (shared-service/js fns) + #(on-become-master graph start-opts) + broadcast-data-types + {:import? (:import-type? start-opts)})] + (assert (p/promise? (get-in service [:status :ready]))) + (reset! *service [graph service]) + service))))) (defn- notify-invalid-data [{:keys [tx-meta]} errors] diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 7bbaf63bec..f84b8e2102 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -129,7 +129,9 @@ (def ^:private non-repo-methods #{:thread-api/init :thread-api/set-db-sync-config + :thread-api/get-db-sync-config :thread-api/db-sync-stop + :thread-api/db-sync-list-remote-graphs :thread-api/db-sync-update-presence :thread-api/db-sync-ensure-user-rsa-keys :thread-api/list-db diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index 9cf77a0e88..64235a62da 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -57,6 +57,22 @@ :remote-tx remote-tx :graph-uuid graph-uuid}))) +(defn status + [repo] + (let [client (current-client repo) + counts (or (sync-counts repo) {}) + ws-url (:ws-url @worker-state/*db-sync-config) + ws-state (or (some-> client :ws-state deref) + (if (seq ws-url) :stopped :inactive))] + {:repo repo + :graph-id (or (:graph-id client) (:graph-uuid counts)) + :ws-state ws-state + :pending-local (or (:pending-local counts) 0) + :pending-asset (or (:pending-asset counts) 0) + :pending-server (or (:pending-server counts) 0) + :local-tx (:local-tx counts) + :remote-tx (:remote-tx counts)})) + (defn- normalize-online-users [users] (->> users @@ -133,7 +149,8 @@ (string/replace base #"/sync/%s$" ""))))) (defn- auth-token [] - (worker-state/get-id-token)) + (or (worker-state/get-id-token) + (:auth-token @worker-state/*db-sync-config))) (defn- id-token-expired? [token] @@ -408,6 +425,102 @@ :url url :body body})))))) +(defn- require-auth-token! + [context] + (when-not (seq (auth-token)) + (fail-fast :db-sync/missing-field (assoc context :field :auth-token)))) + +(defn- ->uint8 + [payload] + (cond + (instance? js/Uint8Array payload) payload + (instance? js/ArrayBuffer payload) (js/Uint8Array. payload) + (string? payload) (.encode text-encoder payload) + :else (js/Uint8Array. payload))) + +(defn- decode-snapshot-rows + [payload] + (sqlite-util/read-transit-str (.decode text-decoder (->uint8 payload)))) + +(defn- frame-len + [^js payload offset] + (let [view (js/DataView. (.-buffer payload) offset 4)] + (.getUint32 view 0 false))) + +(defn- concat-payload + [^js a ^js b] + (cond + (nil? a) b + (nil? b) a + :else + (let [combined (js/Uint8Array. (+ (.-byteLength a) (.-byteLength b)))] + (.set combined a 0) + (.set combined b (.-byteLength a)) + combined))) + +(defn- parse-framed-chunk + [buffer chunk] + (let [payload (concat-payload buffer chunk) + total (.-byteLength payload)] + (loop [offset 0 + rows []] + (if (< (- total offset) 4) + {:rows rows + :buffer (when (< offset total) + (.slice payload offset total))} + (let [len (frame-len payload offset) + next-offset (+ offset 4 len)] + (if (<= next-offset total) + (let [frame-payload (.slice payload (+ offset 4) next-offset) + decoded (decode-snapshot-rows frame-payload)] + (recur next-offset (into rows decoded))) + {:rows rows + :buffer (.slice payload offset total)})))))) + +(defn- finalize-framed-buffer + [buffer] + (if (or (nil? buffer) (zero? (.-byteLength buffer))) + [] + (let [{:keys [rows buffer]} (parse-framed-chunk nil buffer)] + (if (and (seq rows) (or (nil? buffer) (zero? (.-byteLength buffer)))) + rows + (fail-fast :db-sync/incomplete-snapshot-frame + {:rows (count rows) + :remaining-buffer-bytes (some-> buffer .-byteLength)}))))) + +(defn- gzip-payload? + [^js payload] + (and (some? payload) + (>= (.-byteLength payload) 2) + (= 31 (aget payload 0)) + (= 139 (aget payload 1)))) + +(defn- payload->stream + [^js payload] + (js/ReadableStream. + #js {:start (fn [controller] + (.enqueue controller payload) + (.close controller))})) + +(defn- stream payload) + decompressed (.pipeThrough stream (js/DecompressionStream. "gzip")) + resp (js/Response. decompressed) + array-buffer (.arrayBuffer resp)] + (->uint8 array-buffer)) + (p/rejected (ex-info "gzip decompression not supported" + {:type :db-sync/decompression-not-supported})))) + +(defn- uint8 array-buffer)] + (if (gzip-payload? payload) + (js (with-auth-headers {:method "GET"}))) + _ (when-not (.-ok resp) + (fail-fast :db-sync/snapshot-download-failed {:repo repo + :graph-id graph-id + :status (.-status resp)})) + payload ( diff --git a/src/main/frontend/worker/sync/crypt.cljs b/src/main/frontend/worker/sync/crypt.cljs index 2b1d65b31b..d4ed5cec12 100644 --- a/src/main/frontend/worker/sync/crypt.cljs +++ b/src/main/frontend/worker/sync/crypt.cljs @@ -1,7 +1,6 @@ (ns frontend.worker.sync.crypt "E2EE helpers for db-sync." - (:require ["/frontend/idbkv" :as idb-keyval] - [clojure.string :as string] + (:require [clojure.string :as string] [frontend.common.crypt :as crypt] [frontend.common.thread-api :refer [def-thread-api]] [frontend.worker-common.util :as worker-util] @@ -15,7 +14,6 @@ (defonce ^:private *graph->aes-key (atom {})) (defonce ^:private *user-rsa-key-pair-inflight (atom {})) -(defonce ^:private e2ee-store (delay (idb-keyval/newStore "localforage" "keyvaluepairs" 2))) (defonce ^:private e2ee-password-file "e2ee-password") (defonce ^:private native-env? (let [href (try (.. js/self -location -href) @@ -61,7 +59,8 @@ password)) (defn- auth-token [] - (worker-state/get-id-token)) + (or (worker-state/get-id-token) + (:auth-token @worker-state/*db-sync-config))) (defn- auth-headers [] (let [token (auth-token)] @@ -160,8 +159,8 @@ (defn- error ex-message) + data (ex-data error)] + (or (= :thread-api/decrypt-user-e2ee-private-key (:method data)) + (and (string? message) + (string/includes? message "main-thread is not available"))))) + + [] + (seq configured-password) (conj {:source :config + :value configured-password}) + (seq refresh-token) (conj {:source :saved-password + :value refresh-token}) + (and (seq auth-token) + (not= auth-token refresh-token)) + (conj {:source :saved-password + :value auth-token}))] + (letfn [( (p/let [password ( (decrypt-on-main-thread encrypted-private-key) + (p/catch (fn [error] + (if (main-thread-unavailable? error) + (graph repo) :import-type import-type :input input - :allow-missing-graph true}})) + :allow-missing-graph true + :require-missing-graph true}})) (defn execute-graph-list [_action config] diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs new file mode 100644 index 0000000000..8d6aa2602a --- /dev/null +++ b/src/main/logseq/cli/command/sync.cljs @@ -0,0 +1,273 @@ +(ns logseq.cli.command.sync + "Sync-related CLI commands." + (:require [clojure.string :as string] + [logseq.cli.command.core :as core] + [logseq.cli.config :as cli-config] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) + +(def ^:private sync-grant-access-spec + {:graph-id {:desc "Remote graph UUID"} + :email {:desc "Target user email"}}) + +(def entries + [(core/command-entry ["sync" "status"] :sync-status "Show db-sync runtime status" {}) + (core/command-entry ["sync" "start"] :sync-start "Start db-sync client" {}) + (core/command-entry ["sync" "stop"] :sync-stop "Stop db-sync client" {}) + (core/command-entry ["sync" "upload"] :sync-upload "Upload current graph snapshot" {}) + (core/command-entry ["sync" "download"] :sync-download "Download remote graph snapshot" {}) + (core/command-entry ["sync" "remote-graphs"] :sync-remote-graphs "List remote graphs" {}) + (core/command-entry ["sync" "ensure-keys"] :sync-ensure-keys "Ensure user RSA keys for sync/e2ee" {}) + (core/command-entry ["sync" "grant-access"] :sync-grant-access "Grant graph access to an email" sync-grant-access-spec) + (core/command-entry ["sync" "config" "set"] :sync-config-set "Set sync config key" {}) + (core/command-entry ["sync" "config" "get"] :sync-config-get "Get sync config key" {}) + (core/command-entry ["sync" "config" "unset"] :sync-config-unset "Unset sync config key" {})]) + +(def ^:private config-key-map + {"ws-url" :ws-url + "http-base" :http-base + "auth-token" :auth-token + "e2ee-password" :e2ee-password}) + +(defn- missing-repo + [label] + {:ok? false + :error {:code :missing-repo + :message (str "repo is required for " label)}}) + +(defn- invalid-options + [message] + {:ok? false + :error {:code :invalid-options + :message message}}) + +(defn- parse-config-key + [raw-key] + (let [raw-key (some-> raw-key string/trim string/lower-case) + config-key (get config-key-map raw-key)] + (if config-key + {:ok? true + :key config-key} + (invalid-options (str "unknown config key: " raw-key))))) + +(defn build-action + [command options args repo] + (case command + (:sync-status :sync-start :sync-stop :sync-upload) + (if-not (seq repo) + (missing-repo (name command)) + {:ok? true + :action {:type command + :repo repo + :graph (core/repo->graph repo)}}) + + :sync-download + (if-not (seq repo) + (missing-repo (name command)) + {:ok? true + :action {:type :sync-download + :repo repo + :graph (core/repo->graph repo) + :allow-missing-graph true + :require-missing-graph true}}) + + :sync-remote-graphs + {:ok? true + :action {:type :sync-remote-graphs}} + + :sync-ensure-keys + {:ok? true + :action {:type :sync-ensure-keys}} + + :sync-grant-access + (if-not (seq repo) + (missing-repo "sync grant-access") + (let [graph-id (some-> (:graph-id options) string/trim) + email (some-> (:email options) string/trim)] + (cond + (not (seq graph-id)) + (invalid-options "--graph-id is required") + + (not (seq email)) + (invalid-options "--email is required") + + :else + {:ok? true + :action {:type :sync-grant-access + :repo repo + :graph (core/repo->graph repo) + :graph-id graph-id + :email email}}))) + + :sync-config-get + (let [[name] args + key-result (parse-config-key name)] + (if-not (seq (some-> name string/trim)) + (invalid-options "config key is required") + (if-not (:ok? key-result) + key-result + {:ok? true + :action {:type :sync-config-get + :config-key (:key key-result)}}))) + + :sync-config-set + (let [[name value] args + key-result (parse-config-key name)] + (cond + (not (seq (some-> name string/trim))) + (invalid-options "config key is required") + + (not (seq (some-> value str string/trim))) + (invalid-options "config value is required") + + (not (:ok? key-result)) + key-result + + :else + {:ok? true + :action {:type :sync-config-set + :config-key (:key key-result) + :config-value value}})) + + :sync-config-unset + (let [[name] args + key-result (parse-config-key name)] + (cond + (not (seq (some-> name string/trim))) + (invalid-options "config key is required") + + (not (:ok? key-result)) + key-result + + :else + {:ok? true + :action {:type :sync-config-unset + :config-key (:key key-result)}})) + + {:ok? false + :error {:code :unknown-command + :message (str "unknown sync command: " command)}})) + +(defn- invoke-with-repo + [config repo method args] + (let [sync-config {:ws-url (:ws-url config) + :http-base (:http-base config) + :auth-token (:auth-token config) + :e2ee-password (:e2ee-password config)}] + (p/let [cfg (cli-server/ensure-server! config repo) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config]) + result (transport/invoke cfg method false args)] + result))) + +(defn- invoke-global + [config method args] + (let [base-url (:base-url config) + sync-config {:ws-url (:ws-url config) + :http-base (:http-base config) + :auth-token (:auth-token config) + :e2ee-password (:e2ee-password config)}] + (if (seq base-url) + (p/let [_ (transport/invoke config :thread-api/set-db-sync-config false [sync-config])] + (transport/invoke config method false args)) + (p/let [repo (or (core/resolve-repo (:graph config)) + (p/let [graphs (cli-server/list-graphs config)] + (some-> graphs first core/resolve-repo))) + cfg (if (seq repo) + (cli-server/ensure-server! config repo) + (p/rejected (ex-info "graph name is required" + {:code :missing-graph}))) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config])] + (transport/invoke cfg method false args))))) + +(defn execute + [action config] + (case (:type action) + :sync-status + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-status + [(:repo action)])] + {:status :ok + :data result}) + + :sync-start + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-start + [(:repo action)])] + {:status :ok + :data {:result result}}) + + :sync-stop + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-stop + [])] + {:status :ok + :data {:result result}}) + + :sync-upload + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-upload-graph + [(:repo action)])] + {:status :ok + :data (if (map? result) + result + {:result result})}) + + :sync-download + (p/let [remote-graphs (invoke-global config + :thread-api/db-sync-list-remote-graphs + []) + remote-graph (some (fn [graph] + (when (= (:graph action) (:graph-name graph)) + graph)) + remote-graphs)] + (if-not remote-graph + {:status :error + :error {:code :remote-graph-not-found + :message (str "remote graph not found: " (:graph action)) + :graph (:graph action)}} + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-download-graph-by-id + [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] + {:status :ok + :data (if (map? result) + result + {:result result})}))) + + :sync-remote-graphs + (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] + {:status :ok + :data {:graphs (or graphs [])}}) + + :sync-ensure-keys + (p/let [result (invoke-global config :thread-api/db-sync-ensure-user-rsa-keys [])] + {:status :ok + :data {:result result}}) + + :sync-grant-access + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-grant-graph-access + [(:repo action) (:graph-id action) (:email action)])] + {:status :ok + :data {:result result}}) + + :sync-config-get + (p/let [current config] + {:status :ok + :data {:key (:config-key action) + :value (get (or current {}) (:config-key action))}}) + + :sync-config-set + (p/let [_ (cli-config/update-config! config {(:config-key action) (:config-value action)})] + {:status :ok + :data {:key (:config-key action) + :value (:config-value action)}}) + + :sync-config-unset + (p/let [_ (cli-config/update-config! config {(:config-key action) nil})] + {:status :ok + :data {:key (:config-key action)}}) + + (p/resolved {:status :error + :error {:code :unknown-action + :message "unknown sync action"}}))) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index c427971775..70c1b0dd58 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -10,6 +10,7 @@ [logseq.cli.command.remove :as remove-command] [logseq.cli.command.server :as server-command] [logseq.cli.command.show :as show-command] + [logseq.cli.command.sync :as sync-command] [logseq.cli.command.upsert :as upsert-command] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -101,7 +102,8 @@ remove-command/entries query-command/entries show-command/entries - doctor-command/entries))) + doctor-command/entries + sync-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -267,6 +269,10 @@ (not (seq (:graph opts)))) (missing-graph-result summary) + (and (= command :sync-download) + (not (seq (:graph opts)))) + (missing-graph-result summary) + :else (command-core/ok-result command opts args summary)))) @@ -284,7 +290,7 @@ (empty? args) (command-core/help-result summary) - (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query"} (first args))) + (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query" "sync"} (first args))) (command-core/help-result (command-core/group-summary (first args) table)) :else @@ -328,7 +334,9 @@ (defn- ensure-missing-graph [action config] - (if (and (= :graph-import (:type action)) (:repo action)) + (if (and (:repo action) + (or (:require-missing-graph action) + (= :graph-import (:type action)))) (p/let [graphs (cli-server/list-graphs config) graph (command-core/repo->graph (:repo action))] (if (some #(= graph %) graphs) @@ -406,6 +414,11 @@ :doctor (doctor-command/build-action options) + (:sync-status :sync-start :sync-stop :sync-upload :sync-download + :sync-remote-graphs :sync-ensure-keys :sync-grant-access + :sync-config-set :sync-config-get :sync-config-unset) + (sync-command/build-action command options args repo) + {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) @@ -451,6 +464,10 @@ :server-start (server-command/execute-start action config) :server-stop (server-command/execute-stop action config) :server-restart (server-command/execute-restart action config) + (:sync-status :sync-start :sync-stop :sync-upload :sync-download + :sync-remote-graphs :sync-ensure-keys :sync-grant-access + :sync-config-set :sync-config-get :sync-config-unset) + (sync-command/execute action config) {:status :error :error {:code :unknown-action :message "unknown action"}}))] @@ -460,4 +477,5 @@ :schema :source :target :update-tags :update-properties :remove-tags :remove-properties - :export-type :file :import-type :input]))))) + :export-type :file :import-type :input + :graph-id :email :config-key :config-value]))))) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index f27152b557..e050e113f3 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -45,9 +45,16 @@ [{:keys [config-path]} updates] (let [path (or config-path (default-config-path)) current (or (read-config-file path) {}) - filtered-current (dissoc current :auth-token :retries) - filtered-updates (dissoc (or updates {}) :auth-token :retries) - next (merge filtered-current filtered-updates)] + filtered-current (dissoc current :retries) + filtered-updates (dissoc (or updates {}) :retries) + nil-keys (->> filtered-updates + (keep (fn [[k v]] + (when (nil? v) + k)))) + merged (merge filtered-current filtered-updates) + next (if (seq nil-keys) + (apply dissoc merged nil-keys) + merged)] (ensure-config-dir! path) (.writeFileSync fs path (pr-str next)) next)) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index fb8ffd0da4..14dd44e5e0 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -280,6 +280,60 @@ (cond-> [(str "Server " (name status) ": " repo)] (and host port) (conj (str "Host: " host " Port: " port)))))) +(def ^:private redacted-token "[REDACTED]") + +(defn- format-sync-status + [{:keys [repo graph-id ws-state pending-local pending-asset pending-server local-tx remote-tx]}] + (string/join "\n" + [(str "Sync status") + (str "repo: " (or repo "-")) + (str "graph-id: " (or graph-id "-")) + (str "ws-state: " (or ws-state :unknown)) + (str "pending-local: " (or pending-local 0)) + (str "pending-asset: " (or pending-asset 0)) + (str "pending-server: " (or pending-server 0)) + (str "local-tx: " (or local-tx "-")) + (str "remote-tx: " (or remote-tx "-"))])) + +(defn- format-sync-remote-graphs + [graphs] + (format-counted-table + ["GRAPH-ID" "GRAPH-NAME" "ROLE" "E2EE"] + (mapv (fn [{:keys [graph-id graph-name role graph-e2ee?]}] + [graph-id + graph-name + (or role "-") + (if (nil? graph-e2ee?) + "-" + (if graph-e2ee? "true" "false"))]) + (or graphs [])))) + +(defn- format-sync-action + [command {:keys [repo email]}] + (case command + :sync-start (str "Sync started: " repo) + :sync-stop (str "Sync stopped: " repo) + :sync-upload (str "Sync upload requested: " repo) + :sync-download (str "Sync download requested: " repo) + :sync-ensure-keys "Sync keys ensured" + :sync-grant-access (str "Sync access granted: " email " (repo: " repo ")") + "Sync updated")) + +(defn- format-sync-config-get + [{:keys [key value]}] + (let [display-value (if (contains? #{:auth-token :e2ee-password} key) + redacted-token + (if (some? value) value "-"))] + (str "sync config " (name key) ": " display-value))) + +(defn- format-sync-config-set + [{:keys [key]}] + (str "sync config set: " (name key))) + +(defn- format-sync-config-unset + [{:keys [key]}] + (str "sync config unset: " (name key))) + (defn- format-upsert-block [{:keys [repo source target update-tags update-properties remove-tags remove-properties]} result] (if (vector? result) @@ -375,6 +429,13 @@ :server-status (format-server-status data) (:server-start :server-stop :server-restart) (format-server-action command data) + :sync-status (format-sync-status data) + :sync-remote-graphs (format-sync-remote-graphs (:graphs data)) + (:sync-start :sync-stop :sync-upload :sync-download :sync-ensure-keys :sync-grant-access) + (format-sync-action command context) + :sync-config-get (format-sync-config-get data) + :sync-config-set (format-sync-config-set data) + :sync-config-unset (format-sync-config-unset data) :list-page (format-list-page (:items data) now-ms) :list-tag (format-list-tag (:items data) now-ms) :list-property (format-list-property (:items data) now-ms) diff --git a/src/test/frontend/worker/db_core_test.cljs b/src/test/frontend/worker/db_core_test.cljs index bb1462bb81..e1d616adfd 100644 --- a/src/test/frontend/worker/db_core_test.cljs +++ b/src/test/frontend/worker/db_core_test.cljs @@ -1,7 +1,9 @@ (ns frontend.worker.db-core-test - (:require [cljs.test :refer [deftest is]] + (:require [cljs.test :refer [async deftest is]] [frontend.common.thread-api :as thread-api] - [frontend.worker.db-core])) + [frontend.worker.db-core :as db-core] + [frontend.worker.shared-service :as shared-service] + [promesa.core :as p])) (deftest db-core-registers-db-sync-thread-apis (let [api-map @thread-api/*thread-apis] @@ -14,3 +16,29 @@ (is (contains? api-map :thread-api/db-sync-ensure-user-rsa-keys)) (is (contains? api-map :thread-api/db-sync-upload-graph)) (is (contains? api-map :thread-api/db-sync-import-kvs-rows)))) + +(deftest init-service-does-not-close-db-when-graph-unchanged + (async done + (let [service {:status {:ready (p/resolved true)} + :proxy #js {}} + close-calls (atom []) + create-calls (atom 0) + *service @#'db-core/*service + old-service @*service] + (reset! *service ["graph-a" service]) + (with-redefs [db-core/close-db! (fn [repo] + (swap! close-calls conj repo) + nil) + shared-service/ (#'db-core/ (stop!) (p/finally (fn [] (done)))) (done)))))))) +(deftest db-worker-node-sync-status-requires-repo-and-returns-structured-status + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-sync-status") + repo (str "logseq_db_sync_status_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + {:keys [status body]} (invoke-raw host port "thread-api/db-sync-status" []) + parsed (js->clj (js/JSON.parse body) :keywordize-keys true) + _ (is (= 400 status)) + _ (is (= false (:ok parsed))) + _ (is (= "missing-repo" (get-in parsed [:error :code]))) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + status-result (invoke host port "thread-api/db-sync-status" [repo])] + (is (= repo (:repo status-result))) + (is (contains? status-result :ws-state)) + (is (contains? status-result :pending-local)) + (is (contains? status-result :pending-asset)) + (is (contains? status-result :pending-server)) + (is (contains? status-result :local-tx)) + (is (contains? status-result :remote-tx)) + (is (contains? status-result :graph-id))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) diff --git a/src/test/frontend/worker/sync/crypt_test.cljs b/src/test/frontend/worker/sync/crypt_test.cljs index 611306fb04..f3e3c04820 100644 --- a/src/test/frontend/worker/sync/crypt_test.cljs +++ b/src/test/frontend/worker/sync/crypt_test.cljs @@ -166,6 +166,89 @@ (set! js/fetch fetch-prev) (done))))))) +(deftest fetch-graph-aes-key-for-download-uses-platform-kv-clear-test + (async done + (let [fetch-prev js/fetch + graph-id (str (random-uuid)) + expected-key (str "rtc-encrypted-aes-key###" graph-id) + platform-map {:runtime :test} + kv-set-calls (atom [])] + (set! js/fetch + (fn [url _opts] + (cond + (string/includes? url "/e2ee/user-keys") + (js/Promise.resolve + #js {:ok true + :text (fn [] + (js/Promise.resolve + "{\"public-key\":\"public-key\",\"encrypted-private-key\":\"encrypted-private-key\"}"))}) + + (string/includes? url (str "/e2ee/graphs/" graph-id "/aes-key")) + (js/Promise.resolve + #js {:ok true + :text (fn [] + (js/Promise.resolve + "{\"encrypted-aes-key\":\"remote-encrypted\"}"))}) + + :else + (js/Promise.resolve + #js {:ok false + :status 404 + :text (fn [] (js/Promise.resolve "{\"message\":\"not-found\"}"))})))) + (-> (p/with-redefs [sync-crypt/e2ee-base (fn [] "https://example.com") + worker-state/get-id-token (fn [] "token") + worker-state/ (p/with-redefs [ldb/read-transit-str (fn [_] :encrypted-private-key) + worker-state/ (p/let [_ (sync-command/execute {:type :sync-start + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-start false ["logseq_db_demo"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-stop + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-stop + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-stop false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-upload + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-upload-graph false ["logseq_db_demo"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []] + [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download-uses-graph-config-when-base-url-missing + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? false}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:graph "demo" + :data-dir "/tmp"})] + (is (= [[{:graph "demo" + :data-dir "/tmp"} + "logseq_db_demo"] + [{:graph "demo" + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []] + [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-download-remote-graph-not-found + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "other-id" + :graph-name "other-graph"}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :remote-graph-not-found (get-in result [:error :code]))) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-remote-graphs + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved []))) + (-> (p/let [_ (sync-command/execute {:type :sync-remote-graphs} + {:base-url "http://example" + :http-base "https://sync.example.com" + :ws-url "wss://sync.example.com/sync/%s" + :auth-token "test-token" + :e2ee-password "pw" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url "wss://sync.example.com/sync/%s" + :http-base "https://sync.example.com" + :auth-token "test-token" + :e2ee-password "pw"}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-ensure-keys + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-ensure-keys} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-ensure-user-rsa-keys false []]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-grant-access + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-grant-access + :repo "logseq_db_demo" + :graph-id "graph-uuid" + :email "user@example.com"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-config-get + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + ensure-calls (atom []) + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-get + :config-key :auth-token} + {:base-url "http://example" + :auth-token "abc" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-config-set + (async done + (let [orig-invoke transport/invoke + orig-update-config! cli-config/update-config! + invoke-calls (atom []) + update-calls (atom [])] + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil)) + ) + (set! cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (merge {:ws-url "wss://old.example/sync/%s"} updates))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-set + :config-key :auth-token + :config-value "token-value"} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token "token-value"}]] + @update-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (set! cli-config/update-config! orig-update-config!) + (done))))))) + +(deftest test-execute-sync-config-unset + (async done + (let [orig-invoke transport/invoke + orig-update-config! cli-config/update-config! + invoke-calls (atom []) + update-calls (atom [])] + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil))) + (set! cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (dissoc {:ws-url "wss://old.example/sync/%s" + :auth-token "token-value"} + :auth-token))) + (-> (p/let [_ (sync-command/execute {:type :sync-config-unset + :config-key :auth-token} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token nil}]] + @update-calls)) + (is (= [] @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! transport/invoke orig-invoke) + (set! cli-config/update-config! orig-update-config!) + (done))))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index aee1a906c0..44eacce467 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -66,6 +66,7 @@ (is (string/includes? plain-summary "doctor")) (is (string/includes? plain-summary "graph")) (is (string/includes? plain-summary "server")) + (is (string/includes? plain-summary "sync")) (is (string/includes? plain-summary "Path to db-worker data dir (default ~/logseq/graphs)")) (is (contains-bold? summary "list page")) (is (contains-bold? summary "list tag")) @@ -86,6 +87,8 @@ (is (contains-bold? summary "graph create")) (is (contains-bold? summary "server list")) (is (contains-bold? summary "server start")) + (is (contains-bold? summary "sync status")) + (is (contains-bold? summary "sync start")) (is (contains-bold? summary "--help")) (is (contains-bold? summary "--graph")) (is (re-find #"\u001b\[[0-9;]*mCommands\u001b\[[0-9;]*m:" summary)) @@ -205,6 +208,28 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-help-sync-group + (testing "sync group shows subcommands" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["sync"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "sync status")) + (is (string/includes? plain-summary "sync start")) + (is (string/includes? plain-summary "sync stop")) + (is (string/includes? plain-summary "sync upload")) + (is (string/includes? plain-summary "sync download")) + (is (string/includes? plain-summary "sync remote-graphs")) + (is (string/includes? plain-summary "sync ensure-keys")) + (is (string/includes? plain-summary "sync grant-access")) + (is (string/includes? plain-summary "sync config set")) + (is (string/includes? plain-summary "sync config get")) + (is (string/includes? plain-summary "sync config unset")) + (is (contains-bold? summary "sync status")) + (is (contains-bold? summary "sync config set")) + (is (contains-bold? summary "sync grant-access"))))) + (deftest test-parse-args-help-upsert-group (testing "add group is removed" (let [result (commands/parse-args ["add"])] @@ -1217,6 +1242,11 @@ (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) + (testing "sync download requires graph" + (let [result (commands/parse-args ["sync" "download"])] + (is (false? (:ok? result))) + (is (= :missing-graph (get-in result [:error :code]))))) + (testing "graph import rejects unknown type" (let [result (commands/parse-args ["graph" "import" "--type" "zip" @@ -2460,6 +2490,28 @@ (set! cli-server/ensure-server! orig-ensure-server!) (done))))))) +(deftest test-execute-sync-download-rejects-existing-graph + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-ensure-server! cli-server/ensure-server!] + (set! cli-server/list-graphs (fn [_] ["demo"])) + (set! cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))) + (-> (p/let [result (commands/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :allow-missing-graph true + :require-missing-graph true} + {})] + (is (= :error (:status result))) + (is (= :graph-exists (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! cli-server/ensure-server! orig-ensure-server!) + (done))))))) + (deftest test-execute-graph-export (async done (let [invoke-calls (atom []) diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 6c0e91623b..1c461500bf 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -104,5 +104,16 @@ contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] (is (= "new" (:graph parsed))) - (is (not (contains? parsed :auth-token))) + (is (= "secret" (:auth-token parsed))) (is (not (contains? parsed :retries))))) + +(deftest test-update-config-removes-nil-values + (let [dir (node-helper/create-tmp-dir "cli") + cfg-path (node-path/join dir "cli.edn") + _ (fs/writeFileSync cfg-path "{:graph \"old\" :auth-token \"secret\"}") + _ (config/update-config! {:config-path cfg-path} + {:auth-token nil}) + contents (.toString (fs/readFileSync cfg-path) "utf8") + parsed (reader/read-string contents)] + (is (= "old" (:graph parsed))) + (is (not (contains? parsed :auth-token))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 6189b2d7ef..38b3fb6d71 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -235,6 +235,96 @@ {:output-format nil})] (is (= "Imported sqlite from /tmp/import.sqlite" result))))) +(deftest test-human-output-sync-status + (testing "sync status renders runtime and queue fields" + (let [result (format/format-result {:status :ok + :command :sync-status + :data {:repo "demo-graph" + :graph-id "graph-uuid" + :ws-state :open + :pending-local 2 + :pending-asset 1 + :pending-server 3 + :local-tx 10 + :remote-tx 13}} + {:output-format nil})] + (is (string/includes? result "Sync status")) + (is (string/includes? result "demo-graph")) + (is (string/includes? result "graph-uuid")) + (is (string/includes? result "pending-local")) + (is (string/includes? result "pending-asset")) + (is (string/includes? result "pending-server"))))) + +(deftest test-human-output-sync-remote-graphs + (testing "sync remote-graphs renders a table" + (let [result (format/format-result {:status :ok + :command :sync-remote-graphs + :data {:graphs [{:graph-id "graph-1" + :graph-name "Alpha" + :role "manager" + :graph-e2ee? true} + {:graph-id "graph-2" + :graph-name "Beta" + :role "member" + :graph-e2ee? false}]}} + {:output-format nil})] + (is (string/includes? result "GRAPH-ID")) + (is (string/includes? result "GRAPH-NAME")) + (is (string/includes? result "ROLE")) + (is (string/includes? result "Alpha")) + (is (string/includes? result "Beta")) + (is (string/includes? result "Count: 2"))))) + +(deftest test-human-output-sync-actions + (testing "sync start renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-start + :context {:repo "demo-graph"}} + {:output-format nil})] + (is (string/includes? result "Sync started")) + (is (string/includes? result "demo-graph")))) + + (testing "sync upload renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-upload + :context {:repo "demo-graph"} + :data {:graph-id "graph-uuid"}} + {:output-format nil})] + (is (string/includes? result "Sync upload")) + (is (string/includes? result "demo-graph")))) + + (testing "sync download renders a succinct success line" + (let [result (format/format-result {:status :ok + :command :sync-download + :context {:repo "demo-graph"}} + {:output-format nil})] + (is (string/includes? result "Sync download")) + (is (string/includes? result "demo-graph"))))) + +(deftest test-human-output-sync-config-get-token-redaction + (testing "sync config get auth-token redacts value in human output" + (let [token "super-secret-token-value" + result (format/format-result {:status :ok + :command :sync-config-get + :data {:key :auth-token + :value token}} + {:output-format nil})] + (is (string/includes? result "auth-token")) + (is (string/includes? result "[REDACTED]")) + (is (not (string/includes? result token)))))) + +(deftest test-human-output-sync-config-get-e2ee-password-redaction + (testing "sync config get e2ee-password redacts value in human output" + (let [password "super-secret-password" + result (format/format-result {:status :ok + :command :sync-config-get + :data {:key :e2ee-password + :value password}} + {:output-format nil})] + (is (string/includes? result "e2ee-password")) + (is (string/includes? result "[REDACTED]")) + (is (not (string/includes? result password)))))) + (deftest test-human-output-graph-info (testing "graph info includes key metadata lines" (let [result (format/format-result {:status :ok From d74d6b63162230b780012b610b82d97d6859c229 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 5 Mar 2026 08:49:53 -0500 Subject: [PATCH 108/375] 048-sync-download-start-reliability.md --- .../048-sync-download-start-reliability.md | 179 ++++++++++++++ src/main/frontend/common/crypt.cljs | 3 +- src/main/frontend/worker/db_core.cljs | 25 +- src/main/frontend/worker/db_worker_node.cljs | 3 +- src/main/frontend/worker/platform/node.cljs | 47 +++- src/main/frontend/worker/state.cljs | 24 +- src/main/frontend/worker/sync.cljs | 204 +++++++++++----- src/main/frontend/worker/sync/crypt.cljs | 26 +- src/main/logseq/cli/command/sync.cljs | 132 +++++++++-- src/main/logseq/cli/format.cljs | 28 ++- src/test/frontend/worker/db_sync_test.cljs | 222 ++++++++++++++++++ .../frontend/worker/db_worker_node_test.cljs | 28 +++ .../frontend/worker/platform_node_test.cljs | 32 +++ src/test/frontend/worker/state_test.cljs | 32 +++ src/test/frontend/worker/sync/crypt_test.cljs | 69 +++++- src/test/logseq/cli/command/sync_test.cljs | 168 ++++++++++++- src/test/logseq/cli/format_test.cljs | 19 +- src/test/logseq/cli/integration_test.cljs | 75 ++++++ 18 files changed, 1185 insertions(+), 131 deletions(-) create mode 100644 docs/agent-guide/048-sync-download-start-reliability.md create mode 100644 src/test/frontend/worker/state_test.cljs diff --git a/docs/agent-guide/048-sync-download-start-reliability.md b/docs/agent-guide/048-sync-download-start-reliability.md new file mode 100644 index 0000000000..311fe77e04 --- /dev/null +++ b/docs/agent-guide/048-sync-download-start-reliability.md @@ -0,0 +1,179 @@ +# Sync Download And Sync Start Reliability Implementation Plan + +Goal: Validate and fix `logseq sync download` plus `logseq sync start` so download data is complete and start reaches a working sync session. + +Architecture: I will harden behavior with failing tests first at CLI, db-worker-node invoke boundary, and worker sync core, then apply minimal fixes in `logseq.cli.command.sync`, `frontend.worker.db_core`, and `frontend.worker.sync`. +Architecture: The start path will stop reporting false success by checking runtime status after start, and the download path will fail fast on incomplete or inconsistent snapshot import conditions. +Architecture: Manual verification will use a local untracked `cli.edn` built from the exact config block provided in this task. + +Tech Stack: ClojureScript, babashka test runner, db-worker-node daemon, frontend worker thread APIs, websocket plus HTTP sync APIs. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`, and relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md` and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +`/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` currently reports `sync start` success after invoking `:thread-api/db-sync-start`, but it does not verify the websocket state becomes `:open`. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` skips start when config or graph-id is missing, and the CLI currently does not distinguish this from a successful start. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` enforces frame integrity in `finalize-framed-buffer`, but the current tests do not cover incomplete snapshot frame failure through the full download flow. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` imports downloaded rows and returns a summary map, but there is no end-to-end CLI integration test that confirms downloaded graph data is usable and consistent after import. + +This plan uses @planning-documents for naming, @writing-plans for granularity, @test-driven-development for sequence, and @clojure-debug when failures are not immediately obvious. + +## Testing Plan + +I will add CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` for start readiness polling, start timeout behavior, and download error propagation. + +I will add worker sync tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for incomplete framed snapshot behavior and download completeness invariants. + +I will add worker daemon invoke tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` to verify start and status interactions remain stable through `/v1/invoke`. + +I will add CLI integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` to verify `sync download` yields queryable graph data and `sync start` reaches `:open` within timeout in a deterministic mocked sync environment. + +I will run targeted tests after each behavior slice and then run `bb dev:lint-and-test` for regression safety. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current integration map + +```text +logseq sync download/start + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs + -> remote sync HTTP + websocket endpoints +``` + +## Manual test configuration + +Create `/tmp/logseq-sync-cli.edn` with the exact EDN block provided in this task request. + +Keep this file local and untracked, and never commit the auth token to the repository. + +Use `--config /tmp/logseq-sync-cli.edn` for every manual `logseq sync` command in this plan. + +## Implementation plan + +### Phase 0. Baseline and reproducibility. + +1. Create `/tmp/logseq-sync-cli.edn` from the provided config block. +2. Run `bb dev:test -v logseq.cli.command.sync-test` to capture current CLI sync baseline. +3. Run `bb dev:test -v frontend.worker.db-sync-test` to capture current worker sync baseline. +4. Run `bb dev:test -v frontend.worker.db-worker-node-test` to capture current daemon invoke baseline. +5. Run `node ./dist/logseq.js sync remote-graphs --config /tmp/logseq-sync-cli.edn --output json` and record the graph-id and graph-name used for manual verification. + +### Phase 1. RED tests for sync start readiness semantics. + +6. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync start` polls `:thread-api/db-sync-status` and returns success only after `:ws-state` becomes `:open`. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync start` returns `:error` with diagnostic status when `:ws-state` never reaches `:open` before timeout. +8. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that start skipped conditions like missing `:ws-url` are surfaced as actionable CLI errors instead of success. +9. Run `bb dev:test -v logseq.cli.command.sync-test/test-execute-sync-start` and verify RED failures are behavior failures. + +### Phase 2. RED tests for download completeness and failure propagation. + +10. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that incomplete snapshot frames trigger `:db-sync/incomplete-snapshot-frame` behavior through download framing helpers. +11. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that a valid download payload produces stable row batches that can be imported without truncation. +12. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync download` returns `:error` and preserves error code when `:thread-api/db-sync-download-graph-by-id` fails. +13. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that downloaded data is queryable after import and includes expected graph metadata. +14. Run `bb dev:test -v frontend.worker.db-sync-test` and `bb dev:test -v logseq.cli.integration-test` and verify RED failures are behavior failures. + +### Phase 3. GREEN implementation for sync start observability. + +15. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` to add a bounded wait helper that polls `:thread-api/db-sync-status` after `:thread-api/db-sync-start`. +16. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so `:sync-start` returns `{:status :ok}` only when status shows `:ws-state :open`. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so timeout or skipped start returns `{:status :error}` with `:repo`, current `:ws-state`, and remediation hint. +18. If needed, update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` status payload to include one extra diagnostic field for recent start failures. +19. Re-run `bb dev:test -v logseq.cli.command.sync-test` until all start-related tests pass. + +### Phase 4. GREEN implementation for download completeness guarantees. + +20. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to keep strict framed payload validation and expose structured failure details used by CLI. +21. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` download/import path to assert required result fields before import and fail fast on invalid payload. +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` download branch to preserve worker error code and context in CLI output. +23. Re-run `bb dev:test -v frontend.worker.db-sync-test` and `bb dev:test -v logseq.cli.command.sync-test` until download-related tests pass. + +### Phase 5. REFACTOR and regression safety. + +24. Refactor only duplicated sync command helper code in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` without changing behavior. +25. Re-run `bb dev:test -v logseq.cli.command.sync-test` after refactor. +26. Re-run `bb dev:test -v frontend.worker.db-sync-test` after refactor. +27. Re-run `bb dev:test -v frontend.worker.db-worker-node-test` after refactor. + +### Phase 6. Manual end-to-end verification with provided cli.edn. + +28. Run `node ./dist/logseq.js sync download --graph --config /tmp/logseq-sync-cli.edn --output json` and record `graph-id`, `remote-tx`, and `row-count`. +29. Run `node ./dist/logseq.js q --graph --config /tmp/logseq-sync-cli.edn --output json '[:find (count ?e) :where [?e :block/uuid]]'` and confirm graph has non-zero entities. +30. Run `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` and capture pre-start `ws-state`. +31. Run `node ./dist/logseq.js sync start --graph --config /tmp/logseq-sync-cli.edn --output json` and verify command succeeds only when status reaches `:open`. +32. Run `node ./dist/logseq.js upsert block --graph --title "sync smoke $(date +%s)" --config /tmp/logseq-sync-cli.edn --output json` to create a local change. +33. Run `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` repeatedly until `pending-local` trends back to `0`. +34. If step 31 or step 33 fails, inspect db-worker-node log file under the graph repo directory and capture exact error code plus timestamp. + +### Phase 7. Final verification. + +35. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and ensure full suite is green. +36. Re-run manual step 31 one more time to verify no flakiness after full test run. +37. Document final behavior and known limitations in the PR summary. + +## Edge cases to cover explicitly + +`sync start` must return an actionable error when `:ws-url` is missing or empty. + +`sync start` must avoid false success when token exists but handshake never reaches `:open`. + +Repeated `sync start` calls must be idempotent and not create duplicate clients. + +`sync download` must fail clearly when remote graph is not found and must include graph-name in the error. + +`sync download` must fail clearly on incomplete framed snapshot payload. + +`sync download` must not leave a half-imported local graph when import throws mid-batch. + +`sync download` must preserve and report `graph-e2ee?` and remote graph-id in result payload. + +## Verification command matrix + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.command.sync-test` | Sync CLI tests pass, including start readiness and download error propagation. | +| `bb dev:test -v frontend.worker.db-sync-test` | Worker sync framing and download completeness tests pass. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Daemon invoke and status behavior remains green. | +| `bb dev:test -v logseq.cli.integration-test` | CLI integration verifies queryable data after download and start readiness path. | +| `node ./dist/logseq.js sync download --graph --config /tmp/logseq-sync-cli.edn --output json` | Returns `status: ok` with stable `graph-id`, `remote-tx`, and non-negative `row-count`. | +| `node ./dist/logseq.js sync start --graph --config /tmp/logseq-sync-cli.edn --output json` | Returns `status: ok` only when worker status reaches `ws-state: open`. | +| `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` | Shows non-negative queue counters and stable repo graph identifiers. | +| `bb dev:lint-and-test` | Full lint and unit suite passes with exit code `0`. | + +## Testing Details + +The test suite additions validate behavior from user-visible CLI result down to worker framed snapshot handling and daemon invoke boundary. + +The tests verify externally observable behavior such as command status, error code, websocket state progression, and queryable post-download graph data. + +The tests avoid checking private implementation details and focus on functional outcomes. + +## Implementation Details + +- Keep `sync` command API surface unchanged and improve readiness validation inside existing `:sync-start` execution path. +- Keep download protocol unchanged and improve payload validation plus error propagation semantics. +- Reuse `:thread-api/db-sync-status` as the source of readiness truth for CLI start. +- Keep all keyword names kebab-case and avoid introducing underscore keys. +- Ensure any new helper names avoid shadowed locals like `bytes`. +- Keep auth token handling in local config and never persist token values into repository files. +- 在 db-worker-node 中，如果是 desktop app 场景启动，token 读写保持原有代码路径。 +- 在 db-worker-node 中，如果是 cli 场景启动，读取 token 一律从 `cli.edn`（sync config）中获取。 +- 在 db-worker-node 中，如果是 cli 场景启动，写入 token 也走 `cli.edn`（sync config）路径。 +- Preserve existing db-worker-node repo lock and method access rules. +- Keep manual verification commands deterministic by pinning `--config /tmp/logseq-sync-cli.edn`. +- Use a fixed `sync start` wait timeout of `10000` ms with no CLI override option. + +## Question + +No open question. + +--- diff --git a/src/main/frontend/common/crypt.cljs b/src/main/frontend/common/crypt.cljs index a3f4c78a43..1d0e74549d 100644 --- a/src/main/frontend/common/crypt.cljs +++ b/src/main/frontend/common/crypt.cljs @@ -175,7 +175,8 @@ "Decrypts an AES key with a private key." [private-key encrypted-aes-key-data] (assert (and (instance? js/CryptoKey private-key) - (instance? js/Uint8Array encrypted-aes-key-data))) + (instance? js/Uint8Array encrypted-aes-key-data)) + [private-key encrypted-aes-key-data]) (-> (p/let [encrypted-aes-key (js/Uint8Array. encrypted-aes-key-data) decrypted-key-data (.decrypt subtle diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 0f198be4eb..2d98d74522 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -434,9 +434,27 @@ [repo] (db-sync/upload-graph! repo)) +(defn- validate-sync-download-result! + [repo {:keys [rows graph-id remote-tx] :as result}] + (when-not (sequential? rows) + (db-sync/fail-fast :db-sync/invalid-field {:repo repo + :field :rows + :value rows})) + (when-not (and (string? graph-id) (seq graph-id)) + (db-sync/fail-fast :db-sync/invalid-field {:repo repo + :field :graph-id + :value graph-id})) + (when-not (integer? remote-tx) + (db-sync/fail-fast :db-sync/invalid-field {:repo repo + :field :remote-tx + :value remote-tx})) + result) + (def-thread-api :thread-api/db-sync-download-graph [repo] - (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph! repo) + (p/let [download-result (db-sync/download-graph! repo) + {:keys [rows graph-id remote-tx graph-e2ee?]} + (validate-sync-download-result! repo download-result) row-count (count rows) _ (when (seq rows) ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) @@ -449,7 +467,9 @@ (def-thread-api :thread-api/db-sync-download-graph-by-id [repo graph-id graph-e2ee?] - (p/let [{:keys [rows graph-id remote-tx graph-e2ee?]} (db-sync/download-graph-by-id! repo graph-id graph-e2ee?) + (p/let [download-result (db-sync/download-graph-by-id! repo graph-id graph-e2ee?) + {:keys [rows graph-id remote-tx graph-e2ee?]} + (validate-sync-download-result! repo download-result) row-count (count rows) _ (when (seq rows) ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) @@ -656,6 +676,7 @@ :graph-uuid graph-id :message "Graph is ready!"}) ((@thread-api/*thread-apis :thread-api/export-db) repo) + (client-op/update-graph-uuid repo graph-id) (client-op/update-local-tx repo remote-tx) (shared-service/broadcast-to-clients! :add-repo {:repo repo})) (p/catch (fn [error] diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index f84b8e2102..58338f3b6d 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -369,7 +369,8 @@ (db-lock/assert-lock-owner! path lock))) platform (platform-node/node-platform {:data-dir data-dir :event-fn handle-event! - :write-guard-fn write-guard-fn}) + :write-guard-fn write-guard-fn + :owner-source owner-source}) proxy (db-core/init-core! platform) _ (clj (js/Array.from value))))}})) + +(def ^:private kv-transit-reader + (transit/reader + :json + {:handlers + {"uint8array" + (fn [value] + (js/Uint8Array. (clj->js value)))}})) + +(defn- parse-kv-state + [contents] + (try + (let [state (transit/read kv-transit-reader contents)] + (if (map? state) + state + {})) + (catch :default error + (log/warn :db-worker-node-kv-parse-failed + {:error error}) + {}))) + +(defn- serialize-kv-state + [state] + (transit/write kv-transit-writer state)) + (defn- kv-store [data-dir] (let [kv-path (node-path/join data-dir "kv-store.json") @@ -245,8 +280,8 @@ (p/resolved @state) (-> (fs/readFile kv-path "utf8") (p/then (fn [contents] - (let [data (js/JSON.parse contents)] - (reset! state (js->clj data :keywordize-keys false)) + (let [data (parse-kv-state contents)] + (reset! state data) @state))) (p/catch (fn [_] (reset! state {}) @@ -257,19 +292,21 @@ :set! (fn [k value] (p/let [_ (js @state))] + payload (serialize-kv-state @state)] (fs/writeFile kv-path payload "utf8")))})) (defn node-platform - [{:keys [data-dir event-fn write-guard-fn]}] + [{:keys [data-dir event-fn write-guard-fn owner-source]}] (let [data-dir (expand-home (or data-dir "~/logseq/graphs")) + owner-source (db-lock/normalize-owner-source owner-source) kv (kv-store data-dir)] (p/do! (ensure-dir! data-dir) (log/info :db-worker-node-platform {:data-dir data-dir}) {:env {:publishing? false :runtime :node - :data-dir data-dir} + :data-dir data-dir + :owner-source owner-source} :storage {:install-opfs-pool (fn [sqlite-module pool-name] (install-opfs-pool data-dir sqlite-module pool-name)) :list-graphs (fn [] (list-graphs data-dir)) diff --git a/src/main/frontend/worker/state.cljs b/src/main/frontend/worker/state.cljs index 2ecf49d7f7..694c77e47b 100644 --- a/src/main/frontend/worker/state.cljs +++ b/src/main/frontend/worker/state.cljs @@ -1,6 +1,7 @@ (ns frontend.worker.state "State hub for worker" - (:require [logseq.common.util :as common-util])) + (:require [frontend.worker.platform :as platform] + [logseq.common.util :as common-util])) (defonce *main-thread (atom nil)) (defonce *infer-worker (atom nil)) @@ -106,9 +107,28 @@ [] (:auth/id-token @*state)) +(defn- node-runtime? + [] + (try + (= :node (get-in (platform/current) [:env :runtime])) + (catch :default _ + false))) + +(defn- node-online? + [] + (try + (let [online? (some-> js/globalThis .-navigator .-onLine)] + (if (boolean? online?) + online? + true)) + (catch :default _ + true))) + (defn online? [] - @(:thread-atom/online-event @*state)) + (if (node-runtime?) + (node-online?) + @(:thread-atom/online-event @*state))) (comment (defn mobile? diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index 64235a62da..de5d45214e 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -15,6 +15,7 @@ [frontend.worker.sync.const :as rtc-const] [frontend.worker.sync.crypt :as sync-crypt] [lambdaisland.glogi :as log] + [logseq.common.config :as common-config] [logseq.common.util :as common-util] [logseq.db :as ldb] [logseq.db-sync.cycle :as sync-cycle] @@ -63,7 +64,8 @@ counts (or (sync-counts repo) {}) ws-url (:ws-url @worker-state/*db-sync-config) ws-state (or (some-> client :ws-state deref) - (if (seq ws-url) :stopped :inactive))] + (if (seq ws-url) :stopped :inactive)) + last-error (some-> client :last-sync-error deref)] {:repo repo :graph-id (or (:graph-id client) (:graph-uuid counts)) :ws-state ws-state @@ -71,7 +73,8 @@ :pending-asset (or (:pending-asset counts) 0) :pending-server (or (:pending-server counts) 0) :local-tx (:local-tx counts) - :remote-tx (:remote-tx counts)})) + :remote-tx (:remote-tx counts) + :last-error last-error})) (defn- normalize-online-users [users] @@ -148,9 +151,21 @@ :else ws-url)] (string/replace base #"/sync/%s$" ""))))) +(defn- cli-node-owner? + [] + (try + (let [env (:env (platform/current))] + (and (= :node (:runtime env)) + (= :cli (:owner-source env)))) + (catch :default _ + false))) + (defn- auth-token [] - (or (worker-state/get-id-token) - (:auth-token @worker-state/*db-sync-config))) + (let [configured-token (:auth-token @worker-state/*db-sync-config)] + (if (cli-node-owner?) + configured-token + (or (worker-state/get-id-token) + configured-token)))) (defn- id-token-expired? [token] @@ -166,7 +181,8 @@ (defn- (worker-state/get-id-token) + (some-> (auth-token) worker-util/parse-jwt :sub)) @@ -247,16 +263,35 @@ (string? (:asset-type value)))) (defn- get-graph-id [repo] - (when-let [conn (worker-state/get-datascript-conn repo)] - (let [db @conn - graph-uuid (ldb/get-graph-rtc-uuid db)] - (when graph-uuid - (str graph-uuid))))) + (or (when-let [conn (worker-state/get-datascript-conn repo)] + (let [db @conn + graph-uuid (ldb/get-graph-rtc-uuid db)] + (when graph-uuid + (str graph-uuid)))) + (some-> (client-op/get-graph-uuid repo) str))) (defn- ensure-client-graph-uuid! [repo graph-id] (when (seq graph-id) (client-op/update-graph-uuid repo graph-id))) +(declare list-remote-graphs!) + +(defn- repo common-config/strip-leading-db-version-prefix)] + (if-not (seq target-graph-name) + (p/resolved nil) + (p/let [remote-graphs (list-remote-graphs!) + remote-graph-id (some (fn [{:keys [graph-name graph-id]}] + (when (= target-graph-name graph-name) + graph-id)) + remote-graphs)] + (when (seq remote-graph-id) + (ensure-client-graph-uuid! repo remote-graph-id) + remote-graph-id)))))) + (defn- ready-state [ws] (.-readyState ws)) @@ -349,6 +384,33 @@ (.send ws (js/JSON.stringify (clj->js coerced))) (log/error :db-sync/ws-request-invalid {:message message})))) +(defn- ex-message->code + [message] + (when (and (string? message) + (re-matches #"[a-zA-Z0-9._/\-]+" message)) + (keyword message))) + +(defn- error->diagnostic + [error] + (let [data (or (ex-data error) {}) + code (or (:code data) + (ex-message->code (ex-message error)) + :exception)] + {:code code + :message (or (ex-message error) (str error)) + :at (common-util/time-ms) + :data (when (seq data) data)})) + +(defn- set-last-sync-error! + [client error] + (when-let [*last-error (:last-sync-error client)] + (reset! *last-error (error->diagnostic error)))) + +(defn- clear-last-sync-error! + [client] + (when-let [*last-error (:last-sync-error client)] + (reset! *last-error nil))) + (defn update-presence! [editing-block-uuid] (when-let [client @worker-state/*db-sync-client] @@ -1162,6 +1224,7 @@ :send-queue (atom (p/resolved nil)) :asset-queue (atom (p/resolved nil)) :inflight (atom []) + :last-sync-error (atom nil) :reconnect (atom {:attempt 0 :timer nil}) :stale-kill-timer (atom nil) :last-ws-message-ts (atom (common-util/time-ms)) @@ -1758,6 +1821,7 @@ "tx/batch/ok" (do (require-non-negative remote-tx {:repo repo :type "tx/batch/ok"}) (client-op/update-local-tx repo remote-tx) + (clear-last-sync-error! client) (broadcast-rtc-state! client) (remove-pending-txs! repo @(:inflight client)) (reset! (:inflight client) []) @@ -1770,24 +1834,36 @@ txs-data (mapv (fn [data] (parse-transit (:tx data) {:repo repo :type "pull/ok"})) txs)] - (when (seq txs-data) - (p/let [graph-e2ee? (sync-crypt/graph-e2ee? repo) - aes-key (sync-crypt/ (if (seq txs-data) + (p/let [graph-e2ee? (sync-crypt/graph-e2ee? repo) + aes-key (sync-crypt/ - (p/do! - (stop!) - (p/let [client (ensure-client-state! repo) - url (format-ws-url base graph-id) - _ (ensure-client-graph-uuid! repo graph-id) - connected (assoc client :graph-id graph-id) - token ( + (p/do! + (stop!) + (p/let [client (ensure-client-state! repo) + url (format-ws-url base graph-id) + _ (ensure-client-graph-uuid! repo graph-id) + connected (assoc client :graph-id graph-id) + token ( (worker-state/get-id-token) worker-util/parse-jwt :sub)) + (some-> (auth-token) worker-util/parse-jwt :sub)) (defn- r (js->clj :keywordize-keys true)))) + (cond + (instance? js/ArrayBuffer r) (js/Uint8Array. r) + :else r))) (defn- [] (seq configured-password) (conj {:source :config :value configured-password}) (seq refresh-token) (conj {:source :saved-password :value refresh-token}) - (and (seq auth-token) - (not= auth-token refresh-token)) + (and (seq configured-auth-token) + (not= configured-auth-token refresh-token)) (conj {:source :saved-password - :value auth-token}))] + :value configured-auth-token}))] (letfn [(code + [message] + (when (and (string? message) + (re-matches #"[a-zA-Z0-9._/\-]+" message)) + (keyword message))) + +(defn- exception->error + [error extra] + (let [data (or (ex-data error) {}) + code (or (:code data) + (ex-message->code (ex-message error)) + :exception)] + {:status :error + :error (merge {:code code + :message (or (ex-message error) (str error))} + (when (seq data) {:context data}) + extra)})) + (defn- parse-config-key [raw-key] (let [raw-key (some-> raw-key string/trim string/lower-case) @@ -180,6 +204,61 @@ _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config])] (transport/invoke cfg method false args))))) +(defn- wait-sync-start-ready + [config repo action] + (let [timeout-ms (max 0 (or (:wait-timeout-ms action) sync-start-timeout-ms)) + poll-interval-ms (max 0 (or (:wait-poll-interval-ms action) sync-start-poll-interval-ms)) + deadline (+ (js/Date.now) timeout-ms) + config-skipped-hint "Set sync config keys (ws-url/http-base/auth-token) and retry sync start." + graph-id-skipped-hint "Graph-id is missing locally. Run sync download first, then retry sync start." + runtime-error-hint "Run sync status to inspect last-error and fix sync runtime error before retrying." + timeout-hint "Run sync status to inspect ws-state and ensure sync endpoint/token are valid."] + (letfn [(poll! [] + (p/let [status (invoke-with-repo config repo :thread-api/db-sync-status [repo]) + ws-state (:ws-state status) + graph-id (:graph-id status) + last-error (:last-error status) + skipped-hint (if (seq graph-id) + config-skipped-hint + graph-id-skipped-hint)] + (cond + (and (= :open ws-state) (some? last-error)) + {:status :error + :error {:code :sync-start-runtime-error + :message "sync start reached open websocket but runtime sync error is present" + :repo repo + :ws-state ws-state + :status status + :last-error last-error + :hint runtime-error-hint}} + + (= :open ws-state) + {:status :ok + :data status} + + (contains? sync-start-skipped-states ws-state) + {:status :error + :error {:code :sync-start-skipped + :message "sync start was skipped" + :repo repo + :ws-state ws-state + :status status + :hint skipped-hint}} + + (>= (js/Date.now) deadline) + {:status :error + :error {:code :sync-start-timeout + :message "sync start timed out before websocket reached open state" + :repo repo + :ws-state ws-state + :status status + :hint timeout-hint}} + + :else + (p/let [_ (p/delay poll-interval-ms)] + (poll!)))))] + (poll!)))) + (defn execute [action config] (case (:type action) @@ -191,11 +270,13 @@ :data result}) :sync-start - (p/let [result (invoke-with-repo config (:repo action) - :thread-api/db-sync-start - [(:repo action)])] - {:status :ok - :data {:result result}}) + (-> (p/let [_ (invoke-with-repo config (:repo action) + :thread-api/db-sync-start + [(:repo action)]) + result (wait-sync-start-ready config (:repo action) action)] + result) + (p/catch (fn [error] + (exception->error error {:repo (:repo action)})))) :sync-stop (p/let [result (invoke-with-repo config (:repo action) @@ -214,25 +295,28 @@ {:result result})}) :sync-download - (p/let [remote-graphs (invoke-global config - :thread-api/db-sync-list-remote-graphs - []) - remote-graph (some (fn [graph] - (when (= (:graph action) (:graph-name graph)) - graph)) - remote-graphs)] - (if-not remote-graph - {:status :error - :error {:code :remote-graph-not-found - :message (str "remote graph not found: " (:graph action)) - :graph (:graph action)}} - (p/let [result (invoke-with-repo config (:repo action) - :thread-api/db-sync-download-graph-by-id - [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] - {:status :ok - :data (if (map? result) - result - {:result result})}))) + (-> (p/let [remote-graphs (invoke-global config + :thread-api/db-sync-list-remote-graphs + []) + remote-graph (some (fn [graph] + (when (= (:graph action) (:graph-name graph)) + graph)) + remote-graphs)] + (if-not remote-graph + {:status :error + :error {:code :remote-graph-not-found + :message (str "remote graph not found: " (:graph action)) + :graph (:graph action)}} + (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-download-graph-by-id + [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] + {:status :ok + :data (if (map? result) + result + {:result result})}))) + (p/catch (fn [error] + (exception->error error {:repo (:repo action) + :graph (:graph action)})))) :sync-remote-graphs (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 14dd44e5e0..e9261d7dc9 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -283,17 +283,23 @@ (def ^:private redacted-token "[REDACTED]") (defn- format-sync-status - [{:keys [repo graph-id ws-state pending-local pending-asset pending-server local-tx remote-tx]}] - (string/join "\n" - [(str "Sync status") - (str "repo: " (or repo "-")) - (str "graph-id: " (or graph-id "-")) - (str "ws-state: " (or ws-state :unknown)) - (str "pending-local: " (or pending-local 0)) - (str "pending-asset: " (or pending-asset 0)) - (str "pending-server: " (or pending-server 0)) - (str "local-tx: " (or local-tx "-")) - (str "remote-tx: " (or remote-tx "-"))])) + [{:keys [repo graph-id ws-state pending-local pending-asset pending-server local-tx remote-tx last-error]}] + (let [last-error-line (when (map? last-error) + (str "last-error: " + (or (:code last-error) :error) + (when-let [message (:message last-error)] + (str " (" message ")"))))] + (string/join "\n" + (cond-> [(str "Sync status") + (str "repo: " (or repo "-")) + (str "graph-id: " (or graph-id "-")) + (str "ws-state: " (or ws-state :unknown)) + (str "pending-local: " (or pending-local 0)) + (str "pending-asset: " (or pending-asset 0)) + (str "pending-server: " (or pending-server 0)) + (str "local-tx: " (or local-tx "-")) + (str "remote-tx: " (or remote-tx "-"))] + last-error-line (conj last-error-line))))) (defn- format-sync-remote-graphs [graphs] diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index aeb9887660..c3b2cc1032 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -113,6 +113,38 @@ (is nil (str error)) (done)))))))) +(deftest resolve-ws-token-cli-owner-source-prefers-config-token-test + (async done + (let [refresh-calls (atom 0) + config-prev @worker-state/*db-sync-config + state-prev @worker-state/*state + main-thread-prev @worker-state/*main-thread] + (reset! worker-state/*db-sync-config {:auth-token "cli-config-token"}) + (reset! worker-state/*state (assoc state-prev :auth/id-token "state-token")) + (reset! worker-state/*main-thread + (fn [qkw _direct-pass? _args-list] + (when (= qkw :thread-api/ensure-id&access-token) + (swap! refresh-calls inc)) + (p/resolved {:id-token "refreshed-token"}))) + (with-redefs [platform/current (fn [] {:env {:runtime :node + :owner-source :cli}}) + db-sync/id-token-expired? (fn [_token] true)] + (-> (#'db-sync/latest-remote-tx latest-prev)))))))) +(deftest pull-ok-with-empty-txs-still-advances-local-tx-test + (testing "pull/ok with empty tx list should still advance local tx watermark" + (async done + (let [{:keys [conn client-ops-conn]} (setup-parent-child) + raw-message (js/JSON.stringify + (clj->js {:type "pull/ok" + :t 18 + :txs []})) + client {:repo test-repo + :graph-id "graph-1" + :inflight (atom []) + :last-sync-error (atom {:code :previous-error}) + :online-users (atom []) + :ws-state (atom :open)}] + (with-datascript-conns conn client-ops-conn + (fn [] + (client-op/update-local-tx test-repo 16) + (-> (p/let [_ (#'db-sync/handle-message! test-repo client raw-message)] + (is (= 18 (client-op/get-local-tx test-repo))) + (is (nil? @(:last-sync-error client)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))))))) + +(deftest pull-ok-failure-records-last-sync-error-test + (testing "pull/ok failures should surface structured runtime error state" + (async done + (let [tx-payload (sqlite-util/write-transit-str [[:db/add 1 :block/title "remote"]]) + raw-message (js/JSON.stringify + (clj->js {:type "pull/ok" + :t 18 + :txs [{:t 18 :tx tx-payload}]})) + local-tx (atom 16) + orig-get-local-tx client-op/get-local-tx + orig-update-local-tx client-op/update-local-tx + orig-graph-e2ee? sync-crypt/graph-e2ee? + orig-ensure-graph-aes-key sync-crypt/ (p/let [_ (#'db-sync/handle-message! test-repo client raw-message) + error-state @(:last-sync-error client)] + (is (= 16 @local-tx)) + (is (= :missing-field (:code error-state))) + (is (= "missing-field" (:message error-state))) + (is (= :aes-key (get-in error-state [:data :field])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! client-op/get-local-tx orig-get-local-tx) + (set! client-op/update-local-tx orig-update-local-tx) + (set! sync-crypt/graph-e2ee? orig-graph-e2ee?) + (set! sync-crypt/latest-remote-tx latest-prev) (done)))))))))) +(deftest get-graph-id-falls-back-to-client-op-graph-uuid-test + (let [conn (d/create-conn {}) + client-ops-conn (d/create-conn client-op/schema-in-db)] + (with-datascript-conns conn client-ops-conn + (fn [] + (client-op/update-graph-uuid test-repo "graph-from-client-op") + (is (= "graph-from-client-op" + (#'db-sync/get-graph-id test-repo))))))) + +(deftest resolve-start-graph-id-falls-back-to-remote-graph-list-test + (async done + (let [conn (d/create-conn {}) + client-ops-conn (d/create-conn client-op/schema-in-db) + orig-list-remote-graphs! db-sync/list-remote-graphs!] + (set! db-sync/list-remote-graphs! (fn [] + (p/resolved [{:graph-name test-repo + :graph-id "remote-graph-id"}]))) + (with-datascript-conns conn client-ops-conn + (fn [] + (-> (p/let [graph-id (#'db-sync/ (db-sync/start! test-repo) + (p/then (fn [result] + (is (nil? result)) + (is (= 0 @list-calls)))) + (p/catch (fn [error] + (is false (str "unexpected error: " error)))) + (p/finally (fn [] + (reset! worker-state/*db-sync-config config-prev) + (done)))))))) + (deftest connect-uses-platform-websocket-adapter-test (let [ws-ctor-prev js/WebSocket platform-map {:runtime :test} @@ -1068,6 +1211,85 @@ (reset! worker-state/*db-sync-config config-prev) (done)))))))) +(deftest download-graph-by-id-fails-on-incomplete-snapshot-frame-test + (testing "snapshot download rejects incomplete framed payload" + (async done + (let [fetch-prev js/fetch + config-prev @worker-state/*db-sync-config + rows [["addr-1" "content-1" nil]] + frame (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows)) + truncated (.slice frame 0 (dec (.-byteLength frame)))] + (reset! worker-state/*db-sync-config {:http-base "https://example.com" + :auth-token "token-value"}) + (set! js/fetch + (fn [url _opts] + (cond + (>= (.indexOf url "/pull") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":7,\"txs\":[]}"))}) + + (>= (.indexOf url "/snapshot/stream") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :arrayBuffer (fn [] (js/Promise.resolve (.-buffer truncated)))}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " url)))))) + (-> (p/let [_ (db-sync/download-graph-by-id! test-repo "graph-1" false)] + (is false "expected incomplete frame failure")) + (p/catch (fn [e] + (is (= "incomplete-snapshot-frame" (ex-message e))))) + (p/finally + (fn [] + (set! js/fetch fetch-prev) + (reset! worker-state/*db-sync-config config-prev) + (done)))))))) + +(deftest download-graph-by-id-preserves-framed-row-batches-test + (testing "snapshot download merges complete frames without truncating rows" + (async done + (let [fetch-prev js/fetch + config-prev @worker-state/*db-sync-config + rows-a [["addr-1" "content-1" nil] + ["addr-2" "content-2" nil]] + rows-b [["addr-3" "content-3" nil]] + frame-a (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows-a)) + frame-b (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows-b)) + payload (js/Uint8Array. (+ (.-byteLength frame-a) (.-byteLength frame-b)))] + (.set payload frame-a 0) + (.set payload frame-b (.-byteLength frame-a)) + (reset! worker-state/*db-sync-config {:http-base "https://example.com" + :auth-token "token-value"}) + (set! js/fetch + (fn [url _opts] + (cond + (>= (.indexOf url "/pull") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":7,\"txs\":[]}"))}) + + (>= (.indexOf url "/snapshot/stream") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :arrayBuffer (fn [] (js/Promise.resolve (.-buffer payload)))}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " url)))))) + (-> (p/let [result (db-sync/download-graph-by-id! test-repo "graph-1" false)] + (is (= "graph-1" (:graph-id result))) + (is (= 7 (:remote-tx result))) + (is (= false (:graph-e2ee? result))) + (is (= (vec (concat rows-a rows-b)) + (vec (:rows result))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally + (fn [] + (set! js/fetch fetch-prev) + (reset! worker-state/*db-sync-config config-prev) + (done)))))))) + (deftest ^:long rehydrate-large-title-test (testing "rehydrate fills empty title from object storage" (async done diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 8d9a9ae375..225e680c53 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -401,6 +401,34 @@ (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) +(deftest db-worker-node-sync-start-and-status-invoke-path + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-sync-start") + repo (str "logseq_db_sync_start_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [host port stop!]} + (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo}) + _ (reset! daemon {:host host :port port :stop! stop!}) + _ (invoke host port "thread-api/create-or-open-db" [repo {}]) + _ (invoke host port "thread-api/set-db-sync-config" + [{:ws-url nil + :http-base "https://example.com" + :auth-token "token-value"}]) + start-result (invoke host port "thread-api/db-sync-start" [repo]) + status-result (invoke host port "thread-api/db-sync-status" [repo])] + (is (nil? start-result)) + (is (= repo (:repo status-result))) + (is (= :inactive (:ws-state status-result))) + (is (contains? status-result :pending-local)) + (is (contains? status-result :pending-server))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-daemon-smoke-test (async done (let [daemon (atom nil) diff --git a/src/test/frontend/worker/platform_node_test.cljs b/src/test/frontend/worker/platform_node_test.cljs index faedfea3e0..0ce50c008d 100644 --- a/src/test/frontend/worker/platform_node_test.cljs +++ b/src/test/frontend/worker/platform_node_test.cljs @@ -59,6 +59,38 @@ (is (string/includes? source "\"node:sqlite\"")) (is (not (string/includes? source "\"better-sqlite3\""))))) +(deftest node-platform-env-owner-source-is-propagated + (async done + (let [data-dir (node-helper/create-tmp-dir "platform-node-owner-source")] + (-> (p/let [platform-cli (platform-node/node-platform {:data-dir data-dir + :owner-source :cli}) + platform-default (platform-node/node-platform {:data-dir data-dir})] + (is (= :cli (get-in platform-cli [:env :owner-source]))) + (is (= :unknown (get-in platform-default [:env :owner-source])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest kv-store-preserves-uint8array-values-across-reloads-test + (async done + (let [data-dir (node-helper/create-tmp-dir "platform-node-kv-store") + key "rtc-encrypted-aes-key###graph-1" + value (js/Uint8Array. #js [1 2 3 255])] + (-> (p/let [platform-a (platform-node/node-platform {:data-dir data-dir}) + kv-a (:kv platform-a) + _ ((:set! kv-a) key value) + loaded-a ((:get kv-a) key) + platform-b (platform-node/node-platform {:data-dir data-dir}) + kv-b (:kv platform-b) + loaded-b ((:get kv-b) key)] + (is (instance? js/Uint8Array loaded-a)) + (is (= [1 2 3 255] (vec (js->clj loaded-a)))) + (is (instance? js/Uint8Array loaded-b)) + (is (= [1 2 3 255] (vec (js->clj loaded-b))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest exec-sql-string-creates-schema-and-writes-data (async done (let [conn* (atom nil)] diff --git a/src/test/frontend/worker/state_test.cljs b/src/test/frontend/worker/state_test.cljs new file mode 100644 index 0000000000..f056df8a64 --- /dev/null +++ b/src/test/frontend/worker/state_test.cljs @@ -0,0 +1,32 @@ +(ns frontend.worker.state-test + (:require [cljs.test :refer [deftest is testing]] + [frontend.worker.platform :as platform] + [frontend.worker.state :as worker-state])) + +(defn- with-online-event + [value] + (assoc @worker-state/*state :thread-atom/online-event (atom value))) + +(deftest online?-uses-thread-atom-in-non-node-runtime + (let [state-prev @worker-state/*state] + (try + (with-redefs [platform/current (fn [] + {:env {:runtime :web}})] + (reset! worker-state/*state (with-online-event true)) + (testing "web runtime stays compatible with main-thread online-event" + (is (true? (worker-state/online?)))) + (reset! worker-state/*state (with-online-event false)) + (is (false? (worker-state/online?)))) + (finally + (reset! worker-state/*state state-prev))))) + +(deftest online?-node-runtime-does-not-require-main-thread-online-event + (let [state-prev @worker-state/*state] + (try + (with-redefs [platform/current (fn [] + {:env {:runtime :node}})] + (reset! worker-state/*state (with-online-event nil)) + (testing "node runtime should provide its own online detection" + (is (true? (worker-state/online?))))) + (finally + (reset! worker-state/*state state-prev))))) diff --git a/src/test/frontend/worker/sync/crypt_test.cljs b/src/test/frontend/worker/sync/crypt_test.cljs index f3e3c04820..54eaa59121 100644 --- a/src/test/frontend/worker/sync/crypt_test.cljs +++ b/src/test/frontend/worker/sync/crypt_test.cljs @@ -16,6 +16,70 @@ (p/let [encrypted (crypt/ (p/with-redefs [platform/current (fn [] {:runtime :test}) + platform/kv-get (fn [_platform' _k] + (p/resolved expected))] + (#'sync-crypt/clj result)))))) + (p/catch (fn [e] + (is false (str e)))) + (p/finally done))))) + (deftest save-e2ee-password-uses-platform-write-text-when-not-native-test (async done (let [platform-map {:runtime :test} @@ -197,6 +261,7 @@ :text (fn [] (js/Promise.resolve "{\"message\":\"not-found\"}"))})))) (-> (p/with-redefs [sync-crypt/e2ee-base (fn [] "https://example.com") worker-state/get-id-token (fn [] "token") + worker-util/parse-jwt (fn [_] {:sub "user-1"}) worker-state/ (p/let [_ (sync-command/execute {:type :sync-start - :repo "logseq_db_demo"} - {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] - @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-start false ["logseq_db_demo"]]] - @invoke-calls))) + (case method + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved {:repo "logseq_db_demo" + :ws-state (if (= idx 1) :connecting :open) + :pending-local 0 + :pending-asset 0 + :pending-server 0})) + (p/resolved {:ok true})))) + (-> (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo"} + {:data-dir "/tmp"}) + invoked-methods (map first @invoke-calls)] + (is (= :ok (:status result))) + (is (= :open (get-in result [:data :ws-state]))) + (is (<= 1 (count @ensure-calls))) + (is (every? (fn [[_ repo]] (= "logseq_db_demo" repo)) @ensure-calls)) + (is (= 1 (count (filter #(= :thread-api/db-sync-start %) invoked-methods)))) + (is (<= 2 (count (filter #(= :thread-api/db-sync-status %) invoked-methods))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-start-timeout + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-status + (p/resolved {:repo "logseq_db_demo" + :ws-state :connecting + :pending-local 0 + :pending-asset 0 + :pending-server 0}) + (p/resolved {:ok true})))) + (-> (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 20 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-timeout (get-in result [:error :code]))) + (is (= "logseq_db_demo" (get-in result [:error :repo]))) + (is (= :connecting (get-in result [:error :ws-state])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-start-missing-ws-url-is-error + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-status + (p/resolved {:repo "logseq_db_demo" + :ws-state :inactive + :pending-local 0 + :pending-asset 0 + :pending-server 0}) + (p/resolved {:ok true})))) + (-> (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 20 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-skipped (get-in result [:error :code]))) + (is (= :inactive (get-in result [:error :ws-state])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-sync-start-runtime-error-after-open + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + status-calls (atom 0)] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved (if (= idx 1) + {:repo "logseq_db_demo" + :ws-state :connecting + :pending-local 0 + :pending-asset 0 + :pending-server 0} + {:repo "logseq_db_demo" + :ws-state :open + :pending-local 1 + :pending-asset 0 + :pending-server 2 + :last-error {:code :decrypt-aes-key + :message "decrypt-aes-key"}}))) + (p/resolved {:ok true})))) + (-> (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 200 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-runtime-error (get-in result [:error :code]))) + (is (= :decrypt-aes-key (get-in result [:error :last-error :code])))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] @@ -265,6 +376,37 @@ (set! transport/invoke orig-invoke) (done))))))) +(deftest test-execute-sync-download-propagates-worker-error-code + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke] + (set! cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + (set! transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/db-sync-download-graph-by-id + (p/rejected (ex-info "db-sync/incomplete-snapshot-frame" + {:code :db-sync/incomplete-snapshot-frame + :graph-id "remote-graph-id"})) + (p/resolved nil)))) + (-> (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :db-sync/incomplete-snapshot-frame (get-in result [:error :code])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-sync-remote-graphs (async done (let [orig-ensure-server! cli-server/ensure-server! diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 38b3fb6d71..20def3ddb2 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -253,7 +253,24 @@ (is (string/includes? result "graph-uuid")) (is (string/includes? result "pending-local")) (is (string/includes? result "pending-asset")) - (is (string/includes? result "pending-server"))))) + (is (string/includes? result "pending-server")))) + + (testing "sync status renders last error diagnostic when present" + (let [result (format/format-result {:status :ok + :command :sync-status + :data {:repo "demo-graph" + :graph-id "graph-uuid" + :ws-state :open + :pending-local 2 + :pending-asset 1 + :pending-server 3 + :local-tx 10 + :remote-tx 13 + :last-error {:code :decrypt-aes-key + :message "decrypt-aes-key"}}} + {:output-format nil})] + (is (string/includes? result "last-error")) + (is (string/includes? result "decrypt-aes-key"))))) (deftest test-human-output-sync-remote-graphs (testing "sync remote-graphs renders a table" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9623c87913..cb37fd8dcd 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -188,6 +188,81 @@ [payload] (first (get-in payload [:data :result]))) +(deftest test-cli-sync-download-and-start-readiness-with-mocked-sync + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-cli") + download-repo "sync-download-graph" + start-repo "sync-start-graph" + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom []) + status-calls (atom 0)] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" start-repo] data-dir cfg-path) + create-payload (parse-json-output-safe create-result "graph create") + _ (is (= 0 (:exit-code create-result))) + _ (is (= "ok" (:status create-payload))) + _ (set! cli-server/ensure-server! + (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + _ (set! transport/invoke + (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) + + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name download-repo + :graph-e2ee? true}]) + + :thread-api/db-sync-download-graph-by-id + (p/resolved {:repo "logseq_db_sync_integration_graph" + :graph-id "remote-graph-id" + :remote-tx 22 + :graph-e2ee? true + :row-count 3}) + + :thread-api/db-sync-start + (p/resolved nil) + + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved {:repo "logseq_db_sync_integration_graph" + :ws-state (if (= idx 1) :connecting :open) + :pending-local 0 + :pending-asset 0 + :pending-server 0})) + + :thread-api/q + (p/resolved 3) + + (p/resolved nil)))) + download-result (run-cli ["--graph" download-repo "sync" "download"] data-dir cfg-path) + download-payload (parse-json-output-safe download-result "sync download") + start-result (run-cli ["--graph" start-repo "sync" "start"] data-dir cfg-path) + start-payload (parse-json-output-safe start-result "sync start")] + (is (some #(= :thread-api/db-sync-download-graph-by-id (first %)) @invoke-calls)) + (is (some #(= :thread-api/db-sync-status (first %)) @invoke-calls)) + (is (= 0 (:exit-code download-result))) + (is (= "ok" (:status download-payload))) + (is (= "remote-graph-id" (get-in download-payload [:data :graph-id]))) + (is (= 22 (get-in download-payload [:data :remote-tx]))) + (is (= 0 (:exit-code start-result))) + (is (= "ok" (:status start-payload)) + (pr-str start-payload)) + (is (contains? #{"open" :open} + (get-in start-payload [:data :ws-state]))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke))))))) + (deftest ^:long test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From 79446b2c8a13323d6af1165403cc8820d017dfb1 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 5 Mar 2026 17:12:24 +0800 Subject: [PATCH 109/375] 049-logseq-cli-graph-info-kv-display.md --- .../049-logseq-cli-graph-info-kv-display.md | 193 ++++++++++++++++++ src/main/logseq/cli/command/graph.cljs | 33 ++- src/main/logseq/cli/format.cljs | 74 ++++++- src/test/logseq/cli/command/graph_test.cljs | 71 ++++++- src/test/logseq/cli/format_test.cljs | 103 +++++++++- src/test/logseq/cli/integration_test.cljs | 8 + 6 files changed, 464 insertions(+), 18 deletions(-) create mode 100644 docs/agent-guide/049-logseq-cli-graph-info-kv-display.md diff --git a/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md b/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md new file mode 100644 index 0000000000..940048ed24 --- /dev/null +++ b/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md @@ -0,0 +1,193 @@ +# Graph Info Kv Display Implementation Plan + +Goal: Make `logseq graph info` display all persisted `:logseq.kv/<...>` values in a readable and script-friendly way without introducing new db-worker-node protocol endpoints. + +Architecture: Reuse existing `logseq-cli -> transport -> db-worker-node -> :thread-api/q` flow to query kv entities by `:db/ident` namespace `logseq.kv`. + +Architecture: Keep backward compatibility by preserving current summary fields while adding a structured ident-keyed kv map for JSON and EDN output plus a dedicated kv section in human output. + +Architecture: Follow `@test-driven-development` with failing tests first in command, formatter, and integration layers. + +Tech Stack: ClojureScript, Datascript query via existing `:thread-api/q`, db-worker-node HTTP transit transport, Logseq CLI formatter pipeline, Babashka test runner. + +Related: Builds on `docs/agent-guide/001-logseq-cli.md`. + +Related: Builds on `docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md`. + +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +## Problem statement + +Current `execute-graph-info` only fetches `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version` via two `:thread-api/pull` calls. + +Current human output shows only three lines, which is too limited for diagnosing graph state during CLI usage. + +Current JSON and EDN outputs also omit other kv entries such as `:logseq.kv/db-type`, `:logseq.kv/graph-initial-schema-version`, and runtime metadata keys. + +The requested behavior is to show all `:logseq.kv/<...>` kv pairs in a way that is readable for humans and stable for machine consumers. + +The implementation should stay within the current db-worker-node design and avoid adding new RPC methods unless absolutely necessary. + +## Current and target data flow + +```text +Current. +logseq graph info + -> execute-graph-info + -> thread-api/pull (:graph-created-at) + -> thread-api/pull (:schema-version) + -> formatter renders 3 lines. + +Target. +logseq graph info + -> execute-graph-info + -> thread-api/q (query all :db/ident in namespace "logseq.kv" with :kv/value) + -> normalize + sort kv rows + -> data payload keeps summary fields + ident-keyed kv map + -> formatter renders summary + kv section (human) or structured kv payload (json/edn). +``` + +## Target behavior + +`graph info` keeps existing top summary lines so users still see graph name, created-at, and schema version immediately. + +`graph info` adds a complete kv section in human output, sorted by ident for deterministic reading. + +`graph info` adds a structured ident-keyed kv map field in result data for JSON and EDN output so scripts can parse all kv values directly. + +The kv field contains only persisted kv rows, which means keys without `:kv/value` in the DB are not synthesized. + +All output formats redact kv keys matching sensitive patterns like `token`, `secret`, or `password`. + +Human output truncates very long string values with explicit marker text. + +The implementation uses only existing `:thread-api/q` and transport invocation paths. + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before implementation. + +I will add command-level tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` to validate `execute-graph-info` builds kv payload from query rows and preserves existing summary fields. + +I will extend formatting tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to verify human output includes a deterministic kv section and still keeps the existing summary lines. + +I will add JSON and EDN formatting assertions in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for the new structured kv field shape. + +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for sensitive key redaction across human, JSON, and EDN outputs plus long string truncation in human output. + +I will add integration coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a graph and asserts `graph info` returns non-empty `logseq.kv` data end-to-end through db-worker-node. + +I will run focused tests first and use `@clojure-debug` if any failure is non-obvious before changing implementation. + +I will run `bb dev:lint-and-test` as final gate. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests for command payload shape. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that stubs `cli-server/ensure-server!` and `transport/invoke` and asserts `execute-graph-info` issues one `:thread-api/q` request for `logseq.kv` rows. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that asserts result data still includes `:graph`, `:logseq.kv/graph-created-at`, and `:logseq.kv/schema-version`. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that asserts kv rows are normalized into an ident-keyed map for machine output and deterministic sorted entries for human rendering. + +### Phase 2: Add failing tests for human output. + +4. Extend `test-human-output-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to expect the existing summary lines plus a kv section. +5. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies kv rows are ordered lexicographically by ident. +6. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies sensitive kv keys are redacted in human output. +7. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies long string kv values are truncated in human output. + +### Phase 3: Add failing tests for machine outputs and integration. + +8. Add a failing JSON output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting new kv map field is present and parseable. +9. Add a failing EDN output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting kv map preserves keyword idents and applies sensitive key redaction. +10. Add a failing JSON output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting sensitive keys are redacted in machine output. +11. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that runs `graph create` then `graph info` and asserts returned data includes multiple `:logseq.kv/...` keys. +12. Run focused tests and confirm failures are behavior gaps rather than fixture or server boot issues. + +### Phase 4: Implement command-side kv collection with existing db-worker-node API. + +13. Add a private datalog query constant in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/graph.cljs` that finds `?ident` and `?value` where ident namespace is `"logseq.kv"` and entity has `:kv/value`. +14. Update `execute-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/graph.cljs` to call `transport/invoke` once with `:thread-api/q` and query inputs. +15. Normalize query tuples into an ident-keyed kv map for machine output and keep a sorted kv entry list for deterministic human formatting. +16. Derive `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version` from the kv map to keep backward compatibility in `:data`. +17. Add the new structured kv map field to the response payload while keeping existing keys unchanged. + +### Phase 5: Implement formatter changes for reasonable display. + +18. Add a helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to format graph kv rows for human output with a clear header and deterministic ordering. +19. Update `format-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to render summary lines first and kv section second. +20. Add centralized masking for kv keys matching sensitive patterns like `token`, `secret`, or `password` so human, JSON, and EDN output share the same redaction behavior. +21. Add human output truncation for very long string values with an explicit marker. +22. Ensure human output remains readable when kv list is empty by showing an explicit empty state line. + +### Phase 6: Verify behavior and compatibility. + +23. Run `bb dev:test -v 'logseq.cli.command.graph-test'` and verify new command tests pass. +24. Run `bb dev:test -v 'logseq.cli.format-test'` and verify human, JSON, and EDN formatting tests pass. +25. Run `bb dev:test -v 'logseq.cli.integration-test'` and verify graph-info integration behavior across db-worker-node is green. +26. Run `bb dev:lint-and-test` and verify `0 failures, 0 errors`. +27. Review against `/Users/rcmerci/gh-repos/logseq/prompts/review.md` checklist before final submission. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Graph has only default kv rows. | Human output shows summary lines plus kv section with default keys. | +| Graph has additional future `:logseq.kv/*` keys. | All persisted keys are included automatically without code changes. | +| Graph has composite kv values such as map or vector. | Human output renders values safely as printable EDN strings. | +| Graph has boolean and UUID kv values. | JSON output serializes them in existing normalize-json behavior without crashes. | +| Kv key includes `token`, `secret`, or `password`. | Human, JSON, and EDN outputs all show redacted value. | +| Kv value is a very long string. | Human output shows truncated text with explicit marker. | +| Query returns no kv rows due to corrupted graph state. | Command still succeeds with summary placeholders and an explicit empty kv section. | +| Existing script parses `graph-created-at` and `schema-version` fields. | Backward compatible fields remain present in response data. | +| Human output consumers expect old first three lines. | Existing first three lines remain unchanged in order and meaning. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.command.graph-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.integration-test' +bb dev:lint-and-test +``` + +Each command should complete with `0 failures, 0 errors`. + +Integration output should include a non-empty kv field in `graph info` JSON payload. + +Human output should include `Graph`, `Created at`, and `Schema version` before the kv section. + +## Testing Details + +The tests verify behavior at command assembly, output formatting, and end-to-end runtime boundaries. + +The command tests validate query usage and payload compatibility instead of checking internal helper implementation details only. + +The format tests verify visible behavior for human readability and machine parseability for JSON and EDN outputs. + +The integration test verifies real db-worker-node invocation and confirms that kv data is actually surfaced through CLI transport. + +## Implementation Details + +- Reuse `:thread-api/q` and avoid introducing a new db-worker-node endpoint. +- Keep `execute-graph-info` backward compatible for existing top-level summary fields. +- Add one new ident-keyed kv map field for scripts instead of spreading arbitrary keys at top level. +- Keep kv ordering deterministic in CLI code to stabilize human snapshots and tests. +- Render complex values safely in human output with printable EDN formatting. +- Redact sensitive keys in human, JSON, and EDN output for `token`, `secret`, and `password` patterns. +- Truncate very long string values in human output only. +- Preserve existing output pipeline behavior for `:json`, `:edn`, and `:human`. +- Keep command parser and `graph info` CLI flags unchanged for this iteration. +- Follow `@test-driven-development` strictly and use `@clojure-debug` on unexpected failures. + +## Question + +Decision: The new structured kv field is a single ident-keyed map for machine output. + +Decision: Human, JSON, and EDN outputs must redact kv keys matching sensitive patterns like `token`, `secret`, or `password`. + +Decision: Human output must truncate very long string values with an explicit marker. + +--- diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 7d726dfa2c..3aefe9dc6c 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -34,6 +34,14 @@ (def ^:private import-export-types* #{"edn" "sqlite"}) +(def ^:private graph-info-kv-query + '[:find ?ident ?value + :where + [?e :db/ident ?ident] + [(namespace ?ident) ?ns] + [(= "logseq.kv" ?ns)] + [?e :kv/value ?value]]) + (defn import-export-types [] import-export-types*) @@ -203,13 +211,28 @@ (defn execute-graph-info [action config] - (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - created (transport/invoke cfg :thread-api/pull false [(:repo action) [:kv/value] :logseq.kv/graph-created-at]) - schema (transport/invoke cfg :thread-api/pull false [(:repo action) [:kv/value] :logseq.kv/schema-version])] + (let [ident->kv-key (fn [ident] + (if (keyword? ident) + (if-let [ident-ns (namespace ident)] + (str ident-ns "/" (name ident)) + (name ident)) + (str ident))) + kv-lookup (fn [kv key] + (or (get kv key) + (get kv (str ":" key))))] + (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) + rows (transport/invoke cfg :thread-api/q false [(:repo action) [graph-info-kv-query]]) + kv (reduce (fn [acc [ident value]] + (assoc acc (ident->kv-key ident) value)) + {} + (or rows [])) + created-at (kv-lookup kv "logseq.kv/graph-created-at") + schema-version (kv-lookup kv "logseq.kv/schema-version")] {:status :ok :data {:graph (:graph action) - :logseq.kv/graph-created-at (:kv/value created) - :logseq.kv/schema-version (:kv/value schema)}}))) + :logseq.kv/graph-created-at created-at + :logseq.kv/schema-version schema-version + :kv kv}})))) (defn execute-graph-export [action config] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index e9261d7dc9..26034802dc 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -253,14 +253,32 @@ (or doc "-")]) (or queries [])))) +(declare kv-key->string + graph-info-human-max-string-length + graph-info-truncated-suffix) + (defn- format-graph-info - [{:keys [graph logseq.kv/graph-created-at logseq.kv/schema-version]} now-ms] - (string/join "\n" - [(str "Graph: " (or graph "-")) - (str "Created at: " (if (some? graph-created-at) - (human-ago graph-created-at now-ms) - "-")) - (str "Schema version: " (or schema-version "-"))])) + [{:keys [graph kv logseq.kv/graph-created-at logseq.kv/schema-version]} now-ms] + (let [summary-lines [(str "Graph: " (or graph "-")) + (str "Created at: " (if (some? graph-created-at) + (human-ago graph-created-at now-ms) + "-")) + (str "Schema version: " (or schema-version "-"))] + truncate-value (fn [value] + (if (and (string? value) + (> (count value) graph-info-human-max-string-length)) + (str (subs value 0 graph-info-human-max-string-length) + graph-info-truncated-suffix) + value)) + kv-lines (if (seq kv) + (into ["KV:"] + (->> kv + (sort-by (comp kv-key->string key)) + (map (fn [[kv-key kv-value]] + (str " " (kv-key->string kv-key) ": " + (pr-str (truncate-value kv-value))))))) + ["KV:" " (empty)"])] + (string/join "\n" (into summary-lines kv-lines)))) (defn- format-server-status [{:keys [repo status host port]}] @@ -281,6 +299,44 @@ (and host port) (conj (str "Host: " host " Port: " port)))))) (def ^:private redacted-token "[REDACTED]") +(def ^:private graph-info-sensitive-kv-pattern #"(?i)(token|secret|password)") +(def ^:private graph-info-human-max-string-length 120) +(def ^:private graph-info-truncated-suffix "... [truncated]") + +(defn- kv-key->string + [kv-key] + (if (keyword? kv-key) + (if-let [kv-ns (namespace kv-key)] + (str kv-ns "/" (name kv-key)) + (name kv-key)) + (str kv-key))) + +(defn- sensitive-graph-kv-key? + [kv-key] + (boolean (re-find graph-info-sensitive-kv-pattern (kv-key->string kv-key)))) + +(defn- redact-graph-kv + [kv] + (into {} + (map (fn [[kv-key kv-value]] + [kv-key + (if (sensitive-graph-kv-key? kv-key) + redacted-token + kv-value)])) + (or kv {}))) + +(defn- sanitize-graph-info-data + [data] + (if (map? data) + (update data :kv redact-graph-kv) + data)) + +(defn- sanitize-result + [result] + (if (and (= :ok (:status result)) + (= :graph-info (:command result))) + (update result :data sanitize-graph-info-data) + result)) (defn- format-sync-status [{:keys [repo graph-id ws-state pending-local pending-asset pending-server local-tx remote-tx last-error]}] @@ -495,7 +551,9 @@ (defn format-result [result {:keys [output-format] :as opts}] - (let [result (normalize-graph-result result) + (let [result (-> result + normalize-graph-result + sanitize-result) format (cond (= output-format :edn) :edn (= output-format :json) :json diff --git a/src/test/logseq/cli/command/graph_test.cljs b/src/test/logseq/cli/command/graph_test.cljs index 4560d2f804..6f3211c07b 100644 --- a/src/test/logseq/cli/command/graph_test.cljs +++ b/src/test/logseq/cli/command/graph_test.cljs @@ -1,7 +1,10 @@ (ns logseq.cli.command.graph-test - (:require [cljs.test :refer [deftest is]] + (:require [cljs.test :refer [async deftest is]] [clojure.string :as string] - [logseq.cli.command.graph :as graph-command])) + [logseq.cli.command.graph :as graph-command] + [logseq.cli.server :as cli-server] + [logseq.cli.transport :as transport] + [promesa.core :as p])) (deftest test-graph-validate-result (let [graph-validate-result #'graph-command/graph-validate-result @@ -13,3 +16,67 @@ (is (string/includes? (get-in invalid-result [:error :message]) "Found 1 entity with errors:")) (is (= :ok (:status valid-result))))) + +(deftest test-execute-graph-info-queries-kv-rows-with-thread-api-q + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls* (atom []) + action {:repo "demo-repo" + :graph "demo-graph"}] + (set! cli-server/ensure-server! + (fn [_ _] + (p/resolved {:base-url "http://example"}))) + (set! transport/invoke + (fn [_ method _ args] + (swap! invoke-calls* conj [method args]) + (p/resolved [[:logseq.kv/schema-version 7] + [:logseq.kv/graph-created-at 40000] + [:logseq.kv/db-type :sqlite]]))) + (-> (p/let [result (graph-command/execute-graph-info action {})] + (is (= :ok (:status result))) + (is (= 1 (count @invoke-calls*))) + (let [[method [repo query-args]] (first @invoke-calls*)] + (is (= :thread-api/q method)) + (is (= "demo-repo" repo)) + (is (= 1 (count query-args))) + (is (string/includes? (pr-str (first query-args)) "logseq.kv")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + +(deftest test-execute-graph-info-preserves-summary-fields-and-builds-kv-map + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + action {:repo "demo-repo" + :graph "demo-graph"}] + (set! cli-server/ensure-server! + (fn [_ _] + (p/resolved {:base-url "http://example"}))) + (set! transport/invoke + (fn [_ method _ _] + (case method + :thread-api/q + (p/resolved [[:logseq.kv/db-type :sqlite] + [:logseq.kv/graph-created-at 40000] + [:logseq.kv/schema-version 7]]) + (throw (ex-info "unexpected invoke method" {:method method}))))) + (-> (p/let [result (graph-command/execute-graph-info action {})] + (is (= :ok (:status result))) + (is (= "demo-graph" (get-in result [:data :graph]))) + (is (= 40000 (get-in result [:data :logseq.kv/graph-created-at]))) + (is (= 7 (get-in result [:data :logseq.kv/schema-version]))) + (is (= {"logseq.kv/db-type" :sqlite + "logseq.kv/graph-created-at" 40000 + "logseq.kv/schema-version" 7} + (get-in result [:data :kv])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 20def3ddb2..ef517ab758 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -343,19 +343,116 @@ (is (not (string/includes? result password)))))) (deftest test-human-output-graph-info - (testing "graph info includes key metadata lines" + (testing "graph info includes key metadata lines and kv section" (let [result (format/format-result {:status :ok :command :graph-info :data {:graph "demo-graph" :logseq.kv/graph-created-at 40000 - :logseq.kv/schema-version 2}} + :logseq.kv/schema-version 2 + :kv {"logseq.kv/db-type" :sqlite + "logseq.kv/graph-created-at" 40000 + "logseq.kv/schema-version" 2}}} {:output-format nil :now-ms 100000})] (is (= (str "Graph: demo-graph\n" "Created at: 1m ago\n" - "Schema version: 2") + "Schema version: 2\n" + "KV:\n" + " logseq.kv/db-type: :sqlite\n" + " logseq.kv/graph-created-at: 40000\n" + " logseq.kv/schema-version: 2") result))))) +(deftest test-human-output-graph-info-kv-ordering + (testing "graph info kv section is sorted by ident key" + (let [result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/schema-version" 2 + "logseq.kv/db-type" :sqlite + "logseq.kv/graph-created-at" 40000}}} + {:output-format nil + :now-ms 100000}) + idx-db-type (.indexOf result "logseq.kv/db-type") + idx-created (.indexOf result "logseq.kv/graph-created-at") + idx-schema (.indexOf result "logseq.kv/schema-version")] + (is (>= idx-db-type 0)) + (is (< idx-db-type idx-created)) + (is (< idx-created idx-schema))))) + +(deftest test-human-output-graph-info-kv-redaction + (testing "graph info redacts sensitive kv values in human output" + (let [token "secret-token-value" + result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/api-token" token + "logseq.kv/db-type" :sqlite}}} + {:output-format nil})] + (is (string/includes? result "logseq.kv/api-token")) + (is (string/includes? result "[REDACTED]")) + (is (not (string/includes? result token)))))) + +(deftest test-human-output-graph-info-kv-truncates-long-strings + (testing "graph info truncates long string kv values in human output" + (let [long-text (apply str (repeat 180 "a")) + result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/runtime-metadata" long-text}}} + {:output-format nil})] + (is (string/includes? result "... [truncated]")) + (is (not (string/includes? result long-text)))))) + +(deftest test-machine-output-graph-info-kv-structure + (testing "graph info json output includes kv map" + (let [result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/db-type" :sqlite + "logseq.kv/schema-version" 2}}} + {:output-format :json}) + parsed (js->clj (js/JSON.parse result) :keywordize-keys true)] + (is (= "ok" (:status parsed))) + (is (= "demo-graph" (get-in parsed [:data :graph]))) + (is (= "sqlite" (get-in parsed [:data :kv :logseq.kv/db-type]))) + (is (= 2 (get-in parsed [:data :kv :logseq.kv/schema-version]))))) + + (testing "graph info edn output includes kv map" + (let [result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/db-type" :sqlite + "logseq.kv/schema-version" 2}}} + {:output-format :edn}) + parsed (reader/read-string result)] + (is (= :ok (:status parsed))) + (is (= "demo-graph" (get-in parsed [:data :graph]))) + (is (= :sqlite (get-in parsed [:data :kv "logseq.kv/db-type"]))) + (is (= 2 (get-in parsed [:data :kv "logseq.kv/schema-version"])))))) + +(deftest test-machine-output-graph-info-kv-redaction + (testing "graph info redacts sensitive kv values in json and edn output" + (let [token "my-secret-token" + json-result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/api-token" token + "logseq.kv/db-type" :sqlite}}} + {:output-format :json}) + json-parsed (js->clj (js/JSON.parse json-result) :keywordize-keys true) + edn-result (format/format-result {:status :ok + :command :graph-info + :data {:graph "demo-graph" + :kv {"logseq.kv/api-token" token + "logseq.kv/db-type" :sqlite}}} + {:output-format :edn}) + edn-parsed (reader/read-string edn-result)] + (is (= "[REDACTED]" (get-in json-parsed [:data :kv :logseq.kv/api-token]))) + (is (= "[REDACTED]" (get-in edn-parsed [:data :kv "logseq.kv/api-token"]))) + (is (not (string/includes? json-result token))) + (is (not (string/includes? edn-result token)))))) + (deftest test-human-output-server-status (testing "server status includes repo, status, host, port" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index cb37fd8dcd..c97d69aea1 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -331,6 +331,14 @@ (is (= 0 (:exit-code info-result))) (is (= "ok" (:status info-payload))) (is (= "demo-graph" (get-in info-payload [:data :graph]))) + (let [kv (get-in info-payload [:data :kv]) + kv-keys (set (map str (keys (or kv {}))))] + (is (map? kv)) + (is (>= (count kv) 2)) + (is (or (contains? kv-keys "logseq.kv/graph-created-at") + (contains? kv-keys ":logseq.kv/graph-created-at"))) + (is (or (contains? kv-keys "logseq.kv/schema-version") + (contains? kv-keys ":logseq.kv/schema-version")))) (is (= 0 (:exit-code stop-result))) (is (= "ok" (:status stop-payload))) (done)) From 0900b9b41bc85926b2b5e28b7113f61a8e1f49f2 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 5 Mar 2026 10:31:39 -0500 Subject: [PATCH 110/375] enhance: graph list displays current graph like git and npm. Also remove confusing GRAPH header and make validate consistent like other read graph commands --- .gitignore | 1 + src/main/logseq/cli/commands.cljs | 6 ++---- src/main/logseq/cli/format.cljs | 26 ++++++++++++++++++-------- src/test/logseq/cli/format_test.cljs | 23 +++++++++++++++++++++++ 4 files changed, 44 insertions(+), 12 deletions(-) diff --git a/.gitignore b/.gitignore index 888c1ce302..c1a46b6f3b 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,7 @@ resources/electron.js /libs/dist/ charlie/ .vscode +/.claude /.preprocessor-cljs docker android/app/src/main/assets/capacitor.plugin.json diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 70c1b0dd58..5e21a2033d 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -174,7 +174,8 @@ (:help opts) (command-core/help-result cmd-summary) - (and (#{:graph-create :graph-switch :graph-remove :graph-validate} command) + ;; Require graphs when writing to graph + (and (#{:graph-create :graph-switch :graph-remove :graph-import} command) (not (seq graph))) (missing-graph-result summary) @@ -257,9 +258,6 @@ (and (= command :graph-import) (not (seq (:input opts)))) (missing-input-result summary) - (and (= command :graph-import) (not (seq (:graph opts)))) - (missing-graph-result summary) - (and (= command :graph-import) (not (contains? (graph-command/import-export-types) (graph-command/normalize-import-export-type (:type opts))))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 26034802dc..4d267f083e 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -79,8 +79,9 @@ (defn- format-counted-table [headers rows] - (str (render-table headers rows) - "\n" + (str (if headers + (str (render-table headers rows) "\n") + (str (string/join "\n" (map (comp string/trimr first) rows)) "\n")) "Count: " (count rows))) @@ -215,10 +216,19 @@ (mapv #(format-list-property-row % include-ident? now-ms) items)))) (defn- format-graph-list - [graphs] - (format-counted-table - ["GRAPH"] - (mapv (fn [graph] [graph]) (or graphs [])))) + [graphs current-graph] + (let [graphs (or graphs []) + has-current? (and (seq current-graph) + (some #(= % current-graph) graphs))] + (format-counted-table + nil + (mapv (fn [graph] + [(if has-current? + (if (= graph current-graph) + (str "* " graph) + (str " " graph)) + graph)]) + graphs)))) (defn- format-server-list [servers] @@ -478,12 +488,12 @@ (string/join "\n" (into [header] check-lines)))) (defn- ->human - [{:keys [status data error command context]} {:keys [now-ms]}] + [{:keys [status data error command context]} {:keys [now-ms graph]}] (let [now-ms (or now-ms (js/Date.now))] (case status :ok (case command - :graph-list (format-graph-list (:graphs data)) + :graph-list (format-graph-list (:graphs data) graph) :graph-info (format-graph-info data now-ms) (:graph-create :graph-switch :graph-remove :graph-validate) (format-graph-action command context) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index ef517ab758..ca46565244 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -61,6 +61,29 @@ {:output-format nil})] (is (= "Found 1 entity with errors:\n({:entity {:db/id 1}})\n" result))))) +(deftest test-human-output-graph-list + (testing "graph list without current graph shows plain list" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["alpha" "beta"]}} + {:output-format nil})] + (is (= (str "alpha\n" + "beta\n" + "Count: 2") + result)))) + + (testing "graph list with current graph marks it with * and indents others" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["alpha" "beta" "gamma"]}} + {:output-format nil + :graph "beta"})] + (is (= (str " alpha\n" + "* beta\n" + " gamma\n" + "Count: 3") + result))))) + (deftest test-human-output-list-page (testing "list page renders a table with count" (let [result (format/format-result {:status :ok From 2864aab594460d6739df0182ce22138b56f69507 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 5 Mar 2026 10:33:59 -0500 Subject: [PATCH 111/375] fix: cli help double printing usage and meaningless Options --- src/main/logseq/cli/main.cljs | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/src/main/logseq/cli/main.cljs b/src/main/logseq/cli/main.cljs index b069445e7a..f2dd5d447c 100644 --- a/src/main/logseq/cli/main.cljs +++ b/src/main/logseq/cli/main.cljs @@ -1,8 +1,7 @@ (ns logseq.cli.main "CLI entrypoint for invoking db-worker-node." (:refer-clojure :exclude [run!]) - (:require [clojure.string :as string] - [logseq.cli.commands :as commands] + (:require [logseq.cli.commands :as commands] [logseq.cli.config :as config] [logseq.cli.data-dir :as data-dir] [logseq.cli.format :as format] @@ -11,14 +10,6 @@ [lambdaisland.glogi :as log] [promesa.core :as p])) -(defn- usage - [summary] - (string/join "\n" - ["logseq [options]" - "" - "Options:" - summary])) - (defn- result->exit-code [result] (or (:exit-code result) @@ -31,7 +22,7 @@ (cond (:help? parsed) (p/resolved {:exit-code 0 - :output (usage (:summary parsed))}) + :output (:summary parsed)}) (not (:ok? parsed)) (p/resolved {:exit-code 1 From c73fdb8c6f0f122ec280c4f9c7071d7d89af411f Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Thu, 5 Mar 2026 18:38:31 -0500 Subject: [PATCH 112/375] fix: 'Replace graph' command not working --- src/main/frontend/handler/common/developer.cljs | 2 +- src/main/frontend/persist_db.cljs | 7 +++++++ src/main/frontend/worker/db_core.cljs | 5 +++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/main/frontend/handler/common/developer.cljs b/src/main/frontend/handler/common/developer.cljs index 57552effeb..d2a98ec394 100644 --- a/src/main/frontend/handler/common/developer.cljs +++ b/src/main/frontend/handler/common/developer.cljs @@ -85,7 +85,7 @@ (defn import-chosen-graph [repo] - (p/let [_ (persist-db/ From fefab4d00f4c723ce53551bd2ae27d09d037ead3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 6 Mar 2026 21:00:02 +0800 Subject: [PATCH 113/375] 050-sync-download-create-empty-db.md --- .../050-sync-download-create-empty-db.md | 188 ++++++++++++++++++ docs/cli/logseq-cli.md | 2 + src/main/frontend/worker/db_core.cljs | 4 +- src/main/frontend/worker/db_worker_node.cljs | 27 ++- src/main/logseq/cli/command/sync.cljs | 61 ++++-- src/main/logseq/cli/server.cljs | 8 +- src/main/logseq/db_worker/daemon.cljs | 5 +- .../frontend/worker/db_worker_node_test.cljs | 134 +++++++++++++ src/test/logseq/cli/command/sync_test.cljs | 106 ++++++++-- src/test/logseq/cli/commands_test.cljs | 28 +++ src/test/logseq/cli/integration_test.cljs | 3 +- src/test/logseq/cli/server_test.cljs | 48 +++++ src/test/logseq/db_worker/daemon_test.cljs | 39 ++++ 13 files changed, 599 insertions(+), 54 deletions(-) create mode 100644 docs/agent-guide/050-sync-download-create-empty-db.md diff --git a/docs/agent-guide/050-sync-download-create-empty-db.md b/docs/agent-guide/050-sync-download-create-empty-db.md new file mode 100644 index 0000000000..bcd6faf57f --- /dev/null +++ b/docs/agent-guide/050-sync-download-create-empty-db.md @@ -0,0 +1,188 @@ +# Sync Download Create Empty Db Implementation Plan + +Goal: Ensure `sync download` never writes `db-initial-data` before snapshot import by starting `db-worker-node` with an explicit empty-db mode. + +Architecture: Add a new db-worker startup flag `--create-empty-db` and plumb it through CLI server orchestration only for the `sync download` path. + +Architecture: When the flag is enabled, `db-worker-node` will call `:thread-api/create-or-open-db` with `{:datoms []}` during daemon startup so `frontend.worker.db-core/ /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> :thread-api/db-sync-download-graph-by-id import path +``` + +## Scope and non-goals + +In scope are CLI-to-daemon argument plumbing, db-worker startup behavior change behind a new flag, tests, and docs updates. + +Out of scope are sync protocol changes, snapshot payload format changes, and non-download command behavior changes. + +## Implementation plan + +### Phase 0. Baseline and failure capture. + +1. Run `bb dev:test -v logseq.db-worker.daemon-test` and record baseline. +2. Run `bb dev:test -v frontend.worker.db-worker-node-test` and record baseline. +3. Run `bb dev:test -v logseq.cli.server-test` and record baseline. +4. Run `bb dev:test -v logseq.cli.command.sync-test` and record baseline. + +### Phase 1. RED tests for daemon flag plumbing. + +5. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/daemon_test.cljs` that `spawn-server!` appends `--create-empty-db` when `:create-empty-db? true` is passed. +6. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/daemon_test.cljs` that `spawn-server!` keeps default args unchanged when `:create-empty-db?` is absent or false. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `parse-args` recognizes `--create-empty-db` and maps it to `:create-empty-db? true`. +8. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that help output documents `--create-empty-db` as a startup option. +9. Run `bb dev:test -v logseq.db-worker.daemon-test` and `bb dev:test -v frontend.worker.db-worker-node-test` and confirm RED. + +### Phase 2. GREEN implementation for daemon flag plumbing. + +10. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` so `spawn-server!` accepts `:create-empty-db?` and conditionally appends `--create-empty-db`. +11. Keep process scanning in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` compatible by ignoring unknown flags as today. +12. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `parse-args` to set `:create-empty-db? true` when `--create-empty-db` is present. +13. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `show-help!` to include one line for `--create-empty-db`. +14. Re-run `bb dev:test -v logseq.db-worker.daemon-test` and `bb dev:test -v frontend.worker.db-worker-node-test` and confirm green. + +### Phase 3. RED tests for sync download orchestration. + +15. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `:sync-download` calls `cli-server/ensure-server!` with `:create-empty-db? true`. +16. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that `spawn-server!` forwards `:create-empty-db?` from `ensure-server-started!` config when spawning a new daemon. +17. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `sync download` path keeps existing `graph-exists` guard unchanged while still enabling create-empty mode on success paths. +18. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync download` fails fast with a dedicated error when the graph DB is not empty. +19. Run `bb dev:test -v logseq.cli.command.sync-test` and `bb dev:test -v logseq.cli.server-test` and confirm RED. + +### Phase 4. GREEN implementation for sync download orchestration. + +20. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so the `:sync-download` execution path enriches config with `:create-empty-db? true` before `invoke-global` and `invoke-with-repo`. +21. Add a DB emptiness guard in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` or worker thread-api path that explicitly validates target graph DB is empty before calling `:thread-api/db-sync-download-graph-by-id`. +22. Keep non-download sync actions in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` unchanged so default startup remains backward compatible. +23. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` and its internal `spawn-server!` wrapper to pass through `:create-empty-db?` to daemon spawn. +24. Re-run `bb dev:test -v logseq.cli.command.sync-test` and `bb dev:test -v logseq.cli.server-test` and confirm green. + +### Phase 5. RED tests for startup create-or-open behavior. + +25. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that startup invokes `thread-api/create-or-open-db` with `[repo {:datoms []}]` when `:create-empty-db? true`. +26. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that default startup still invokes `[repo {}]` when flag is not set. +27. Add a failing behavior test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` or `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` validating download bootstrap path does not depend on locally generated initial data. +28. Add a failing behavior test validating non-empty graph DB state is rejected before import starts. +29. Run targeted tests and confirm RED failures are behavior-based. + +### Phase 6. GREEN implementation for empty-db startup. + +30. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `start-daemon!` to pass startup opts `{:datoms []}` to `:thread-api/create-or-open-db` only when `:create-empty-db? true`. +31. Keep lock lifecycle and health endpoints unchanged in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`. +32. Add a tiny helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` if needed to keep startup opts construction testable. +33. Implement or expose a reusable DB emptiness check in worker/CLI path and return a stable error code for non-empty DB. +34. Re-run `bb dev:test -v frontend.worker.db-worker-node-test` and the selected download behavior test namespace until green. + +### Phase 7. Docs and regression. + +35. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document that `sync download` starts db-worker in create-empty mode and rejects non-empty DB state. +36. Update `/Users/rcmerci/gh-repos/logseq/docs/developers/desktop-db-worker-node.md` with a note that create-empty mode is CLI download bootstrap behavior and does not change desktop default startup. +37. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and ensure full suite passes. +38. Run final review checklist in `/Users/rcmerci/gh-repos/logseq/prompts/review.md` before merge. + +## Edge cases to cover explicitly + +| Scenario | Expected behavior | +|---|---| +| `sync download` for a brand-new graph. | Daemon starts with `--create-empty-db` and startup `create-or-open-db` uses `{:datoms []}`. | +| `sync start`, `sync upload`, and graph CRUD commands. | No `--create-empty-db` flag is used, and existing startup behavior stays unchanged. | +| Existing local graph (`graph-exists`). | Command fails before daemon startup as it does today. | +| Existing ready daemon lock for the same repo. | Command must validate DB emptiness before download and fail fast when DB is not empty. | +| Missing remote graph. | `remote-graph-not-found` response remains unchanged, and no behavior regression in error shape occurs. | +| Flag typo or absent flag. | Startup remains backward compatible with default `create-or-open-db` opts `{}`. | + +## Verification command matrix + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.db-worker.daemon-test` | Spawn arg and parser tests for `--create-empty-db` pass. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Startup invoke args and help text tests pass. | +| `bb dev:test -v logseq.cli.server-test` | CLI server pass-through tests for `:create-empty-db?` pass. | +| `bb dev:test -v logseq.cli.command.sync-test` | `sync download` ensures server with create-empty mode. | +| `bb dev:test -v logseq.cli.commands-test` | Existing `graph-exists` guard and command dispatch remain stable. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-sync-download-and-start-readiness-with-mocked-sync` | Integration path remains green with download plus start readiness behavior. | +| `bb dev:lint-and-test` | Full lint and unit test suite exits with code `0`. | + +## Rollout notes + +This change is intentionally scoped to startup behavior for `sync download` and does not alter runtime sync protocol. + +If regressions appear, rollback is to remove `:create-empty-db?` wiring in sync command and keep daemon/parser support dormant until retried. + +## Testing Details + +Tests focus on externally visible behavior, namely which startup args are passed and whether startup create-or-open uses empty datoms in download bootstrap mode. + +Tests also verify `sync download` rejects non-empty graph DB state before any import side effect. + +Tests avoid asserting private implementation internals except where argument boundaries are the behavior contract. + +Integration coverage confirms existing sync download and sync start semantics remain stable after plumbing changes. + +## Implementation Details + +- Add `:create-empty-db?` option plumbing from sync command to CLI server spawn path. +- Add `--create-empty-db` process arg emission in daemon spawn helper. +- Parse `--create-empty-db` in db-worker-node entrypoint args. +- When create-empty is enabled, startup invoke payload uses `[repo {:datoms []}]`. +- Add a strict pre-download emptiness check and fail with a stable error code when DB is not empty. +- Keep default startup invoke payload `[repo {}]` for all other flows. +- Keep lock ownership and stale-lock cleanup behavior unchanged. +- Keep command-level `graph-exists` and `remote-graph-not-found` behaviors unchanged. +- Update CLI and developer docs with exact scope of the new flag. +- Run targeted tests first, then full `bb dev:lint-and-test`. +- Use `@test-driven-development` workflow and `@clojure-debug` when failures are non-obvious. + +## Question + +No open question. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 7f7c908c40..16b4deb0c1 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -107,6 +107,8 @@ Sync download behavior: - `sync download` requires `--graph `. - If a local graph with the same name already exists, the CLI returns `graph-exists`. - If no remote graph with that name exists, the CLI returns `remote-graph-not-found`. +- `sync download` starts `db-worker-node` in create-empty mode so local startup does not write `db-initial-data` before snapshot import. +- If the target graph DB is not empty at download time, the CLI returns `graph-db-not-empty` and aborts before import. - For e2ee remote graphs in headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before download. Sync config persistence: diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 9575537022..d9eb15f09e 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -332,6 +332,7 @@ client-ops-conn (when-not @*publishing? (common-sqlite/get-storage-conn client-ops-storage client-op/schema-in-db)) + empty-startup? (and (some? datoms) (empty? datoms)) initial-data-exists? (when (nil? datoms) (and (d/entity @conn :logseq.class/Root) (= "db" (:kv/value (d/entity @conn :logseq.kv/db-type)))))] @@ -347,7 +348,8 @@ (gc-sqlite-dbs! db client-ops-db conn {}) - (db-migrate/migrate conn) + (when-not empty-startup? + (db-migrate/migrate conn)) (db-listener/listen-db-changes! repo (get @*datascript-conns repo)))))) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 58338f3b6d..47add71381 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -46,14 +46,15 @@ opts {}] (if (empty? args) opts - (let [[flag value & remaining] args] + (let [flag (first args)] (case flag - "--data-dir" (recur remaining (assoc opts :data-dir value)) - "--repo" (recur remaining (assoc opts :repo value)) - "--owner-source" (recur remaining (assoc opts :owner-source value)) - "--log-level" (recur remaining (assoc opts :log-level value)) - "--help" (recur remaining (assoc opts :help? true)) - (recur remaining opts)))))) + "--data-dir" (recur (subvec args 2) (assoc opts :data-dir (second args))) + "--repo" (recur (subvec args 2) (assoc opts :repo (second args))) + "--owner-source" (recur (subvec args 2) (assoc opts :owner-source (second args))) + "--log-level" (recur (subvec args 2) (assoc opts :log-level (second args))) + "--create-empty-db" (recur (subvec args 1) (assoc opts :create-empty-db? true)) + "--help" (recur (subvec args 1) (assoc opts :help? true)) + (recur (subvec args 1) opts)))))) (defn- normalize-owner-source [owner-source] @@ -283,9 +284,16 @@ (println (str (style/bold "db-worker-node") " " (style/bold "options") ":")) (println (str " " (style/bold "--data-dir") " (default ~/logseq/graphs)")) (println (str " " (style/bold "--repo") " (required)")) + (println (str " " (style/bold "--create-empty-db") " (start with empty initial datoms)")) (println (str " " (style/bold "--log-level") " (default info)")) (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)")) +(defn- startup-db-opts + [{:keys [create-empty-db?]}] + (if create-empty-db? + {:datoms []} + {})) + (defn- pad2 [value] (if (< value 10) @@ -350,7 +358,7 @@ file-path)) (defn start-daemon! - [{:keys [data-dir repo log-level owner-source]}] + [{:keys [data-dir repo log-level owner-source] :as opts}] (let [host "127.0.0.1" port 0 owner-source (normalize-owner-source owner-source)] @@ -381,7 +389,7 @@ _ (reset! *lock-info {:path path :lock lock}) _ (let [method-kw :thread-api/create-or-open-db method-str (normalize-method-str method-kw)] - ( (p/let [{:keys [stop!] :as daemon} (start-daemon! {:data-dir data-dir :repo repo + :create-empty-db? (:create-empty-db? opts) :owner-source owner-source :log-level (:log-level opts)})] (log/info :db-worker-node-ready {:host (:host daemon) :port (:port daemon)}) diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index a265e20d01..3f34ae92fc 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -36,6 +36,16 @@ (def ^:private sync-start-skipped-states #{:inactive :stopped}) +(def ^:private sync-download-non-empty-query + '[:find (count ?e) . + :where + (or [?e :block/name] + [?e :block/page _] + [?e :block/parent _]) + (not [?e :logseq.property/built-in? true]) + (not [?e :db/ident]) + (not [?e :file/path])]) + (defn- missing-repo [label] {:ok? false @@ -169,30 +179,31 @@ :action {:type :sync-config-unset :config-key (:key key-result)}})) - {:ok? false + {:ok? false :error {:code :unknown-command :message (str "unknown sync command: " command)}})) +(defn- sync-config + [config] + {:ws-url (:ws-url config) + :http-base (:http-base config) + :auth-token (:auth-token config) + :e2ee-password (:e2ee-password config)}) + (defn- invoke-with-repo [config repo method args] - (let [sync-config {:ws-url (:ws-url config) - :http-base (:http-base config) - :auth-token (:auth-token config) - :e2ee-password (:e2ee-password config)}] + (let [sync-cfg (sync-config config)] (p/let [cfg (cli-server/ensure-server! config repo) - _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config]) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-cfg]) result (transport/invoke cfg method false args)] result))) (defn- invoke-global [config method args] (let [base-url (:base-url config) - sync-config {:ws-url (:ws-url config) - :http-base (:http-base config) - :auth-token (:auth-token config) - :e2ee-password (:e2ee-password config)}] + sync-cfg (sync-config config)] (if (seq base-url) - (p/let [_ (transport/invoke config :thread-api/set-db-sync-config false [sync-config])] + (p/let [_ (transport/invoke config :thread-api/set-db-sync-config false [sync-cfg])] (transport/invoke config method false args)) (p/let [repo (or (core/resolve-repo (:graph config)) (p/let [graphs (cli-server/list-graphs config)] @@ -201,9 +212,24 @@ (cli-server/ensure-server! config repo) (p/rejected (ex-info "graph name is required" {:code :missing-graph}))) - _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-config])] + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-cfg])] (transport/invoke cfg method false args))))) +(defn- download-config + [config] + (assoc config :create-empty-db? true)) + +(defn- ensure-empty-download-db! + [cfg repo] + (p/let [non-empty-entity-count (transport/invoke cfg :thread-api/q false [repo [sync-download-non-empty-query]])] + (when (and (number? non-empty-entity-count) + (pos? non-empty-entity-count)) + (throw (ex-info "graph db is not empty" + {:code :graph-db-not-empty + :repo repo + :non-empty-entity-count non-empty-entity-count}))) + nil)) + (defn- wait-sync-start-ready [config repo action] (let [timeout-ms (max 0 (or (:wait-timeout-ms action) sync-start-timeout-ms)) @@ -295,7 +321,8 @@ {:result result})}) :sync-download - (-> (p/let [remote-graphs (invoke-global config + (let [config' (download-config config)] + (-> (p/let [remote-graphs (invoke-global config' :thread-api/db-sync-list-remote-graphs []) remote-graph (some (fn [graph] @@ -307,8 +334,10 @@ :error {:code :remote-graph-not-found :message (str "remote graph not found: " (:graph action)) :graph (:graph action)}} - (p/let [result (invoke-with-repo config (:repo action) - :thread-api/db-sync-download-graph-by-id + (p/let [cfg (cli-server/ensure-server! config' (:repo action)) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) + _ (ensure-empty-download-db! cfg (:repo action)) + result (transport/invoke cfg :thread-api/db-sync-download-graph-by-id false [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] {:status :ok :data (if (map? result) @@ -316,7 +345,7 @@ {:result result})}))) (p/catch (fn [error] (exception->error error {:repo (:repo action) - :graph (:graph action)})))) + :graph (:graph action)}))))) :sync-remote-graphs (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 3b6f981e51..f5fb089003 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -130,11 +130,12 @@ (daemon/ready? lock)) (defn- spawn-server! - [{:keys [repo data-dir owner-source]}] + [{:keys [repo data-dir owner-source create-empty-db?]}] (daemon/spawn-server! {:script (db-worker-script-path) :repo repo :data-dir data-dir - :owner-source owner-source})) + :owner-source owner-source + :create-empty-db? create-empty-db?})) (defn- rewrite-lock-owner-source! [path lock owner-source] @@ -155,7 +156,8 @@ :data-dir data-dir}) (spawn-server! {:repo repo :data-dir data-dir - :owner-source requester-owner}) + :owner-source requester-owner + :create-empty-db? (:create-empty-db? config)}) (-> (wait-for-lock path) (p/catch (fn [e] (if (= :timeout (:code (ex-data e))) diff --git a/src/main/logseq/db_worker/daemon.cljs b/src/main/logseq/db_worker/daemon.cljs index 4885c2cce8..55f233dfbc 100644 --- a/src/main/logseq/db_worker/daemon.cljs +++ b/src/main/logseq/db_worker/daemon.cljs @@ -254,9 +254,10 @@ :interval-ms 250})) (defn spawn-server! - [{:keys [script repo data-dir owner-source]}] + [{:keys [script repo data-dir owner-source create-empty-db?]}] (let [owner-source (normalize-owner-source owner-source) - args #js [script "--repo" repo "--data-dir" data-dir "--owner-source" (name owner-source)] + args (clj->js (cond-> [script "--repo" repo "--data-dir" data-dir "--owner-source" (name owner-source)] + create-empty-db? (conj "--create-empty-db"))) env (js/Object.assign #js {} (.-env js/process) #js {:ELECTRON_RUN_AS_NODE "1"})] (if-not script (do diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 225e680c53..dd4e30c4a2 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -5,11 +5,14 @@ [cljs.test :refer [async deftest is use-fixtures]] [clojure.string :as string] [frontend.test.node-helper :as node-helper] + [frontend.worker.db-core :as db-core] [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.db-worker-node :as db-worker-node] + [frontend.worker.platform.node :as platform-node] [goog.object :as gobj] [logseq.cli.server :as cli-server] [logseq.cli.style :as style] + [logseq.common.config :as common-config] [logseq.db :as ldb] [promesa.core :as p])) @@ -249,6 +252,14 @@ (is (nil? (:rtc-ws-url result))) (is (= "logseq_db_parse_args" (:repo result))))) +(deftest db-worker-node-parse-args-recognizes-create-empty-db + (let [parse-args #'db-worker-node/parse-args + result (parse-args #js ["node" "dist/db-worker-node.js" + "--repo" "logseq_db_parse_args" + "--create-empty-db"])] + (is (= "logseq_db_parse_args" (:repo result))) + (is (= true (:create-empty-db? result))))) + (deftest db-worker-node-owner-source-cli-is-written-into-lock (async done (let [daemon (atom nil) @@ -323,9 +334,108 @@ (is (contains-bold? output "db-worker-node")) (is (contains-bold? output "--data-dir")) (is (contains-bold? output "--repo")) + (is (string/includes? plain-output "--create-empty-db")) + (is (contains-bold? output "--create-empty-db")) (is (not (contains-bold? output "--rtc-ws-url"))) (is (contains-bold? output "--log-level")))) +(deftest db-worker-node-start-daemon-uses-empty-datoms-when-create-empty-enabled + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-create-empty-start") + repo (str "logseq_db_create_empty_start_" (subs (str (random-uuid)) 0 8)) + lock-file-path (lock-path data-dir repo) + invoke-calls (atom []) + original-platform platform-node/node-platform + original-init-core db-core/init-core! + original-ensure-lock db-lock/ensure-lock! + original-update-lock db-lock/update-lock!] + (set! platform-node/node-platform (fn [_opts] #js {})) + (set! db-core/init-core! + (fn [_platform] + #js {:remoteInvoke (fn [method direct-pass? args] + (swap! invoke-calls conj + [method + direct-pass? + (if direct-pass? + (vec (js->clj args)) + (ldb/read-transit-str args))]) + (p/resolved nil))})) + (set! db-lock/ensure-lock! + (fn [_] + (p/resolved {:path lock-file-path + :lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :lock-id "create-empty-lock"}}))) + (set! db-lock/update-lock! (fn [_path lock] lock)) + (-> (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :create-empty-db? true + :log-level "error"}) + _ (is (= ["thread-api/init" true []] + (first @invoke-calls))) + _ (is (= ["thread-api/create-or-open-db" false [repo {:datoms []}]] + (second @invoke-calls))) + _ (stop!)] + true) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! platform-node/node-platform original-platform) + (set! db-core/init-core! original-init-core) + (set! db-lock/ensure-lock! original-ensure-lock) + (set! db-lock/update-lock! original-update-lock) + (done))))))) + +(deftest db-worker-node-start-daemon-uses-default-startup-opts-without-create-empty + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-default-start") + repo (str "logseq_db_default_start_" (subs (str (random-uuid)) 0 8)) + lock-file-path (lock-path data-dir repo) + invoke-calls (atom []) + original-platform platform-node/node-platform + original-init-core db-core/init-core! + original-ensure-lock db-lock/ensure-lock! + original-update-lock db-lock/update-lock!] + (set! platform-node/node-platform (fn [_opts] #js {})) + (set! db-core/init-core! + (fn [_platform] + #js {:remoteInvoke (fn [method direct-pass? args] + (swap! invoke-calls conj + [method + direct-pass? + (if direct-pass? + (vec (js->clj args)) + (ldb/read-transit-str args))]) + (p/resolved nil))})) + (set! db-lock/ensure-lock! + (fn [_] + (p/resolved {:path lock-file-path + :lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :lock-id "default-lock"}}))) + (set! db-lock/update-lock! (fn [_path lock] lock)) + (-> (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :log-level "error"}) + _ (is (= ["thread-api/init" true []] + (first @invoke-calls))) + _ (is (= ["thread-api/create-or-open-db" false [repo {}]] + (second @invoke-calls))) + _ (stop!)] + true) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! platform-node/node-platform original-platform) + (set! db-core/init-core! original-init-core) + (set! db-lock/ensure-lock! original-ensure-lock) + (set! db-lock/update-lock! original-update-lock) + (done))))))) + (deftest db-worker-node-repo-error-handles-keyword-methods (let [repo-error #'db-worker-node/repo-error bound-repo "logseq_db_bound"] @@ -370,6 +480,30 @@ (-> (stop!) (p/finally (fn [] (done)))) (done)))))))) +(deftest db-worker-node-create-empty-startup-skips-built-in-initial-data + (async done + (let [daemon (atom nil) + data-dir (node-helper/create-tmp-dir "db-worker-empty-initial-data") + repo (str "logseq_db_empty_initial_" (subs (str (random-uuid)) 0 8))] + (-> (p/let [{:keys [host port stop!]} + (start-daemon! {:data-dir data-dir + :repo repo + :create-empty-db? true}) + _ (reset! daemon {:stop! stop!}) + library-result (invoke host port "thread-api/q" + [repo + ['[:find ?e + :in $ ?title + :where [?e :block/title ?title]] + common-config/library-page-name]])] + (is (empty? library-result))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (if-let [stop! (:stop! @daemon)] + (-> (stop!) (p/finally (fn [] (done)))) + (done)))))))) + (deftest db-worker-node-sync-status-requires-repo-and-returns-structured-status (async done (let [daemon (atom nil) diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index ef4273564c..a8600182b8 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -261,6 +261,8 @@ (p/resolved [{:graph-id "remote-graph-id" :graph-name "demo" :graph-e2ee? true}]) + :thread-api/q + (p/resolved 0) :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))) @@ -270,20 +272,28 @@ {:base-url "http://example" :data-dir "/tmp"})] (is (= [[{:base-url "http://example" + :create-empty-db? true :data-dir "/tmp"} "logseq_db_demo"]] @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-list-remote-graphs false []] - [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]]] - @invoke-calls))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 0))) + (is (= [:thread-api/db-sync-list-remote-graphs false []] + (nth @invoke-calls 1))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 2))) + (let [[method direct-pass? args] (nth @invoke-calls 3)] + (is (= :thread-api/q method)) + (is (= false direct-pass?)) + (is (= "logseq_db_demo" (first args)))) + (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]] + (nth @invoke-calls 4)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] @@ -307,6 +317,8 @@ (p/resolved [{:graph-id "remote-graph-id" :graph-name "demo" :graph-e2ee? false}]) + :thread-api/q + (p/resolved 0) :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))) @@ -316,23 +328,32 @@ {:graph "demo" :data-dir "/tmp"})] (is (= [[{:graph "demo" + :create-empty-db? true :data-dir "/tmp"} "logseq_db_demo"] [{:graph "demo" + :create-empty-db? true :data-dir "/tmp"} "logseq_db_demo"]] @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-list-remote-graphs false []] - [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]]] - @invoke-calls))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 0))) + (is (= [:thread-api/db-sync-list-remote-graphs false []] + (nth @invoke-calls 1))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 2))) + (let [[method direct-pass? args] (nth @invoke-calls 3)] + (is (= :thread-api/q method)) + (is (= false direct-pass?)) + (is (= "logseq_db_demo" (first args)))) + (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]] + (nth @invoke-calls 4)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] @@ -388,6 +409,8 @@ (p/resolved [{:graph-id "remote-graph-id" :graph-name "demo" :graph-e2ee? true}]) + :thread-api/q + (p/resolved 0) :thread-api/db-sync-download-graph-by-id (p/rejected (ex-info "db-sync/incomplete-snapshot-frame" {:code :db-sync/incomplete-snapshot-frame @@ -407,6 +430,45 @@ (set! transport/invoke orig-invoke) (done))))))) +(deftest test-execute-sync-download-fails-fast-when-db-is-not-empty + (async done + (let [orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (set! cli-server/ensure-server! (fn [config repo] + (p/resolved (assoc config + :repo repo + :base-url "http://example")))) + (set! transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/q + (p/resolved 2) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))) + (-> (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :graph-db-not-empty (get-in result [:error :code]))) + (is (= "logseq_db_demo" (get-in result [:error :repo]))) + (is (= 2 (get-in result [:error :context :non-empty-entity-count]))) + (is (not-any? (fn [[method _ _]] + (= :thread-api/db-sync-download-graph-by-id method)) + @invoke-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-sync-remote-graphs (async done (let [orig-ensure-server! cli-server/ensure-server! diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 44eacce467..65ec027e3a 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -3,6 +3,7 @@ [clojure.string :as string] [logseq.cli.command.add :as add-command] [logseq.cli.command.show :as show-command] + [logseq.cli.command.sync :as sync-command] [logseq.cli.commands :as commands] [logseq.cli.server :as cli-server] [logseq.cli.style :as style] @@ -2512,6 +2513,33 @@ (set! cli-server/ensure-server! orig-ensure-server!) (done))))))) +(deftest test-execute-sync-download-runs-command-when-graph-is-missing + (async done + (let [orig-list-graphs cli-server/list-graphs + orig-execute sync-command/execute + captured (atom nil)] + (set! cli-server/list-graphs (fn [_] [])) + (set! sync-command/execute + (fn [action config] + (reset! captured [action config]) + (p/resolved {:status :ok + :data {:repo (:repo action)}}))) + (-> (p/let [result (commands/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :allow-missing-graph true + :require-missing-graph true} + {:data-dir "/tmp"})] + (is (= :ok (:status result))) + (is (= "logseq_db_demo" (get-in result [:data :repo]))) + (is (= :sync-download (get-in @captured [0 :type])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/list-graphs orig-list-graphs) + (set! sync-command/execute orig-execute) + (done))))))) + (deftest test-execute-graph-export (async done (let [invoke-calls (atom []) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index c97d69aea1..10d3a32cfa 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -237,7 +237,7 @@ :pending-server 0})) :thread-api/q - (p/resolved 3) + (p/resolved 0) (p/resolved nil)))) download-result (run-cli ["--graph" download-repo "sync" "download"] data-dir cfg-path) @@ -245,6 +245,7 @@ start-result (run-cli ["--graph" start-repo "sync" "start"] data-dir cfg-path) start-payload (parse-json-output-safe start-result "sync start")] (is (some #(= :thread-api/db-sync-download-graph-by-id (first %)) @invoke-calls)) + (is (some #(= :thread-api/q (first %)) @invoke-calls)) (is (some #(= :thread-api/db-sync-status (first %)) @invoke-calls)) (is (= 0 (:exit-code download-result))) (is (= "ok" (:status download-payload))) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index 92918fb799..c659056625 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -259,3 +259,51 @@ (set! daemon/wait-for-lock original-wait-lock) (set! daemon/find-orphan-processes original-find-orphans) (done))))))) + +(deftest ensure-server-forwards-create-empty-db-flag-when-spawning-daemon + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-create-empty") + repo (str "logseq_db_create_empty_" (subs (str (random-uuid)) 0 8)) + captured (atom nil) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9301} + read-lock-calls (atom 0) + original-read-lock daemon/read-lock + original-cleanup-stale daemon/cleanup-stale-lock! + original-cleanup-orphans daemon/cleanup-orphan-processes! + original-spawn daemon/spawn-server! + original-wait-lock daemon/wait-for-lock + original-wait-ready daemon/wait-for-ready] + (set! daemon/read-lock + (fn [_] + (if (= 1 (swap! read-lock-calls inc)) + nil + lock))) + (set! daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil))) + (set! daemon/cleanup-orphan-processes! (fn [_] {:orphans [] :killed-pids []})) + (set! daemon/spawn-server! + (fn [opts] + (reset! captured opts) + nil)) + (set! daemon/wait-for-lock (fn [_] (p/resolved true))) + (set! daemon/wait-for-ready (fn [_] (p/resolved true))) + (-> (cli-server/ensure-server! {:data-dir data-dir + :create-empty-db? true} + repo) + (p/then (fn [_] + (is (= repo (:repo @captured))) + (is (= (cli-server/resolve-data-dir {:data-dir data-dir}) + (:data-dir @captured))) + (is (= true (:create-empty-db? @captured))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! daemon/read-lock original-read-lock) + (set! daemon/cleanup-stale-lock! original-cleanup-stale) + (set! daemon/cleanup-orphan-processes! original-cleanup-orphans) + (set! daemon/spawn-server! original-spawn) + (set! daemon/wait-for-lock original-wait-lock) + (set! daemon/wait-for-ready original-wait-ready) + (done))))))) diff --git a/src/test/logseq/db_worker/daemon_test.cljs b/src/test/logseq/db_worker/daemon_test.cljs index 07242aa6e7..9093e88ac8 100644 --- a/src/test/logseq/db_worker/daemon_test.cljs +++ b/src/test/logseq/db_worker/daemon_test.cljs @@ -30,6 +30,45 @@ (finally (set! (.-spawn child-process) original-spawn))))) +(deftest spawn-server-appends-create-empty-db-flag-when-enabled + (let [captured (atom nil) + original-spawn (.-spawn child-process)] + (set! (.-spawn child-process) + (fn [cmd args opts] + (reset! captured {:cmd cmd + :args (vec (js->clj args)) + :opts (js->clj opts :keywordize-keys true)}) + (js-obj "unref" (fn [] nil)))) + (try + (daemon/spawn-server! {:script "/tmp/db-worker-node.js" + :repo "logseq_db_spawn_helper_test" + :data-dir "/tmp/logseq-db-worker" + :create-empty-db? true}) + (is (some #{"--create-empty-db"} (:args @captured))) + (finally + (set! (.-spawn child-process) original-spawn))))) + +(deftest spawn-server-omits-create-empty-db-flag-by-default + (let [captured (atom []) + original-spawn (.-spawn child-process) + capture! (fn [_cmd args _opts] + (swap! captured conj (vec (js->clj args))) + (js-obj "unref" (fn [] nil)))] + (set! (.-spawn child-process) capture!) + (try + (daemon/spawn-server! {:script "/tmp/db-worker-node.js" + :repo "logseq_db_spawn_helper_test" + :data-dir "/tmp/logseq-db-worker"}) + (daemon/spawn-server! {:script "/tmp/db-worker-node.js" + :repo "logseq_db_spawn_helper_test" + :data-dir "/tmp/logseq-db-worker" + :create-empty-db? false}) + (is (every? (fn [args] + (not-any? #{"--create-empty-db"} args)) + @captured)) + (finally + (set! (.-spawn child-process) original-spawn))))) + (deftest cleanup-stale-lock-removes-invalid-lock (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-daemon-helper") From c8da78efd1c1635f72d235c1ac4af4d53c30199b Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Mar 2026 10:32:29 +0800 Subject: [PATCH 114/375] 051-logseq-cli-sync-upload-fix.md --- .../051-logseq-cli-sync-upload-fix.md | 347 ++++++++++++++++++ docs/cli/logseq-cli.md | 10 + src/main/frontend/handler/db_based/sync.cljs | 5 +- src/main/frontend/worker/sync.cljs | 261 ++++++++----- src/main/frontend/worker_common/util.cljc | 2 +- src/main/logseq/cli/command/sync.cljs | 74 ++-- .../frontend/handler/db_based/sync_test.cljs | 34 ++ src/test/frontend/worker/db_sync_test.cljs | 270 ++++++++++++-- src/test/logseq/cli/command/sync_test.cljs | 32 ++ src/test/logseq/cli/integration_test.cljs | 46 +++ 10 files changed, 927 insertions(+), 154 deletions(-) create mode 100644 docs/agent-guide/051-logseq-cli-sync-upload-fix.md diff --git a/docs/agent-guide/051-logseq-cli-sync-upload-fix.md b/docs/agent-guide/051-logseq-cli-sync-upload-fix.md new file mode 100644 index 0000000000..e4c3c241ee --- /dev/null +++ b/docs/agent-guide/051-logseq-cli-sync-upload-fix.md @@ -0,0 +1,347 @@ +# Logseq CLI Sync Upload Fix Plan + +Goal: Fix `logseq sync upload` so a CLI-managed local graph can be uploaded successfully with the current `logseq-cli` + `db-worker-node` architecture, including the first upload of a graph that does not yet have a local `graph-id`. + +Architecture: Keep the CLI surface small and move the real fix into the worker-side sync layer so `:thread-api/db-sync-upload-graph` becomes self-sufficient: it should resolve or create the remote graph when needed, persist the resulting graph metadata locally, and then perform snapshot upload. + +Architecture: Remove the current error-swallowing behavior in worker upload so failures propagate back through `db-worker-node` and the CLI can report an actual error instead of a false success. + +Tech Stack: ClojureScript, `logseq-cli`, `db-worker-node`, `frontend.worker.sync`, db-sync HTTP endpoints, promesa, babashka test runner. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`. + +Related: Relates to `docs/cli/logseq-cli.md` and `docs/developers/desktop-db-worker-node.md`. + +## Problem statement + +With the current implementation, `logseq sync upload --graph ` is wired only as: + +- CLI `sync upload` +- `logseq.cli.command.sync/execute` +- `:thread-api/db-sync-upload-graph` +- `frontend.worker.sync/upload-graph!` + +That path assumes the local graph already has a usable remote `graph-id`. + +However, `frontend.worker.sync/upload-graph!` currently fails when either `http-base` or `graph-id` is missing: + +- `http-base` comes from CLI sync config and is present in the provided `cli.edn`. +- `graph-id` is read from local graph metadata via `get-graph-id`. +- For a fresh local graph that has never been uploaded or downloaded before, `graph-id` is typically missing. + +The desktop/UI flow does not have this problem because `frontend.handler.db_based.sync/` should: + +1. start or reuse the graph's `db-worker-node` +2. apply sync config to the worker +3. detect whether the local graph already has a remote `graph-id` +4. if missing, resolve or create the remote graph +5. persist the resolved/created graph id and sync metadata locally +6. upload the graph snapshot +7. return a real success result only when upload actually succeeds +8. return a real error with context when any step fails + +This should work both for: + +- an already-linked graph that has a local `graph-id` +- a fresh local graph being uploaded for the first time + +## Recommended design + +### Option A: Fix inside worker upload orchestration + +Preferred approach: make `frontend.worker.sync/upload-graph!` capable of ensuring remote graph identity before snapshot upload. + +This keeps the CLI simple and makes `:thread-api/db-sync-upload-graph` a complete unit of behavior. + +Suggested worker-side flow: + +1. Read local `graph-id`. +2. If present, continue with existing upload logic. +3. If absent, list remote graphs visible to the current auth context. +4. Match by canonical graph name. +5. If a matching remote graph exists, bind local graph to that `graph-id`. +6. If no matching remote graph exists, create one with the current graph name and schema version. +7. Persist returned graph metadata locally. +8. Continue snapshot upload using the resolved `graph-id`. + +This mirrors the desktop behavior conceptually while avoiding duplicated orchestration in CLI code. + +### Option B: Add CLI-specific preflight before invoking upload + +Alternative approach: keep worker upload low-level and add a new CLI-side preflight that creates or resolves the remote graph before invoking `:thread-api/db-sync-upload-graph`. + +I do not recommend this because it duplicates sync orchestration across CLI and desktop-adjacent code and makes the worker thread API less useful as a stable contract. + +## Scope + +### In scope + +- Fix first-time `sync upload` for CLI-managed graphs. +- Make worker upload propagate failures. +- Reuse existing remote graph APIs (`GET /graphs`, `POST /graphs`) instead of inventing a new cloud protocol. +- Add tests for first-time upload, already-linked upload, and failure propagation. +- Update CLI docs so `sync upload` behavior is explicit. + +### Out of scope + +- Changing the cloud snapshot upload protocol. +- Adding a new top-level CLI command for graph creation. +- Reworking websocket sync start/stop behavior. +- Large refactors unrelated to upload bootstrap. + +## Implementation plan + +### Phase 1. Add failing tests for the broken behavior + +1. Add a worker sync test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for first-time upload when local graph has no graph id. +2. Stub remote graph listing to return no match, stub graph creation to return a new id, and assert upload continues with that id. +3. Add a companion test where remote graph listing already contains the same graph name and assert upload reuses the existing graph id instead of creating a duplicate graph. +4. Add a failure test that simulates graph creation failure and asserts the promise rejects rather than resolving silently. +5. Add a failure test that simulates snapshot upload failure and asserts the CLI-visible result is an error, not success. +6. Add a CLI command execution/integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` or CLI integration tests that covers the real first-upload path, not just method wiring. + +### Phase 2. Move remote graph bootstrap into worker sync code + +7. Add a worker helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to ensure a usable remote graph id before snapshot upload. +8. That helper should read the local graph name from repo name using the same canonicalization rules already used by CLI and desktop sync. +9. If local graph id exists, return it immediately. +10. If local graph id is missing, call existing remote graph list API. +11. If a graph with the same canonical name exists, reuse its `graph-id` and persist it locally. +12. If no match exists, call the existing graph creation API and persist the new `graph-id` locally. +13. Keep local metadata writes inside worker code so later sync commands (`sync start`, asset upload, etc.) see a consistent graph identity. + +### Phase 3. Reuse or extract desktop graph-creation logic + +14. Avoid leaving two divergent graph-creation implementations. +15. Extract the parts of `frontend.handler.db_based.sync/`. +3. Confirm the command returns success only after remote graph bootstrap + snapshot upload finish. +4. Run `logseq sync status --graph ` and confirm the graph now has stable remote metadata. +5. Re-run `logseq sync upload --graph ` and confirm it reuses the existing graph id. +6. Force a failing auth token and confirm the CLI returns an error instead of false success. +7. Force a server-side snapshot upload failure and confirm the CLI returns an error with useful context. + +## Acceptance criteria + +The fix is complete when all of the following are true: + +- `logseq sync upload --graph ` works for a fresh local graph with no prior `graph-id`. +- The command reuses an existing same-name remote graph when appropriate. +- Worker upload failures propagate to CLI callers and no longer appear as success. +- Desktop and CLI upload bootstrap logic no longer diverge in a risky way. +- Test coverage includes first-time upload and failure propagation. +- CLI docs describe first-time upload behavior and required config without exposing secrets. + +## Risks and mitigations + +### Risk: duplicate remote graphs + +If upload always creates a graph when local `graph-id` is missing, we may create duplicates. + +Mitigation: list remote graphs first and only create when there is no exact name match. + +### Risk: inconsistent E2EE defaults + +Fresh graph creation from CLI may accidentally create an unencrypted graph while desktop defaults to encrypted. + +Mitigation: make fresh-upload `graph-e2ee?` behavior explicit and test it. + +### Risk: hidden regressions in desktop flow + +If desktop and worker share more code, UI expectations could shift. + +Mitigation: keep only transport/bootstrap logic shared; keep UI state refresh and notifications in frontend handler code. + +## Suggested file touch list + +Primary implementation files: + +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/db_based/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` + +Primary test files: + +- `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` + +Primary docs: + +- `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` + +## Question + +No open question. The current implementation gap is clear enough to proceed with a worker-centered fix. diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 16b4deb0c1..961e274c2e 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -103,6 +103,16 @@ Sync commands: - `sync config get [--graph ] ws-url|http-base|auth-token|e2ee-password` - get db-sync runtime config key - `sync config unset [--graph ] ws-url|http-base|auth-token|e2ee-password` - remove db-sync runtime config key +Sync upload behavior: +- `sync upload` requires `--graph `. +- The CLI starts or reuses that graph's `db-worker-node`, applies the current sync config, and uploads the local snapshot only after the worker has resolved a usable remote graph id. +- If the local graph already has sync metadata, upload reuses the stored remote `graph-id`. +- If the local graph does not have a stored remote `graph-id`, upload first lists visible remote graphs and reuses an exact same-name match when one exists. +- If no same-name remote graph exists, upload creates a new remote graph and persists the returned remote metadata locally before snapshot transfer. +- Fresh uploads default to encrypted remote graph creation unless local sync metadata explicitly marks the graph as non-e2ee. In headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before uploading encrypted graphs. +- `sync upload` returns a real error instead of false success when auth, remote graph bootstrap, or snapshot upload fails. +- Common upload failures include missing/invalid `auth-token`, missing `http-base`, remote graph creation failure, snapshot upload failure, and local DB/worker startup failure. + Sync download behavior: - `sync download` requires `--graph `. - If a local graph with the same name already exists, the CLI returns `graph-exists`. diff --git a/src/main/frontend/handler/db_based/sync.cljs b/src/main/frontend/handler/db_based/sync.cljs index 7d4e0544b3..b5718e3bfb 100644 --- a/src/main/frontend/handler/db_based/sync.cljs +++ b/src/main/frontend/handler/db_based/sync.cljs @@ -488,10 +488,9 @@ (defn js (with-auth-headers opts))) - text (.text resp) +(defn- parse-json-response-body + [resp url response-schema error-schema] + (p/let [text (.text resp) data (when (seq text) (js/JSON.parse text))] (if (.-ok resp) (let [body (js->clj data :keywordize-keys true) @@ -487,6 +499,11 @@ :url url :body body})))))) +(defn- fetch-json + [url opts {:keys [response-schema error-schema] :or {error-schema :error}}] + (p/let [resp (js/fetch url (clj->js (with-auth-headers opts)))] + (parse-json-response-body resp url response-schema error-schema))) + (defn- require-auth-token! [context] (when-not (seq (auth-token)) @@ -625,18 +642,20 @@ (-restore [_ addr] (restore-data-from-addr db addr)))) -(defn- create-temp-sqlite-db +(defn- clj result)))) +(defn- normalize-snapshot-row-value [value] + (cond + (nil? value) nil + (string? value) value + (number? value) value + (boolean? value) value + (= "bigint" (js* "typeof ~{}" value)) (str value) + (instance? js/Uint8Array value) (.decode text-decoder value) + (instance? js/ArrayBuffer value) (.decode text-decoder (js/Uint8Array. value)) + :else (str value))) + (defn- normalize-snapshot-rows [rows] - (mapv (fn [row] (vec row)) (array-seq rows))) + (mapv (fn [row] + (mapv normalize-snapshot-row-value (vec row))) + (array-seq rows))) (defn- encode-snapshot-rows [rows] (.encode snapshot-text-encoder (sqlite-util/write-transit-str rows))) @@ -2121,17 +2153,8 @@ (defn- repo common-config/strip-leading-db-version-prefix) + schema-version (some-> (worker-state/get-datascript-conn repo) + deref + ldb/get-graph-schema-version + :major + str) + graph-e2ee? (normalize-graph-e2ee? graph-e2ee?)] + (cond + (not (seq base)) + (fail-fast :db-sync/missing-field {:repo repo :field :http-base}) + + (not (seq graph-name)) + (fail-fast :db-sync/missing-field {:repo repo :field :graph-name}) + + :else + (do + (require-auth-token! {:repo repo :field :auth-token}) + (p/let [body (coerce-http-request :graphs/create + {:graph-name graph-name + :schema-version schema-version + :graph-e2ee? graph-e2ee?}) + _ (when (nil? body) + (fail-fast :db-sync/invalid-field {:repo repo + :field :create-graph-body})) + result (fetch-json (str base "/graphs") + {:method "POST" + :headers {"content-type" "application/json"} + :body (js/JSON.stringify (clj->js body))} + {:response-schema :graphs/create}) + graph-id (:graph-id result) + graph-e2ee? (normalize-graph-e2ee? (if (contains? result :graph-e2ee?) + (:graph-e2ee? result) + graph-e2ee?))] + (when-not (seq graph-id) + (fail-fast :db-sync/missing-field {:repo repo + :field :graph-id + :op :create-graph})) + (persist-upload-graph-identity! repo graph-id graph-e2ee?)))))) + +(defn- repo common-config/strip-leading-db-version-prefix) + local-graph-e2ee? (normalize-graph-e2ee? (sync-crypt/graph-e2ee? repo))] + (if-not (seq target-graph-name) + (fail-fast :db-sync/missing-field {:repo repo :field :graph-name}) + (p/let [remote-graphs (list-remote-graphs!) + matching-graphs (filterv (fn [{:keys [graph-name]}] + (= target-graph-name graph-name)) + remote-graphs)] + (cond + (> (count matching-graphs) 1) + (fail-fast :db-sync/ambiguous-graph-match {:repo repo + :graph-name target-graph-name + :match-count (count matching-graphs)}) + + (= 1 (count matching-graphs)) + (let [{:keys [graph-id graph-e2ee?]} (first matching-graphs)] + (persist-upload-graph-identity! repo graph-id (if (contains? (first matching-graphs) :graph-e2ee?) + graph-e2ee? + local-graph-e2ee?))) + + :else + ( - (let [base (http-base-url) - graph-id (get-graph-id repo) - update-progress (fn [payload] - (worker-util/post-message :rtc-log - (merge {:type :rtc.log/upload - :graph-uuid graph-id} - payload)))] - (if (and (seq base) (seq graph-id)) - (if-let [source-conn (worker-state/get-datascript-conn repo)] - (let [graph-e2ee? (true? (sync-crypt/graph-e2ee? repo))] - (p/let [aes-key (when graph-e2ee? - (sync-crypt/ - (p/loop [last-addr -1 - first-batch? true - loaded 0] - (let [rows (fetch-kvs-rows db last-addr upload-kvs-batch-size)] - (if (empty? rows) - (do - (client-op/remove-local-tx repo) - (client-op/update-local-tx repo 0) - (client-op/add-all-exists-asset-as-ops repo) - (update-progress {:sub-type :upload-completed - :message "Graph upload finished!"}) - {:graph-id graph-id}) - (let [max-addr (apply max (map first rows)) - rows (normalize-snapshot-rows rows) - upload-url (str base "/sync/" graph-id "/snapshot/upload?reset=" (if first-batch? "true" "false"))] - (p/let [{:keys [body encoding]} ( {"content-type" snapshot-content-type} - (string? encoding) (assoc "content-encoding" encoding)) - _ (fetch-json upload-url - {:method "POST" - :headers headers - :body body} - {:response-schema :sync/snapshot-upload})] - (let [loaded' (+ loaded (count rows))] - (update-progress {:sub-type :upload-progress - :message (str "Uploading " loaded' "/" total-rows)}) - (p/recur max-addr false loaded'))))))) - (p/finally - (fn [] - (cleanup-temp-sqlite! temp))))))) - (p/rejected (ex-info "db-sync missing datascript conn" - {:repo repo :graph-id graph-id}))) - (p/rejected (ex-info "db-sync missing upload info" - {:repo repo :base base :graph-id graph-id})))) - (p/catch (fn [error] - (js/console.error error))))) + (let [base (http-base-url)] + (if-not (seq base) + (p/rejected (ex-info "db-sync missing upload info" + {:repo repo :base base :graph-id nil})) + (if-let [source-conn (worker-state/get-datascript-conn repo)] + (p/let [{:keys [graph-id graph-e2ee?]} ( + (p/loop [last-addr -1 + first-batch? true + loaded 0] + (let [rows (fetch-kvs-rows db last-addr upload-kvs-batch-size)] + (if (empty? rows) + (do + (client-op/remove-local-tx repo) + (client-op/update-local-tx repo 0) + (client-op/add-all-exists-asset-as-ops repo) + (update-progress {:sub-type :upload-completed + :message "Graph upload finished!"}) + {:graph-id graph-id}) + (let [max-addr (apply max (map first rows)) + rows (normalize-snapshot-rows rows) + upload-url (str base "/sync/" graph-id "/snapshot/upload?reset=" (if first-batch? "true" "false"))] + (p/let [{:keys [body encoding]} ( {"content-type" snapshot-content-type} + (string? encoding) (assoc "content-encoding" encoding)) + resp (js/fetch upload-url + #js {:method "POST" + :headers (clj->js (merge (or (auth-headers) {}) headers)) + :body body}) + _ (parse-json-response-body resp upload-url :sync/snapshot-upload :error)] + (let [loaded' (+ loaded (count rows))] + (update-progress {:sub-type :upload-progress + :message (str "Uploading " loaded' "/" total-rows)}) + (p/recur max-addr false loaded'))))))) + (p/finally + (fn [] + (cleanup-temp-sqlite! temp))))))) + (p/rejected (ex-info "db-sync missing datascript conn" + {:repo repo})))))) diff --git a/src/main/frontend/worker_common/util.cljc b/src/main/frontend/worker_common/util.cljc index ec9dab4695..1a6767ec7e 100644 --- a/src/main/frontend/worker_common/util.cljc +++ b/src/main/frontend/worker_common/util.cljc @@ -28,7 +28,7 @@ (do (defn post-message [type data & {:keys [port]}] - (when-let [worker (or port js/self)] + (when-let [worker (or port (when (exists? js/self) js/self))] (.postMessage worker (ldb/write-transit-str [type data])))) (defn encode-graph-dir-name diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index 3f34ae92fc..2be65fa2ca 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -285,6 +285,46 @@ (poll!)))))] (poll!)))) +(defn- execute-sync-upload + [action config] + (-> (p/let [result (invoke-with-repo config (:repo action) + :thread-api/db-sync-upload-graph + [(:repo action)])] + {:status :ok + :data (if (map? result) + result + {:result result})}) + (p/catch (fn [error] + (exception->error error {:repo (:repo action)}))))) + +(defn- execute-sync-download + [action config] + (let [config' (download-config config)] + (-> (p/let [remote-graphs (invoke-global config' + :thread-api/db-sync-list-remote-graphs + []) + remote-graph (some (fn [graph] + (when (= (:graph action) (:graph-name graph)) + graph)) + remote-graphs)] + (if-not remote-graph + {:status :error + :error {:code :remote-graph-not-found + :message (str "remote graph not found: " (:graph action)) + :graph (:graph action)}} + (p/let [cfg (cli-server/ensure-server! config' (:repo action)) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) + _ (ensure-empty-download-db! cfg (:repo action)) + result (transport/invoke cfg :thread-api/db-sync-download-graph-by-id false + [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] + {:status :ok + :data (if (map? result) + result + {:result result})}))) + (p/catch (fn [error] + (exception->error error {:repo (:repo action) + :graph (:graph action)})))))) + (defn execute [action config] (case (:type action) @@ -312,40 +352,10 @@ :data {:result result}}) :sync-upload - (p/let [result (invoke-with-repo config (:repo action) - :thread-api/db-sync-upload-graph - [(:repo action)])] - {:status :ok - :data (if (map? result) - result - {:result result})}) + (execute-sync-upload action config) :sync-download - (let [config' (download-config config)] - (-> (p/let [remote-graphs (invoke-global config' - :thread-api/db-sync-list-remote-graphs - []) - remote-graph (some (fn [graph] - (when (= (:graph action) (:graph-name graph)) - graph)) - remote-graphs)] - (if-not remote-graph - {:status :error - :error {:code :remote-graph-not-found - :message (str "remote graph not found: " (:graph action)) - :graph (:graph action)}} - (p/let [cfg (cli-server/ensure-server! config' (:repo action)) - _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) - _ (ensure-empty-download-db! cfg (:repo action)) - result (transport/invoke cfg :thread-api/db-sync-download-graph-by-id false - [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] - {:status :ok - :data (if (map? result) - result - {:result result})}))) - (p/catch (fn [error] - (exception->error error {:repo (:repo action) - :graph (:graph action)}))))) + (execute-sync-download action config) :sync-remote-graphs (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] diff --git a/src/test/frontend/handler/db_based/sync_test.cljs b/src/test/frontend/handler/db_based/sync_test.cljs index 59b815747f..189856d5d4 100644 --- a/src/test/frontend/handler/db_based/sync_test.cljs +++ b/src/test/frontend/handler/db_based/sync_test.cljs @@ -153,6 +153,40 @@ (is false (str e)) (done))))))) +(deftest rtc-upload-graph-persists-local-e2ee-choice-and-defers-bootstrap-to-worker-test + (async done + (let [tx-called (atom nil) + worker-calls (atom []) + get-remote-graphs-calls (atom 0) + rtc-start-calls (atom 0)] + (-> (p/with-redefs [ldb/transact! (fn [repo tx-data] + (reset! tx-called {:repo repo :tx-data tx-data}) + nil) + state/js body)) + "")] + (js/Promise.resolve + #js {:ok (<= 200 status 299) + :status status + :text (fn [] (js/Promise.resolve payload))}))) + (deftest resolve-ws-token-refreshes-when-token-expired-test (async done (let [refresh-calls (atom 0) @@ -264,41 +291,41 @@ (deftest pull-ok-failure-records-last-sync-error-test (testing "pull/ok failures should surface structured runtime error state" (async done - (let [tx-payload (sqlite-util/write-transit-str [[:db/add 1 :block/title "remote"]]) - raw-message (js/JSON.stringify - (clj->js {:type "pull/ok" - :t 18 - :txs [{:t 18 :tx tx-payload}]})) - local-tx (atom 16) - orig-get-local-tx client-op/get-local-tx - orig-update-local-tx client-op/update-local-tx - orig-graph-e2ee? sync-crypt/graph-e2ee? - orig-ensure-graph-aes-key sync-crypt/ (p/let [_ (#'db-sync/handle-message! test-repo client raw-message) - error-state @(:last-sync-error client)] - (is (= 16 @local-tx)) - (is (= :missing-field (:code error-state))) - (is (= "missing-field" (:message error-state))) - (is (= :aes-key (get-in error-state [:data :field])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! client-op/get-local-tx orig-get-local-tx) - (set! client-op/update-local-tx orig-update-local-tx) - (set! sync-crypt/graph-e2ee? orig-graph-e2ee?) - (set! sync-crypt/js {:type "pull/ok" + :t 18 + :txs [{:t 18 :tx tx-payload}]})) + local-tx (atom 16) + orig-get-local-tx client-op/get-local-tx + orig-update-local-tx client-op/update-local-tx + orig-graph-e2ee? sync-crypt/graph-e2ee? + orig-ensure-graph-aes-key sync-crypt/ (p/let [_ (#'db-sync/handle-message! test-repo client raw-message) + error-state @(:last-sync-error client)] + (is (= 16 @local-tx)) + (is (= :missing-field (:code error-state))) + (is (= "missing-field" (:message error-state))) + (is (= :aes-key (get-in error-state [:data :field])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! client-op/get-local-tx orig-get-local-tx) + (set! client-op/update-local-tx orig-update-local-tx) + (set! sync-crypt/graph-e2ee? orig-graph-e2ee?) + (set! sync-crypt/ opts .-method) "GET")] + (swap! fetch-calls conj {:url url :method method}) + (cond + (and (= "https://example.com/graphs" url) + (= "GET" method)) + (json-response 200 {:graphs []}) + + (and (= "https://example.com/graphs" url) + (= "POST" method)) + (json-response 200 {:graph-id "created-graph-id" + :graph-e2ee? false}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) + (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] + (-> (p/let [result (#'db-sync/ opts .-method) "GET")] + (swap! fetch-calls conj {:url url :method method}) + (cond + (and (= "https://example.com/graphs" url) + (= "GET" method)) + (json-response 200 {:graphs [{:graph-name test-repo + :graph-id "remote-graph-id" + :graph-e2ee? false + :created-at 1 + :updated-at 1}]}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) + (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (-> (p/let [result (#'db-sync/ opts .-method) "GET")] + (cond + (and (= "https://example.com/graphs" url) + (= "GET" method)) + (json-response 200 {:graphs []}) + + (and (= "https://example.com/graphs" url) + (= "POST" method)) + (json-response 500 {:error "create failed"}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) + (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] + (-> (p/let [_ (#'db-sync/ opts .-method) "GET")] + (cond + (and (= "https://example.com/graphs" url) + (= "GET" method)) + (json-response 200 {:graphs []}) + + (and (= "https://example.com/graphs" url) + (= "POST" method)) + (do + (reset! create-body (-> opts .-body js/JSON.parse (js->clj :keywordize-keys true))) + (json-response 200 {:graph-id "encrypted-graph-id"})) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) + (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body) + sync-crypt/graph-e2ee? (fn [_repo] nil)] + (-> (p/let [result (#'db-sync/ (p/let [result (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :snapshot-upload-failed (get-in result [:error :code]))) + (is (= 500 (get-in result [:error :context :status]))) + (is (= "graph-1" (get-in result [:error :context :graph-id])))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest test-execute-sync-download (async done (let [orig-ensure-server! cli-server/ensure-server! diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 10d3a32cfa..22a1ccfa70 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -264,6 +264,52 @@ (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke))))))) +(deftest test-cli-sync-upload-with-mocked-worker-bootstrap + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-cli") + upload-repo "sync-upload-graph" + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" upload-repo] data-dir cfg-path) + create-payload (parse-json-output-safe create-result "graph create") + _ (is (= 0 (:exit-code create-result))) + _ (is (= "ok" (:status create-payload))) + _ (set! cli-server/ensure-server! + (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + _ (set! transport/invoke + (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) + + :thread-api/db-sync-upload-graph + (p/resolved {:graph-id "created-graph-id"}) + + (p/resolved nil)))) + upload-result (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path) + upload-payload (parse-json-output-safe upload-result "sync upload")] + (is (= 0 (:exit-code upload-result))) + (is (= "ok" (:status upload-payload))) + (is (= "created-graph-id" (get-in upload-payload [:data :graph-id]))) + (is (= [[:thread-api/set-db-sync-config [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-upload-graph ["logseq_db_sync-upload-graph"]]] + @invoke-calls)) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke))))))) + (deftest ^:long test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From 4fe1dde02ffabd81719d94b58d4ee2faa78d8e89 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Mar 2026 16:50:06 +0800 Subject: [PATCH 115/375] 052-logseq-cli-sync-upload-graph-uuid-alignment.md --- ...eq-cli-sync-upload-graph-uuid-alignment.md | 172 ++++++++++++++++++ docs/cli/logseq-cli.md | 2 + src/main/frontend/worker/sync.cljs | 40 ++-- src/test/frontend/worker/db_sync_test.cljs | 104 +++++++---- src/test/logseq/cli/integration_test.cljs | 55 ++++++ 5 files changed, 319 insertions(+), 54 deletions(-) create mode 100644 docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md diff --git a/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md b/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md new file mode 100644 index 0000000000..288acc2032 --- /dev/null +++ b/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md @@ -0,0 +1,172 @@ +# CLI Sync Upload Graph UUID Alignment Implementation Plan + +Goal: Make `logseq sync upload` persist local graph UUID metadata so CLI and web app upload flows leave the graph in the same sync-ready state. + +Architecture: Keep `:thread-api/db-sync-upload-graph` as the single upload contract used by both CLI and web app. +Architecture: Align identity persistence inside `frontend.worker.sync` so every resolved graph id is written to both client-op storage and graph KV metadata. +Architecture: Use web app graph identity persistence semantics from `frontend.handler.db_based.sync/` succeeds, local graph metadata must contain a stable `:logseq.kv/graph-uuid` value in graph KV and the same graph id in client-op storage. + +The persistence rule must be enforced by worker sync code so both CLI and web app callers inherit identical behavior. + +When upload finds graph id from client-op fallback, worker must backfill missing graph KV UUID before returning success. + +Repeated uploads must be idempotent and must not create a new graph id or rewrite to a different value unexpectedly. + +## Architecture and integration points + +```text +CLI path + logseq.cli.command.sync/execute-sync-upload + -> logseq.cli.transport/invoke POST /v1/invoke + -> frontend.worker.db_worker_node /v1/invoke + -> :thread-api/db-sync-upload-graph in frontend.worker.db_core + -> frontend.worker.sync/ frontend.worker.sync/upload-graph! + +Web app path + frontend.handler.db_based.sync/ state/ frontend.worker.sync/ frontend.worker.sync/upload-graph! + +Web app source-of-truth identity persistence reference + frontend.handler.db_based.sync/ ldb/transact! with :logseq.kv/graph-uuid +``` + +## Testing Plan + +I will add worker unit tests that fail first when upload identity resolution does not persist `:logseq.kv/graph-uuid` into the graph database. + +I will add worker unit tests that fail first for the three identity branches, which are graph id from remote create, graph id from remote name match, and graph id from client-op fallback. + +I will add regression assertions that verify the persisted graph UUID is readable through `logseq.db/get-graph-rtc-uuid` and matches `client-op/get-graph-uuid`. + +I will add a CLI-facing regression check that validates upload success is followed by graph info data containing `logseq.kv/graph-uuid` in real or staged integration coverage. + +I will run targeted tests after each micro-change, then run broad lint and test commands before final review. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1. Add failing worker tests for UUID persistence parity. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for remote-create upload bootstrap to assert `ldb/get-graph-rtc-uuid` is set after ` --output json` and confirm `data.kv.logseq.kv/graph-uuid` is present. If it is missing, rerun `sync upload` for the same graph to trigger identity backfill. Sync download behavior: - `sync download` requires `--graph `. diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index 3f436daa78..29cac24c00 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -207,7 +207,6 @@ (def ^:private max-asset-size (* 100 1024 1024)) (def ^:private upload-kvs-batch-size 500) (def ^:private snapshot-content-type "application/transit+json") -(def ^:private snapshot-content-encoding "gzip") (def ^:private snapshot-text-encoder (js/TextEncoder.)) (def ^:private reconnect-base-delay-ms 1000) (def ^:private reconnect-max-delay-ms 30000) @@ -2140,32 +2139,37 @@ (.set out data 4) out)) -(defn- maybe-compress-stream [stream] - (if (exists? js/CompressionStream) - (.pipeThrough stream (js/CompressionStream. "gzip")) - stream)) - -(defn- uuid + [repo graph-id] + (when-not (seq graph-id) + (fail-fast :db-sync/missing-field {:repo repo :field :graph-id})) + (try + (uuid graph-id) + (catch :default e + (fail-fast :db-sync/invalid-field {:repo repo + :field :graph-id + :value graph-id + :error e})))) + (defn- set-graph-sync-metadata! - [repo graph-e2ee?] + [repo graph-id graph-e2ee?] (when-let [conn (worker-state/get-datascript-conn repo)] - (ldb/transact! conn [(ldb/kv :logseq.kv/graph-remote? true) + (ldb/transact! conn [(ldb/kv :logseq.kv/graph-uuid (graph-id->uuid repo graph-id)) + (ldb/kv :logseq.kv/graph-remote? true) (ldb/kv :logseq.kv/graph-rtc-e2ee? (true? graph-e2ee?))]))) (defn- persist-upload-graph-identity! [repo graph-id graph-e2ee?] - (let [graph-e2ee? (normalize-graph-e2ee? graph-e2ee?)] - (set-graph-sync-metadata! repo graph-e2ee?) + (let [graph-id (some-> graph-id str) + graph-e2ee? (normalize-graph-e2ee? graph-e2ee?)] + (when-not (seq graph-id) + (fail-fast :db-sync/missing-field {:repo repo :field :graph-id})) + (set-graph-sync-metadata! repo graph-id graph-e2ee?) (ensure-client-graph-uuid! repo graph-id) {:graph-id graph-id :graph-e2ee? graph-e2ee?})) @@ -2227,8 +2231,8 @@ (defn- repo common-config/strip-leading-db-version-prefix) local-graph-e2ee? (normalize-graph-e2ee? (sync-crypt/graph-e2ee? repo))] (if-not (seq target-graph-name) diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index 317428726b..1b8646bd69 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -297,35 +297,26 @@ :t 18 :txs [{:t 18 :tx tx-payload}]})) local-tx (atom 16) - orig-get-local-tx client-op/get-local-tx - orig-update-local-tx client-op/update-local-tx - orig-graph-e2ee? sync-crypt/graph-e2ee? - orig-ensure-graph-aes-key sync-crypt/ (p/let [_ (#'db-sync/handle-message! test-repo client raw-message) - error-state @(:last-sync-error client)] - (is (= 16 @local-tx)) - (is (= :missing-field (:code error-state))) - (is (= "missing-field" (:message error-state))) - (is (= :aes-key (get-in error-state [:data :field])))) + (-> (p/with-redefs [client-op/get-local-tx (fn [_repo] @local-tx) + client-op/update-local-tx (fn [_repo tx] (reset! local-tx tx)) + sync-crypt/graph-e2ee? (fn [_repo] true) + sync-crypt/latest-remote-tx {}) (with-datascript-conns conn client-ops-conn (fn [] - (-> (p/let [_ (#'db-sync/handle-message! test-repo client raw-new) - _ (#'db-sync/handle-message! test-repo client raw-stale) - parent' (d/entity @conn parent-id)] - (is (= "remote-new-title" (:block/title parent'))) - (is (= 2 (client-op/get-local-tx test-repo)))) + (-> (p/with-redefs [sync-crypt/graph-e2ee? (fn [_repo] false)] + (p/let [_ (client-op/update-local-tx test-repo 0) + _ (#'db-sync/handle-message! test-repo client raw-new) + _ (#'db-sync/handle-message! test-repo client raw-stale) + parent' (d/entity @conn parent-id)] + (is (= "remote-new-title" (:block/title parent'))) + (is (= 2 (client-op/get-local-tx test-repo))))) (p/finally (fn [] (reset! db-sync/*repo->latest-remote-tx latest-prev) (done)))))))))) @@ -1320,10 +1313,12 @@ (deftest ensure-upload-graph-identity-creates-remote-graph-when-local-graph-id-missing-test (testing "first upload bootstrap creates and persists a remote graph when local metadata is missing" (async done - (let [client-ops-conn (d/create-conn client-op/schema-in-db) + (let [conn (d/create-conn {}) + client-ops-conn (d/create-conn client-op/schema-in-db) self-prev (js* "globalThis.self") fetch-prev js/fetch config-prev @worker-state/*db-sync-config + created-graph-id "c52bf9d4-3a95-4f17-9c8c-7f86eef5f75d" fetch-calls (atom [])] (aset js/globalThis "self" #js {:postMessage (fn [_] nil)}) (reset! worker-state/*db-sync-config {:http-base "https://example.com" @@ -1339,19 +1334,21 @@ (and (= "https://example.com/graphs" url) (= "POST" method)) - (json-response 200 {:graph-id "created-graph-id" + (json-response 200 {:graph-id created-graph-id :graph-e2ee? false}) :else (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) - (with-worker-conns {} {test-repo client-ops-conn} + (with-worker-conns {test-repo conn} {test-repo client-ops-conn} (fn [] (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] (-> (p/let [result (#'db-sync/ (ldb/get-graph-rtc-uuid @conn) str))) (is (= [{:url "https://example.com/graphs" :method "GET"} {:url "https://example.com/graphs" :method "POST"}] @fetch-calls))) @@ -1366,10 +1363,12 @@ (deftest ensure-upload-graph-identity-reuses-existing-remote-graph-when-local-graph-id-missing-test (testing "first upload bootstrap reuses a same-name remote graph instead of creating a duplicate" (async done - (let [client-ops-conn (d/create-conn client-op/schema-in-db) + (let [conn (d/create-conn {}) + client-ops-conn (d/create-conn client-op/schema-in-db) self-prev (js* "globalThis.self") fetch-prev js/fetch config-prev @worker-state/*db-sync-config + remote-graph-id "56cfcfc5-035a-4c6a-a5ba-70fe9cc27530" fetch-calls (atom [])] (aset js/globalThis "self" #js {:postMessage (fn [_] nil)}) (reset! worker-state/*db-sync-config {:http-base "https://example.com" @@ -1382,20 +1381,22 @@ (and (= "https://example.com/graphs" url) (= "GET" method)) (json-response 200 {:graphs [{:graph-name test-repo - :graph-id "remote-graph-id" + :graph-id remote-graph-id :graph-e2ee? false :created-at 1 :updated-at 1}]}) :else (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) - (with-worker-conns {} {test-repo client-ops-conn} + (with-worker-conns {test-repo conn} {test-repo client-ops-conn} (fn [] (-> (p/let [result (#'db-sync/ (ldb/get-graph-rtc-uuid @conn) str))) (is (= [{:url "https://example.com/graphs" :method "GET"}] @fetch-calls))) (p/catch (fn [e] @@ -1490,6 +1491,37 @@ (reset! worker-state/*db-sync-config config-prev) (done))))))))))) +(deftest ensure-upload-graph-identity-backfills-graph-kv-from-client-op-fallback-test + (testing "upload identity fallback backfills graph kv and stays idempotent" + (async done + (let [conn (d/create-conn {}) + client-ops-conn (d/create-conn client-op/schema-in-db) + fallback-graph-id "6f3b0d47-4a62-4da7-8e56-7e5dbfbe3f47" + list-calls (atom 0)] + (with-worker-conns {test-repo conn} {test-repo client-ops-conn} + (fn [] + (with-redefs [sync-crypt/graph-e2ee? (fn [_repo] nil) + db-sync/get-graph-id (fn [_repo] fallback-graph-id) + db-sync/list-remote-graphs! (fn [] + (swap! list-calls inc) + (p/resolved []))] + (is (nil? (client-op/get-graph-uuid test-repo))) + (is (nil? (ldb/get-graph-rtc-uuid @conn))) + (let [first-result-p (#'db-sync/ (p/let [first-result first-result-p + second-result second-result-p] + (is (= fallback-graph-id (:graph-id first-result))) + (is (= fallback-graph-id (:graph-id second-result))) + (is (= (:graph-e2ee? first-result) + (:graph-e2ee? second-result))) + (is (= fallback-graph-id + (some-> (ldb/get-graph-rtc-uuid @conn) str))) + (is (= 0 @list-calls))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))))))))) + (deftest ^:long rehydrate-large-title-test (testing "rehydrate fills empty title from object storage" (async done diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 22a1ccfa70..746e3b65ca 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -310,6 +310,61 @@ (set! cli-server/ensure-server! orig-ensure-server!) (set! transport/invoke orig-invoke))))))) +(deftest test-cli-sync-upload-followed-by-graph-info-shows-graph-uuid-test + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-info-cli") + upload-repo "sync-upload-graph-info" + uploaded-graph-id "0f64b4a9-6f31-4f35-a83c-6b16f9ddf1ff" + orig-ensure-server! cli-server/ensure-server! + orig-invoke transport/invoke + invoke-calls (atom [])] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" upload-repo] data-dir cfg-path) + create-payload (parse-json-output-safe create-result "graph create") + _ (is (= 0 (:exit-code create-result))) + _ (is (= "ok" (:status create-payload))) + _ (set! cli-server/ensure-server! + (fn [config _repo] + (p/resolved (assoc config :base-url "http://example")))) + _ (set! transport/invoke + (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) + + :thread-api/db-sync-upload-graph + (p/resolved {:graph-id uploaded-graph-id}) + + :thread-api/q + (p/resolved [[:logseq.kv/graph-uuid uploaded-graph-id] + [:logseq.kv/schema-version "65"]]) + + (p/resolved nil)))) + upload-result (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path) + upload-payload (parse-json-output-safe upload-result "sync upload") + info-result (run-cli ["--graph" upload-repo "graph" "info"] data-dir cfg-path) + info-payload (parse-json-output-safe info-result "graph info after upload") + q-call (some (fn [[method args]] + (when (= :thread-api/q method) + args)) + @invoke-calls)] + (is (= 0 (:exit-code upload-result))) + (is (= "ok" (:status upload-payload))) + (is (= uploaded-graph-id (get-in upload-payload [:data :graph-id]))) + (is (= 0 (:exit-code info-result))) + (is (= "ok" (:status info-payload))) + (is (= uploaded-graph-id + (get-in info-payload [:data :kv :logseq.kv/graph-uuid]))) + (is (= "logseq_db_sync-upload-graph-info" (first q-call)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (set! cli-server/ensure-server! orig-ensure-server!) + (set! transport/invoke orig-invoke) + (done))))))) + (deftest ^:long test-cli-graph-list (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From 900ebf210e01f4fa7c261763257b80a2588226d7 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Mar 2026 21:53:35 +0800 Subject: [PATCH 116/375] 053-logseq-cli-async-test-isolation.md --- .../053-logseq-cli-async-test-isolation.md | 288 ++++ src/main/logseq/cli/command/show.cljs | 4 +- .../frontend/worker/db_worker_node_test.cljs | 187 +-- src/test/logseq/cli/command/doctor_test.cljs | 172 +- src/test/logseq/cli/command/graph_test.cljs | 88 +- src/test/logseq/cli/command/show_test.cljs | 29 +- src/test/logseq/cli/command/sync_test.cljs | 1081 ++++++------ src/test/logseq/cli/commands_test.cljs | 1480 +++++++---------- src/test/logseq/cli/integration_test.cljs | 182 +- src/test/logseq/cli/server_test.cljs | 486 +++--- src/test/logseq/cli/test_helper.cljs | 30 + src/test/logseq/cli/transport_test.cljs | 33 +- 12 files changed, 1927 insertions(+), 2133 deletions(-) create mode 100644 docs/agent-guide/053-logseq-cli-async-test-isolation.md create mode 100644 src/test/logseq/cli/test_helper.cljs diff --git a/docs/agent-guide/053-logseq-cli-async-test-isolation.md b/docs/agent-guide/053-logseq-cli-async-test-isolation.md new file mode 100644 index 0000000000..2f5dcb2ccd --- /dev/null +++ b/docs/agent-guide/053-logseq-cli-async-test-isolation.md @@ -0,0 +1,288 @@ +# Logseq CLI and db-worker-node Async Test Isolation Plan + +Goal: Audit all CLI-related async test cases and ensure every mocked function, overridden global binding, and shared test state is reset after each test case so no test leaks state into later tests. + +Architecture: Prefer per-test scoped mocking (`with-redefs` / `p/with-redefs`) and explicit `:each` fixtures for mutable globals, while keeping real process/server lifecycle cleanup in `p/finally` so async resources are always released. + +Tech Stack: ClojureScript, `cljs.test`, `promesa`, Node.js-based test runtime. + +Related: Builds on `docs/agent-guide/001-logseq-cli.md`, `docs/agent-guide/003-db-worker-node-cli-orchestration.md`, `docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md`, and `docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md`. + +## Problem statement + +Current CLI-related async tests rely heavily on mutating namespace vars and process globals with `set!`, then restoring them manually in `p/finally` blocks. + +That pattern works only if every async branch reaches cleanup. It is easy to miss one restore path, and a failed or aborted test can leave mutated globals behind for later tests. + +The most leak-prone coverage today is concentrated in these files: + +- `src/test/logseq/cli/command/sync_test.cljs` — many async tests mutate `cli-server/ensure-server!`, `transport/invoke`, and `cli-config/update-config!` with `set!`. +- `src/test/logseq/cli/commands_test.cljs` — many async command execution tests mutate `cli-server/*`, `transport/invoke`, and add-command resolver vars. +- `src/test/logseq/cli/server_test.cljs` — async tests mutate daemon fns plus process globals like `js/process.kill` and `child_process.spawn`. +- `src/test/logseq/cli/command/doctor_test.cljs` and `src/test/logseq/cli/command/graph_test.cljs` — smaller but repeated manual global override patterns. +- `src/test/logseq/cli/integration_test.cljs` — a mix of safe `p/with-redefs` usage and unsafe direct `set!` of `cli-server/ensure-server!`, `transport/invoke`, and `process.stderr.write` interception. +- `src/test/logseq/cli/transport_test.cljs` — mutates `url.parse` and depends on manual restoration around async flows. +- `src/test/frontend/worker/db_worker_node_test.cljs` — async tests mutate platform/db-core/db-lock vars and also interact with module-level `defonce` atoms such as `*sse-clients`, `*ready?`, `*lock-info`, and `*file-handler` in `src/main/frontend/worker/db_worker_node.cljs`. + +The risk is not theoretical: these tests are asynchronous, share a single Node process, and run against real namespace vars or process objects. Any missing reset can change behavior of unrelated tests that run later in the same suite. + +## Current leak vectors + +### 1. Manual `set!` / restore in async tests + +The dominant pattern is: + +1. capture original var value, +2. `set!` the var to a mock, +3. run async code, +4. restore in `p/finally`. + +This appears throughout `sync_test.cljs`, `commands_test.cljs`, `server_test.cljs`, `doctor_test.cljs`, `graph_test.cljs`, and parts of `integration_test.cljs`. + +This pattern is brittle because cleanup is duplicated in every test and can drift over time. + +### 2. Process-global overrides + +Some tests override Node globals directly, for example: + +- `child_process.spawn` in `src/test/logseq/cli/server_test.cljs` +- `js/process.kill` in `src/test/logseq/cli/server_test.cljs` +- `js/process.stderr.write` via `capture-stderr!` in `src/test/logseq/cli/integration_test.cljs` +- `url.parse` in `src/test/logseq/cli/transport_test.cljs` + +These are more dangerous than namespace-local mocks because they affect any code running in the same process while the override is active. + +### 3. Module-level mutable atoms in db-worker-node + +`src/main/frontend/worker/db_worker_node.cljs` keeps daemon state in `defonce` atoms: + +- `*ready?` +- `*sse-clients` +- `*lock-info` +- `*file-handler` + +`db_worker_node_test.cljs` already has one `:each` fixture for print suppression, but the file still contains tests that interact with or depend on mutable daemon state. Some tests locally save and restore `*sse-clients`; others rely on daemon stop paths to reset internals. + +This should be normalized into a single fixture-backed reset strategy so every test starts from a known baseline even if a previous test fails unexpectedly. + +### 4. Mixed mocking styles across files + +Some tests already use safer scoped mocking: + +- `p/with-redefs` in `src/test/logseq/cli/integration_test.cljs` +- `with-redefs` in targeted synchronous tests + +But many nearby tests still use raw `set!`. The inconsistency makes future maintenance error-prone and makes it harder to know which tests are safely isolated. + +## Scope + +This plan covers async tests for current `logseq-cli` and `db-worker-node` behavior. + +In scope: + +- `src/test/logseq/cli/*.cljs` +- `src/test/logseq/cli/command/*.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` if follow-up cleanup is needed for consistency + +Out of scope: + +- Non-CLI frontend async tests unless a reusable helper extracted here is intentionally shared +- Refactoring production runtime logic unless needed only to expose test-reset hooks for daemon state +- Rewriting all sync tests that already use safe scoped mocks + +## Desired end state + +After this work: + +- every async test that mocks a namespace var uses scoped mocking or a standardized helper that guarantees restoration; +- every process-global override has one canonical helper with guaranteed teardown; +- every db-worker-node mutable singleton used by tests is reset in an `:each` fixture; +- no CLI-related async test relies on state left behind by a previous test; +- the suite can run target namespaces together, repeatedly, and in different orders without order-dependent failures. + +## Testing Plan + +I will first lock in the current risk surface by inventorying every CLI-related async test that uses `set!`, process-global mutation, or mutable singleton state. + +I will convert one representative file in each category to a safer reset pattern before touching the rest, so the approach is proven incrementally. + +I will run targeted test namespaces repeatedly and in grouped combinations to detect order-dependent leakage. + +NOTE: I will write or adjust tests/helpers before broad refactors where possible, and any unexpected async failure will be debugged before continuing wide mechanical conversion. + +## Implementation plan + +1. Create an audit checklist of all CLI-related async tests that currently use `set!`, process-global mutation, manual restore logic, or mutable singleton state. + +2. Group those findings by leak type: + - namespace-var overrides, + - Node process/global overrides, + - db-worker-node singleton atoms, + - real server/daemon lifecycle cleanup. + +3. Add a small shared test helper namespace for CLI async isolation, e.g. `src/test/logseq/cli/test_helper.cljs`, to centralize patterns that should no longer be repeated inline. + +4. In that helper, add wrappers/macros/functions for scoped async mocking around Promesa flows so tests can replace repeated `orig-*` / `set!` / `p/finally restore` boilerplate. + +5. Prefer `p/with-redefs` for namespace vars that are only needed during the async body, especially in: + - `src/test/logseq/cli/command/sync_test.cljs` + - `src/test/logseq/cli/commands_test.cljs` + - `src/test/logseq/cli/command/doctor_test.cljs` + - `src/test/logseq/cli/command/graph_test.cljs` + - targeted sections of `src/test/logseq/cli/integration_test.cljs` + +6. Convert low-risk files first (`graph_test.cljs`, `doctor_test.cljs`, `main_test.cljs` if needed) to establish the preferred style with minimal surface area. + +7. Convert `src/test/logseq/cli/command/sync_test.cljs` from repeated manual `set!` restoration to scoped mocks. This file should become the reference pattern for async command tests. + +8. Convert `src/test/logseq/cli/commands_test.cljs` in batches, prioritizing the async sections that currently mutate `cli-server/list-graphs`, `cli-server/ensure-server!`, `transport/invoke`, and add-command resolution vars. + +9. For `src/test/logseq/cli/server_test.cljs`, separate two concerns: + - keep real filesystem/server cleanup in `p/finally`, + - move mock restoration for functions/globals into scoped helper wrappers wherever possible. + +10. For process-global overrides that cannot use ordinary `with-redefs` (for example `child_process.spawn`, `js/process.kill`, `process.stderr.write`, `url.parse`), add dedicated helper functions that use `try`/`finally` or Promesa-aware wrappers so restoration is guaranteed from one place. + +11. Refactor `src/test/logseq/cli/integration_test.cljs` to replace the remaining direct `set!` mocks with `p/with-redefs` where possible, since that file already demonstrates the safer pattern in other tests. + +12. Add or reuse an `:each` fixture in `src/test/frontend/worker/db_worker_node_test.cljs` to reset db-worker-node mutable singleton atoms before and after every test: + - `*ready?` + - `*sse-clients` + - `*lock-info` + - `*file-handler` + +13. Keep the existing quiet-print fixture in `db_worker_node_test.cljs`, but merge or compose it with the new daemon-state reset fixture so both print behavior and daemon singletons are normalized per test. + +14. Review tests in `db_worker_node_test.cljs` that directly save/restore individual atoms (for example `*sse-clients`) and simplify them once the shared fixture guarantees a clean baseline. + +15. If a daemon stop path is currently relied on for cleanup, keep asserting `stop!` behavior but do not rely on it as the only test isolation mechanism. + +16. Add comments in the helper or fixture code documenting the rule: mock restoration belongs in scoped helpers/fixtures, while resource cleanup (servers, files, daemons) belongs in `p/finally`. + +17. Update any contributor-facing guidance near the modified tests if a recurring pattern should be preserved, especially when replacing repetitive manual restore code with helper abstractions. + +18. Run targeted namespaces individually after each batch of refactors: + - `logseq.cli.command.graph-test` + - `logseq.cli.command.doctor-test` + - `logseq.cli.command.sync-test` + - `logseq.cli.server-test` + - `logseq.cli.transport-test` + - `logseq.cli.commands-test` + - `logseq.cli.integration-test` + - `frontend.worker.db-worker-node-test` + +19. Run grouped combinations of the above namespaces multiple times in the same process to catch order-dependent leaks. + +20. Finish with `bb dev:lint-and-test` if the changed surface is stable enough and runtime permits. + +## Verification strategy + +Use three levels of verification. + +### A. Static inventory verification + +Re-scan CLI-related test files and confirm that leak-prone raw patterns are either gone or intentionally wrapped: + +- direct `set!` to namespace vars in async tests should be eliminated or heavily reduced; +- any remaining direct process-global mutation must be routed through a dedicated helper with centralized restore logic; +- db-worker-node singleton atoms should be covered by fixtures rather than ad hoc local resets. + +### B. Repetition and order verification + +Run the target namespaces repeatedly and in different orders. + +At minimum: + +- run each namespace alone; +- run `sync-test`, `commands-test`, and `integration-test` together; +- run `server-test`, `transport-test`, and `db_worker_node_test` together; +- re-run the same grouped command at least twice to catch leaked state from the previous run. + +### C. Behavioral smoke verification + +Confirm that tests still validate real async behavior rather than only helper abstractions: + +- server startup/shutdown tests still exercise real server lifecycle paths; +- integration tests still verify CLI flow outputs; +- db-worker-node tests still verify daemon and HTTP behavior; +- helper conversion does not weaken assertions or skip cleanup-sensitive paths. + +## Proposed helper patterns + +### Pattern 1: Scoped Promesa var redefs + +For ordinary vars such as `cli-server/ensure-server!`, `transport/invoke`, `cli-config/update-config!`, and `db-lock/update-lock!`, prefer `p/with-redefs` around the async body. + +This should replace bespoke `orig-*` capture and restoration in most test files. + +### Pattern 2: Process-global guard helpers + +For globals like `process.stderr.write`, `process.kill`, or JS module properties such as `child_process.spawn` and `url.parse`, create helper wrappers such as: + +- `with-stderr-write-capture` +- `with-process-kill-mock` +- `with-child-process-spawn-mock` +- `with-js-property-override` + +These helpers should always restore the original value in a single `finally` path. + +### Pattern 3: db-worker-node state reset fixture + +Create a fixture that snapshots and resets db-worker-node singleton atoms around each test. The fixture should not depend on individual tests remembering which atoms they touched. + +If some atoms contain resources such as handlers or open references, the fixture should also null them out after test completion. + +## Edge cases to validate during implementation + +If an async test fails before reaching the happy path, mock restoration must still happen. + +If a mocked function is rebound inside nested async calls, the outer helper should still restore the original after the Promise chain settles. + +If real daemon startup fails, fixture teardown must not throw while resetting singleton atoms. + +If `capture-stderr!` or similar helpers are used concurrently in the future, helper APIs should make nesting/ownership behavior explicit. + +If a test still needs local save/restore of atom contents for assertion purposes, that local logic should compose safely with the global fixture baseline. + +## Expected file touch points + +Primary expected edits: + +- `src/test/logseq/cli/command/graph_test.cljs` +- `src/test/logseq/cli/command/doctor_test.cljs` +- `src/test/logseq/cli/command/sync_test.cljs` +- `src/test/logseq/cli/commands_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `src/test/logseq/cli/transport_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/logseq/cli/test_helper.cljs` or similar new helper namespace + +Possible follow-up edits: + +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` if helper reuse or fixture alignment is beneficial +- any nearby docs/comments that explain the approved async mocking pattern + +## Non-goals + +This plan does not require rewriting working synchronous tests just to match style. + +This plan does not require changing production APIs unless a tiny testability hook is necessary for deterministic reset behavior. + +This plan does not require introducing a new test framework; it should stay within current `cljs.test` and `promesa` patterns. + +## Decision + +Do not add a new lint rule or automated check for this work. + +Enforce the no-leaky-mocks rule by convention, shared helpers, and fixtures only. + +Adopt a two-layer strategy: + +1. use scoped redefs/helpers for mocked functions and process globals; +2. use `:each` fixtures for mutable singleton state owned by `db-worker-node`. + +This keeps the rollout focused on improving test isolation directly, avoids adding maintenance burden for a custom check, and still addresses both function mocking leaks and shared-state leaks. + +--- diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index a22aebdd2c..a65203e50a 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -52,11 +52,9 @@ (defn- resolve-stdin-id [options] - (let [id-present? (or (contains? options :id) (some? (:id options))) - stdin (cond + (let [stdin (cond (contains? options :stdin) (:stdin options) (:id-from-stdin? options) (read-stdin) - (and id-present? (stdin-available?)) (read-stdin) :else nil) normalized (normalize-stdin-id stdin)] (if (string/blank? normalized) diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index dd4e30c4a2..0f6de7c617 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -137,8 +137,25 @@ (when-let [orig @*orig-print-fn] (set-print-fn! orig))) -(use-fixtures :each {:before quiet-debug-output-before - :after quiet-debug-output-after}) +(defn- reset-daemon-state! + [] + (reset! @#'db-worker-node/*ready? false) + (reset! @#'db-worker-node/*sse-clients #{}) + (reset! @#'db-worker-node/*lock-info nil) + (reset! @#'db-worker-node/*file-handler nil)) + +(defn- normalize-db-worker-state-before + [] + (quiet-debug-output-before) + (reset-daemon-state!)) + +(defn- normalize-db-worker-state-after + [] + (reset-daemon-state!) + (quiet-debug-output-after)) + +(use-fixtures :each {:before normalize-db-worker-state-before + :after normalize-db-worker-state-after}) (deftest db-worker-node-data-dir-permission-error (async done @@ -303,24 +320,20 @@ (deftest db-worker-node-handle-event-encodes-sse-json-payload (let [handle-event! #'db-worker-node/handle-event! *sse-clients @#'db-worker-node/*sse-clients - old-clients @*sse-clients writes (atom []) fake-res #js {:write (fn [message] (swap! writes conj message))}] - (try - (reset! *sse-clients #{fake-res}) - (handle-event! "sync-db-changes" {:repo "graph-a"}) - (is (= 1 (count @writes))) - (let [raw-message (first @writes) - event-json (-> raw-message - (string/replace-first #"^data: " "") - (string/replace #"\n\n$" "")) - parsed (js->clj (js/JSON.parse event-json) :keywordize-keys true)] - (is (= "sync-db-changes" (:type parsed))) - (is (= {:repo "graph-a"} - (ldb/read-transit-str (:payload parsed))))) - (finally - (reset! *sse-clients old-clients))))) + (reset! *sse-clients #{fake-res}) + (handle-event! "sync-db-changes" {:repo "graph-a"}) + (is (= 1 (count @writes))) + (let [raw-message (first @writes) + event-json (-> raw-message + (string/replace-first #"^data: " "") + (string/replace #"\n\n$" "")) + parsed (js->clj (js/JSON.parse event-json) :keywordize-keys true)] + (is (= "sync-db-changes" (:type parsed))) + (is (= {:repo "graph-a"} + (ldb/read-transit-str (:payload parsed))))))) (deftest db-worker-node-help-omits-auth-token (let [show-help! #'db-worker-node/show-help! @@ -344,97 +357,75 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-create-empty-start") repo (str "logseq_db_create_empty_start_" (subs (str (random-uuid)) 0 8)) lock-file-path (lock-path data-dir repo) - invoke-calls (atom []) - original-platform platform-node/node-platform - original-init-core db-core/init-core! - original-ensure-lock db-lock/ensure-lock! - original-update-lock db-lock/update-lock!] - (set! platform-node/node-platform (fn [_opts] #js {})) - (set! db-core/init-core! - (fn [_platform] - #js {:remoteInvoke (fn [method direct-pass? args] - (swap! invoke-calls conj - [method - direct-pass? - (if direct-pass? - (vec (js->clj args)) - (ldb/read-transit-str args))]) - (p/resolved nil))})) - (set! db-lock/ensure-lock! - (fn [_] - (p/resolved {:path lock-file-path - :lock {:repo repo - :pid (.-pid js/process) - :host "127.0.0.1" - :port 0 - :lock-id "create-empty-lock"}}))) - (set! db-lock/update-lock! (fn [_path lock] lock)) - (-> (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo - :create-empty-db? true - :log-level "error"}) - _ (is (= ["thread-api/init" true []] - (first @invoke-calls))) - _ (is (= ["thread-api/create-or-open-db" false [repo {:datoms []}]] - (second @invoke-calls))) - _ (stop!)] - true) + invoke-calls (atom [])] + (-> (p/with-redefs [platform-node/node-platform (fn [_opts] #js {}) + db-core/init-core! (fn [_platform] + #js {:remoteInvoke (fn [method direct-pass? args] + (swap! invoke-calls conj + [method + direct-pass? + (if direct-pass? + (vec (js->clj args)) + (ldb/read-transit-str args))]) + (p/resolved nil))}) + db-lock/ensure-lock! (fn [_] + (p/resolved {:path lock-file-path + :lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :lock-id "create-empty-lock"}})) + db-lock/update-lock! (fn [_path lock] lock)] + (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :create-empty-db? true + :log-level "error"}) + _ (is (= ["thread-api/init" true []] + (first @invoke-calls))) + _ (is (= ["thread-api/create-or-open-db" false [repo {:datoms []}]] + (second @invoke-calls))) + _ (stop!)] + true)) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! platform-node/node-platform original-platform) - (set! db-core/init-core! original-init-core) - (set! db-lock/ensure-lock! original-ensure-lock) - (set! db-lock/update-lock! original-update-lock) - (done))))))) + (p/finally done))))) (deftest db-worker-node-start-daemon-uses-default-startup-opts-without-create-empty (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-default-start") repo (str "logseq_db_default_start_" (subs (str (random-uuid)) 0 8)) lock-file-path (lock-path data-dir repo) - invoke-calls (atom []) - original-platform platform-node/node-platform - original-init-core db-core/init-core! - original-ensure-lock db-lock/ensure-lock! - original-update-lock db-lock/update-lock!] - (set! platform-node/node-platform (fn [_opts] #js {})) - (set! db-core/init-core! - (fn [_platform] - #js {:remoteInvoke (fn [method direct-pass? args] - (swap! invoke-calls conj - [method - direct-pass? - (if direct-pass? - (vec (js->clj args)) - (ldb/read-transit-str args))]) - (p/resolved nil))})) - (set! db-lock/ensure-lock! - (fn [_] - (p/resolved {:path lock-file-path - :lock {:repo repo - :pid (.-pid js/process) - :host "127.0.0.1" - :port 0 - :lock-id "default-lock"}}))) - (set! db-lock/update-lock! (fn [_path lock] lock)) - (-> (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo - :log-level "error"}) - _ (is (= ["thread-api/init" true []] - (first @invoke-calls))) - _ (is (= ["thread-api/create-or-open-db" false [repo {}]] - (second @invoke-calls))) - _ (stop!)] - true) + invoke-calls (atom [])] + (-> (p/with-redefs [platform-node/node-platform (fn [_opts] #js {}) + db-core/init-core! (fn [_platform] + #js {:remoteInvoke (fn [method direct-pass? args] + (swap! invoke-calls conj + [method + direct-pass? + (if direct-pass? + (vec (js->clj args)) + (ldb/read-transit-str args))]) + (p/resolved nil))}) + db-lock/ensure-lock! (fn [_] + (p/resolved {:path lock-file-path + :lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :lock-id "default-lock"}})) + db-lock/update-lock! (fn [_path lock] lock)] + (p/let [{:keys [stop!]} (db-worker-node/start-daemon! {:data-dir data-dir + :repo repo + :log-level "error"}) + _ (is (= ["thread-api/init" true []] + (first @invoke-calls))) + _ (is (= ["thread-api/create-or-open-db" false [repo {}]] + (second @invoke-calls))) + _ (stop!)] + true)) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! platform-node/node-platform original-platform) - (set! db-core/init-core! original-init-core) - (set! db-lock/ensure-lock! original-ensure-lock) - (set! db-lock/update-lock! original-update-lock) - (done))))))) + (p/finally done))))) (deftest db-worker-node-repo-error-handles-keyword-methods (let [repo-error #'db-worker-node/repo-error diff --git a/src/test/logseq/cli/command/doctor_test.cljs b/src/test/logseq/cli/command/doctor_test.cljs index 497a9fd904..37b27984a5 100644 --- a/src/test/logseq/cli/command/doctor_test.cljs +++ b/src/test/logseq/cli/command/doctor_test.cljs @@ -8,71 +8,59 @@ (deftest test-execute-doctor-script-missing (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers - ensure-data-dir-called? (atom false) + (let [ensure-data-dir-called? (atom false) list-servers-called? (atom false)] - (set! data-dir/ensure-data-dir! (fn [_] - (reset! ensure-data-dir-called? true) - "/tmp/logseq-doctor")) - (set! cli-server/list-servers (fn [_] - (reset! list-servers-called? true) - (p/resolved []))) - (-> (p/let [result (commands/execute {:type :doctor - :script-path "/tmp/logseq-cli-missing-db-worker-node.js"} - {})] - (is (= :error (:status result))) - (is (= :doctor-script-missing (get-in result [:error :code]))) - (is (= :db-worker-script - (get-in result [:error :checks 0 :id]))) - (is (= :error - (get-in result [:error :checks 0 :status]))) - (is (false? @ensure-data-dir-called?)) - (is (false? @list-servers-called?))) + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] + (reset! ensure-data-dir-called? true) + "/tmp/logseq-doctor") + cli-server/list-servers (fn [_] + (reset! list-servers-called? true) + (p/resolved []))] + (p/let [result (commands/execute {:type :doctor + :script-path "/tmp/logseq-cli-missing-db-worker-node.js"} + {})] + (is (= :error (:status result))) + (is (= :doctor-script-missing (get-in result [:error :code]))) + (is (= :db-worker-script + (get-in result [:error :checks 0 :id]))) + (is (= :error + (get-in result [:error :checks 0 :status]))) + (is (false? @ensure-data-dir-called?)) + (is (false? @list-servers-called?)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (p/finally done))))) (deftest test-execute-doctor-data-dir-permission (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers - list-servers-called? (atom false)] - (set! data-dir/ensure-data-dir! (fn [_] - (throw (ex-info "data-dir is not readable/writable: /tmp/nope" - {:code :data-dir-permission - :path "/tmp/nope" - :cause "EACCES"})))) - (set! cli-server/list-servers (fn [_] - (reset! list-servers-called? true) - (p/resolved []))) - (-> (p/let [result (commands/execute {:type :doctor - :script-path "src/main/logseq/cli/commands.cljs"} - {:data-dir "/tmp/nope"})] - (is (= :error (:status result))) - (is (= :data-dir-permission (get-in result [:error :code]))) - (is (= [:db-worker-script :data-dir] - (mapv :id (get-in result [:error :checks])))) - (is (= [:ok :error] - (mapv :status (get-in result [:error :checks])))) - (is (false? @list-servers-called?))) + (let [list-servers-called? (atom false)] + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] + (throw (ex-info "data-dir is not readable/writable: /tmp/nope" + {:code :data-dir-permission + :path "/tmp/nope" + :cause "EACCES"}))) + cli-server/list-servers (fn [_] + (reset! list-servers-called? true) + (p/resolved []))] + (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/nope"})] + (is (= :error (:status result))) + (is (= :data-dir-permission (get-in result [:error :code]))) + (is (= [:db-worker-script :data-dir] + (mapv :id (get-in result [:error :checks])))) + (is (= [:ok :error] + (mapv :status (get-in result [:error :checks])))) + (is (false? @list-servers-called?)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (p/finally done))))) (deftest test-execute-doctor-all-checks-pass (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers] - (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) - (set! cli-server/list-servers (fn [_] (p/resolved []))) - (-> (p/let [result (commands/execute {:type :doctor + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-server/list-servers (fn [_] (p/resolved []))] + (p/let [result (commands/execute {:type :doctor :script-path "src/main/logseq/cli/commands.cljs"} {:data-dir "/tmp/logseq-doctor"})] (is (= :ok (:status result))) @@ -80,25 +68,20 @@ (is (= [:db-worker-script :data-dir :running-servers] (mapv :id (get-in result [:data :checks])))) (is (= [:ok :ok :ok] - (mapv :status (get-in result [:data :checks]))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (mapv :status (get-in result [:data :checks])))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-doctor-starting-server-warning (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers] - (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) - (set! cli-server/list-servers (fn [_] - (p/resolved [{:repo "logseq_db_demo" - :status :starting - :host "127.0.0.1" - :port 9010}]))) - (-> (p/let [result (commands/execute {:type :doctor + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-server/list-servers (fn [_] + (p/resolved [{:repo "logseq_db_demo" + :status :starting + :host "127.0.0.1" + :port 9010}]))] + (p/let [result (commands/execute {:type :doctor :script-path "src/main/logseq/cli/commands.cljs"} {:data-dir "/tmp/logseq-doctor"})] (is (= :ok (:status result))) @@ -108,49 +91,36 @@ (is (= :warning (get-in result [:data :checks 2 :status]))) (is (= :doctor-server-not-ready - (get-in result [:data :checks 2 :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (get-in result [:data :checks 2 :code]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-doctor-default-script-checks-packaged-runtime-target (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers] - (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) - (set! cli-server/list-servers (fn [_] (p/resolved []))) - (-> (p/let [result (commands/execute {:type :doctor} + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-server/list-servers (fn [_] (p/resolved []))] + (p/let [result (commands/execute {:type :doctor} {:data-dir "/tmp/logseq-doctor"}) checked-path (get-in result [:data :checks 0 :path])] (is (= :ok (:status result))) (is (= (cli-server/db-worker-script-path) checked-path)) - (is (string/ends-with? checked-path "/dist/db-worker-node.js"))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (is (string/ends-with? checked-path "/dist/db-worker-node.js")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-doctor-explicit-script-path-checks-static-runtime-target (async done - (let [orig-ensure-data-dir! data-dir/ensure-data-dir! - orig-list-servers cli-server/list-servers] - (set! data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor")) - (set! cli-server/list-servers (fn [_] (p/resolved []))) - (-> (p/let [result (commands/execute {:type :doctor + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-server/list-servers (fn [_] (p/resolved []))] + (p/let [result (commands/execute {:type :doctor :script-path (cli-server/db-worker-dev-script-path)} {:data-dir "/tmp/logseq-doctor"}) checked-path (get-in result [:data :checks 0 :path])] (is (= :ok (:status result))) (is (= (cli-server/db-worker-dev-script-path) checked-path)) - (is (string/ends-with? checked-path "/static/db-worker-node.js"))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! data-dir/ensure-data-dir! orig-ensure-data-dir!) - (set! cli-server/list-servers orig-list-servers) - (done))))))) + (is (string/ends-with? checked-path "/static/db-worker-node.js")))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) diff --git a/src/test/logseq/cli/command/graph_test.cljs b/src/test/logseq/cli/command/graph_test.cljs index 6f3211c07b..1acad2f1b9 100644 --- a/src/test/logseq/cli/command/graph_test.cljs +++ b/src/test/logseq/cli/command/graph_test.cljs @@ -19,64 +19,50 @@ (deftest test-execute-graph-info-queries-kv-rows-with-thread-api-q (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - invoke-calls* (atom []) + (let [invoke-calls* (atom []) action {:repo "demo-repo" :graph "demo-graph"}] - (set! cli-server/ensure-server! - (fn [_ _] - (p/resolved {:base-url "http://example"}))) - (set! transport/invoke - (fn [_ method _ args] - (swap! invoke-calls* conj [method args]) - (p/resolved [[:logseq.kv/schema-version 7] - [:logseq.kv/graph-created-at 40000] - [:logseq.kv/db-type :sqlite]]))) - (-> (p/let [result (graph-command/execute-graph-info action {})] - (is (= :ok (:status result))) - (is (= 1 (count @invoke-calls*))) - (let [[method [repo query-args]] (first @invoke-calls*)] - (is (= :thread-api/q method)) - (is (= "demo-repo" repo)) - (is (= 1 (count query-args))) - (is (string/includes? (pr-str (first query-args)) "logseq.kv")))) + (-> (p/with-redefs [cli-server/ensure-server! (fn [_ _] + (p/resolved {:base-url "http://example"})) + transport/invoke (fn [_ method _ args] + (swap! invoke-calls* conj [method args]) + (p/resolved [[:logseq.kv/schema-version 7] + [:logseq.kv/graph-created-at 40000] + [:logseq.kv/db-type :sqlite]]))] + (p/let [result (graph-command/execute-graph-info action {})] + (is (= :ok (:status result))) + (is (= 1 (count @invoke-calls*))) + (let [[method [repo query-args]] (first @invoke-calls*)] + (is (= :thread-api/q method)) + (is (= "demo-repo" repo)) + (is (= 1 (count query-args))) + (is (string/includes? (pr-str (first query-args)) "logseq.kv"))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-graph-info-preserves-summary-fields-and-builds-kv-map (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - action {:repo "demo-repo" + (let [action {:repo "demo-repo" :graph "demo-graph"}] - (set! cli-server/ensure-server! - (fn [_ _] - (p/resolved {:base-url "http://example"}))) - (set! transport/invoke - (fn [_ method _ _] - (case method - :thread-api/q - (p/resolved [[:logseq.kv/db-type :sqlite] - [:logseq.kv/graph-created-at 40000] - [:logseq.kv/schema-version 7]]) - (throw (ex-info "unexpected invoke method" {:method method}))))) - (-> (p/let [result (graph-command/execute-graph-info action {})] - (is (= :ok (:status result))) - (is (= "demo-graph" (get-in result [:data :graph]))) - (is (= 40000 (get-in result [:data :logseq.kv/graph-created-at]))) - (is (= 7 (get-in result [:data :logseq.kv/schema-version]))) - (is (= {"logseq.kv/db-type" :sqlite - "logseq.kv/graph-created-at" 40000 - "logseq.kv/schema-version" 7} - (get-in result [:data :kv])))) + (-> (p/with-redefs [cli-server/ensure-server! (fn [_ _] + (p/resolved {:base-url "http://example"})) + transport/invoke (fn [_ method _ _] + (case method + :thread-api/q + (p/resolved [[:logseq.kv/db-type :sqlite] + [:logseq.kv/graph-created-at 40000] + [:logseq.kv/schema-version 7]]) + (throw (ex-info "unexpected invoke method" {:method method}))))] + (p/let [result (graph-command/execute-graph-info action {})] + (is (= :ok (:status result))) + (is (= "demo-graph" (get-in result [:data :graph]))) + (is (= 40000 (get-in result [:data :logseq.kv/graph-created-at]))) + (is (= 7 (get-in result [:data :logseq.kv/schema-version]))) + (is (= {"logseq.kv/db-type" :sqlite + "logseq.kv/graph-created-at" 40000 + "logseq.kv/schema-version" 7} + (get-in result [:data :kv]))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) diff --git a/src/test/logseq/cli/command/show_test.cljs b/src/test/logseq/cli/command/show_test.cljs index bf8c326c6c..8d60c487b8 100644 --- a/src/test/logseq/cli/command/show_test.cljs +++ b/src/test/logseq/cli/command/show_test.cljs @@ -1,5 +1,6 @@ (ns logseq.cli.command.show-test - (:require [cljs.test :refer [deftest is testing]] + (:require ["fs" :as fs] + [cljs.test :refer [deftest is testing]] [clojure.string :as string] [logseq.cli.command.show :as show-command])) @@ -23,7 +24,7 @@ (is (= [1 2 3] (get-in result [:action :ids]))) (is (true? (get-in result [:action :multi-id?]))))) - (testing "stdin overrides explicit id when present" + (testing "explicit stdin still overrides id when provided in options" (let [result (show-command/build-action {:id "99" :stdin "[1 2]"} "logseq_db_demo")] @@ -31,6 +32,30 @@ (is (= [1 2] (get-in result [:action :ids]))) (is (true? (get-in result [:action :multi-id?]))))) + (testing "pipe stdin does not override explicit id unless stdin mode is requested" + (let [orig-fstat-sync (.-fstatSync fs) + orig-read-file-sync (.-readFileSync fs) + read-count* (atom 0)] + (set! (.-fstatSync fs) + (fn [_] + #js {:isFIFO (fn [] true) + :isFile (fn [] false)})) + (set! (.-readFileSync fs) + (fn [fd] + (when (= fd 0) + (swap! read-count* inc) + "[1 2]"))) + (try + (let [result (show-command/build-action {:id "99"} + "logseq_db_demo")] + (is (true? (:ok? result))) + (is (= 99 (get-in result [:action :id]))) + (is (= [99] (get-in result [:action :ids]))) + (is (zero? @read-count*))) + (finally + (set! (.-fstatSync fs) orig-fstat-sync) + (set! (.-readFileSync fs) orig-read-file-sync))))) + (testing "blank stdin falls back to explicit id" (let [result (show-command/build-action {:id "99" :stdin " "} diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index d8a8077cbb..4278ed9a67 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -1,7 +1,7 @@ (ns logseq.cli.command.sync-test (:require [cljs.test :refer [async deftest is testing]] - [logseq.cli.config :as cli-config] [logseq.cli.command.sync :as sync-command] + [logseq.cli.config :as cli-config] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [promesa.core :as p])) @@ -46,641 +46,550 @@ (deftest test-execute-sync-start (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom []) - status-calls (atom 0)] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-status - (let [idx (swap! status-calls inc)] - (p/resolved {:repo "logseq_db_demo" - :ws-state (if (= idx 1) :connecting :open) - :pending-local 0 - :pending-asset 0 - :pending-server 0})) - (p/resolved {:ok true})))) - (-> (p/let [result (sync-command/execute {:type :sync-start - :repo "logseq_db_demo"} - {:data-dir "/tmp"}) - invoked-methods (map first @invoke-calls)] - (is (= :ok (:status result))) - (is (= :open (get-in result [:data :ws-state]))) - (is (<= 1 (count @ensure-calls))) - (is (every? (fn [[_ repo]] (= "logseq_db_demo" repo)) @ensure-calls)) - (is (= 1 (count (filter #(= :thread-api/db-sync-start %) invoked-methods)))) - (is (<= 2 (count (filter #(= :thread-api/db-sync-status %) invoked-methods))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom []) + status-calls (atom 0)] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved {:repo "logseq_db_demo" + :ws-state (if (= idx 1) :connecting :open) + :pending-local 0 + :pending-asset 0 + :pending-server 0})) + (p/resolved {:ok true})))] + (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo"} + {:data-dir "/tmp"}) + invoked-methods (map first @invoke-calls)] + (is (= :ok (:status result))) + (is (= :open (get-in result [:data :ws-state]))) + (is (<= 1 (count @ensure-calls))) + (is (every? (fn [[_ repo]] (= "logseq_db_demo" repo)) @ensure-calls)) + (is (= 1 (count (filter #(= :thread-api/db-sync-start %) invoked-methods)))) + (is (<= 2 (count (filter #(= :thread-api/db-sync-status %) invoked-methods)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-start-timeout (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-status - (p/resolved {:repo "logseq_db_demo" - :ws-state :connecting - :pending-local 0 - :pending-asset 0 - :pending-server 0}) - (p/resolved {:ok true})))) - (-> (p/let [result (sync-command/execute {:type :sync-start - :repo "logseq_db_demo" - :wait-timeout-ms 20 - :wait-poll-interval-ms 0} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :sync-start-timeout (get-in result [:error :code]))) - (is (= "logseq_db_demo" (get-in result [:error :repo]))) - (is (= :connecting (get-in result [:error :ws-state])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-status + (p/resolved {:repo "logseq_db_demo" + :ws-state :connecting + :pending-local 0 + :pending-asset 0 + :pending-server 0}) + (p/resolved {:ok true})))] + (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 20 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-timeout (get-in result [:error :code]))) + (is (= "logseq_db_demo" (get-in result [:error :repo]))) + (is (= :connecting (get-in result [:error :ws-state]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-start-missing-ws-url-is-error (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method _direct-pass? _args] - (case method - :thread-api/db-sync-status - (p/resolved {:repo "logseq_db_demo" - :ws-state :inactive - :pending-local 0 - :pending-asset 0 - :pending-server 0}) - (p/resolved {:ok true})))) - (-> (p/let [result (sync-command/execute {:type :sync-start - :repo "logseq_db_demo" - :wait-timeout-ms 20 - :wait-poll-interval-ms 0} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :sync-start-skipped (get-in result [:error :code]))) - (is (= :inactive (get-in result [:error :ws-state])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-status + (p/resolved {:repo "logseq_db_demo" + :ws-state :inactive + :pending-local 0 + :pending-asset 0 + :pending-server 0}) + (p/resolved {:ok true})))] + (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 20 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-skipped (get-in result [:error :code]))) + (is (= :inactive (get-in result [:error :ws-state]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-sync-start-runtime-error-after-open (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - status-calls (atom 0)] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method _direct-pass? _args] - (case method - :thread-api/db-sync-status - (let [idx (swap! status-calls inc)] - (p/resolved (if (= idx 1) - {:repo "logseq_db_demo" - :ws-state :connecting - :pending-local 0 - :pending-asset 0 - :pending-server 0} - {:repo "logseq_db_demo" - :ws-state :open - :pending-local 1 - :pending-asset 0 - :pending-server 2 - :last-error {:code :decrypt-aes-key - :message "decrypt-aes-key"}}))) - (p/resolved {:ok true})))) - (-> (p/let [result (sync-command/execute {:type :sync-start - :repo "logseq_db_demo" - :wait-timeout-ms 200 - :wait-poll-interval-ms 0} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :sync-start-runtime-error (get-in result [:error :code]))) - (is (= :decrypt-aes-key (get-in result [:error :last-error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [status-calls (atom 0)] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved (if (= idx 1) + {:repo "logseq_db_demo" + :ws-state :connecting + :pending-local 0 + :pending-asset 0 + :pending-server 0} + {:repo "logseq_db_demo" + :ws-state :open + :pending-local 1 + :pending-asset 0 + :pending-server 2 + :last-error {:code :decrypt-aes-key + :message "decrypt-aes-key"}}))) + (p/resolved {:ok true})))] + (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo" + :wait-timeout-ms 200 + :wait-poll-interval-ms 0} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :sync-start-runtime-error (get-in result [:error :code]))) + (is (= :decrypt-aes-key (get-in result [:error :last-error :code]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-stop (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))) - (-> (p/let [_ (sync-command/execute {:type :sync-stop - :repo "logseq_db_demo"} - {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] - @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-stop false []]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [_ (sync-command/execute {:type :sync-stop + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-stop false []]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-upload (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))) - (-> (p/let [_ (sync-command/execute {:type :sync-upload - :repo "logseq_db_demo"} - {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] - @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-upload-graph false ["logseq_db_demo"]]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [_ (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-upload-graph false ["logseq_db_demo"]]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-upload-propagates-worker-error (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method _direct-pass? _args] - (case method - :thread-api/set-db-sync-config - (p/resolved nil) + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/set-db-sync-config + (p/resolved nil) - :thread-api/db-sync-upload-graph - (p/rejected (ex-info "snapshot upload failed" - {:code :snapshot-upload-failed - :status 500 - :graph-id "graph-1"})) + :thread-api/db-sync-upload-graph + (p/rejected (ex-info "snapshot upload failed" + {:code :snapshot-upload-failed + :status 500 + :graph-id "graph-1"})) - (p/resolved nil)))) - (-> (p/let [result (sync-command/execute {:type :sync-upload - :repo "logseq_db_demo"} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :snapshot-upload-failed (get-in result [:error :code]))) - (is (= 500 (get-in result [:error :context :status]))) - (is (= "graph-1" (get-in result [:error :context :graph-id])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :snapshot-upload-failed (get-in result [:error :code]))) + (is (= 500 (get-in result [:error :context :status]))) + (is (= "graph-1" (get-in result [:error :context :graph-id]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-sync-download (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "remote-graph-id" - :graph-name "demo" - :graph-e2ee? true}]) - :thread-api/q - (p/resolved 0) - :thread-api/db-sync-download-graph-by-id - (p/resolved {:ok true}) - (p/resolved nil)))) - (-> (p/let [_ (sync-command/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo"} - {:base-url "http://example" - :data-dir "/tmp"})] - (is (= [[{:base-url "http://example" - :create-empty-db? true - :data-dir "/tmp"} - "logseq_db_demo"]] - @ensure-calls)) - (is (= [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - (nth @invoke-calls 0))) - (is (= [:thread-api/db-sync-list-remote-graphs false []] - (nth @invoke-calls 1))) - (is (= [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - (nth @invoke-calls 2))) - (let [[method direct-pass? args] (nth @invoke-calls 3)] - (is (= :thread-api/q method)) - (is (= false direct-pass?)) - (is (= "logseq_db_demo" (first args)))) - (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]] - (nth @invoke-calls 4)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/q + (p/resolved 0) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :create-empty-db? true + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 0))) + (is (= [:thread-api/db-sync-list-remote-graphs false []] + (nth @invoke-calls 1))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 2))) + (let [[method direct-pass? args] (nth @invoke-calls 3)] + (is (= :thread-api/q method)) + (is (= false direct-pass?)) + (is (= "logseq_db_demo" (first args)))) + (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" true]] + (nth @invoke-calls 4))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-download-uses-graph-config-when-base-url-missing (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "remote-graph-id" - :graph-name "demo" - :graph-e2ee? false}]) - :thread-api/q - (p/resolved 0) - :thread-api/db-sync-download-graph-by-id - (p/resolved {:ok true}) - (p/resolved nil)))) - (-> (p/let [_ (sync-command/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo"} - {:graph "demo" - :data-dir "/tmp"})] - (is (= [[{:graph "demo" - :create-empty-db? true - :data-dir "/tmp"} - "logseq_db_demo"] - [{:graph "demo" - :create-empty-db? true - :data-dir "/tmp"} - "logseq_db_demo"]] - @ensure-calls)) - (is (= [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - (nth @invoke-calls 0))) - (is (= [:thread-api/db-sync-list-remote-graphs false []] - (nth @invoke-calls 1))) - (is (= [:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - (nth @invoke-calls 2))) - (let [[method direct-pass? args] (nth @invoke-calls 3)] - (is (= :thread-api/q method)) - (is (= false direct-pass?)) - (is (= "logseq_db_demo" (first args)))) - (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]] - (nth @invoke-calls 4)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? false}]) + :thread-api/q + (p/resolved 0) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [_ (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:graph "demo" + :data-dir "/tmp"})] + (is (= [[{:graph "demo" + :create-empty-db? true + :data-dir "/tmp"} + "logseq_db_demo"] + [{:graph "demo" + :create-empty-db? true + :data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 0))) + (is (= [:thread-api/db-sync-list-remote-graphs false []] + (nth @invoke-calls 1))) + (is (= [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + (nth @invoke-calls 2))) + (let [[method direct-pass? args] (nth @invoke-calls 3)] + (is (= :thread-api/q method)) + (is (= false direct-pass?)) + (is (= "logseq_db_demo" (first args)))) + (is (= [:thread-api/db-sync-download-graph-by-id false ["logseq_db_demo" "remote-graph-id" false]] + (nth @invoke-calls 4))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-download-remote-graph-not-found (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "other-id" - :graph-name "other-graph"}]) - :thread-api/db-sync-download-graph-by-id - (p/resolved {:ok true}) - (p/resolved nil)))) - (-> (p/let [result (sync-command/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo"} - {:base-url "http://example" - :data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :remote-graph-not-found (get-in result [:error :code]))) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-list-remote-graphs false []]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "other-id" + :graph-name "other-graph"}]) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :remote-graph-not-found (get-in result [:error :code]))) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-download-propagates-worker-error-code (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method _direct-pass? _args] - (case method - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "remote-graph-id" - :graph-name "demo" - :graph-e2ee? true}]) - :thread-api/q - (p/resolved 0) - :thread-api/db-sync-download-graph-by-id - (p/rejected (ex-info "db-sync/incomplete-snapshot-frame" - {:code :db-sync/incomplete-snapshot-frame - :graph-id "remote-graph-id"})) - (p/resolved nil)))) - (-> (p/let [result (sync-command/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo"} - {:base-url "http://example" - :data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :db-sync/incomplete-snapshot-frame (get-in result [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/q + (p/resolved 0) + :thread-api/db-sync-download-graph-by-id + (p/rejected (ex-info "db-sync/incomplete-snapshot-frame" + {:code :db-sync/incomplete-snapshot-frame + :graph-id "remote-graph-id"})) + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :db-sync/incomplete-snapshot-frame (get-in result [:error :code]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-sync-download-fails-fast-when-db-is-not-empty (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (p/resolved (assoc config - :repo repo - :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (case method - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "remote-graph-id" - :graph-name "demo" - :graph-e2ee? true}]) - :thread-api/q - (p/resolved 2) - :thread-api/db-sync-download-graph-by-id - (p/resolved {:ok true}) - (p/resolved nil)))) - (-> (p/let [result (sync-command/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo"} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :graph-db-not-empty (get-in result [:error :code]))) - (is (= "logseq_db_demo" (get-in result [:error :repo]))) - (is (= 2 (get-in result [:error :context :non-empty-entity-count]))) - (is (not-any? (fn [[method _ _]] - (= :thread-api/db-sync-download-graph-by-id method)) - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (p/resolved (assoc config + :repo repo + :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + :thread-api/q + (p/resolved 2) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :graph-db-not-empty (get-in result [:error :code]))) + (is (= "logseq_db_demo" (get-in result [:error :repo]))) + (is (= 2 (get-in result [:error :context :non-empty-entity-count]))) + (is (not-any? (fn [[method _ _]] + (= :thread-api/db-sync-download-graph-by-id method)) + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-remote-graphs (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved []))) - (-> (p/let [_ (sync-command/execute {:type :sync-remote-graphs} - {:base-url "http://example" - :http-base "https://sync.example.com" - :ws-url "wss://sync.example.com/sync/%s" - :auth-token "test-token" - :e2ee-password "pw" - :data-dir "/tmp"})] - (is (= [] @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url "wss://sync.example.com/sync/%s" - :http-base "https://sync.example.com" - :auth-token "test-token" - :e2ee-password "pw"}]] - [:thread-api/db-sync-list-remote-graphs false []]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved []))] + (p/let [_ (sync-command/execute {:type :sync-remote-graphs} + {:base-url "http://example" + :http-base "https://sync.example.com" + :ws-url "wss://sync.example.com/sync/%s" + :auth-token "test-token" + :e2ee-password "pw" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url "wss://sync.example.com/sync/%s" + :http-base "https://sync.example.com" + :auth-token "test-token" + :e2ee-password "pw"}]] + [:thread-api/db-sync-list-remote-graphs false []]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-ensure-keys (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))) - (-> (p/let [_ (sync-command/execute {:type :sync-ensure-keys} - {:base-url "http://example" - :data-dir "/tmp"})] - (is (= [] @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-ensure-user-rsa-keys false []]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [_ (sync-command/execute {:type :sync-ensure-keys} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-ensure-user-rsa-keys false []]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-grant-access (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))) - (-> (p/let [_ (sync-command/execute {:type :sync-grant-access - :repo "logseq_db_demo" - :graph-id "graph-uuid" - :email "user@example.com"} - {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] - @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil - :http-base nil - :auth-token nil - :e2ee-password nil}]] - [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] - @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [_ (sync-command/execute {:type :sync-grant-access + :repo "logseq_db_demo" + :graph-id "graph-uuid" + :email "user@example.com"} + {:data-dir "/tmp"})] + (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token nil + :e2ee-password nil}]] + [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-config-get (async done - (let [orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - ensure-calls (atom []) - invoke-calls (atom [])] - (set! cli-server/ensure-server! (fn [config repo] - (swap! ensure-calls conj [config repo]) - (p/resolved (assoc config :base-url "http://example")))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))) - (-> (p/let [_ (sync-command/execute {:type :sync-config-get - :config-key :auth-token} - {:base-url "http://example" - :auth-token "abc" - :data-dir "/tmp"})] - (is (= [] @ensure-calls)) - (is (= [] @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [_ (sync-command/execute {:type :sync-config-get + :config-key :auth-token} + {:base-url "http://example" + :auth-token "abc" + :data-dir "/tmp"})] + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-config-set (async done - (let [orig-invoke transport/invoke - orig-update-config! cli-config/update-config! - invoke-calls (atom []) - update-calls (atom [])] - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved nil)) - ) - (set! cli-config/update-config! (fn [config updates] - (swap! update-calls conj [config updates]) - (merge {:ws-url "wss://old.example/sync/%s"} updates))) - (-> (p/let [_ (sync-command/execute {:type :sync-config-set - :config-key :auth-token - :config-value "token-value"} - {:base-url "http://example" - :config-path "/tmp/cli.edn" - :data-dir "/tmp"})] - (is (= [[{:base-url "http://example" - :config-path "/tmp/cli.edn" - :data-dir "/tmp"} - {:auth-token "token-value"}]] - @update-calls)) - (is (= [] @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! transport/invoke orig-invoke) - (set! cli-config/update-config! orig-update-config!) - (done))))))) + (let [invoke-calls (atom []) + update-calls (atom [])] + (-> (p/with-redefs [transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil)) + cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (merge {:ws-url "wss://old.example/sync/%s"} updates))] + (p/let [_ (sync-command/execute {:type :sync-config-set + :config-key :auth-token + :config-value "token-value"} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token "token-value"}]] + @update-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-config-unset (async done - (let [orig-invoke transport/invoke - orig-update-config! cli-config/update-config! - invoke-calls (atom []) - update-calls (atom [])] - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved nil))) - (set! cli-config/update-config! (fn [config updates] - (swap! update-calls conj [config updates]) - (dissoc {:ws-url "wss://old.example/sync/%s" - :auth-token "token-value"} - :auth-token))) - (-> (p/let [_ (sync-command/execute {:type :sync-config-unset - :config-key :auth-token} - {:base-url "http://example" - :config-path "/tmp/cli.edn" - :data-dir "/tmp"})] - (is (= [[{:base-url "http://example" - :config-path "/tmp/cli.edn" - :data-dir "/tmp"} - {:auth-token nil}]] - @update-calls)) - (is (= [] @invoke-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! transport/invoke orig-invoke) - (set! cli-config/update-config! orig-update-config!) - (done))))))) + (let [invoke-calls (atom []) + update-calls (atom [])] + (-> (p/with-redefs [transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved nil)) + cli-config/update-config! (fn [config updates] + (swap! update-calls conj [config updates]) + (dissoc {:ws-url "wss://old.example/sync/%s" + :auth-token "token-value"} + :auth-token))] + (p/let [_ (sync-command/execute {:type :sync-config-unset + :config-key :auth-token} + {:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"})] + (is (= [[{:base-url "http://example" + :config-path "/tmp/cli.edn" + :data-dir "/tmp"} + {:auth-token nil}]] + @update-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) \ No newline at end of file diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 65ec027e3a..4414037676 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -696,101 +696,86 @@ (deftest test-show-selectors-include-db-ident (async done - (let [selectors (atom []) - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _] config)) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ selector _] args] - (swap! selectors conj selector) - (p/resolved {:db/id 1 - :block/page {:db/id 2}})) - :thread-api/q (let [[_ [query _]] args - pull-form (second query) - selector (when (and (seq? pull-form) - (= 'pull (first pull-form))) - (nth pull-form 2))] - (when selector - (swap! selectors conj selector)) - (p/resolved [])) - :thread-api/get-block-refs (p/resolved [{:db/id 10}]) - (p/resolved nil)))) - (-> (p/let [_ (show-command/execute-show {:type :show - :repo "demo" - :id 1} - {:output-format :json})] - (is (some #(some #{:db/ident} %) @selectors)) - (is (some #(and (some #{:db/ident} %) - (some (fn [entry] - (and (map? entry) - (contains? entry :block/page))) - %)) - @selectors))) + (let [selectors (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _] config) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ selector _] args] + (swap! selectors conj selector) + (p/resolved {:db/id 1 + :block/page {:db/id 2}})) + :thread-api/q (let [[_ [query _]] args + pull-form (second query) + selector (when (and (seq? pull-form) + (= 'pull (first pull-form))) + (nth pull-form 2))] + (when selector + (swap! selectors conj selector)) + (p/resolved [])) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + (p/resolved nil)))] + (p/let [_ (show-command/execute-show {:type :show + :repo "demo" + :id 1} + {:output-format :json})] + (is (some #(some #{:db/ident} %) @selectors)) + (is (some #(and (some #{:db/ident} %) + (some (fn [entry] + (and (map? entry) + (contains? entry :block/page))) + %)) + @selectors)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-show-linked-references-disabled (async done - (let [method-calls (atom []) - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _] config)) - (set! transport/invoke (fn [_ method _ _] - (swap! method-calls conj method) - (case method - :thread-api/pull (p/resolved {:db/id 1 - :block/page {:db/id 2}}) - :thread-api/q (p/resolved []) - :thread-api/get-block-refs (p/resolved [{:db/id 10}]) - (p/resolved nil)))) - (-> (p/let [result (show-command/execute-show {:type :show - :repo "demo" - :id 1 - :linked-references? false} - {:output-format :json})] - (is (= :ok (:status result))) - (is (not (contains? (:data result) :linked-references))) - (is (not (some #{:thread-api/get-block-refs} @method-calls)))) + (let [method-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _] config) + transport/invoke (fn [_ method _ _] + (swap! method-calls conj method) + (case method + :thread-api/pull (p/resolved {:db/id 1 + :block/page {:db/id 2}}) + :thread-api/q (p/resolved []) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + (p/resolved nil)))] + (p/let [result (show-command/execute-show {:type :show + :repo "demo" + :id 1 + :linked-references? false} + {:output-format :json})] + (is (= :ok (:status result))) + (is (not (contains? (:data result) :linked-references))) + (is (not (some #{:thread-api/get-block-refs} @method-calls))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-show-linked-references-enabled (async done - (let [method-calls (atom []) - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/ensure-server! (fn [config _] config)) - (set! transport/invoke (fn [_ method _ _] - (swap! method-calls conj method) - (case method - :thread-api/pull (p/resolved {:db/id 1 - :block/page {:db/id 2}}) - :thread-api/q (p/resolved []) - :thread-api/get-block-refs (p/resolved [{:db/id 10}]) - (p/resolved nil)))) - (-> (p/let [result (show-command/execute-show {:type :show - :repo "demo" - :id 1 - :linked-references? true} - {:output-format :json})] - (is (= :ok (:status result))) - (is (contains? (:data result) :linked-references)) - (is (some #{:thread-api/get-block-refs} @method-calls))) + (let [method-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _] config) + transport/invoke (fn [_ method _ _] + (swap! method-calls conj method) + (case method + :thread-api/pull (p/resolved {:db/id 1 + :block/page {:db/id 2}}) + :thread-api/q (p/resolved []) + :thread-api/get-block-refs (p/resolved [{:db/id 10}]) + (p/resolved nil)))] + (p/let [result (show-command/execute-show {:type :show + :repo "demo" + :id 1 + :linked-references? true} + {:output-format :json})] + (is (= :ok (:status result))) + (is (contains? (:data result) :linked-references)) + (is (some #{:thread-api/get-block-refs} @method-calls)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-tree->text-uuid-ref-recursion-limit (testing "show tree text limits uuid ref replacement depth" @@ -1671,435 +1656,345 @@ (async done (let [ops* (atom nil) created?* (atom false) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :repo "demo" :name "Quote"}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (= lookup [:block/name "quote"]) - (if @created?* - {:db/id 4242 - :block/name "quote" - :block/title "Quote" - :block/tags [{:db/ident :logseq.class/Tag}]} - {}) - {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! created?* true) - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [4242] (get-in result [:data :result]))) - (is (= [[:create-page ["Quote" {:class? true}]]] - @ops*))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "quote"]) + (if @created?* + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {}) + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! created?* true) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= [[:create-page ["Quote" {:class? true}]]] + @ops*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-rejects-existing-non-tag-page (async done (let [action {:type :upsert-tag :repo "demo" - :name "Home"} - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (= lookup [:block/name "home"]) - {:db/id 99 - :block/name "home" - :block/title "Home" - :block/tags [{:db/ident :logseq.class/Page}]} - {})) - :thread-api/apply-outliner-ops - (throw (ex-info "should not create tag" {:args args})) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :error (:status result))) - (is (= :tag-name-conflict (get-in result [:error :code])))) + :name "Home"}] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "home"]) + {:db/id 99 + :block/name "home" + :block/title "Home" + :block/tags [{:db/ident :logseq.class/Page}]} + {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not create tag" {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-name-conflict (get-in result [:error :code]))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-idempotent-when-tag-exists (async done (let [apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :repo "demo" :name "Quote"}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (= lookup [:block/name "quote"]) - {:db/id 4242 - :block/name "quote" - :block/title "Quote" - :block/tags [{:db/ident :logseq.class/Tag}]} - {})) - :thread-api/apply-outliner-ops (do - (swap! apply-calls* inc) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [4242] (get-in result [:data :result]))) - (is (= 0 @apply-calls*))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup [:block/name "quote"]) + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-property-emits-upsert-op (async done (let [ops* (atom nil) created?* (atom false) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-property :repo "demo" :name "owner" :schema {:logseq.property/type :node :db/cardinality :db.cardinality/many}}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (if @created?* - {:db/id 654 - :db/ident :user.property/owner - :block/name "owner" - :block/title "owner" - :logseq.property/type :node} - {}) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! created?* true) - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [[:upsert-property [nil - {:logseq.property/type :node - :db/cardinality :db.cardinality/many} - {:property-name "owner"}]]] - @ops*))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (if @created?* + {:db/id 654 + :db/ident :user.property/owner + :block/name "owner" + :block/title "owner" + :logseq.property/type :node} + {}) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! created?* true) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [[:upsert-property [nil + {:logseq.property/type :node + :db/cardinality :db.cardinality/many} + {:property-name "owner"}]]] + @ops*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-by-id-no-op (async done (let [apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :mode :update :repo "demo" :id 4242}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (= lookup 4242) - {:db/id 4242 - :block/name "quote" - :block/title "Quote" - :block/tags [{:db/ident :logseq.class/Tag}]} - {})) - :thread-api/apply-outliner-ops (do - (swap! apply-calls* inc) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [4242] (get-in result [:data :result]))) - (is (= 0 @apply-calls*))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup 4242) + {:db/id 4242 + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-by-id-with-name-emits-rename-op (async done (let [ops* (atom nil) apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :mode :update :repo "demo" :id 4242 :name "Project Renamed"} tag-uuid (uuid "00000000-0000-0000-0000-000000004242")] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup 4242) - {:db/id 4242 - :block/uuid tag-uuid - :block/name "project" - :block/title "Project" - :block/tags [{:db/ident :logseq.class/Tag}]} + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid tag-uuid + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} - (= lookup [:block/name "project renamed"]) - {} + (= lookup [:block/name "project renamed"]) + {} - :else - {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (swap! apply-calls* inc) - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [4242] (get-in result [:data :result]))) - (is (= 1 @apply-calls*)) - (is (= [[:rename-page [tag-uuid "Project Renamed"]]] - @ops*))) + :else + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (swap! apply-calls* inc) + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 1 @apply-calls*)) + (is (= [[:rename-page [tag-uuid "Project Renamed"]]] + @ops*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-by-id-with-name-no-op-when-normalized-name-matches (async done (let [apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :mode :update :repo "demo" :id 4242 :name " #QUOTE "}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (= lookup 4242) - {:db/id 4242 - :block/uuid (uuid "00000000-0000-0000-0000-000000004242") - :block/name "quote" - :block/title "Quote" - :block/tags [{:db/ident :logseq.class/Tag}]} - {})) - :thread-api/apply-outliner-ops (do - (swap! apply-calls* inc) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [4242] (get-in result [:data :result]))) - (is (= 0 @apply-calls*))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "quote" + :block/title "Quote" + :block/tags [{:db/ident :logseq.class/Tag}]} + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [4242] (get-in result [:data :result]))) + (is (= 0 @apply-calls*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-by-id-with-name-rejects-existing-non-tag-page (async done (let [apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :mode :update :repo "demo" :id 4242 :name "Project Renamed"}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup 4242) - {:db/id 4242 - :block/uuid (uuid "00000000-0000-0000-0000-000000004242") - :block/name "project" - :block/title "Project" - :block/tags [{:db/ident :logseq.class/Tag}]} + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} - (= lookup [:block/name "project renamed"]) - {:db/id 5000 - :block/name "project renamed" - :block/title "Project Renamed" - :block/tags [{:db/ident :logseq.class/Page}]} + (= lookup [:block/name "project renamed"]) + {:db/id 5000 + :block/name "project renamed" + :block/title "Project Renamed" + :block/tags [{:db/ident :logseq.class/Page}]} - :else - {})) - :thread-api/apply-outliner-ops (do - (swap! apply-calls* inc) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :error (:status result))) - (is (= :tag-name-conflict (get-in result [:error :code]))) - (is (= 0 @apply-calls*))) + :else + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-name-conflict (get-in result [:error :code]))) + (is (= 0 @apply-calls*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-tag-by-id-with-name-rejects-existing-tag (async done (let [apply-calls* (atom 0) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke action {:type :upsert-tag :mode :update :repo "demo" :id 4242 :name "Project Renamed"}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup 4242) - {:db/id 4242 - :block/uuid (uuid "00000000-0000-0000-0000-000000004242") - :block/name "project" - :block/title "Project" - :block/tags [{:db/ident :logseq.class/Tag}]} + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 4242) + {:db/id 4242 + :block/uuid (uuid "00000000-0000-0000-0000-000000004242") + :block/name "project" + :block/title "Project" + :block/tags [{:db/ident :logseq.class/Tag}]} - (= lookup [:block/name "project renamed"]) - {:db/id 9001 - :block/uuid (uuid "00000000-0000-0000-0000-000000009001") - :block/name "project renamed" - :block/title "Project Renamed" - :block/tags [{:db/ident :logseq.class/Tag}]} + (= lookup [:block/name "project renamed"]) + {:db/id 9001 + :block/uuid (uuid "00000000-0000-0000-0000-000000009001") + :block/name "project renamed" + :block/title "Project Renamed" + :block/tags [{:db/ident :logseq.class/Tag}]} - :else - {})) - :thread-api/apply-outliner-ops (do - (swap! apply-calls* inc) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :error (:status result))) - (is (= :tag-rename-conflict (get-in result [:error :code]))) - (is (= 0 @apply-calls*))) + :else + {})) + :thread-api/apply-outliner-ops (do + (swap! apply-calls* inc) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :tag-rename-conflict (get-in result [:error :code]))) + (is (= 0 @apply-calls*)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest test-execute-upsert-id-mode-validates-target-entity (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (case lookup - 100 {} - 101 {:db/id 101 - :block/uuid (uuid "00000000-0000-0000-0000-000000000101")} - 200 {} - 201 {:db/id 201 - :block/name "not-a-tag" - :block/title "Not a tag" - :block/tags [{:db/ident :logseq.class/Page}]} - 300 {} - 301 {:db/id 301 - :block/name "not-a-property" - :block/title "Not a property"} - {})) - :thread-api/apply-outliner-ops - (throw (ex-info "should not mutate on invalid id update mode" {:args args})) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [page-missing (commands/execute {:type :upsert-page - :mode :update - :repo "demo" - :id 100} - {}) - page-mismatch (commands/execute {:type :upsert-page - :mode :update - :repo "demo" - :id 101} - {}) - tag-missing (commands/execute {:type :upsert-tag - :mode :update - :repo "demo" - :id 200} - {}) - tag-mismatch (commands/execute {:type :upsert-tag - :mode :update - :repo "demo" - :id 201} - {}) - property-missing (commands/execute {:type :upsert-property - :mode :update - :repo "demo" - :id 300} - {}) - property-mismatch (commands/execute {:type :upsert-property - :mode :update - :repo "demo" - :id 301} - {})] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (case lookup + 100 {} + 101 {:db/id 101 + :block/uuid (uuid "00000000-0000-0000-0000-000000000101")} + 200 {} + 201 {:db/id 201 + :block/name "not-a-tag" + :block/title "Not a tag" + :block/tags [{:db/ident :logseq.class/Page}]} + 300 {} + 301 {:db/id 301 + :block/name "not-a-property" + :block/title "Not a property"} + {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not mutate on invalid id update mode" {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [page-missing (commands/execute {:type :upsert-page :mode :update :repo "demo" :id 100} {}) + page-mismatch (commands/execute {:type :upsert-page :mode :update :repo "demo" :id 101} {}) + tag-missing (commands/execute {:type :upsert-tag :mode :update :repo "demo" :id 200} {}) + tag-mismatch (commands/execute {:type :upsert-tag :mode :update :repo "demo" :id 201} {}) + property-missing (commands/execute {:type :upsert-property :mode :update :repo "demo" :id 300} {}) + property-mismatch (commands/execute {:type :upsert-property :mode :update :repo "demo" :id 301} {})] (is (= :error (:status page-missing))) (is (= :upsert-id-not-found (get-in page-missing [:error :code]))) (is (= :error (:status page-mismatch))) @@ -2111,572 +2006,337 @@ (is (= :error (:status property-missing))) (is (= :upsert-id-not-found (get-in property-missing [:error :code]))) (is (= :error (:status property-mismatch))) - (is (= :upsert-id-type-mismatch (get-in property-mismatch [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (is (= :upsert-id-type-mismatch (get-in property-mismatch [:error :code]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-upsert-block-create-applies-extra-tag-property-ops (async done (let [ops* (atom nil) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-execute-add-block add-command/execute-add-block - orig-resolve-tags add-command/resolve-tags - orig-resolve-properties add-command/resolve-properties - orig-resolve-property-identifiers add-command/resolve-property-identifiers - orig-invoke transport/invoke - action {:type :upsert-block - :mode :create - :repo "demo" + action {:type :upsert-block :mode :create :repo "demo" :update-tags [:tag/new] :remove-tags [:tag/old] :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} :remove-properties [:logseq.property/publishing-public?]}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! add-command/execute-add-block (fn [_ _] - (p/resolved {:status :ok - :data {:result [11 12]}}))) - (set! add-command/resolve-tags (fn [_ _ tags] - (p/resolved (cond - (= tags [:tag/new]) [{:db/id 101}] - (= tags [:tag/old]) [{:db/id 202}] - :else nil)))) - (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) - (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (if (and (vector? lookup) - (= :db/ident (first lookup))) - {:db/id 99} - {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [11 12] (get-in result [:data :result]))) - (is (= [[:batch-delete-property-value [[11 12] :block/tags 202]] - [:batch-remove-property [[11 12] :logseq.property/publishing-public?]] - [:batch-set-property [[11 12] :block/tags 101 {}]] - [:batch-set-property [[11 12] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] - @ops*))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! add-command/execute-add-block orig-execute-add-block) - (set! add-command/resolve-tags orig-resolve-tags) - (set! add-command/resolve-properties orig-resolve-properties) - (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + add-command/execute-add-block (fn [_ _] (p/resolved {:status :ok :data {:result [11 12]}})) + add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil))) + add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties)) + add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties)) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (if (and (vector? lookup) (= :db/ident (first lookup))) + {:db/id 99} + {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [11 12] (get-in result [:data :result]))) + (is (= [[:batch-delete-property-value [[11 12] :block/tags 202]] + [:batch-remove-property [[11 12] :logseq.property/publishing-public?]] + [:batch-set-property [[11 12] :block/tags 101 {}]] + [:batch-set-property [[11 12] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + @ops*)))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-upsert-page-applies-ops-on-existing-page (async done (let [ops* (atom nil) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-resolve-tags add-command/resolve-tags - orig-resolve-properties add-command/resolve-properties - orig-resolve-property-identifiers add-command/resolve-property-identifiers - orig-invoke transport/invoke - action {:type :upsert-page - :repo "demo" - :page "Home" + action {:type :upsert-page :repo "demo" :page "Home" :update-tags [:tag/next] :remove-tags [:tag/old] :update-properties {:logseq.property/publishing-public? true} :remove-properties [:logseq.property/deadline]}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! add-command/resolve-tags (fn [_ _ tags] - (p/resolved (cond - (= tags [:tag/next]) [{:db/id 303}] - (= tags [:tag/old]) [{:db/id 202}] - :else nil)))) - (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) - (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup [:block/name "home"]) - {:db/id 50 - :block/uuid (uuid "00000000-0000-0000-0000-000000000050")} - - (and (vector? lookup) (= :db/ident (first lookup))) - {:db/id 888} - - :else {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {}) - ops @ops*] - (is (= :ok (:status result))) - (is (= [50] (get-in result [:data :result]))) - (is (= 4 (count ops))) - (is (some #(= [:batch-delete-property-value [[50] :block/tags 202]] %) ops)) - (is (some #(= [:batch-remove-property [[50] :logseq.property/deadline]] %) ops)) - (is (some #(= [:batch-set-property [[50] :block/tags 303 {}]] %) ops)) - (is (some #(= [:batch-set-property [[50] :logseq.property/publishing-public? true {}]] %) ops))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! add-command/resolve-tags orig-resolve-tags) - (set! add-command/resolve-properties orig-resolve-properties) - (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond (= tags [:tag/next]) [{:db/id 303}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil))) + add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties)) + add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties)) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup [:block/name "home"]) + {:db/id 50 :block/uuid (uuid "00000000-0000-0000-0000-000000000050")} + (and (vector? lookup) (= :db/ident (first lookup))) + {:db/id 888} + :else {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {}) + ops @ops*] + (is (= :ok (:status result))) + (is (= [50] (get-in result [:data :result]))) + (is (= 4 (count ops))) + (is (some #(= [:batch-delete-property-value [[50] :block/tags 202]] %) ops)) + (is (some #(= [:batch-remove-property [[50] :logseq.property/deadline]] %) ops)) + (is (some #(= [:batch-set-property [[50] :block/tags 303 {}]] %) ops)) + (is (some #(= [:batch-set-property [[50] :logseq.property/publishing-public? true {}]] %) ops)))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-upsert-page-errors-when-property-does-not-exist (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-resolve-tags add-command/resolve-tags - orig-resolve-properties add-command/resolve-properties - orig-resolve-property-identifiers add-command/resolve-property-identifiers - orig-invoke transport/invoke - action {:type :upsert-page - :repo "demo" - :page "Home" + (let [action {:type :upsert-page :repo "demo" :page "Home" :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"}}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! add-command/resolve-tags (fn [_ _ _] (p/resolved nil))) - (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) - (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) - (set! transport/invoke (fn [_ method _ args] - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup [:block/name "home"]) - {:db/id 50} - - (and (vector? lookup) (= :db/ident (first lookup))) - {} - - :else {})) - :thread-api/apply-outliner-ops - (throw (ex-info "should not apply ops when property lookup fails" - {:args args})) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :error (:status result))) - (is (= :property-not-found (get-in result [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! add-command/resolve-tags orig-resolve-tags) - (set! add-command/resolve-properties orig-resolve-properties) - (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + add-command/resolve-tags (fn [_ _ _] (p/resolved nil)) + add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties)) + add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties)) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup [:block/name "home"]) {:db/id 50} + (and (vector? lookup) (= :db/ident (first lookup))) {} + :else {})) + :thread-api/apply-outliner-ops + (throw (ex-info "should not apply ops when property lookup fails" {:args args})) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [result (commands/execute action {})] + (is (= :error (:status result))) + (is (= :property-not-found (get-in result [:error :code]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-remove-tag-property (async done - (let [ops* (atom []) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke - (fn [_ method _ args] - (case method - :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"}] - :thread-api/api-list-properties [{:db/id 2 :block/title "owner"}] - :thread-api/pull (let [[_ selector lookup] args] - (cond - (= lookup 1) - {:db/id 1 - :block/title "Quote" - :block/uuid (uuid "00000000-0000-0000-0000-000000000011") - :block/tags [{:db/ident :logseq.class/Tag}] - :logseq.property/public? true} - - (= lookup 2) - {:db/id 2 - :db/ident :user.property/owner - :block/title "owner" - :block/uuid (uuid "00000000-0000-0000-0000-000000000022") - :logseq.property/type :node - :logseq.property/public? true} - - (= lookup [:block/name "quote"]) - {:db/id 1 - :block/title "Quote" - :block/uuid (uuid "00000000-0000-0000-0000-000000000011") - :block/tags [{:db/ident :logseq.class/Tag}] - :logseq.property/public? true} - - (= lookup [:block/name "owner"]) - {:db/id 2 - :db/ident :user.property/owner - :block/title "owner" - :block/uuid (uuid "00000000-0000-0000-0000-000000000022") - :logseq.property/type :node - :logseq.property/public? true} - - :else - (throw (ex-info "unexpected pull lookup" - {:lookup lookup :selector selector})))) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (swap! ops* conj ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :args args}))))) - (-> (p/let [tag-result (commands/execute {:type :remove-tag - :repo "demo" - :name "Quote"} - {}) - property-result (commands/execute {:type :remove-property - :repo "demo" - :id 2} - {})] - (is (= :ok (:status tag-result))) - (is (= :ok (:status property-result))) - (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000011")]]] - (first @ops*))) - (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000022")]]] - (second @ops*)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (let [ops* (atom [])] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ args] + (case method + :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"}] + :thread-api/api-list-properties [{:db/id 2 :block/title "owner"}] + :thread-api/pull (let [[_ selector lookup] args] + (cond + (= lookup 1) {:db/id 1 :block/title "Quote" :block/uuid (uuid "00000000-0000-0000-0000-000000000011") :block/tags [{:db/ident :logseq.class/Tag}] :logseq.property/public? true} + (= lookup 2) {:db/id 2 :db/ident :user.property/owner :block/title "owner" :block/uuid (uuid "00000000-0000-0000-0000-000000000022") :logseq.property/type :node :logseq.property/public? true} + (= lookup [:block/name "quote"]) {:db/id 1 :block/title "Quote" :block/uuid (uuid "00000000-0000-0000-0000-000000000011") :block/tags [{:db/ident :logseq.class/Tag}] :logseq.property/public? true} + (= lookup [:block/name "owner"]) {:db/id 2 :db/ident :user.property/owner :block/title "owner" :block/uuid (uuid "00000000-0000-0000-0000-000000000022") :logseq.property/type :node :logseq.property/public? true} + :else (throw (ex-info "unexpected pull lookup" {:lookup lookup :selector selector})))) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (swap! ops* conj ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :args args}))))] + (p/let [tag-result (commands/execute {:type :remove-tag :repo "demo" :name "Quote"} {}) + property-result (commands/execute {:type :remove-property :repo "demo" :id 2} {})] + (is (= :ok (:status tag-result))) + (is (= :ok (:status property-result))) + (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000011")]]] + (first @ops*))) + (is (= [[:delete-page [(uuid "00000000-0000-0000-0000-000000000022")]]] + (second @ops*))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-remove-tag-ambiguous-name (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! transport/invoke - (fn [_ method _ _] - (case method - :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"} - {:db/id 2 :block/title "QUOTE"}] - (throw (ex-info "unexpected invoke" {:method method}))))) - (-> (p/let [result (commands/execute {:type :remove-tag - :repo "demo" - :name "Quote"} - {})] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ _] + (case method + :thread-api/api-list-tags [{:db/id 1 :block/title "Quote"} + {:db/id 2 :block/title "QUOTE"}] + (throw (ex-info "unexpected invoke" {:method method}))))] + (p/let [result (commands/execute {:type :remove-tag :repo "demo" :name "Quote"} {})] (is (= :error (:status result))) (is (= :ambiguous-tag-name (get-in result [:error :code]))) - (is (= 2 (count (get-in result [:error :candidates]))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (is (= 2 (count (get-in result [:error :candidates])))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-update-builds-batch-ops (async done (let [ops* (atom nil) calls* (atom []) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-resolve-tags add-command/resolve-tags - orig-resolve-properties add-command/resolve-properties - orig-resolve-property-identifiers add-command/resolve-property-identifiers - orig-invoke transport/invoke - action {:type :upsert-block - :mode :update - :repo "demo" - :id 1 - :target-id 2 - :pos "last-child" + action {:type :upsert-block :mode :update :repo "demo" :id 1 :target-id 2 :pos "last-child" :update-tags [:tag/new] :remove-tags [:tag/old] :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} :remove-properties [:logseq.property/publishing-public?]}] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] {:base-url "http://example"})) - (set! add-command/resolve-tags (fn [_ _ tags] - (p/resolved (cond - (= tags [:tag/new]) [{:db/id 101}] - (= tags [:tag/old]) [{:db/id 202}] - :else nil)))) - (set! add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties))) - (set! add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties))) - (set! transport/invoke (fn [_ method _ args] - (swap! calls* conj {:method method :args args}) - (case method - :thread-api/pull (let [[_ _ lookup] args] - (cond - (= lookup 1) - {:db/id 1 - :block/name nil - :block/uuid (uuid "00000000-0000-0000-0000-000000000001")} - (= lookup 2) - {:db/id 2 - :block/name nil - :block/uuid (uuid "00000000-0000-0000-0000-000000000002")} - :else {})) - :thread-api/apply-outliner-ops (let [[_ ops _] args] - (reset! ops* ops) - {:result :ok}) - (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))) - (-> (p/let [result (commands/execute action {})] - (is (= :ok (:status result))) - (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] - [:batch-delete-property-value [[1] :block/tags 202]] - [:batch-remove-property [[1] :logseq.property/publishing-public?]] - [:batch-set-property [[1] :block/tags 101 {}]] - [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] - @ops*))) - (p/catch (fn [e] - (is false (str "unexpected error: " e " calls: " @calls*)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! add-command/resolve-tags orig-resolve-tags) - (set! add-command/resolve-properties orig-resolve-properties) - (set! add-command/resolve-property-identifiers orig-resolve-property-identifiers) - (set! transport/invoke orig-invoke) - (done))))))) + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + add-command/resolve-tags (fn [_ _ tags] + (p/resolved (cond (= tags [:tag/new]) [{:db/id 101}] + (= tags [:tag/old]) [{:db/id 202}] + :else nil))) + add-command/resolve-properties (fn [_ _ properties & _] (p/resolved properties)) + add-command/resolve-property-identifiers (fn [_ _ properties & _] (p/resolved properties)) + transport/invoke (fn [_ method _ args] + (swap! calls* conj {:method method :args args}) + (case method + :thread-api/pull (let [[_ _ lookup] args] + (cond + (= lookup 1) {:db/id 1 :block/name nil :block/uuid (uuid "00000000-0000-0000-0000-000000000001")} + (= lookup 2) {:db/id 2 :block/name nil :block/uuid (uuid "00000000-0000-0000-0000-000000000002")} + :else {})) + :thread-api/apply-outliner-ops (let [[_ ops _] args] + (reset! ops* ops) + {:result :ok}) + (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))] + (p/let [result (commands/execute action {})] + (is (= :ok (:status result))) + (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] + [:batch-delete-property-value [[1] :block/tags 202]] + [:batch-remove-property [[1] :logseq.property/publishing-public?]] + [:batch-set-property [[1] :block/tags 101 {}]] + [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + @ops*)))) + (p/catch (fn [e] (is false (str "unexpected error: " e " calls: " @calls*)))) + (p/finally done))))) (deftest test-execute-requires-existing-graph (async done - (with-redefs [cli-server/list-graphs (fn [_] []) - cli-server/ensure-server! (fn [_ _] - (throw (ex-info "should not start server" {})))] - (-> (p/let [result (commands/execute {:type :list-page - :repo "logseq_db_missing"} - {})] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] []) + cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))] + (p/let [result (commands/execute {:type :list-page :repo "logseq_db_missing"} {})] (is (= :error (:status result))) (is (= :graph-not-exists (get-in result [:error :code]))) - (is (= "graph not exists" (get-in result [:error :message]))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (is (= "graph not exists" (get-in result [:error :message]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-graph-import-rejects-existing-graph (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server!] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] - (throw (ex-info "should not start server" {})))) - (-> (p/let [result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :allow-missing-graph true} - {})] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))] + (p/let [result (commands/execute {:type :graph-import :repo "logseq_db_demo" :allow-missing-graph true} {})] (is (= :error (:status result))) - (is (= :graph-exists (get-in result [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (done))))))) + (is (= :graph-exists (get-in result [:error :code]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-sync-download-rejects-existing-graph (async done - (let [orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server!] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [_ _] - (throw (ex-info "should not start server" {})))) - (-> (p/let [result (commands/execute {:type :sync-download + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [_ _] + (throw (ex-info "should not start server" {})))] + (p/let [result (commands/execute {:type :sync-download :repo "logseq_db_demo" :graph "demo" :allow-missing-graph true :require-missing-graph true} {})] (is (= :error (:status result))) - (is (= :graph-exists (get-in result [:error :code])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (done))))))) + (is (= :graph-exists (get-in result [:error :code]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) (deftest test-execute-sync-download-runs-command-when-graph-is-missing (async done - (let [orig-list-graphs cli-server/list-graphs - orig-execute sync-command/execute - captured (atom nil)] - (set! cli-server/list-graphs (fn [_] [])) - (set! sync-command/execute - (fn [action config] - (reset! captured [action config]) - (p/resolved {:status :ok - :data {:repo (:repo action)}}))) - (-> (p/let [result (commands/execute {:type :sync-download - :repo "logseq_db_demo" - :graph "demo" - :allow-missing-graph true - :require-missing-graph true} - {:data-dir "/tmp"})] - (is (= :ok (:status result))) - (is (= "logseq_db_demo" (get-in result [:data :repo]))) - (is (= :sync-download (get-in @captured [0 :type])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! sync-command/execute orig-execute) - (done))))))) + (let [captured (atom nil)] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] []) + sync-command/execute (fn [action config] + (reset! captured [action config]) + (p/resolved {:status :ok + :data {:repo (:repo action)}}))] + (p/let [result (commands/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :allow-missing-graph true + :require-missing-graph true} + {:data-dir "/tmp"})] + (is (= :ok (:status result))) + (is (= "logseq_db_demo" (get-in result [:data :repo]))) + (is (= :sync-download (get-in @captured [0 :type]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-graph-export (async done (let [invoke-calls (atom []) - write-calls (atom []) - orig-list-graphs cli-server/list-graphs - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke - orig-write-output transport/write-output] - (set! cli-server/list-graphs (fn [_] ["demo"])) - (set! cli-server/ensure-server! (fn [config _] - (assoc config :base-url "http://127.0.0.1:9999"))) - (set! transport/invoke (fn [_ method direct-pass? args] - (swap! invoke-calls conj [method direct-pass? args]) - (if (= method :thread-api/export-db-base64) - "c3FsaXRl" - {:exported true}))) - (set! transport/write-output (fn [opts] - (swap! write-calls conj opts))) - (-> (p/let [edn-result (commands/execute {:type :graph-export - :repo "logseq_db_demo" - :graph "demo" - :export-type "edn" - :file "/tmp/export.edn" - :allow-missing-graph true} - {}) - sqlite-result (commands/execute {:type :graph-export - :repo "logseq_db_demo" - :graph "demo" - :export-type "sqlite" - :file "/tmp/export.sqlite" - :allow-missing-graph true} - {})] - (is (= :ok (:status edn-result))) - (is (= :ok (:status sqlite-result))) - (is (= "edn" (get-in edn-result [:context :export-type]))) - (is (= "/tmp/export.edn" (get-in edn-result [:context :file]))) - (is (= "sqlite" (get-in sqlite-result [:context :export-type]))) - (is (= "/tmp/export.sqlite" (get-in sqlite-result [:context :file]))) - (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] - [:thread-api/export-db-base64 true ["logseq_db_demo"]]] - @invoke-calls)) - (is (= 2 (count @write-calls))) - (let [[edn-write sqlite-write] @write-calls] - (is (= {:format :edn :path "/tmp/export.edn" :data {:exported true}} - edn-write)) - (is (= :sqlite (:format sqlite-write))) - (is (= "/tmp/export.sqlite" (:path sqlite-write))) - (is (= "sqlite" (.toString (:data sqlite-write) "utf8"))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (set! transport/write-output orig-write-output) - (done))))))) + write-calls (atom [])] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) + cli-server/ensure-server! (fn [config _] + (assoc config :base-url "http://127.0.0.1:9999")) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (if (= method :thread-api/export-db-base64) + "c3FsaXRl" + {:exported true})) + transport/write-output (fn [opts] + (swap! write-calls conj opts))] + (p/let [edn-result (commands/execute {:type :graph-export :repo "logseq_db_demo" :graph "demo" :export-type "edn" :file "/tmp/export.edn" :allow-missing-graph true} {}) + sqlite-result (commands/execute {:type :graph-export :repo "logseq_db_demo" :graph "demo" :export-type "sqlite" :file "/tmp/export.sqlite" :allow-missing-graph true} {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= "edn" (get-in edn-result [:context :export-type]))) + (is (= "/tmp/export.edn" (get-in edn-result [:context :file]))) + (is (= "sqlite" (get-in sqlite-result [:context :export-type]))) + (is (= "/tmp/export.sqlite" (get-in sqlite-result [:context :file]))) + (is (= [[:thread-api/export-edn false ["logseq_db_demo" {:export-type :graph}]] + [:thread-api/export-db-base64 true ["logseq_db_demo"]]] + @invoke-calls)) + (is (= 2 (count @write-calls))) + (let [[edn-write sqlite-write] @write-calls] + (is (= {:format :edn :path "/tmp/export.edn" :data {:exported true}} edn-write)) + (is (= :sqlite (:format sqlite-write))) + (is (= "/tmp/export.sqlite" (:path sqlite-write))) + (is (= "sqlite" (.toString (:data sqlite-write) "utf8")))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-graph-import (async done (let [invoke-calls (atom []) read-calls (atom []) stop-calls (atom []) - restart-calls (atom []) - orig-list-graphs cli-server/list-graphs - orig-stop-server! cli-server/stop-server! - orig-restart-server! cli-server/restart-server! - orig-ensure-server! cli-server/ensure-server! - orig-read-input transport/read-input - orig-invoke transport/invoke] - (set! cli-server/list-graphs (fn [_] [])) - (set! cli-server/stop-server! (fn [_ repo] - (swap! stop-calls conj repo) - (p/resolved {:ok? true}))) - (set! cli-server/restart-server! (fn [_ repo] - (swap! restart-calls conj repo) - (p/resolved {:ok? true}))) - (set! cli-server/ensure-server! (fn [config _] - (assoc config :base-url "http://127.0.0.1:9999"))) - (set! transport/read-input (fn [{:keys [format path]}] - (swap! read-calls conj [format path]) - (if (= format :edn) - {:page "Import Page"} - (js/Buffer.from "sqlite" "utf8")))) - (set! transport/invoke (fn [_ method _ args] - (swap! invoke-calls conj [method args]) - {:ok true})) - (-> (p/let [edn-result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :graph "demo" - :import-type "edn" - :input "/tmp/import.edn" - :allow-missing-graph true} - {}) - sqlite-result (commands/execute {:type :graph-import - :repo "logseq_db_demo" - :graph "demo" - :import-type "sqlite" - :input "/tmp/import.sqlite" - :allow-missing-graph true} - {})] - (is (= :ok (:status edn-result))) - (is (= :ok (:status sqlite-result))) - (is (= "edn" (get-in edn-result [:context :import-type]))) - (is (= "/tmp/import.edn" (get-in edn-result [:context :input]))) - (is (= "sqlite" (get-in sqlite-result [:context :import-type]))) - (is (= "/tmp/import.sqlite" (get-in sqlite-result [:context :input]))) - (is (= [[:edn "/tmp/import.edn"] - [:sqlite "/tmp/import.sqlite"]] - @read-calls)) - (is (= [[:thread-api/import-edn ["logseq_db_demo" {:page "Import Page"}]] - [:thread-api/import-db-base64 ["logseq_db_demo" "c3FsaXRl"]]] - @invoke-calls)) - (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) - (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (set! cli-server/stop-server! orig-stop-server!) - (set! cli-server/restart-server! orig-restart-server!) - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/read-input orig-read-input) - (set! transport/invoke orig-invoke) - (done))))))) + restart-calls (atom [])] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] []) + cli-server/stop-server! (fn [_ repo] (swap! stop-calls conj repo) (p/resolved {:ok? true})) + cli-server/restart-server! (fn [_ repo] (swap! restart-calls conj repo) (p/resolved {:ok? true})) + cli-server/ensure-server! (fn [config _] (assoc config :base-url "http://127.0.0.1:9999")) + transport/read-input (fn [{:keys [format path]}] + (swap! read-calls conj [format path]) + (if (= format :edn) + {:page "Import Page"} + (js/Buffer.from "sqlite" "utf8"))) + transport/invoke (fn [_ method _ args] + (swap! invoke-calls conj [method args]) + {:ok true})] + (p/let [edn-result (commands/execute {:type :graph-import :repo "logseq_db_demo" :graph "demo" :import-type "edn" :input "/tmp/import.edn" :allow-missing-graph true} {}) + sqlite-result (commands/execute {:type :graph-import :repo "logseq_db_demo" :graph "demo" :import-type "sqlite" :input "/tmp/import.sqlite" :allow-missing-graph true} {})] + (is (= :ok (:status edn-result))) + (is (= :ok (:status sqlite-result))) + (is (= "edn" (get-in edn-result [:context :import-type]))) + (is (= "/tmp/import.edn" (get-in edn-result [:context :input]))) + (is (= "sqlite" (get-in sqlite-result [:context :import-type]))) + (is (= "/tmp/import.sqlite" (get-in sqlite-result [:context :input]))) + (is (= [[:edn "/tmp/import.edn"] [:sqlite "/tmp/import.sqlite"]] @read-calls)) + (is (= [[:thread-api/import-edn ["logseq_db_demo" {:page "Import Page"}]] + [:thread-api/import-db-base64 ["logseq_db_demo" "c3FsaXRl"]]] + @invoke-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @stop-calls)) + (is (= ["logseq_db_demo" "logseq_db_demo"] @restart-calls)))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-graph-list-strips-db-prefix (async done - (let [orig-list-graphs cli-server/list-graphs] - (set! cli-server/list-graphs (fn [_] ["logseq_db_demo" - "logseq_db_logseq_db_other" - "my_logseq_db_notes"])) - (-> (p/let [result (commands/execute {:type :graph-list} {})] + (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["logseq_db_demo" + "logseq_db_logseq_db_other" + "my_logseq_db_notes"])] + (p/let [result (commands/execute {:type :graph-list} {})] (is (= :ok (:status result))) (is (= ["demo" "logseq_db_other" "my_logseq_db_notes"] - (get-in result [:data :graphs])))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/list-graphs orig-list-graphs) - (done))))))) + (get-in result [:data :graphs]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 746e3b65ca..fca33a55dd 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -13,6 +13,7 @@ [logseq.cli.main :as cli-main] [logseq.cli.server :as cli-server] [logseq.cli.style :as style] + [logseq.cli.test-helper :as test-helper] [logseq.cli.transport :as transport] [logseq.common.util :as common-util] [logseq.db.frontend.property :as db-property] @@ -86,16 +87,12 @@ e)))))) (defn- capture-stderr! - [] - (let [stderr (.-stderr js/process) - original-write (.-write stderr) - buffer (atom "")] - (set! (.-write stderr) - (fn [chunk] - (swap! buffer str chunk) - true)) - {:buffer buffer - :restore! (fn [] (set! (.-write stderr) original-write))})) + [f] + (test-helper/with-stderr-write-capture + (fn [buffer] + (p/let [result (f)] + {:buffer buffer + :result result})))) (defn- node-title [node] @@ -193,8 +190,6 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-cli") download-repo "sync-download-graph" start-repo "sync-start-graph" - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke invoke-calls (atom []) status-calls (atom 0)] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -203,46 +198,46 @@ create-payload (parse-json-output-safe create-result "graph create") _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) - _ (set! cli-server/ensure-server! - (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - _ (set! transport/invoke - (fn [_ method _direct-pass? args] - (swap! invoke-calls conj [method args]) - (case method - :thread-api/set-db-sync-config - (p/resolved nil) + [download-result start-result] + (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) - :thread-api/db-sync-list-remote-graphs - (p/resolved [{:graph-id "remote-graph-id" - :graph-name download-repo - :graph-e2ee? true}]) + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name download-repo + :graph-e2ee? true}]) - :thread-api/db-sync-download-graph-by-id - (p/resolved {:repo "logseq_db_sync_integration_graph" - :graph-id "remote-graph-id" - :remote-tx 22 - :graph-e2ee? true - :row-count 3}) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:repo "logseq_db_sync_integration_graph" + :graph-id "remote-graph-id" + :remote-tx 22 + :graph-e2ee? true + :row-count 3}) - :thread-api/db-sync-start - (p/resolved nil) + :thread-api/db-sync-start + (p/resolved nil) - :thread-api/db-sync-status - (let [idx (swap! status-calls inc)] - (p/resolved {:repo "logseq_db_sync_integration_graph" - :ws-state (if (= idx 1) :connecting :open) - :pending-local 0 - :pending-asset 0 - :pending-server 0})) + :thread-api/db-sync-status + (let [idx (swap! status-calls inc)] + (p/resolved {:repo "logseq_db_sync_integration_graph" + :ws-state (if (= idx 1) :connecting :open) + :pending-local 0 + :pending-asset 0 + :pending-server 0})) - :thread-api/q - (p/resolved 0) + :thread-api/q + (p/resolved 0) - (p/resolved nil)))) - download-result (run-cli ["--graph" download-repo "sync" "download"] data-dir cfg-path) + (p/resolved nil)))] + (p/let [download-result (run-cli ["--graph" download-repo "sync" "download"] data-dir cfg-path) + start-result (run-cli ["--graph" start-repo "sync" "start"] data-dir cfg-path)] + [download-result start-result])) download-payload (parse-json-output-safe download-result "sync download") - start-result (run-cli ["--graph" start-repo "sync" "start"] data-dir cfg-path) start-payload (parse-json-output-safe start-result "sync start")] (is (some #(= :thread-api/db-sync-download-graph-by-id (first %)) @invoke-calls)) (is (some #(= :thread-api/q (first %)) @invoke-calls)) @@ -259,17 +254,12 @@ (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) - (done))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke))))))) + (done))))))) (deftest test-cli-sync-upload-with-mocked-worker-bootstrap (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-cli") upload-repo "sync-upload-graph" - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke invoke-calls (atom [])] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") @@ -277,21 +267,20 @@ create-payload (parse-json-output-safe create-result "graph create") _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) - _ (set! cli-server/ensure-server! - (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - _ (set! transport/invoke - (fn [_ method _direct-pass? args] - (swap! invoke-calls conj [method args]) - (case method - :thread-api/set-db-sync-config - (p/resolved nil) + upload-result (p/with-redefs + [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) - :thread-api/db-sync-upload-graph - (p/resolved {:graph-id "created-graph-id"}) + :thread-api/db-sync-upload-graph + (p/resolved {:graph-id "created-graph-id"}) - (p/resolved nil)))) - upload-result (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path) + (p/resolved nil)))] + (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path)) upload-payload (parse-json-output-safe upload-result "sync upload")] (is (= 0 (:exit-code upload-result))) (is (= "ok" (:status upload-payload))) @@ -301,22 +290,16 @@ :auth-token nil :e2ee-password nil}]] [:thread-api/db-sync-upload-graph ["logseq_db_sync-upload-graph"]]] - @invoke-calls)) - (done)) + @invoke-calls))) (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke))))))) + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-cli-sync-upload-followed-by-graph-info-shows-graph-uuid-test (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-info-cli") upload-repo "sync-upload-graph-info" uploaded-graph-id "0f64b4a9-6f31-4f35-a83c-6b16f9ddf1ff" - orig-ensure-server! cli-server/ensure-server! - orig-invoke transport/invoke invoke-calls (atom [])] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") @@ -324,27 +307,27 @@ create-payload (parse-json-output-safe create-result "graph create") _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) - _ (set! cli-server/ensure-server! - (fn [config _repo] - (p/resolved (assoc config :base-url "http://example")))) - _ (set! transport/invoke - (fn [_ method _direct-pass? args] - (swap! invoke-calls conj [method args]) - (case method - :thread-api/set-db-sync-config - (p/resolved nil) + [upload-result info-result] + (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method _direct-pass? args] + (swap! invoke-calls conj [method args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) - :thread-api/db-sync-upload-graph - (p/resolved {:graph-id uploaded-graph-id}) + :thread-api/db-sync-upload-graph + (p/resolved {:graph-id uploaded-graph-id}) - :thread-api/q - (p/resolved [[:logseq.kv/graph-uuid uploaded-graph-id] - [:logseq.kv/schema-version "65"]]) + :thread-api/q + (p/resolved [[:logseq.kv/graph-uuid uploaded-graph-id] + [:logseq.kv/schema-version "65"]]) - (p/resolved nil)))) - upload-result (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path) + (p/resolved nil)))] + (p/let [upload-result (run-cli ["--graph" upload-repo "sync" "upload"] data-dir cfg-path) + info-result (run-cli ["--graph" upload-repo "graph" "info"] data-dir cfg-path)] + [upload-result info-result])) upload-payload (parse-json-output-safe upload-result "sync upload") - info-result (run-cli ["--graph" upload-repo "graph" "info"] data-dir cfg-path) info-payload (parse-json-output-safe info-result "graph info after upload") q-call (some (fn [[method args]] (when (= :thread-api/q method) @@ -360,10 +343,7 @@ (is (= "logseq_db_sync-upload-graph-info" (first q-call)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! cli-server/ensure-server! orig-ensure-server!) - (set! transport/invoke orig-invoke) - (done))))))) + (p/finally done))))) (deftest ^:long test-cli-graph-list (async done @@ -1151,9 +1131,9 @@ (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") _ (fs/writeFileSync cfg-path "{:output-format :json}") _ (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) - {:keys [buffer restore!]} (capture-stderr!) - result (-> (run-cli ["--verbose" "--graph" repo "graph" "info"] data-dir cfg-path) - (p/finally (fn [] (restore!)))) + {:keys [buffer result]} (capture-stderr! + (fn [] + (run-cli ["--verbose" "--graph" repo "graph" "info"] data-dir cfg-path))) payload (parse-json-output-safe result "verbose graph info") stderr-text @buffer _ (run-cli ["server" "stop" "--graph" repo] data-dir cfg-path)] @@ -1212,10 +1192,10 @@ (let [data-dir (node-helper/create-tmp-dir "db-worker-upsert-block-custom-property")] (-> (p/let [{:keys [cfg-path repo]} (setup-tags-graph data-dir) upsert-property-result (run-cli ["--graph" repo - "upsert" "property" - "--name" "owner" - "--type" "default"] - data-dir cfg-path) + "upsert" "property" + "--name" "owner" + "--type" "default"] + data-dir cfg-path) upsert-property-payload (parse-json-output upsert-property-result) add-block-result (run-cli ["--graph" repo "upsert" "block" diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index c659056625..b321c7ea9f 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -1,40 +1,46 @@ (ns logseq.cli.server-test {:clj-kondo/config '{:linters {:private-var-access {:level :off}}}} - (:require [cljs.test :refer [async deftest is]] - [frontend.test.node-helper :as node-helper] - [logseq.cli.server :as cli-server] - [logseq.db-worker.daemon :as daemon] - [promesa.core :as p] + (:require ["child_process" :as child-process] ["fs" :as fs] ["http" :as http] ["path" :as node-path] - ["child_process" :as child-process])) + [cljs.test :refer [async deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.server :as cli-server] + [logseq.cli.test-helper :as test-helper] + [logseq.db-worker.daemon :as daemon] + [promesa.core :as p])) (deftest spawn-server-omits-host-and-port-flags (let [spawn-server! #'cli-server/spawn-server! captured (atom nil) - original-spawn (.-spawn child-process) original-cwd (.cwd js/process)] - (set! (.-spawn child-process) - (fn [cmd args opts] - (reset! captured {:cmd cmd - :args (vec (js->clj args)) - :opts (js->clj opts :keywordize-keys true)}) - (js-obj "unref" (fn [] nil)))) (try (.chdir js/process "/") - (spawn-server! {:repo "logseq_db_spawn_test" - :data-dir "/tmp/logseq-db-worker"}) - (is (= (.-execPath js/process) - (:cmd @captured))) - (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") - (first (:args @captured)))) - (is (some #{"--repo"} (:args @captured))) - (is (some #{"--data-dir"} (:args @captured))) - (is (not-any? #{"--host" "--port"} (:args @captured))) + (-> (test-helper/with-js-property-override + child-process + "spawn" + (fn [cmd args opts] + (reset! captured {:cmd cmd + :args (vec (js->clj args)) + :opts (js->clj opts :keywordize-keys true)}) + (js-obj "unref" (fn [] nil))) + (fn [] + (spawn-server! {:repo "logseq_db_spawn_test" + :data-dir "/tmp/logseq-db-worker"}) + (p/resolved true))) + (p/then (fn [_] + (is (= (.-execPath js/process) + (:cmd @captured))) + (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") + (first (:args @captured)))) + (is (some #{"--repo"} (:args @captured))) + (is (some #{"--data-dir"} (:args @captured))) + (is (not-any? #{"--host" "--port"} (:args @captured))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e))))) (finally - (.chdir js/process original-cwd) - (set! (.-spawn child-process) original-spawn))))) + (.chdir js/process original-cwd))))) (deftest lock-path-uses-canonical-graph-dir (let [data-dir "/tmp/logseq-db-worker" @@ -46,264 +52,214 @@ (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") (cli-server/db-worker-runtime-script-path)))) - (deftest ensure-server-repairs-stale-lock (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server") - repo (str "logseq_db_stale_" (subs (str (random-uuid)) 0 8)) - path (cli-server/lock-path data-dir repo) - cleanup-stale-lock! #'cli-server/cleanup-stale-lock! - lock {:repo repo - :pid (.-pid js/process) - :host "127.0.0.1" - :port 0 - :startedAt (.toISOString (js/Date.))}] - (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) - (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) - (-> (p/let [_ (cleanup-stale-lock! path lock)] - (is (not (fs/existsSync path))) - (done)) - (p/catch (fn [e] - (is false (str "unexpected error: " e)) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server") + repo (str "logseq_db_stale_" (subs (str (random-uuid)) 0 8)) + path (cli-server/lock-path data-dir repo) + cleanup-stale-lock! #'cli-server/cleanup-stale-lock! + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 0 + :startedAt (.toISOString (js/Date.))}] + (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) + (fs/writeFileSync path (js/JSON.stringify (clj->js lock))) + (-> (p/let [_ (cleanup-stale-lock! path lock)] + (is (not (fs/existsSync path))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) (deftest ensure-server-reuses-existing-running-daemon-lock (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-reuse") - repo (str "logseq_db_reuse_" (subs (str (random-uuid)) 0 8)) - lock-file (cli-server/lock-path data-dir repo) - host "127.0.0.1" - spawn-calls (atom 0) - server (http/createServer - (fn [^js req ^js res] - (case (.-url req) - "/healthz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) - (.end res "ok")) - "/readyz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) - (.end res "ok")) - (do (.writeHead res 404 #js {"Content-Type" "text/plain"}) - (.end res "not-found")))))] - (.listen server 0 host - (fn [] - (let [address (.address server) - port (if (number? address) address (.-port address)) - lock {:repo repo - :pid (.-pid js/process) - :host host - :port port} - original-spawn! daemon/spawn-server!] - (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) - (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) - (set! daemon/spawn-server! - (fn [_opts] - (swap! spawn-calls inc) - (throw (ex-info "should not spawn when lock is ready" {})))) - (-> (cli-server/ensure-server! {:data-dir data-dir} repo) - (p/then (fn [config] - (is (= (str "http://" host ":" port) (:base-url config))) - (is (= 0 @spawn-calls)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/spawn-server! original-spawn!) - (.close server (fn [] (done)))))))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-reuse") + repo (str "logseq_db_reuse_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + host "127.0.0.1" + spawn-calls (atom 0) + server (http/createServer + (fn [^js req ^js res] + (case (.-url req) + "/healthz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")) + "/readyz" (do (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")) + (do (.writeHead res 404 #js {"Content-Type" "text/plain"}) + (.end res "not-found")))))] + (.listen server 0 host + (fn [] + (let [address (.address server) + port (if (number? address) address (.-port address)) + lock {:repo repo + :pid (.-pid js/process) + :host host + :port port}] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (-> (p/with-redefs [daemon/spawn-server! (fn [_opts] + (swap! spawn-calls inc) + (throw (ex-info "should not spawn when lock is ready" {})))] + (cli-server/ensure-server! {:data-dir data-dir} repo)) + (p/then (fn [config] + (is (= (str "http://" host ":" port) (:base-url config))) + (is (= 0 @spawn-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (.close server (fn [] (done)))))))))))) (deftest start-server-reports-repo-locked-error-stably (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-repo-locked") - repo (str "logseq_db_locked_" (subs (str (random-uuid)) 0 8)) - lock-file (cli-server/lock-path data-dir repo) - lock {:repo repo - :pid 999999 - :host "127.0.0.1" - :port 55555} - original-cleanup daemon/cleanup-stale-lock! - original-ready daemon/wait-for-ready] - (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) - (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) - (set! daemon/cleanup-stale-lock! (fn [_path _lock] (p/resolved nil))) - (set! daemon/wait-for-ready - (fn [_lock] - (p/rejected (ex-info "graph already locked" - {:code :repo-locked - :lock lock})))) - (-> (cli-server/start-server! {:data-dir data-dir} repo) - (p/then (fn [result] - (is (= false (:ok? result))) - (is (= :repo-locked (get-in result [:error :code]))) - (is (= lock (get-in result [:error :lock]))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/cleanup-stale-lock! original-cleanup) - (set! daemon/wait-for-ready original-ready) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-repo-locked") + repo (str "logseq_db_locked_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid 999999 + :host "127.0.0.1" + :port 55555}] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (-> (p/with-redefs [daemon/cleanup-stale-lock! (fn [_path _lock] (p/resolved nil)) + daemon/wait-for-ready (fn [_lock] + (p/rejected (ex-info "graph already locked" + {:code :repo-locked + :lock lock})))] + (cli-server/start-server! {:data-dir data-dir} repo)) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :repo-locked (get-in result [:error :code]))) + (is (= lock (get-in result [:error :lock]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest stop-server-denies-owner-mismatch (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-stop") - repo (str "logseq_db_owner_stop_" (subs (str (random-uuid)) 0 8)) - lock-file (cli-server/lock-path data-dir repo) - lock {:repo repo - :pid (.-pid js/process) - :host "127.0.0.1" - :port 9101 - :owner-source :electron} - original-http daemon/http-request - original-wait daemon/wait-for] - (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) - (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) - (set! daemon/http-request (fn [_] (p/resolved {:status 200 :body ""}))) - (set! daemon/wait-for (fn [_ _] (p/resolved true))) - (-> (cli-server/stop-server! {:data-dir data-dir - :owner-source :cli} - repo) - (p/then (fn [result] - (is (= false (:ok? result))) - (is (= :server-owned-by-other (get-in result [:error :code]))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/http-request original-http) - (set! daemon/wait-for original-wait) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-stop") + repo (str "logseq_db_owner_stop_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9101 + :owner-source :electron}] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (-> (p/with-redefs [daemon/http-request (fn [_] (p/resolved {:status 200 :body ""})) + daemon/wait-for (fn [_ _] (p/resolved true))] + (cli-server/stop-server! {:data-dir data-dir + :owner-source :cli} + repo)) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-owned-by-other (get-in result [:error :code]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest restart-server-does-not-sigterm-external-owner-daemon (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-restart") - repo (str "logseq_db_owner_restart_" (subs (str (random-uuid)) 0 8)) - lock-file (cli-server/lock-path data-dir repo) - lock {:repo repo - :pid 424242 - :host "127.0.0.1" - :port 9102 - :owner-source :electron} - start-calls (atom 0) - kill-calls (atom []) - original-start cli-server/start-server! - original-http daemon/http-request - original-wait daemon/wait-for - original-pid-status daemon/pid-status - original-kill (.-kill js/process)] - (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) - (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) - (set! daemon/http-request (fn [_] (p/resolved {:status 200 :body ""}))) - (set! daemon/wait-for (fn [_ _] (p/rejected (ex-info "timeout" {:code :timeout})))) - (set! daemon/pid-status (fn [_] :alive)) - (set! (.-kill js/process) - (fn [pid signal] - (swap! kill-calls conj [pid signal]) - true)) - (set! cli-server/start-server! - (fn [_ _] - (swap! start-calls inc) - (p/resolved {:ok? true - :data {:repo repo}}))) - (-> (cli-server/restart-server! {:data-dir data-dir - :owner-source :cli} - repo) - (p/then (fn [result] - (is (= false (:ok? result))) - (is (= :server-owned-by-other (get-in result [:error :code]))) - (is (zero? @start-calls)) - (is (empty? @kill-calls)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/http-request original-http) - (set! daemon/wait-for original-wait) - (set! daemon/pid-status original-pid-status) - (set! (.-kill js/process) original-kill) - (set! cli-server/start-server! original-start) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-owner-restart") + repo (str "logseq_db_owner_restart_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid 424242 + :host "127.0.0.1" + :port 9102 + :owner-source :electron} + start-calls (atom 0) + kill-calls (atom [])] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (-> (test-helper/with-js-property-override + js/process + "kill" + (fn [pid signal] + (swap! kill-calls conj [pid signal]) + true) + (fn [] + (p/with-redefs [daemon/http-request (fn [_] (p/resolved {:status 200 :body ""})) + daemon/wait-for (fn [_ _] (p/rejected (ex-info "timeout" {:code :timeout}))) + daemon/pid-status (fn [_] :alive) + cli-server/start-server! (fn [_ _] + (swap! start-calls inc) + (p/resolved {:ok? true + :data {:repo repo}}))] + (cli-server/restart-server! {:data-dir data-dir + :owner-source :cli} + repo)))) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-owned-by-other (get-in result [:error :code]))) + (is (zero? @start-calls)) + (is (empty? @kill-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest start-server-returns-timeout-orphan-error-with-pids (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-orphan-timeout") - repo (str "logseq_db_orphan_timeout_" (subs (str (random-uuid)) 0 8)) - cleanup-calls (atom 0) - spawn-calls (atom 0) - original-cleanup-stale daemon/cleanup-stale-lock! - original-cleanup-orphans daemon/cleanup-orphan-processes! - original-spawn daemon/spawn-server! - original-wait-lock daemon/wait-for-lock - original-find-orphans daemon/find-orphan-processes] - (set! daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil))) - (set! daemon/cleanup-orphan-processes! (fn [_] - (swap! cleanup-calls inc) - {:killed-pids [111]})) - (set! daemon/spawn-server! (fn [_] - (swap! spawn-calls inc) - nil)) - (set! daemon/wait-for-lock (fn [_] - (p/rejected (ex-info "timeout" - {:code :timeout})))) - (set! daemon/find-orphan-processes (fn [_] - [{:pid 111} - {:pid 222}])) - (-> (cli-server/start-server! {:data-dir data-dir - :owner-source :cli} - repo) - (p/then (fn [result] - (is (= false (:ok? result))) - (is (= :server-start-timeout-orphan (get-in result [:error :code]))) - (is (= [111 222] (get-in result [:error :pids]))) - (is (= 1 @cleanup-calls)) - (is (= 1 @spawn-calls)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/cleanup-stale-lock! original-cleanup-stale) - (set! daemon/cleanup-orphan-processes! original-cleanup-orphans) - (set! daemon/spawn-server! original-spawn) - (set! daemon/wait-for-lock original-wait-lock) - (set! daemon/find-orphan-processes original-find-orphans) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-orphan-timeout") + repo (str "logseq_db_orphan_timeout_" (subs (str (random-uuid)) 0 8)) + cleanup-calls (atom 0) + spawn-calls (atom 0)] + (-> (p/with-redefs [daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil)) + daemon/cleanup-orphan-processes! (fn [_] + (swap! cleanup-calls inc) + {:killed-pids [111]}) + daemon/spawn-server! (fn [_] + (swap! spawn-calls inc) + nil) + daemon/wait-for-lock (fn [_] + (p/rejected (ex-info "timeout" + {:code :timeout}))) + daemon/find-orphan-processes (fn [_] + [{:pid 111} + {:pid 222}])] + (cli-server/start-server! {:data-dir data-dir + :owner-source :cli} + repo)) + (p/then (fn [result] + (is (= false (:ok? result))) + (is (= :server-start-timeout-orphan (get-in result [:error :code]))) + (is (= [111 222] (get-in result [:error :pids]))) + (is (= 1 @cleanup-calls)) + (is (= 1 @spawn-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest ensure-server-forwards-create-empty-db-flag-when-spawning-daemon (async done - (let [data-dir (node-helper/create-tmp-dir "cli-server-create-empty") - repo (str "logseq_db_create_empty_" (subs (str (random-uuid)) 0 8)) - captured (atom nil) - lock {:repo repo - :pid (.-pid js/process) - :host "127.0.0.1" - :port 9301} - read-lock-calls (atom 0) - original-read-lock daemon/read-lock - original-cleanup-stale daemon/cleanup-stale-lock! - original-cleanup-orphans daemon/cleanup-orphan-processes! - original-spawn daemon/spawn-server! - original-wait-lock daemon/wait-for-lock - original-wait-ready daemon/wait-for-ready] - (set! daemon/read-lock - (fn [_] - (if (= 1 (swap! read-lock-calls inc)) - nil - lock))) - (set! daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil))) - (set! daemon/cleanup-orphan-processes! (fn [_] {:orphans [] :killed-pids []})) - (set! daemon/spawn-server! - (fn [opts] - (reset! captured opts) - nil)) - (set! daemon/wait-for-lock (fn [_] (p/resolved true))) - (set! daemon/wait-for-ready (fn [_] (p/resolved true))) - (-> (cli-server/ensure-server! {:data-dir data-dir - :create-empty-db? true} - repo) - (p/then (fn [_] - (is (= repo (:repo @captured))) - (is (= (cli-server/resolve-data-dir {:data-dir data-dir}) - (:data-dir @captured))) - (is (= true (:create-empty-db? @captured))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (set! daemon/read-lock original-read-lock) - (set! daemon/cleanup-stale-lock! original-cleanup-stale) - (set! daemon/cleanup-orphan-processes! original-cleanup-orphans) - (set! daemon/spawn-server! original-spawn) - (set! daemon/wait-for-lock original-wait-lock) - (set! daemon/wait-for-ready original-wait-ready) - (done))))))) + (let [data-dir (node-helper/create-tmp-dir "cli-server-create-empty") + repo (str "logseq_db_create_empty_" (subs (str (random-uuid)) 0 8)) + captured (atom nil) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9301} + read-lock-calls (atom 0)] + (-> (p/with-redefs [daemon/read-lock (fn [_] + (if (= 1 (swap! read-lock-calls inc)) + nil + lock)) + daemon/cleanup-stale-lock! (fn [_ _] (p/resolved nil)) + daemon/cleanup-orphan-processes! (fn [_] {:orphans [] :killed-pids []}) + daemon/spawn-server! (fn [opts] + (reset! captured opts) + nil) + daemon/wait-for-lock (fn [_] (p/resolved true)) + daemon/wait-for-ready (fn [_] (p/resolved true))] + (cli-server/ensure-server! {:data-dir data-dir + :create-empty-db? true} + repo)) + (p/then (fn [_] + (is (= repo (:repo @captured))) + (is (= (cli-server/resolve-data-dir {:data-dir data-dir}) + (:data-dir @captured))) + (is (= true (:create-empty-db? @captured))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) diff --git a/src/test/logseq/cli/test_helper.cljs b/src/test/logseq/cli/test_helper.cljs new file mode 100644 index 0000000000..bbc4849423 --- /dev/null +++ b/src/test/logseq/cli/test_helper.cljs @@ -0,0 +1,30 @@ +(ns logseq.cli.test-helper + (:require [goog.object :as gobj] + [promesa.core :as p])) + +(defn with-js-property-override + "Temporarily override a JS object property for the duration of an async body. + + Restores the original property value after the returned promise settles. Use + this for process/module globals that cannot be handled by `with-redefs`. + Resource cleanup still belongs in outer `p/finally` blocks." + [obj prop value f] + (let [original (gobj/get obj prop)] + (gobj/set obj prop value) + (-> (f) + (p/finally (fn [] + (gobj/set obj prop original)))))) + +(defn with-stderr-write-capture + "Capture writes to `process.stderr.write` for the duration of an async body. + + Calls `f` with an atom containing the accumulated stderr output." + [f] + (let [stderr (.-stderr js/process) + buffer (atom "")] + (with-js-property-override stderr "write" + (fn [chunk] + (swap! buffer str chunk) + true) + (fn [] + (f buffer))))) \ No newline at end of file diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index 5402a487ef..e23bb44fd2 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -1,5 +1,6 @@ (ns logseq.cli.transport-test (:require [cljs.test :refer [deftest is async testing]] + [logseq.cli.test-helper :as test-helper] [logseq.cli.transport :as transport] [promesa.core :as p])) @@ -34,25 +35,25 @@ (let [url-module (js/require "url") original-parse (.-parse url-module) parse-calls (atom 0)] - (set! (.-parse url-module) + (-> (test-helper/with-js-property-override + url-module + "parse" (fn [& args] (swap! parse-calls inc) - (.apply original-parse url-module (to-array args)))) - (-> (p/let [{:keys [url stop!]} (start-server - (fn [_req ^js res] - (.writeHead res 200 #js {"Content-Type" "text/plain"}) - (.end res "ok")))] - (p/let [response (transport/request {:method "GET" - :url (str url "/status") - :timeout-ms 1000})] - (is (= 200 (:status response))) - (is (= 0 @parse-calls)) - (p/let [_ (stop!)] true))) - (p/then (fn [_] - (set! (.-parse url-module) original-parse) - (done))) + (.apply original-parse url-module (to-array args))) + (fn [] + (p/let [{:keys [url stop!]} (start-server + (fn [_req ^js res] + (.writeHead res 200 #js {"Content-Type" "text/plain"}) + (.end res "ok")))] + (p/let [response (transport/request {:method "GET" + :url (str url "/status") + :timeout-ms 1000})] + (is (= 200 (:status response))) + (is (= 0 @parse-calls)) + (p/let [_ (stop!)] true))))) + (p/then (fn [_] (done))) (p/catch (fn [e] - (set! (.-parse url-module) original-parse) (is false (str "unexpected error: " e)) (done))))))) From f0cabc65e9124fa9503cd18bbea27a56bf213501 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 7 Mar 2026 21:53:47 +0800 Subject: [PATCH 117/375] chore: remove unused fn --- src/main/logseq/cli/command/show.cljs | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index a65203e50a..0ab5106279 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -29,13 +29,6 @@ [] (.toString (fs/readFileSync 0) "utf8")) -(defn- stdin-available? - [] - (try - (let [stat (fs/fstatSync 0)] - (or (.isFIFO stat) (.isFile stat))) - (catch :default _ false))) - (defn- normalize-stdin-id [value] (let [text (string/trim (or value ""))] From 9502f57dbedd15bdab18424b4bfe7dd850058daa Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 09:44:06 -0400 Subject: [PATCH 118/375] 054-db-sync-test-isolation.md --- .../agent-guide/054-db-sync-test-isolation.md | 166 +++++++++++++ src/test/frontend/worker/db_sync_test.cljs | 226 ++++++++++-------- 2 files changed, 286 insertions(+), 106 deletions(-) create mode 100644 docs/agent-guide/054-db-sync-test-isolation.md diff --git a/docs/agent-guide/054-db-sync-test-isolation.md b/docs/agent-guide/054-db-sync-test-isolation.md new file mode 100644 index 0000000000..b54c5f0b9c --- /dev/null +++ b/docs/agent-guide/054-db-sync-test-isolation.md @@ -0,0 +1,166 @@ +# 054-db-sync-test-isolation + +## Summary + +`bb dev:lint-and-test` would currently fail in CLJS runtime tests, while lint and test compilation would still pass. The observed failures would point to test isolation problems inside `frontend.worker.db-sync-test`, not to a stable regression in db-sync production logic. + +## Background + +The failing command would be: + +- `bb dev:lint-and-test` + +Its execution flow would be: + +1. lint tasks +2. CLJS test compilation +3. CLJS runtime test execution + +Observed behavior from the captured run: + +- lint would pass +- test compilation would pass +- runtime tests would fail with 3 failures + +The failing tests would include: + +- `frontend.worker.db-sync-test/ensure-upload-graph-identity-defaults-e2ee-enabled-test` +- `frontend.worker.db-sync-test/download-graph-by-id-fails-on-incomplete-snapshot-frame-test` + +A follow-up run of the full namespace would fail on a different test: + +- `frontend.worker.db-sync-test/pull-ok-out-of-order-stale-response-is-ignored-test` + +This changing failure surface would strongly suggest flaky shared state rather than a deterministic implementation bug. + +## Evidence + +### 1. Failing tests would pass in isolation + +Running these tests individually would pass: + +- `frontend.worker.db-sync-test/ensure-upload-graph-identity-defaults-e2ee-enabled-test` +- `frontend.worker.db-sync-test/download-graph-by-id-fails-on-incomplete-snapshot-frame-test` + +This would indicate that the underlying code paths can behave correctly when the test environment is clean. + +### 2. Full-namespace execution would produce a different failure + +Running the whole namespace would fail in `pull-ok-out-of-order-stale-response-is-ignored-test` instead of the two failures seen in the full suite. + +This would indicate order dependence or leaked global state between tests in the same namespace. + +### 3. Failure payloads would match leaked state + +`ensure-upload-graph-identity-defaults-e2ee-enabled-test` would unexpectedly receive an existing graph identity with: + +- `:graph-id "c52bf9d4-3a95-4f17-9c8c-7f86eef5f75d"` +- `:graph-e2ee? false` + +That graph id would match a value used by another test in the same file. This would suggest that the later test is reusing state left behind by an earlier test instead of executing its own intended bootstrap path. + +`download-graph-by-id-fails-on-incomplete-snapshot-frame-test` would expect `incomplete-snapshot-frame` but would sometimes receive an unrelated assertion failure: + +- `Assert failed: test-db-sync-repo (some? conn)` + +This would suggest that another asynchronous code path is still interacting with shared worker state while the test is running. + +## Root Cause + +The most likely root cause would be incomplete per-test cleanup of shared global state in: + +- `frontend.worker.state` +- `frontend.worker.sync` + +The highest-risk shared atoms would include: + +- `worker-state/*datascript-conns` +- `worker-state/*client-ops-conns` +- `worker-state/*db-sync-client` +- `worker-state/*db-sync-config` +- `db-sync/*repo->latest-remote-tx` +- `db-sync/*start-inflight-target` + +The test file would also rely heavily on: + +- `async done` +- nested `promesa` chains +- cleanup inside `p/finally` + +That structure would make it possible for a test to signal completion before all outer cleanup has fully restored shared state, especially when asynchronous callbacks or listeners are still active. + +## Proposed Changes + +### 1. Add a per-test fixture for global state reset + +A `:each` fixture in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` would reset the shared worker and db-sync atoms before and after every test. + +The fixture would reset at least: + +- `worker-state/*datascript-conns` +- `worker-state/*client-ops-conns` +- `worker-state/*db-sync-client` +- `worker-state/*db-sync-config` +- `db-sync/*repo->latest-remote-tx` +- `db-sync/*start-inflight-target` + +### 2. Review high-risk async tests + +If fixture-based isolation is not sufficient, the next step would be to tighten async structure in the tests most likely to leak state. + +Priority candidates would include: + +- upload identity tests +- snapshot download tests +- pull/ok ordering tests + +The goal would be to ensure that: + +- cleanup happens after the actual promise chain settles +- `done` is called only after cleanup boundaries are satisfied +- any listener or temporary override is fully restored before the next test starts + +## Why Production Code Is Less Likely To Be At Fault + +The current evidence would not strongly support a production db-sync bug because: + +1. the affected tests would pass when run alone +2. the failing test would change depending on suite shape +3. the observed incorrect values would match data seeded by neighboring tests +4. the expected production code path for incomplete snapshot handling already appears to raise the correct error in isolated execution + +Because of that, modifying production sync logic first would risk masking a test harness problem instead of fixing the actual instability. + +## Verification Plan + +After applying the test isolation changes, verification would include: + +1. `bb dev:test -v frontend.worker.db-sync-test` +2. `bb dev:lint-and-test` + +Success criteria would be: + +- individual failing tests still pass +- the full `frontend.worker.db-sync-test` namespace passes reliably +- the full `bb dev:lint-and-test` command no longer fails on these db-sync tests + +## Risks And Follow-ups + +If failures remain after the fixture reset, the next investigation would focus on: + +- leaked `d/listen!` listeners +- background tasks still running across tests +- `js/fetch` restoration timing +- promise chains whose cleanup is attached at the wrong level +- other shared atoms not yet covered by the fixture + +## Files + +Would modify: + +- `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` + +Would not modify initially: + +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/client_op.cljs` diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index 1b8646bd69..0b1060bb84 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -1,5 +1,5 @@ (ns frontend.worker.db-sync-test - (:require [cljs.test :refer [deftest is testing async]] + (:require [cljs.test :refer [deftest is testing async use-fixtures]] [datascript.core :as d] [frontend.common.crypt :as crypt] [frontend.worker-common.util :as worker-util] @@ -21,6 +21,19 @@ (def ^:private test-repo "test-db-sync-repo") +(defn- reset-db-sync-test-state! + [] + (db-sync/stop!) + (reset! worker-state/*datascript-conns nil) + (reset! worker-state/*client-ops-conns nil) + (reset! worker-state/*db-sync-client nil) + (reset! worker-state/*db-sync-config {:ws-url nil}) + (reset! db-sync/*repo->latest-remote-tx {}) + (reset! db-sync/*start-inflight-target nil)) + +(use-fixtures :each {:before reset-db-sync-test-state! + :after reset-db-sync-test-state!}) + (defn- with-datascript-conns [db-conn ops-conn f] (let [db-prev @worker-state/*datascript-conns @@ -278,15 +291,15 @@ :last-sync-error (atom {:code :previous-error}) :online-users (atom []) :ws-state (atom :open)}] - (with-datascript-conns conn client-ops-conn - (fn [] - (client-op/update-local-tx test-repo 16) - (-> (p/let [_ (#'db-sync/handle-message! test-repo client raw-message)] + (-> (with-datascript-conns conn client-ops-conn + (fn [] + (client-op/update-local-tx test-repo 16) + (p/let [_ (#'db-sync/handle-message! test-repo client raw-message)] (is (= 18 (client-op/get-local-tx test-repo))) - (is (nil? @(:last-sync-error client)))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally done)))))))) + (is (nil? @(:last-sync-error client)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))))) (deftest pull-ok-failure-records-last-sync-error-test (testing "pull/ok failures should surface structured runtime error state" @@ -340,18 +353,18 @@ :online-users (atom []) :ws-state (atom :open)}] (reset! db-sync/*repo->latest-remote-tx {}) - (with-datascript-conns conn client-ops-conn - (fn [] - (-> (p/with-redefs [sync-crypt/graph-e2ee? (fn [_repo] false)] + (-> (with-datascript-conns conn client-ops-conn + (fn [] + (p/with-redefs [sync-crypt/graph-e2ee? (fn [_repo] false)] (p/let [_ (client-op/update-local-tx test-repo 0) _ (#'db-sync/handle-message! test-repo client raw-new) _ (#'db-sync/handle-message! test-repo client raw-stale) parent' (d/entity @conn parent-id)] (is (= "remote-new-title" (:block/title parent'))) - (is (= 2 (client-op/get-local-tx test-repo))))) - (p/finally (fn [] - (reset! db-sync/*repo->latest-remote-tx latest-prev) - (done)))))))))) + (is (= 2 (client-op/get-local-tx test-repo))))))) + (p/finally (fn [] + (reset! db-sync/*repo->latest-remote-tx latest-prev))) + (p/finally done)))))) (deftest pull-ok-batched-txs-preserve-tempid-boundaries-test (testing "pull/ok applies tx batches without cross-tx tempid collisions" @@ -388,17 +401,17 @@ :inflight (atom []) :online-users (atom []) :ws-state (atom :open)}] - (with-datascript-conns conn client-ops-conn - (fn [] - (reset! db-sync/*repo->latest-remote-tx {}) - (-> (p/let [_ (client-op/update-local-tx test-repo 0) + (-> (with-datascript-conns conn client-ops-conn + (fn [] + (reset! db-sync/*repo->latest-remote-tx {}) + (p/let [_ (client-op/update-local-tx test-repo 0) _ (#'db-sync/handle-message! test-repo client raw-message)] ;; TODO(db-sync): re-enable after batched pull tempid-boundary behavior is stabilized. #_(is (= "remote-a" (:block/title (d/entity @conn [:block/uuid block-uuid-a])))) - #_(is (= "remote-b" (:block/title (d/entity @conn [:block/uuid block-uuid-b]))))) - (p/finally (fn [] - (reset! db-sync/*repo->latest-remote-tx latest-prev) - (done)))))))))) + #_(is (= "remote-b" (:block/title (d/entity @conn [:block/uuid block-uuid-b]))))))) + (p/finally (fn [] + (reset! db-sync/*repo->latest-remote-tx latest-prev))) + (p/finally done)))))) (deftest get-graph-id-falls-back-to-client-op-graph-uuid-test (let [conn (d/create-conn {}) @@ -417,16 +430,16 @@ (set! db-sync/list-remote-graphs! (fn [] (p/resolved [{:graph-name test-repo :graph-id "remote-graph-id"}]))) - (with-datascript-conns conn client-ops-conn - (fn [] - (-> (p/let [graph-id (#'db-sync/ (with-datascript-conns conn client-ops-conn + (fn [] + (p/let [graph-id (#'db-sync/ (p/let [result (#'db-sync/ (with-worker-conns {test-repo conn} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] + (p/let [result (#'db-sync/ (ldb/get-graph-rtc-uuid @conn) str))) (is (= [{:url "https://example.com/graphs" :method "GET"} {:url "https://example.com/graphs" :method "POST"}] - @fetch-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (aset js/globalThis "self" self-prev) - (set! js/fetch fetch-prev) - (reset! worker-state/*db-sync-config config-prev) - (done))))))))))) + @fetch-calls)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (aset js/globalThis "self" self-prev) + (set! js/fetch fetch-prev) + (reset! worker-state/*db-sync-config config-prev))) + (p/finally done)))))) (deftest ensure-upload-graph-identity-reuses-existing-remote-graph-when-local-graph-id-missing-test (testing "first upload bootstrap reuses a same-name remote graph instead of creating a duplicate" @@ -1388,9 +1401,9 @@ :else (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) - (with-worker-conns {test-repo conn} {test-repo client-ops-conn} - (fn [] - (-> (p/let [result (#'db-sync/ (with-worker-conns {test-repo conn} {test-repo client-ops-conn} + (fn [] + (p/let [result (#'db-sync/ (ldb/get-graph-rtc-uuid @conn) str))) (is (= [{:url "https://example.com/graphs" :method "GET"}] - @fetch-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally (fn [] - (aset js/globalThis "self" self-prev) - (set! js/fetch fetch-prev) - (reset! worker-state/*db-sync-config config-prev) - (done)))))))))) + @fetch-calls))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (aset js/globalThis "self" self-prev) + (set! js/fetch fetch-prev) + (reset! worker-state/*db-sync-config config-prev))) + (p/finally done)))))) (deftest ensure-upload-graph-identity-propagates-create-graph-failures-test (testing "first upload bootstrap rejects when remote graph creation fails" @@ -1431,19 +1444,19 @@ :else (js/Promise.reject (js/Error. (str "unexpected fetch url: " method " " url))))))) - (with-worker-conns {} {test-repo client-ops-conn} - (fn [] - (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] - (-> (p/let [_ (#'db-sync/ (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body)] + (p/let [_ (#'db-sync/ (p/let [result (#'db-sync/ (with-worker-conns {} {test-repo client-ops-conn} + (fn [] + (with-redefs [db-sync/coerce-http-request (fn [_schema-key body] body) + sync-crypt/graph-e2ee? (fn [_repo] nil)] + (p/let [result (#'db-sync/ (p/let [first-result first-result-p - second-result second-result-p] - (is (= fallback-graph-id (:graph-id first-result))) - (is (= fallback-graph-id (:graph-id second-result))) - (is (= (:graph-e2ee? first-result) - (:graph-e2ee? second-result))) - (is (= fallback-graph-id - (some-> (ldb/get-graph-rtc-uuid @conn) str))) - (is (= 0 @list-calls))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally done)))))))))) + list-calls (atom 0) + result-p (with-worker-conns {test-repo conn} {test-repo client-ops-conn} + (fn [] + (with-redefs [sync-crypt/graph-e2ee? (fn [_repo] nil) + db-sync/get-graph-id (fn [_repo] fallback-graph-id) + db-sync/list-remote-graphs! (fn [] + (swap! list-calls inc) + (p/resolved []))] + (is (nil? (client-op/get-graph-uuid test-repo))) + (is (nil? (ldb/get-graph-rtc-uuid @conn))) + (let [first-result-p (#'db-sync/ (ldb/get-graph-rtc-uuid @conn) str))) + (is (= 0 @list-calls)))))))] + (-> result-p + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))))) (deftest ^:long rehydrate-large-title-test (testing "rehydrate fills empty title from object storage" From 674e80887ebc3fae362c0e1aa396439461b059a3 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 8 Mar 2026 21:43:33 +0800 Subject: [PATCH 119/375] 055-logseq-cli-login-logout.md --- .../src/logseq/common/cognito_config.cljs | 41 ++ .../055-logseq-cli-login-logout.md | 317 ++++++++++++ docs/cli/logseq-cli.md | 49 +- shadow-cljs.edn | 7 +- src/main/frontend/config.cljs | 21 +- src/main/logseq/cli/auth.cljs | 466 ++++++++++++++++++ src/main/logseq/cli/command/auth.cljs | 56 +++ src/main/logseq/cli/command/core.cljs | 4 +- src/main/logseq/cli/command/sync.cljs | 70 ++- src/main/logseq/cli/commands.cljs | 9 +- src/main/logseq/cli/config.cljs | 22 +- src/main/logseq/cli/format.cljs | 40 +- src/test/logseq/cli/auth_test.cljs | 102 ++++ src/test/logseq/cli/command/auth_test.cljs | 215 ++++++++ src/test/logseq/cli/command/sync_test.cljs | 137 +++-- src/test/logseq/cli/commands_test.cljs | 26 + src/test/logseq/cli/config_test.cljs | 20 +- src/test/logseq/cli/format_test.cljs | 52 +- src/test/logseq/cli/integration_test.cljs | 145 +++++- .../logseq/common/cognito_config_test.cljs | 15 + 20 files changed, 1702 insertions(+), 112 deletions(-) create mode 100644 deps/common/src/logseq/common/cognito_config.cljs create mode 100644 docs/agent-guide/055-logseq-cli-login-logout.md create mode 100644 src/main/logseq/cli/auth.cljs create mode 100644 src/main/logseq/cli/command/auth.cljs create mode 100644 src/test/logseq/cli/auth_test.cljs create mode 100644 src/test/logseq/cli/command/auth_test.cljs create mode 100644 src/test/logseq/common/cognito_config_test.cljs diff --git a/deps/common/src/logseq/common/cognito_config.cljs b/deps/common/src/logseq/common/cognito_config.cljs new file mode 100644 index 0000000000..71824e50d5 --- /dev/null +++ b/deps/common/src/logseq/common/cognito_config.cljs @@ -0,0 +1,41 @@ +(ns logseq.common.cognito-config + "Shared Cognito configuration for frontend and CLI-safe consumers." + (:require [clojure.string :as string])) + +(goog-define ENABLE-FILE-SYNC-PRODUCTION false) + +(def ^:private prod-login-url + "https://logseq-prod.auth.us-east-1.amazoncognito.com/login?client_id=3c7np6bjtb4r1k1bi9i049ops5&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") + +(def ^:private test-login-url + "https://logseq-test2.auth.us-east-2.amazoncognito.com/login?client_id=3ji1a0059hspovjq5fhed3uil8&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") + +(def LOGIN-URL + (if ENABLE-FILE-SYNC-PRODUCTION + prod-login-url + test-login-url)) + +(def COGNITO-CLIENT-ID + (or (some-> (js/URL. LOGIN-URL) + .-searchParams + (.get "client_id")) + (if ENABLE-FILE-SYNC-PRODUCTION + "69cs1lgme7p8kbgld8n5kseii6" + "1qi1uijg8b6ra70nejvbptis0q"))) + +(def CLI-COGNITO-CLIENT-ID + (if ENABLE-FILE-SYNC-PRODUCTION + "69cs1lgme7p8kbgld8n5kseii6" + "1qi1uijg8b6ra70nejvbptis0q")) + +(def OAUTH-DOMAIN + (if ENABLE-FILE-SYNC-PRODUCTION + "logseq-prod.auth.us-east-1.amazoncognito.com" + "logseq-test2.auth.us-east-2.amazoncognito.com")) + +(def OAUTH-SCOPE + (or (some-> (js/URL. LOGIN-URL) + .-searchParams + (.get "scope") + (string/replace #"\+" " ")) + "email openid phone")) diff --git a/docs/agent-guide/055-logseq-cli-login-logout.md b/docs/agent-guide/055-logseq-cli-login-logout.md new file mode 100644 index 0000000000..978976febe --- /dev/null +++ b/docs/agent-guide/055-logseq-cli-login-logout.md @@ -0,0 +1,317 @@ +# Logseq CLI Login and Logout Implementation Plan + +Goal: Add `logseq login` and `logseq logout`, persist Cognito auth in `/Users/rcmerci/logseq/auth.json`, and remove `:auth-token` persistence from `/Users/rcmerci/logseq/cli.edn`. + +Architecture: The CLI will get a dedicated auth module that owns loopback OAuth login, token persistence, token refresh, and logout file cleanup. +Architecture: Sync commands will continue to pass `:auth-token` to db-sync at runtime, but that token will be resolved from `auth.json` instead of being edited through `sync config set|get|unset`. +Architecture: The flow will reuse existing Cognito constants and token refresh semantics from the frontend user code, while following the ECA browser plus localhost callback pattern for headless login. + +Tech Stack: ClojureScript, babashka.cli, Node.js HTTP server APIs, Cognito Hosted UI OAuth, Promesa, JSON file persistence. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`, `docs/agent-guide/048-sync-download-start-reliability.md`, `docs/agent-guide/051-logseq-cli-sync-upload-fix.md`, and `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +## Problem statement + +The current CLI expects headless sync authentication to be provided as `:auth-token` inside `/Users/rcmerci/logseq/cli.edn`. + +That approach is awkward for users, leaks auth concerns into general CLI config, and does not provide a first-class login or logout flow. + +The current sync code path in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` still reads `:auth-token` directly from resolved config and writes it through `sync config set|get|unset`. + +The db-sync worker then consumes that runtime token through `worker-state/*db-sync-config` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +Logseq user management already uses AWS Cognito in the frontend app, with Cognito constants in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` and refresh-token logic in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs`. + +The ECA repository shows the closest CLI-friendly reference implementation for this feature, because it opens a browser, listens on localhost for the callback, exchanges the authorization code for tokens, and persists auth state locally. + +This plan keeps the existing db-sync runtime contract stable by continuing to inject `:auth-token` into the worker config map, while changing only how the CLI obtains and persists that token. + +I will use @planning-documents for naming, @writing-plans for task granularity, @test-driven-development for implementation order, and the current `logseq-cli` plus `db-worker-node` implementation as the baseline architecture. + +## Testing Plan + +I will add unit tests for auth file path resolution, JSON persistence, token refresh, and auth file deletion before adding implementation behavior. + +I will add command parser tests for the new `login` and `logout` commands before wiring them into the CLI. + +I will add execution tests for `login` and `logout` that verify browser launch, callback handling, Cognito token exchange, auth file writes, and logout cleanup. + +I will add sync command regression tests that fail first and verify `sync config set|get|unset` no longer accept `auth-token`. + +I will add config resolution tests that fail first and verify `resolve-config` no longer loads or persists `:auth-token` from `cli.edn`, while a new auth resolver loads the effective token from `auth.json`. + +I will add worker-facing tests that fail first and verify CLI sync runtime still injects an `:auth-token` into `worker-state/*db-sync-config` when `auth.json` exists. + +I will add integration tests that fail first and cover `login`, `logout`, and one authenticated sync command using a stubbed Cognito token exchange. + +I will run targeted tests after each slice, and I will finish with `bb dev:lint-and-test`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Proposed CLI surface + +| Command | Purpose | Persistence effect | Notes | +|---|---|---|---| +| `logseq login` | Authenticate the current machine against Logseq cloud. | Creates or updates `/Users/rcmerci/logseq/auth.json`. | Opens browser and completes a localhost callback flow. | +| `logseq logout` | Remove locally persisted cloud auth for the CLI. | Deletes or clears `/Users/rcmerci/logseq/auth.json`. | Idempotent when the file does not exist. | +| `logseq sync config set ws-url|http-base|e2ee-password ` | Persist non-auth sync config. | Updates `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | +| `logseq sync config get ws-url|http-base|e2ee-password` | Read non-auth sync config. | Reads `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | +| `logseq sync config unset ws-url|http-base|e2ee-password` | Remove non-auth sync config. | Updates `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | + +The runtime db-sync config map will still contain `:auth-token` when the CLI invokes worker methods. + +Only the persistence source changes from `cli.edn` to `auth.json`. + +## Auth file design + +The new auth file will live at `/Users/rcmerci/logseq/auth.json` by default. + +The implementation should expose an internal override path for tests, but it should not add a new public CLI flag unless later required. + +The file format should be JSON rather than EDN so it is easy to inspect and delete manually. + +The file should contain enough information to refresh expired tokens without asking the user to log in again. + +A good starting shape is the following. + +```json +{ + "provider": "cognito", + "id-token": "", + "access-token": "", + "refresh-token": "", + "expires-at": 1735689600000, + "sub": "", + "email": "", + "updated-at": 1735686000000 +} +``` + +`id-token` should remain the value injected into db-sync as runtime `:auth-token`, because current CLI and worker behavior already assumes the sync token is the same value as `state/get-auth-id-token` on desktop. + +`refresh-token` is required so the CLI can refresh auth non-interactively before sync commands. + +`access-token` can be stored for parity with the existing frontend token model, even if the first CLI release does not consume it directly. + +The file should be written with restrictive permissions on Unix when feasible, and tests should verify the implementation does not fail on platforms where chmod semantics differ. + +## OAuth flow design + +The login flow should follow the ECA pattern more than the current browser app pattern. + +The CLI should start a temporary localhost callback server, generate a PKCE verifier and challenge, build a Cognito Hosted UI authorization URL, and then open the browser. + +The callback should validate `state`, read the authorization `code`, exchange it against Cognito `/oauth2/token`, persist the resulting tokens, and print a concise success result. + +A suggested flow is shown below. + +```text +logseq login + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs + -> start localhost callback server on an ephemeral port + -> build Cognito authorize URL with PKCE and redirect_uri http://127.0.0.1:/auth/callback + -> open browser with system launcher, or print URL fallback + -> receive code on callback server + -> POST code exchange to https:///oauth2/token + -> persist /Users/rcmerci/logseq/auth.json + -> future sync commands refresh token if needed and inject runtime :auth-token +``` + +The callback server should prefer an ephemeral port rather than a fixed port, because that avoids collisions during local development and test runs. + +The login result should include non-sensitive metadata such as email, subject, auth file path, and whether the token was freshly created or refreshed. + +The CLI should not print raw tokens in human output. + +## Refresh and runtime token resolution + +The current worker code does not refresh tokens in CLI-owned node mode. + +`frontend.worker.sync//oauth2/token` using `grant_type=refresh_token` and `client_id`. + +`/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` should stop reading `:auth-token` from resolved config and instead call the new auth helper when building the sync runtime config sent through `:thread-api/set-db-sync-config`. + +If no valid auth file exists, authenticated sync commands should return a dedicated error with a hint such as `Run logseq login first.`. + +## Implementation plan + +### Phase 1. Add failing tests for auth file helpers. + +1. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/auth_test.cljs`. +2. Add a failing test that the default auth path resolves to `/Users/rcmerci/logseq/auth.json`. +3. Add a failing test that writing auth data creates the parent directory when missing. +4. Add a failing test that reading a missing auth file returns `nil` instead of throwing. +5. Add a failing test that deleting auth data is idempotent when the file does not exist. +6. Add a failing test that expired `id-token` plus valid `refresh-token` triggers refresh and persists updated JSON. +7. Add a failing test that malformed JSON returns a stable `invalid-auth-file` error code. +8. Run `bb dev:test -v logseq.cli.auth-test` and confirm only auth helper tests fail. + +### Phase 2. Add failing parser and help tests for `login` and `logout`. + +9. Update `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with a failing assertion that top-level help lists `login` and `logout`. +10. Add a failing assertion that `logseq login --help` and `logseq logout --help` produce command-specific help. +11. Add a failing assertion that `sync config set|get|unset` help no longer mentions `auth-token`. +12. Run `bb dev:test -v logseq.cli.commands-test` and confirm the failures reflect missing auth command wiring and old sync config text. + +### Phase 3. Add failing command execution tests for auth commands. + +13. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/auth_test.cljs`. +14. Add a failing test that `login` starts a callback server and opens the browser with a Cognito authorize URL. +15. Add a failing test that `login` validates `state` before exchanging the authorization code. +16. Add a failing test that a successful code exchange writes `/Users/rcmerci/logseq/auth.json` through the auth persistence helper. +17. Add a failing test that `logout` removes the auth file and returns success when the file existed. +18. Add a failing test that `logout` still succeeds when the auth file was already absent. +19. Add a failing test that `login` returns a timeout error when no browser callback arrives. +20. Run `bb dev:test -v logseq.cli.command.auth-test` and confirm the tests fail for missing implementation only. + +### Phase 4. Implement auth helpers and OAuth plumbing. + +21. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs` for auth path resolution, JSON read and write helpers, token expiry checks, refresh logic, and auth file deletion. +22. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs` for command entries, action builders, and command execution. +23. Implement PKCE helpers and loopback callback server support inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs`, unless the file becomes too large, in which case split transport helpers into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth_oauth.cljs`. +24. Build the Cognito authorize URL using constants from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` so the CLI and app share the same cloud environment. +25. Implement the token exchange POST against `https:///oauth2/token` and map the response into the `auth.json` shape. +26. Implement refresh-token exchange using the same Cognito domain and client id. +27. Add best-effort browser launching for macOS and Linux, and always print the URL so the flow remains usable when auto-open fails. +28. Run `bb dev:test -v logseq.cli.auth-test` and `bb dev:test -v logseq.cli.command.auth-test` until green. + +### Phase 5. Wire `login` and `logout` into the CLI parser and help output. + +29. Register `auth-command/entries` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +30. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` so top-level summaries include `login` and `logout` in a dedicated auth section or the existing management section. +31. Extend action building and execute dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to route `:login` and `:logout`. +32. Re-run `bb dev:test -v logseq.cli.commands-test` until the new parser and help output tests pass. + +### Phase 6. Remove `:auth-token` persistence from `cli.edn`. + +33. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs` that `resolve-config` no longer returns `:auth-token` from file config. +34. Add a failing test that `update-config!` strips `:auth-token` from updates instead of persisting it. +35. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` so file-backed config excludes `:auth-token` while still preserving `ws-url`, `http-base`, `e2ee-password`, graph selection, and output settings. +36. Update any config-related tests that currently expect `:auth-token` round-tripping through `cli.edn`. +37. Run `bb dev:test -v logseq.cli.config-test` until green. + +### Phase 7. Update sync commands to resolve auth from `auth.json`. + +38. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync config set|get|unset auth-token` is rejected as an unknown key. +39. Add a failing test that sync execution uses the auth helper to resolve a runtime `:auth-token` before invoking `:thread-api/set-db-sync-config`. +40. Add a failing test that missing auth file returns a `missing-auth` style error for authenticated sync operations such as `sync remote-graphs`. +41. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so `config-key-map` removes `auth-token`. +42. Update sync execution to call the auth helper and merge the effective runtime token into the in-memory sync config map. +43. Update human-readable hints that currently say `Set sync config keys (ws-url/http-base/auth-token)` so they instead mention login plus the remaining sync config keys. +44. Run `bb dev:test -v logseq.cli.command.sync-test` until green. + +### Phase 8. Keep worker behavior stable while clarifying CLI ownership of auth. + +45. Add a failing regression test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that CLI node mode still prefers the runtime `:auth-token` provided by sync config over renderer state. +46. Decide whether an additional worker test is needed to prove no worker-side changes are required beyond existing runtime behavior. +47. Make only the smallest worker change necessary, and prefer no production worker changes if CLI runtime injection remains sufficient. +48. Run `bb dev:test -v frontend.worker.sync.crypt-test` and any related worker namespace tests. + +### Phase 9. Update output formatting and docs. + +49. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `login` and `logout`. +50. Remove or rewrite any format tests that currently mention `sync config get auth-token` redaction. +51. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` so auth command output reports file path, email, and status without printing tokens. +52. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document `login`, `logout`, `auth.json`, and the reduced `sync config` key set. +53. Add one short troubleshooting section telling users to delete `/Users/rcmerci/logseq/auth.json` or run `logseq logout` when local auth becomes invalid. +54. Run `bb dev:test -v logseq.cli.format-test` after formatter updates. + +### Phase 10. Add integration coverage and perform final verification. + +55. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that simulates a successful `login` callback and verifies `auth.json` contents. +56. Add a failing integration test that `logout` removes the auth file. +57. Add a failing integration test that an authenticated sync command reads `auth.json`, refreshes if needed, and sends runtime `:auth-token` to the worker. +58. Stub browser opening and Cognito HTTP responses so the integration tests remain deterministic and offline. +59. Run `bb dev:test -v logseq.cli.integration-test` until the new coverage is green. +60. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and confirm exit code `0`. + +## File map + +| Area | Files to update or add | Reason | +|---|---|---| +| Auth persistence and OAuth flow | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs`, optionally `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth_oauth.cljs` | Own `auth.json`, login callback server, PKCE, refresh, and logout deletion. | +| CLI command wiring | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs` | Expose `login` and `logout` and update help text. | +| Sync runtime auth | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` | Replace file-backed `:auth-token` lookup with auth helper resolution. | +| Config cleanup | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` | Stop round-tripping `:auth-token` in `cli.edn`. | +| Cloud constants reuse | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs` | Reference existing Cognito endpoints and refresh semantics. | +| Tests | `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/auth_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/auth_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`, and possibly `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` | Cover behavior before implementation. | +| User-facing docs | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` | Explain the new auth workflow and removal of `auth-token` from sync config commands. | + +## Edge cases and failure handling + +`logseq login` must fail cleanly when the localhost callback port cannot be opened. + +`logseq login` must reject callbacks with mismatched `state` or missing `code`. + +`logseq login` must surface Cognito code-exchange failures without writing a partial auth file. + +`logseq logout` must succeed even when `/Users/rcmerci/logseq/auth.json` is already missing. + +Malformed or partially written `auth.json` must produce a deterministic error code and an actionable hint rather than a generic JSON parse exception. + +Expired `id-token` with a valid `refresh-token` should auto-refresh before sync commands instead of forcing an immediate re-login. + +Expired `id-token` with an invalid or missing `refresh-token` should fail with a hint to run `logseq login`. + +`sync start`, `sync remote-graphs`, `sync upload`, `sync download`, `sync ensure-keys`, and `sync grant-access` should all use the same auth resolution path so behavior is consistent. + +The CLI must not print raw JWTs or refresh tokens in human output, logs, or test snapshots. + +The implementation should preserve the current worker contract by continuing to pass `:auth-token` in runtime sync config, because changing the worker key name would expand scope without user benefit. + +## Verification commands + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.auth-test` | Auth helper tests pass. | +| `bb dev:test -v logseq.cli.command.auth-test` | Login and logout command tests pass. | +| `bb dev:test -v logseq.cli.commands-test` | Help output includes `login` and `logout`, and sync config no longer mentions `auth-token`. | +| `bb dev:test -v logseq.cli.config-test` | `cli.edn` no longer persists `:auth-token`. | +| `bb dev:test -v logseq.cli.command.sync-test` | Sync command tests pass with auth resolved from `auth.json`. | +| `bb dev:test -v logseq.cli.integration-test` | End-to-end auth flow tests pass with stubs. | +| `bb dev:lint-and-test` | Full repo lint and test suite passes with exit code `0`. | +| `node ./dist/logseq.js login` | Opens browser or prints login URL, then writes `/Users/rcmerci/logseq/auth.json` after callback. | +| `node ./dist/logseq.js logout` | Removes local auth state and reports success. | +| `node ./dist/logseq.js sync remote-graphs --output json` | Reads auth from `auth.json` and returns remote graphs without requiring `sync config set auth-token`. | + +## Testing Details + +The tests focus on externally observable behavior, including CLI help text, browser launch requests, callback validation, auth file contents, refresh-on-expiry behavior, sync runtime payloads, and user-facing error hints. + +The tests should not assert private helper structure unless that structure is itself the behavior contract, such as the persisted JSON keys or the generated Cognito callback URL. + +## Implementation Details + +- Keep `auth-token` as an in-memory db-sync runtime key, but remove it from persistent `cli.edn` config and from `sync config` command parsing. +- Add a dedicated CLI auth module instead of scattering login and refresh logic across `sync.cljs` and `config.cljs`. +- Reuse Cognito constants from `frontend.config` so the CLI always targets the same environment as the app. +- Reuse the refresh grant semantics from `frontend.handler.user.cljs` instead of inventing a second refresh protocol. +- Prefer an ephemeral localhost callback port and PKCE-based authorization code flow. +- Always print a copyable login URL even when automatic browser opening succeeds. +- Make `logout` local and idempotent first, and treat remote Cognito session invalidation as optional future scope. +- Keep the worker contract stable and prefer solving expiration entirely in the CLI auth module. +- Add an internal test-only auth path override rather than a new public CLI flag. +- Update user-facing docs and error hints so `Run logseq login first.` becomes the primary recovery path. + +## Decision + +The implementation will use a loopback redirect URI such as `http://127.0.0.1:/auth/callback`. + +If the current Cognito app client does not yet allow that redirect URI pattern, updating the Cognito app-client configuration is part of the implementation prerequisite rather than a reason to change the CLI design. + +`logout` will only clear `/Users/rcmerci/logseq/auth.json` in the first release. + +Best-effort browser sign-out against the Cognito Hosted UI logout endpoint is explicitly out of scope for this iteration. + +`auth.json` will persist `id-token`, `access-token`, and `refresh-token`, plus non-sensitive metadata such as `expires-at`, `sub`, `email`, and `updated-at`. + +This keeps the first CLI implementation aligned with the existing frontend token model while still using `id-token` as the runtime `:auth-token` injected into db-sync. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index c4ae857c39..b900b63274 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -56,13 +56,33 @@ Supported keys include: - `:data-dir` - `:timeout-ms` - `:output-format` (use `:json` or `:edn` for scripting) +- sync config persisted via `sync config set|get|unset`: `:ws-url`, `:http-base`, `:e2ee-password` + +`cli.edn` no longer persists cloud auth tokens. CLI login state is stored separately in `~/logseq/auth.json`. CLI flags take precedence over environment variables, which take precedence over the config file. +## Authentication + +Use `logseq login` to authenticate the current machine with Logseq cloud. + +- `logseq login` starts a temporary callback server at `http://localhost:8765/auth/callback`, opens a browser to the Logseq Cognito Hosted UI, exchanges the returned authorization code, and writes `~/logseq/auth.json`. +- `logseq logout` removes `~/logseq/auth.json`, opens a browser to the Cognito Hosted UI logout endpoint, and completes the browser logout flow at `http://localhost:8765/logout-complete`. +- Sync commands still pass an in-memory runtime `:auth-token` to db-sync, but that token is now resolved from `auth.json` instead of `cli.edn`. + +Default auth file: `~/logseq/auth.json` + +Auth file contents include the persisted Cognito `id-token`, `access-token`, `refresh-token`, `expires-at`, `sub`, `email`, and `updated-at` values needed for headless refresh. + Verbose logging: - `--verbose` enables structured debug logs to stderr for CLI option parsing and db-worker-node API calls. - stdout remains reserved for command output; large payloads are truncated in debug previews. +Timeouts: +- `--timeout-ms` continues to control request timeout behavior for CLI transport. +- Login callback timeout is controlled separately by `:login-timeout-ms` / `LOGSEQ_CLI_LOGIN_TIMEOUT_MS` and defaults to 5 minutes. +- Logout callback timeout is controlled separately by `:logout-timeout-ms` / `LOGSEQ_CLI_LOGOUT_TIMEOUT_MS` and defaults to 2 minutes. + ## Commands Graph commands: @@ -85,6 +105,10 @@ Server commands: - `server restart --graph ` - restart db-worker-node for a graph - `doctor [--dev-script]` - run runtime diagnostics for `db-worker-node.js`, `data-dir` permissions, and running server readiness (`--dev-script` checks `static/db-worker-node.js` explicitly) +Auth commands: +- `login` - authenticate this machine and create/update `~/logseq/auth.json` +- `logout` - remove persisted CLI auth from `~/logseq/auth.json` + Server ownership behavior: - `server stop` and `server restart` can return `server-owned-by-other` if the daemon was started by another owner source. - `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. @@ -96,12 +120,12 @@ Sync commands: - `sync stop --graph ` - stop db-sync client on a graph daemon - `sync upload --graph ` - upload local graph snapshot to remote - `sync download --graph ` - download remote graph `` into a same-name local graph directory -- `sync remote-graphs [--graph ]` - list remote graphs visible to the current auth context +- `sync remote-graphs [--graph ]` - list remote graphs visible to the current login context - `sync ensure-keys [--graph ]` - ensure user RSA keys for sync/e2ee - `sync grant-access --graph --graph-id --email ` - grant encrypted graph access to a user -- `sync config set [--graph ] ws-url|http-base|auth-token|e2ee-password ` - set db-sync runtime config key -- `sync config get [--graph ] ws-url|http-base|auth-token|e2ee-password` - get db-sync runtime config key -- `sync config unset [--graph ] ws-url|http-base|auth-token|e2ee-password` - remove db-sync runtime config key +- `sync config set [--graph ] ws-url|http-base|e2ee-password ` - set non-auth db-sync runtime config key +- `sync config get [--graph ] ws-url|http-base|e2ee-password` - get non-auth db-sync runtime config key +- `sync config unset [--graph ] ws-url|http-base|e2ee-password` - remove non-auth db-sync runtime config key Sync upload behavior: - `sync upload` requires `--graph `. @@ -110,9 +134,9 @@ Sync upload behavior: - If the local graph does not have a stored remote `graph-id`, upload first lists visible remote graphs and reuses an exact same-name match when one exists. - If no same-name remote graph exists, upload creates a new remote graph and persists the returned remote metadata locally before snapshot transfer. - Successful upload persists graph identity metadata locally in both client-op state and graph KV (`logseq.kv/graph-uuid`, `logseq.kv/graph-remote?`, and `logseq.kv/graph-rtc-e2ee?`) so CLI and web upload/bootstrap flows stay aligned. -- Fresh uploads default to encrypted remote graph creation unless local sync metadata explicitly marks the graph as non-e2ee. In headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before uploading encrypted graphs. -- `sync upload` returns a real error instead of false success when auth, remote graph bootstrap, or snapshot upload fails. -- Common upload failures include missing/invalid `auth-token`, missing `http-base`, remote graph creation failure, snapshot upload failure, and local DB/worker startup failure. +- Fresh uploads default to encrypted remote graph creation unless local sync metadata explicitly marks the graph as non-e2ee. In headless CLI mode, run `logseq login` first and set `e2ee-password` via `sync config set` (or in `--config`) before uploading encrypted graphs. +- `sync upload` returns a real error instead of false success when login state, remote graph bootstrap, or snapshot upload fails. +- Common upload failures include missing/invalid CLI login state, missing `http-base`, remote graph creation failure, snapshot upload failure, and local DB/worker startup failure. - Troubleshooting: after a successful upload, run `graph info --graph --output json` and confirm `data.kv.logseq.kv/graph-uuid` is present. If it is missing, rerun `sync upload` for the same graph to trigger identity backfill. Sync download behavior: @@ -121,12 +145,13 @@ Sync download behavior: - If no remote graph with that name exists, the CLI returns `remote-graph-not-found`. - `sync download` starts `db-worker-node` in create-empty mode so local startup does not write `db-initial-data` before snapshot import. - If the target graph DB is not empty at download time, the CLI returns `graph-db-not-empty` and aborts before import. -- For e2ee remote graphs in headless CLI mode, set `e2ee-password` via `sync config set` (or in `--config`) before download. +- For e2ee remote graphs in headless CLI mode, run `logseq login` first and set `e2ee-password` via `sync config set` (or in `--config`) before download. Sync config persistence: -- `sync config set/unset` writes to the CLI config file selected by `--config`. +- `sync config set/unset` writes non-auth sync config to the CLI config file selected by `--config`. - If `--config` is not provided, the default config path is `~/logseq/cli.edn`. - `sync config get` reads from that same config source. +- Cloud auth is persisted separately in `~/logseq/auth.json`. Inspect and edit commands: - `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages @@ -222,9 +247,14 @@ id7 │ └── b7 id8 └── b8 ``` +Troubleshooting: +- If authenticated sync commands fail with missing or invalid local auth, run `logseq logout` and then `logseq login` again. +- You can also manually remove `~/logseq/auth.json` and repeat `logseq login`. + Examples: ```bash +node ./dist/logseq.js login node ./dist/logseq.js graph create --graph demo node ./dist/logseq.js graph export --type edn --file /tmp/demo.edn --graph demo node ./dist/logseq.js graph import --type edn --input /tmp/demo.edn --graph demo-import @@ -236,4 +266,5 @@ node ./dist/logseq.js server list node ./dist/logseq.js doctor node ./dist/logseq.js doctor --dev-script node ./dist/logseq.js doctor --output json +node ./dist/logseq.js logout ``` diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 33645efdfc..289a2e2cff 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -42,8 +42,7 @@ frontend.modules.instrumentation.posthog/POSTHOG-TOKEN #shadow/env "LOGSEQ_POSTHOG_TOKEN" frontend.config/ENABLE-PLUGINS #shadow/env ["ENABLE_PLUGINS" :as :bool :default true] ;; Set to switch file sync server to dev, set this to false in `yarn watch` - frontend.config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true] - frontend.config/ENABLE-RTC-SYNC-PRODUCTION #shadow/env ["ENABLE_RTC_SYNC_PRODUCTION" :as :bool :default true] + logseq.common.cognito-config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true] frontend.config/ENABLE-DB-SYNC-LOCAL #shadow/env ["ENABLE_DB_SYNC_LOCAL" :as :bool :default false] frontend.config/REVISION #shadow/env ["LOGSEQ_REVISION" :default "dev"]} ;; set by git-revision-hook @@ -101,6 +100,7 @@ :output-to "static/logseq-cli.js" :main logseq.cli.main/main :build-hooks [(shadow.hooks/logseq-cli-metadata-hook "--long --always --dirty")] + :closure-defines {logseq.common.cognito-config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true]} :compiler-options {:infer-externs :auto :source-map true :externs ["datascript/externs.js" @@ -167,8 +167,7 @@ frontend.modules.instrumentation.posthog/POSTHOG-TOKEN #shadow/env "LOGSEQ_POSTHOG_TOKEN" ;; frontend.config/ENABLE-PLUGINS #shadow/env ["ENABLE_PLUGINS" :as :bool :default true] ;; Set to switch file sync server to dev, set this to false in `yarn watch` - frontend.config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true] - frontend.config/ENABLE-RTC-SYNC-PRODUCTION #shadow/env ["ENABLE_RTC_SYNC_PRODUCTION" :as :bool :default true] + logseq.common.cognito-config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true] frontend.config/ENABLE-DB-SYNC-LOCAL #shadow/env ["ENABLE_DB_SYNC_LOCAL" :as :bool :default false] frontend.config/REVISION #shadow/env ["LOGSEQ_REVISION" :default "dev"]} diff --git a/src/main/frontend/config.cljs b/src/main/frontend/config.cljs index 9396d9c4c3..0539a99b61 100644 --- a/src/main/frontend/config.cljs +++ b/src/main/frontend/config.cljs @@ -5,6 +5,7 @@ [frontend.state :as state] [frontend.util :as util] [goog.crypt.Md5] + [logseq.common.cognito-config :as cognito-config] [logseq.common.config :as common-config] [logseq.common.path :as path] [logseq.db.sqlite.util :as sqlite-util] @@ -19,33 +20,27 @@ (goog-define REVISION "unknown") (defonce revision REVISION) -(goog-define ENABLE-FILE-SYNC-PRODUCTION false) - ;; this is a feature flag to enable the account tab ;; when it launches (when pro plan launches) it should be removed (def ENABLE-SETTINGS-ACCOUNT-TAB false) -(if ENABLE-FILE-SYNC-PRODUCTION - (do (def LOGIN-URL - "https://logseq-prod.auth.us-east-1.amazoncognito.com/login?client_id=3c7np6bjtb4r1k1bi9i049ops5&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") - (def API-DOMAIN "api.logseq.com") +(def LOGIN-URL cognito-config/LOGIN-URL) +(def COGNITO-CLIENT-ID cognito-config/COGNITO-CLIENT-ID) +(def OAUTH-DOMAIN cognito-config/OAUTH-DOMAIN) + +(if cognito-config/ENABLE-FILE-SYNC-PRODUCTION + (do (def API-DOMAIN "api.logseq.com") (def COGNITO-IDP "https://cognito-idp.us-east-1.amazonaws.com/") - (def COGNITO-CLIENT-ID "69cs1lgme7p8kbgld8n5kseii6") (def REGION "us-east-1") (def USER-POOL-ID "us-east-1_dtagLnju8") (def IDENTITY-POOL-ID "us-east-1:d6d3b034-1631-402b-b838-b44513e93ee0") - (def OAUTH-DOMAIN "logseq-prod.auth.us-east-1.amazoncognito.com") (def PUBLISH-API-BASE "https://logseq.io")) - (do (def LOGIN-URL - "https://logseq-test2.auth.us-east-2.amazoncognito.com/login?client_id=3ji1a0059hspovjq5fhed3uil8&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") - (def API-DOMAIN "api-dev.logseq.com") + (do (def API-DOMAIN "api-dev.logseq.com") (def COGNITO-IDP "https://cognito-idp.us-east-2.amazonaws.com/") - (def COGNITO-CLIENT-ID "1qi1uijg8b6ra70nejvbptis0q") (def REGION "us-east-2") (def USER-POOL-ID "us-east-2_kAqZcxIeM") (def IDENTITY-POOL-ID "us-east-2:cc7d2ad3-84d0-4faf-98fe-628f6b52c0a5") - (def OAUTH-DOMAIN "logseq-test2.auth.us-east-2.amazoncognito.com") (def PUBLISH-API-BASE "https://logseq-publish-staging.logseq.workers.dev"))) ;; Enable for local development diff --git a/src/main/logseq/cli/auth.cljs b/src/main/logseq/cli/auth.cljs new file mode 100644 index 0000000000..9c8dd95032 --- /dev/null +++ b/src/main/logseq/cli/auth.cljs @@ -0,0 +1,466 @@ +(ns logseq.cli.auth + "CLI auth helpers for persisted login state." + (:require [clojure.string :as string] + [logseq.cli.transport :as transport] + [logseq.common.cognito-config :as cognito-config] + [promesa.core :as p] + ["child_process" :as child-process] + ["crypto" :as crypto] + ["fs" :as fs] + ["http" :as http] + ["os" :as os] + ["path" :as node-path])) + +(def ^:private default-login-timeout-ms 300000) +(def ^:private default-logout-timeout-ms 120000) +(def ^:private redirect-path "/auth/callback") +(def ^:private logout-complete-path "/logout-complete") +(def ^:private callback-host "localhost") +(def ^:private callback-port 8765) +(def ^:private auth-provider "cognito") +(def ^:private default-scope "email openid phone") +(def ^:private token-endpoint-path "/oauth2/token") +(def ^:private authorize-endpoint-path "/oauth2/authorize") +(def ^:private logout-endpoint-path "/logout") + +(defn default-auth-path + [] + (node-path/join (.homedir os) "logseq" "auth.json")) + +(defn auth-path + [{custom-auth-path :auth-path}] + (or custom-auth-path (default-auth-path))) + +(defn- ensure-auth-dir! + [path] + (let [dir (node-path/dirname path)] + (when (and (seq dir) (not (fs/existsSync dir))) + (.mkdirSync fs dir #js {:recursive true})))) + +(defn- try-chmod! + [path] + (try + (.chmodSync fs path 384) + (catch :default _ + nil))) + +(defn- parse-json + [text] + (js->clj (js/JSON.parse text) :keywordize-keys true)) + +(defn- login-url + [] + (js/URL. cognito-config/LOGIN-URL)) + +(defn- oauth-client-id + [] + cognito-config/CLI-COGNITO-CLIENT-ID) + +(defn- oauth-scope + [] + (or (.get (.-searchParams (login-url)) "scope") + cognito-config/OAUTH-SCOPE + default-scope)) + +(defn- oauth-domain + [] + cognito-config/OAUTH-DOMAIN) + +(defn- logout-complete-uri + [] + (str "http://" callback-host ":" callback-port logout-complete-path)) + +(defn- token-endpoint + [] + (str "https://" (oauth-domain) token-endpoint-path)) + +(defn- authorize-endpoint + [] + (str "https://" (oauth-domain) authorize-endpoint-path)) + +(defn- logout-endpoint + [] + (str "https://" (oauth-domain) logout-endpoint-path)) + +(defn- build-logout-url + [] + (let [params (doto (js/URLSearchParams.) + (.set "client_id" (oauth-client-id)) + (.set "logout_uri" (logout-complete-uri)))] + (str (logout-endpoint) "?" (.toString params)))) + +(defn- parse-jwt + [jwt] + (when (seq jwt) + (try + (let [parts (string/split jwt #"\.") + payload (second parts)] + (when (seq payload) + (-> (js/Buffer.from payload "base64url") + (.toString "utf8") + parse-json))) + (catch :default _ + nil)))) + +(defn write-auth-file! + [opts auth-data] + (let [path (auth-path opts) + payload (js/JSON.stringify (clj->js auth-data) nil 2)] + (ensure-auth-dir! path) + (.writeFileSync fs path payload "utf8") + (try-chmod! path) + auth-data)) + +(defn read-auth-file + [opts] + (let [path (auth-path opts)] + (when (fs/existsSync path) + (try + (-> (fs/readFileSync path) + (.toString "utf8") + parse-json) + (catch :default e + (throw (ex-info "invalid auth file" + {:code :invalid-auth-file + :auth-path path} + e))))))) + +(defn delete-auth-file! + [opts] + (let [path (auth-path opts)] + (when (fs/existsSync path) + (.unlinkSync fs path)) + nil)) + +(declare start-logout-complete-server! open-browser!) + +(defn logout! + [opts] + (let [path (auth-path opts) + existed? (fs/existsSync path) + logout-url (build-logout-url)] + (delete-auth-file! opts) + (-> (p/let [callback-server (start-logout-complete-server! opts)] + (-> (p/let [open-result (open-browser! logout-url) + logout-completed? (if (:opened? open-result) + (-> ((:wait! callback-server)) + (p/then (constantly true)) + (p/catch (fn [_] + false))) + false)] + {:auth-path path + :deleted? existed? + :logout-url logout-url + :opened? (:opened? open-result) + :logout-completed? logout-completed?}) + (p/finally (fn [] + ((:stop! callback-server)))))) + (p/catch (fn [_] + {:auth-path path + :deleted? existed? + :logout-url logout-url + :opened? false + :logout-completed? false}))))) + +(defn expired-auth? + [{:keys [expires-at]}] + (or (not (number? expires-at)) + (<= expires-at (js/Date.now)))) + +(defn- random-base64url + [size] + (.toString (.randomBytes crypto size) "base64url")) + +(defn- code-challenge + [code-verifier] + (-> (.createHash crypto "sha256") + (.update code-verifier) + (.digest "base64url"))) + +(defn build-authorize-url + [{:keys [redirect-uri state pkce-challenge]}] + (let [params (doto (js/URLSearchParams.) + (.set "response_type" "code") + (.set "client_id" (oauth-client-id)) + (.set "scope" (oauth-scope)) + (.set "redirect_uri" redirect-uri) + (.set "state" state) + (.set "code_challenge" pkce-challenge) + (.set "code_challenge_method" "S256"))] + (str (authorize-endpoint) "?" (.toString params)))) + +(defn- stop-server! + [server] + (if (some? server) + (p/create (fn [resolve _reject] + (.close server (fn [] + (resolve true))))) + (p/resolved true))) + +(defn start-login-callback-server! + [{:keys [state login-timeout-ms]}] + (p/create + (fn [resolve reject] + (let [callback-handlers (atom nil) + settled? (atom false) + callback-promise (p/create (fn [resolve' reject'] + (reset! callback-handlers {:resolve resolve' + :reject reject'}))) + finish! (fn [kind payload] + (when-not @settled? + (reset! settled? true) + (when-let [{:keys [resolve reject]} @callback-handlers] + ((if (= kind :resolve) resolve reject) payload)))) + server (.createServer http + (fn [^js req ^js res] + (let [url (js/URL. (str "http://" callback-host (.-url req))) + pathname (.-pathname url) + params (.-searchParams url) + code (.get params "code") + callback-state (.get params "state") + error-code (.get params "error")] + (cond + (not= redirect-path pathname) + (do + (.writeHead res 404 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Not found")) + + (seq error-code) + (do + (.writeHead res 400 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Login failed. You can return to the CLI.") + (finish! :reject (ex-info "login callback returned oauth error" + {:code :login-callback-error + :oauth-error error-code}))) + + (not= state callback-state) + (do + (.writeHead res 400 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Login failed due to state mismatch. Return to the CLI and retry.") + (finish! :reject (ex-info "login callback state mismatch" + {:code :invalid-callback-state}))) + + (not (seq code)) + (do + (.writeHead res 400 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Login failed because the callback did not include a code.") + (finish! :reject (ex-info "missing authorization code" + {:code :missing-callback-code}))) + + :else + (do + (.writeHead res 200 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Login successful. You can return to the CLI.") + (finish! :resolve {:code code})))))) + timeout-id (js/setTimeout (fn [] + (finish! :reject (ex-info "login callback timed out" + {:code :login-timeout}))) + (or login-timeout-ms default-login-timeout-ms))] + (.on server "error" (fn [error] + (js/clearTimeout timeout-id) + (reject (ex-info "failed to start login callback server" + {:code :login-callback-server-start-failed} + error)))) + (.listen server callback-port callback-host + (fn [] + (let [address (.address server) + port (.-port address) + redirect-uri (str "http://" callback-host ":" port redirect-path)] + (resolve {:port port + :redirect-uri redirect-uri + :wait! (fn [] + (-> callback-promise + (p/finally (fn [] + (js/clearTimeout timeout-id))))) + :stop! (fn [] + (js/clearTimeout timeout-id) + (stop-server! server))})))))))) + +(defn start-logout-complete-server! + [{:keys [logout-timeout-ms]}] + (p/create + (fn [resolve reject] + (let [settled? (atom false) + callback-handlers (atom nil) + callback-promise (p/create (fn [resolve' reject'] + (reset! callback-handlers {:resolve resolve' + :reject reject'}))) + finish! (fn [kind payload] + (when-not @settled? + (reset! settled? true) + (when-let [{:keys [resolve reject]} @callback-handlers] + ((if (= kind :resolve) resolve reject) payload)))) + server (.createServer http + (fn [^js req ^js res] + (let [url (js/URL. (str "http://" callback-host (.-url req))) + pathname (.-pathname url)] + (if (= logout-complete-path pathname) + (do + (.writeHead res 200 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Logout successful. You can return to the CLI.") + (finish! :resolve true)) + (do + (.writeHead res 404 #js {"Content-Type" "text/plain; charset=utf-8"}) + (.end res "Not found")))))) + timeout-id (js/setTimeout (fn [] + (finish! :reject (ex-info "logout callback timed out" + {:code :logout-timeout}))) + (or logout-timeout-ms default-logout-timeout-ms))] + (.on server "error" (fn [error] + (js/clearTimeout timeout-id) + (reject (ex-info "failed to start logout callback server" + {:code :logout-callback-server-start-failed} + error)))) + (.listen server callback-port callback-host + (fn [] + (resolve {:logout-uri (logout-complete-uri) + :wait! (fn [] + (-> callback-promise + (p/finally (fn [] + (js/clearTimeout timeout-id))))) + :stop! (fn [] + (js/clearTimeout timeout-id) + (stop-server! server))}))))))) + +(defn open-browser! + [url] + (let [platform (.-platform js/process) + [command args] (case platform + "darwin" ["open" [url]] + "linux" ["xdg-open" [url]] + "win32" ["cmd" ["/c" "start" "" url]] + [nil nil])] + (if-not (seq command) + (p/resolved {:opened? false}) + (p/create + (fn [resolve _reject] + (try + (let [child (.spawn child-process command (clj->js args) + #js {:detached true + :stdio "ignore" + :shell false})] + (.unref child) + (resolve {:opened? true + :command command})) + (catch :default e + (resolve {:opened? false + :command command + :error (or (.-message e) (str e))})))))))) + +(defn- oauth-token-request! + [params] + (let [search-params (js/URLSearchParams.)] + (.set search-params "client_id" (oauth-client-id)) + (doseq [[k v] params] + (.set search-params k (str v))) + (let [body (.toString search-params)] + (-> (transport/request {:method "POST" + :url (token-endpoint) + :headers {"Content-Type" "application/x-www-form-urlencoded" + "Accept" "application/json"} + :body body + :timeout-ms 10000}) + (p/then (fn [{:keys [body]}] + (parse-json body))))))) + +(defn- token-body->auth-data + [token-body current-auth] + (let [id-token (:id_token token-body) + claims (parse-jwt id-token) + refresh-token (or (:refresh_token token-body) + (:refresh-token current-auth))] + {:provider auth-provider + :id-token id-token + :access-token (:access_token token-body) + :refresh-token refresh-token + :expires-at (some-> (:exp claims) (* 1000)) + :sub (:sub claims) + :email (:email claims) + :updated-at (js/Date.now)})) + +(defn exchange-code-for-auth! + [_opts {:keys [code redirect-uri code-verifier]}] + (-> (oauth-token-request! {"grant_type" "authorization_code" + "code" code + "redirect_uri" redirect-uri + "code_verifier" code-verifier}) + (p/then (fn [token-body] + (token-body->auth-data token-body nil))) + (p/catch (fn [error] + (let [data (ex-data error)] + (p/rejected + (ex-info "authorization code exchange failed" + (merge {:code :auth-code-exchange-failed} + (when data {:context data})) + error))))))) + +(defn refresh-auth! + [opts auth-data] + (let [refresh-token (:refresh-token auth-data)] + (if (seq refresh-token) + (-> (oauth-token-request! {"grant_type" "refresh_token" + "refresh_token" refresh-token}) + (p/then (fn [token-body] + (token-body->auth-data token-body auth-data))) + (p/catch (fn [error] + (let [data (ex-data error) + parsed-body (try + (some-> (:body data) parse-json) + (catch :default _ + nil))] + (if (= "invalid_grant" (:error parsed-body)) + (p/rejected + (ex-info "refresh token is invalid" + {:code :missing-auth + :hint "Run logseq login first." + :auth-path (auth-path opts)} + error)) + (p/rejected + (ex-info "auth refresh failed" + {:code :auth-refresh-failed + :hint "Run logseq login first." + :auth-path (auth-path opts) + :context data} + error))))))) + (p/rejected (ex-info "missing refresh token" + {:code :missing-auth + :hint "Run logseq login first." + :auth-path (auth-path opts)}))))) + +(defn login! + [opts] + (let [state (or (:state opts) (random-base64url 24)) + code-verifier (or (:code-verifier opts) (random-base64url 48)) + authorize-payload {:state state + :pkce-challenge (code-challenge code-verifier)}] + (p/let [callback-server (start-login-callback-server! (merge opts {:state state})) + redirect-uri (:redirect-uri callback-server) + authorize-url (build-authorize-url (assoc authorize-payload :redirect-uri redirect-uri))] + (-> (p/let [open-result (open-browser! authorize-url) + callback-result ((:wait! callback-server)) + auth-data (exchange-code-for-auth! opts {:code (:code callback-result) + :redirect-uri redirect-uri + :code-verifier code-verifier}) + _ (write-auth-file! opts auth-data)] + {:auth-path (auth-path opts) + :authorize-url authorize-url + :opened? (:opened? open-result) + :email (:email auth-data) + :sub (:sub auth-data) + :updated-at (:updated-at auth-data)}) + (p/finally (fn [] + ((:stop! callback-server)))))))) + +(defn resolve-auth-token! + [opts] + (if-let [current-auth (read-auth-file opts)] + (if (expired-auth? current-auth) + (p/let [refreshed-auth (refresh-auth! opts current-auth) + next-auth (merge current-auth refreshed-auth)] + (write-auth-file! opts next-auth) + (:id-token next-auth)) + (p/resolved (:id-token current-auth))) + (p/rejected (ex-info "missing auth" + {:code :missing-auth + :hint "Run logseq login first." + :auth-path (auth-path opts)})))) diff --git a/src/main/logseq/cli/command/auth.cljs b/src/main/logseq/cli/command/auth.cljs new file mode 100644 index 0000000000..628573232a --- /dev/null +++ b/src/main/logseq/cli/command/auth.cljs @@ -0,0 +1,56 @@ +(ns logseq.cli.command.auth + "Authentication-related CLI commands." + (:require [logseq.cli.auth :as cli-auth] + [logseq.cli.command.core :as core] + [promesa.core :as p])) + +(def entries + [(core/command-entry ["login"] + :login + "Authenticate this machine with Logseq cloud" + {}) + (core/command-entry ["logout"] + :logout + "Remove persisted CLI auth" + {})]) + +(defn build-action + [command] + {:ok? true + :action {:type command}}) + +(defn- ex-message->code + [message] + (when (and (string? message) + (re-matches #"[a-zA-Z0-9._/\-]+" message)) + (keyword message))) + +(defn- exception->error + [error] + (let [data (or (ex-data error) {}) + code (or (:code data) + (ex-message->code (ex-message error)) + :exception)] + {:status :error + :error (merge {:code code + :message (or (ex-message error) (str error))} + (when (seq data) {:context data}))})) + +(defn execute + [action config] + (case (:type action) + :login + (-> (p/let [data (cli-auth/login! config)] + {:status :ok + :data data}) + (p/catch exception->error)) + + :logout + (-> (p/let [data (cli-auth/logout! config)] + {:status :ok + :data data}) + (p/catch exception->error)) + + (p/resolved {:status :error + :error {:code :unknown-action + :message "unknown auth action"}}))) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 05059c6642..c0134a6807 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -99,7 +99,9 @@ (let [groups [{:title "Graph Inspect and Edit" :commands #{"list" "upsert" "remove" "query" "show"}} {:title "Graph Management" - :commands #{"graph" "server" "doctor" "sync"}}] + :commands #{"graph" "server" "doctor" "sync"}} + {:title "Authentication" + :commands #{"login" "logout"}}] render-group (fn [{:keys [title commands]}] (let [entries (filter #(contains? commands (first (:cmds %))) table)] (string/join "\n" [title (format-commands entries)])))] diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index 2be65fa2ca..bdbe2f8c72 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -1,6 +1,7 @@ (ns logseq.cli.command.sync "Sync-related CLI commands." (:require [clojure.string :as string] + [logseq.cli.auth :as cli-auth] [logseq.cli.command.core :as core] [logseq.cli.config :as cli-config] [logseq.cli.server :as cli-server] @@ -27,9 +28,16 @@ (def ^:private config-key-map {"ws-url" :ws-url "http-base" :http-base - "auth-token" :auth-token "e2ee-password" :e2ee-password}) +(def ^:private authenticated-sync-actions + #{:sync-start + :sync-upload + :sync-download + :sync-remote-graphs + :sync-ensure-keys + :sync-grant-access}) + (def ^:private sync-start-timeout-ms 10000) (def ^:private sync-start-poll-interval-ms 100) @@ -190,6 +198,15 @@ :auth-token (:auth-token config) :e2ee-password (:e2ee-password config)}) +(defn- resolve-runtime-config! + [action config] + (if (contains? authenticated-sync-actions (:type action)) + (if (seq (:auth-token config)) + (p/resolved config) + (p/let [auth-token (cli-auth/resolve-auth-token! config)] + (assoc config :auth-token auth-token))) + (p/resolved config))) + (defn- invoke-with-repo [config repo method args] (let [sync-cfg (sync-config config)] @@ -235,7 +252,7 @@ (let [timeout-ms (max 0 (or (:wait-timeout-ms action) sync-start-timeout-ms)) poll-interval-ms (max 0 (or (:wait-poll-interval-ms action) sync-start-poll-interval-ms)) deadline (+ (js/Date.now) timeout-ms) - config-skipped-hint "Set sync config keys (ws-url/http-base/auth-token) and retry sync start." + config-skipped-hint "Run logseq login, set sync config keys (ws-url/http-base), and retry sync start." graph-id-skipped-hint "Graph-id is missing locally. Run sync download first, then retry sync start." runtime-error-hint "Run sync status to inspect last-error and fix sync runtime error before retrying." timeout-hint "Run sync status to inspect ws-state and ensure sync endpoint/token are valid."] @@ -336,10 +353,11 @@ :data result}) :sync-start - (-> (p/let [_ (invoke-with-repo config (:repo action) + (-> (p/let [config' (resolve-runtime-config! action config) + _ (invoke-with-repo config' (:repo action) :thread-api/db-sync-start [(:repo action)]) - result (wait-sync-start-ready config (:repo action) action)] + result (wait-sync-start-ready config' (:repo action) action)] result) (p/catch (fn [error] (exception->error error {:repo (:repo action)})))) @@ -352,27 +370,45 @@ :data {:result result}}) :sync-upload - (execute-sync-upload action config) + (-> (p/let [config' (resolve-runtime-config! action config)] + (execute-sync-upload action config')) + (p/catch (fn [error] + (exception->error error {:repo (:repo action)})))) :sync-download - (execute-sync-download action config) + (-> (p/let [config' (resolve-runtime-config! action config)] + (execute-sync-download action config')) + (p/catch (fn [error] + (exception->error error {:repo (:repo action) + :graph (:graph action)})))) :sync-remote-graphs - (p/let [graphs (invoke-global config :thread-api/db-sync-list-remote-graphs [])] - {:status :ok - :data {:graphs (or graphs [])}}) + (-> (p/let [config' (resolve-runtime-config! action config) + graphs (invoke-global config' :thread-api/db-sync-list-remote-graphs [])] + {:status :ok + :data {:graphs (or graphs [])}}) + (p/catch (fn [error] + (exception->error error nil)))) :sync-ensure-keys - (p/let [result (invoke-global config :thread-api/db-sync-ensure-user-rsa-keys [])] - {:status :ok - :data {:result result}}) + (-> (p/let [config' (resolve-runtime-config! action config) + result (invoke-global config' :thread-api/db-sync-ensure-user-rsa-keys [])] + {:status :ok + :data {:result result}}) + (p/catch (fn [error] + (exception->error error nil)))) :sync-grant-access - (p/let [result (invoke-with-repo config (:repo action) - :thread-api/db-sync-grant-graph-access - [(:repo action) (:graph-id action) (:email action)])] - {:status :ok - :data {:result result}}) + (-> (p/let [config' (resolve-runtime-config! action config) + result (invoke-with-repo config' (:repo action) + :thread-api/db-sync-grant-graph-access + [(:repo action) (:graph-id action) (:email action)])] + {:status :ok + :data {:result result}}) + (p/catch (fn [error] + (exception->error error {:repo (:repo action) + :graph-id (:graph-id action) + :email (:email action)})))) :sync-config-get (p/let [current config] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 5e21a2033d..21b178250e 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -2,6 +2,7 @@ "Command parsing and action building for the Logseq CLI." (:require [babashka.cli :as cli] [clojure.string :as string] + [logseq.cli.command.auth :as auth-command] [logseq.cli.command.core :as command-core] [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] @@ -103,7 +104,8 @@ query-command/entries show-command/entries doctor-command/entries - sync-command/entries))) + sync-command/entries + auth-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -417,6 +419,9 @@ :sync-config-set :sync-config-get :sync-config-unset) (sync-command/build-action command options args repo) + (:login :logout) + (auth-command/build-action command) + {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) @@ -466,6 +471,8 @@ :sync-remote-graphs :sync-ensure-keys :sync-grant-access :sync-config-set :sync-config-get :sync-config-unset) (sync-command/execute action config) + (:login :logout) + (auth-command/execute action config) {:status :error :error {:code :unknown-action :message "unknown action"}}))] diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index e050e113f3..4874c7d6c6 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -28,11 +28,19 @@ [] (node-path/join (.homedir os) "logseq" "cli.edn")) +(def ^:private removed-config-keys + #{:auth-token :retries}) + +(defn- sanitize-file-config + [config] + (apply dissoc (or config {}) removed-config-keys)) + (defn- read-config-file [config-path] (when (and (some? config-path) (fs/existsSync config-path)) (let [contents (.toString (fs/readFileSync config-path) "utf8")] - (reader/read-string contents)))) + (-> (reader/read-string contents) + sanitize-file-config)))) (defn- ensure-config-dir! [config-path] @@ -45,8 +53,8 @@ [{:keys [config-path]} updates] (let [path (or config-path (default-config-path)) current (or (read-config-file path) {}) - filtered-current (dissoc current :retries) - filtered-updates (dissoc (or updates {}) :retries) + filtered-current (sanitize-file-config current) + filtered-updates (sanitize-file-config updates) nil-keys (->> filtered-updates (keep (fn [[k v]] (when (nil? v) @@ -72,6 +80,12 @@ (seq (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS")) (assoc :timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_TIMEOUT_MS"))) + (seq (gobj/get env "LOGSEQ_CLI_LOGIN_TIMEOUT_MS")) + (assoc :login-timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_LOGIN_TIMEOUT_MS"))) + + (seq (gobj/get env "LOGSEQ_CLI_LOGOUT_TIMEOUT_MS")) + (assoc :logout-timeout-ms (parse-int (gobj/get env "LOGSEQ_CLI_LOGOUT_TIMEOUT_MS"))) + (seq (gobj/get env "LOGSEQ_CLI_OUTPUT")) (assoc :output-format (parse-output-format (gobj/get env "LOGSEQ_CLI_OUTPUT"))) @@ -81,6 +95,8 @@ (defn resolve-config [opts] (let [defaults {:timeout-ms 10000 + :login-timeout-ms 300000 + :logout-timeout-ms 120000 :output-format nil :data-dir "~/logseq/graphs" :config-path (default-config-path)} diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 4d267f083e..57f3445987 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -341,11 +341,23 @@ (update data :kv redact-graph-kv) data)) +(defn- sanitize-auth-data + [data] + (if (map? data) + (apply dissoc data [:id-token :access-token :refresh-token]) + data)) + (defn- sanitize-result [result] - (if (and (= :ok (:status result)) - (= :graph-info (:command result))) + (cond + (and (= :ok (:status result)) + (= :graph-info (:command result))) (update result :data sanitize-graph-info-data) + + (= :login (:command result)) + (update result :data sanitize-auth-data) + + :else result)) (defn- format-sync-status @@ -406,6 +418,28 @@ [{:keys [key]}] (str "sync config unset: " (name key))) +(defn- format-login + [{:keys [auth-path email sub]}] + (string/join "\n" + (cond-> ["Login successful" + (str "Auth file: " (or auth-path "-"))] + (seq email) (conj (str "Email: " email)) + (seq sub) (conj (str "User: " sub))))) + +(defn- format-logout + [{:keys [auth-path deleted? opened? logout-completed?]}] + (string/join "\n" + (cond-> [(str (if deleted? + "Logged out" + "Already logged out") + ": " + (or auth-path "-"))] + logout-completed? (conj "Cognito logout: completed") + (and (not logout-completed?) (true? opened?)) + (conj "Cognito logout: browser opened, completion not confirmed") + (false? opened?) + (conj "Cognito logout: could not open browser")))) + (defn- format-upsert-block [{:keys [repo source target update-tags update-properties remove-tags remove-properties]} result] (if (vector? result) @@ -508,6 +542,8 @@ :sync-config-get (format-sync-config-get data) :sync-config-set (format-sync-config-set data) :sync-config-unset (format-sync-config-unset data) + :login (format-login data) + :logout (format-logout data) :list-page (format-list-page (:items data) now-ms) :list-tag (format-list-tag (:items data) now-ms) :list-property (format-list-property (:items data) now-ms) diff --git a/src/test/logseq/cli/auth_test.cljs b/src/test/logseq/cli/auth_test.cljs new file mode 100644 index 0000000000..b5b25a96c9 --- /dev/null +++ b/src/test/logseq/cli/auth_test.cljs @@ -0,0 +1,102 @@ +(ns logseq.cli.auth-test + (:require [cljs.test :refer [async deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.auth :as auth] + [promesa.core :as p] + ["fs" :as fs] + ["os" :as os] + ["path" :as node-path])) + +(defn- sample-auth + ([] + (sample-auth {})) + ([overrides] + (merge {:provider "cognito" + :id-token "id-token-1" + :access-token "access-token-1" + :refresh-token "refresh-token-1" + :expires-at (+ (js/Date.now) 3600000) + :sub "user-123" + :email "user@example.com" + :updated-at 1735686000000} + overrides))) + +(defn- read-json-file + [path] + (-> (fs/readFileSync path) + (.toString "utf8") + js/JSON.parse + (js->clj :keywordize-keys true))) + +(deftest test-default-auth-path + (is (= (node-path/join (.homedir os) "logseq" "auth.json") + (auth/default-auth-path)))) + +(deftest test-write-auth-file-creates-parent-dir + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-dir (node-path/join dir "nested" "tokens") + auth-path (node-path/join auth-dir "auth.json") + payload (sample-auth)] + (is (not (fs/existsSync auth-dir))) + (auth/write-auth-file! {:auth-path auth-path} payload) + (is (fs/existsSync auth-dir)) + (is (fs/existsSync auth-path)) + (when (fs/existsSync auth-path) + (is (= payload (read-json-file auth-path)))))) + +(deftest test-read-auth-file-returns-nil-when-missing + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "missing" "auth.json")] + (is (nil? (auth/read-auth-file {:auth-path auth-path}))))) + +(deftest test-delete-auth-file-is-idempotent + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json")] + (auth/delete-auth-file! {:auth-path auth-path}) + (is (not (fs/existsSync auth-path))) + (auth/write-auth-file! {:auth-path auth-path} (sample-auth)) + (is (fs/existsSync auth-path)) + (auth/delete-auth-file! {:auth-path auth-path}) + (is (not (fs/existsSync auth-path))))) + +(deftest test-read-auth-file-invalid-json + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json")] + (fs/writeFileSync auth-path "{\"provider\":") + (try + (auth/read-auth-file {:auth-path auth-path}) + (is false "expected invalid-auth-file error") + (catch :default e + (is (= :invalid-auth-file (-> e ex-data :code))) + (is (= auth-path (-> e ex-data :auth-path))))))) + +(deftest test-resolve-auth-token-refreshes-expired-token + (async done + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json") + refresh-calls (atom []) + expired-auth (sample-auth {:id-token "expired-id-token" + :access-token "expired-access-token" + :expires-at 0}) + refreshed-auth (sample-auth {:id-token "fresh-id-token" + :access-token "fresh-access-token" + :refresh-token "refresh-token-1" + :expires-at (+ (js/Date.now) 7200000) + :updated-at 1735689600000})] + (auth/write-auth-file! {:auth-path auth-path} expired-auth) + (let [result-promise + (p/with-redefs [auth/refresh-auth! (fn [opts auth-data] + (swap! refresh-calls conj [opts auth-data]) + (p/resolved refreshed-auth))] + (p/let [token (auth/resolve-auth-token! {:auth-path auth-path}) + stored (auth/read-auth-file {:auth-path auth-path})] + (is (= [[{:auth-path auth-path} expired-auth]] @refresh-calls)) + (is (= "fresh-id-token" token)) + (is (= refreshed-auth stored)) + (when (fs/existsSync auth-path) + (is (= refreshed-auth (read-json-file auth-path))))))] + (-> result-promise + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (done)))))))) diff --git a/src/test/logseq/cli/command/auth_test.cljs b/src/test/logseq/cli/command/auth_test.cljs new file mode 100644 index 0000000000..4ced86b4b4 --- /dev/null +++ b/src/test/logseq/cli/command/auth_test.cljs @@ -0,0 +1,215 @@ +(ns logseq.cli.command.auth-test + (:require [cljs.test :refer [async deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.auth :as cli-auth] + [logseq.cli.command.auth :as auth-command] + [logseq.common.cognito-config :as cognito-config] + [promesa.core :as p] + ["fs" :as fs] + ["path" :as node-path])) + +(defn- sample-auth + ([] + (sample-auth {})) + ([overrides] + (merge {:provider "cognito" + :id-token "id-token-1" + :access-token "access-token-1" + :refresh-token "refresh-token-1" + :expires-at (+ (js/Date.now) 3600000) + :sub "user-123" + :email "user@example.com" + :updated-at 1735686000000} + overrides))) + +(deftest test-login-opens-browser-with-authorize-url-and-persists-auth + (async done + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json") + open-calls (atom []) + exchange-calls (atom []) + write-calls (atom []) + auth-data (sample-auth) + callback-server {:port 8765 + :redirect-uri "http://localhost:8765/auth/callback" + :wait! (fn [] (p/resolved {:code "oauth-code"})) + :stop! (fn [] (p/resolved true))}] + (-> (p/with-redefs [cognito-config/CLI-COGNITO-CLIENT-ID "69cs1lgme7p8kbgld8n5kseii6" + cli-auth/start-login-callback-server! (fn [_opts] + (p/resolved callback-server)) + cli-auth/open-browser! (fn [url] + (swap! open-calls conj url) + (p/resolved {:opened? true})) + cli-auth/exchange-code-for-auth! (fn [opts payload] + (swap! exchange-calls conj [opts payload]) + (p/resolved auth-data)) + cli-auth/write-auth-file! (fn [opts payload] + (swap! write-calls conj [opts payload]) + payload)] + (p/let [result (cli-auth/login! {:auth-path auth-path}) + authorize-url (first @open-calls)] + (is (= 1 (count @open-calls))) + (is (string? authorize-url)) + (is (re-find #"/oauth2/authorize" authorize-url)) + (is (re-find #"response_type=code" authorize-url)) + (is (re-find #"client_id=69cs1lgme7p8kbgld8n5kseii6" authorize-url)) + (is (re-find #"redirect_uri=http%3A%2F%2Flocalhost%3A8765%2Fauth%2Fcallback" authorize-url)) + (is (re-find #"state=" authorize-url)) + (is (re-find #"code_challenge=" authorize-url)) + (is (= 1 (count @exchange-calls))) + (let [[exchange-opts exchange-payload] (first @exchange-calls)] + (is (= {:auth-path auth-path} exchange-opts)) + (is (= "oauth-code" (:code exchange-payload))) + (is (= "http://localhost:8765/auth/callback" (:redirect-uri exchange-payload))) + (is (string? (:code-verifier exchange-payload)))) + (is (= [[{:auth-path auth-path} auth-data]] @write-calls)) + (is (= auth-path (:auth-path result))) + (is (= "user@example.com" (:email result))) + (is (= "user-123" (:sub result))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-login-validates-state-before-code-exchange + (async done + (let [exchange-calls (atom [])] + (-> (p/with-redefs [cli-auth/open-browser! (fn [authorize-url] + (let [parsed (js/URL. authorize-url) + redirect-uri (.get (.-searchParams parsed) "redirect_uri")] + (-> (js/fetch (str redirect-uri "?code=oauth-code&state=wrong-state")) + (p/then (fn [_] + {:opened? true})))) ) + cli-auth/exchange-code-for-auth! (fn [opts payload] + (swap! exchange-calls conj [opts payload]) + (p/resolved (sample-auth)))] + (-> (cli-auth/login! {:timeout-ms 200}) + (p/then (fn [_] + (is false "expected invalid callback state error"))) + (p/catch (fn [e] + (is (= :invalid-callback-state (-> e ex-data :code))) + (is (= [] @exchange-calls)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-login-timeout-when-no-browser-callback-arrives + (async done + (let [exchange-calls (atom [])] + (-> (p/with-redefs [cli-auth/open-browser! (fn [_authorize-url] + (p/resolved {:opened? false})) + cli-auth/exchange-code-for-auth! (fn [opts payload] + (swap! exchange-calls conj [opts payload]) + (p/resolved (sample-auth)))] + (-> (cli-auth/login! {:login-timeout-ms 10}) + (p/then (fn [_] + (is false "expected login timeout error"))) + (p/catch (fn [e] + (is (= :login-timeout (-> e ex-data :code))) + (is (= [] @exchange-calls)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-login-ignores-global-request-timeout-for-callback-wait + (async done + (let [exchange-calls (atom [])] + (-> (p/with-redefs [cli-auth/open-browser! (fn [_authorize-url] + (p/resolved {:opened? false})) + cli-auth/exchange-code-for-auth! (fn [opts payload] + (swap! exchange-calls conj [opts payload]) + (p/resolved (sample-auth)))] + (-> (cli-auth/login! {:timeout-ms 10 + :login-timeout-ms 200}) + (p/then (fn [_] + (is false "expected login timeout error"))) + (p/catch (fn [e] + (is (= :login-timeout (-> e ex-data :code))) + (is (= [] @exchange-calls)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-logout-removes-auth-file-and-completes-cognito-logout-when-file-existed + (async done + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json") + open-calls (atom [])] + (cli-auth/write-auth-file! {:auth-path auth-path} (sample-auth)) + (is (fs/existsSync auth-path)) + (-> (p/with-redefs [cognito-config/CLI-COGNITO-CLIENT-ID "69cs1lgme7p8kbgld8n5kseii6" + cli-auth/open-browser! (fn [url] + (swap! open-calls conj url) + (let [parsed (js/URL. url) + logout-uri (.get (.-searchParams parsed) "logout_uri")] + (-> (js/fetch logout-uri) + (p/then (fn [_] + {:opened? true})))))] + (p/let [result (cli-auth/logout! {:auth-path auth-path}) + logout-url (first @open-calls)] + (is (= 1 (count @open-calls))) + (is (= auth-path (:auth-path result))) + (is (true? (:deleted? result))) + (is (true? (:opened? result))) + (is (true? (:logout-completed? result))) + (is (not (fs/existsSync auth-path))) + (is (string? logout-url)) + (when (string? logout-url) + (is (re-find #"/logout\?" logout-url)) + (is (re-find #"client_id=69cs1lgme7p8kbgld8n5kseii6" logout-url)) + (is (re-find #"logout_uri=http%3A%2F%2Flocalhost%3A8765%2Flogout-complete" logout-url))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-logout-completes-cognito-logout-when-auth-file-is-already-absent + (async done + (let [dir (node-helper/create-tmp-dir "cli-auth") + auth-path (node-path/join dir "auth.json") + open-calls (atom [])] + (-> (p/with-redefs [cognito-config/CLI-COGNITO-CLIENT-ID "69cs1lgme7p8kbgld8n5kseii6" + cli-auth/open-browser! (fn [url] + (swap! open-calls conj url) + (let [parsed (js/URL. url) + logout-uri (.get (.-searchParams parsed) "logout_uri")] + (-> (js/fetch logout-uri) + (p/then (fn [_] + {:opened? true})))))] + (p/let [result (cli-auth/logout! {:auth-path auth-path}) + logout-url (first @open-calls)] + (is (= 1 (count @open-calls))) + (is (= auth-path (:auth-path result))) + (is (false? (:deleted? result))) + (is (true? (:opened? result))) + (is (true? (:logout-completed? result))) + (is (not (fs/existsSync auth-path))) + (is (string? logout-url)) + (when (string? logout-url) + (is (re-find #"logout_uri=http%3A%2F%2Flocalhost%3A8765%2Flogout-complete" logout-url))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) + +(deftest test-command-execute-login-and-logout + (async done + (let [login-calls (atom []) + logout-calls (atom [])] + (-> (p/with-redefs [cli-auth/login! (fn [config] + (swap! login-calls conj config) + (p/resolved {:auth-path "/tmp/auth.json" + :email "user@example.com" + :sub "user-123"})) + cli-auth/logout! (fn [config] + (swap! logout-calls conj config) + {:auth-path "/tmp/auth.json" + :deleted? true})] + (p/let [login-result (auth-command/execute {:type :login} {:auth-path "/tmp/auth.json"}) + logout-result (auth-command/execute {:type :logout} {:auth-path "/tmp/auth.json"})] + (is (= [{:auth-path "/tmp/auth.json"}] @login-calls)) + (is (= [{:auth-path "/tmp/auth.json"}] @logout-calls)) + (is (= :ok (:status login-result))) + (is (= "user@example.com" (get-in login-result [:data :email]))) + (is (= :ok (:status logout-result))) + (is (true? (get-in logout-result [:data :deleted?]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] (done))))))) diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index 4278ed9a67..4e9f83791a 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -1,11 +1,16 @@ (ns logseq.cli.command.sync-test (:require [cljs.test :refer [async deftest is testing]] + [logseq.cli.auth :as cli-auth] [logseq.cli.command.sync :as sync-command] [logseq.cli.config :as cli-config] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] [promesa.core :as p])) +(defn- execute-with-runtime-auth + [action config] + (sync-command/execute action (assoc config :auth-token "runtime-token"))) + (deftest test-build-action-validation (testing "sync status requires repo" (let [result (sync-command/build-action :sync-status {} [] nil)] @@ -25,12 +30,23 @@ (testing "sync config set requires name and value" (let [missing-both (sync-command/build-action :sync-config-set {} [] nil) - missing-value (sync-command/build-action :sync-config-set {} ["auth-token"] nil)] + missing-value (sync-command/build-action :sync-config-set {} ["ws-url"] nil)] (is (false? (:ok? missing-both))) (is (= :invalid-options (get-in missing-both [:error :code]))) (is (false? (:ok? missing-value))) (is (= :invalid-options (get-in missing-value [:error :code]))))) + (testing "sync config rejects auth-token key" + (let [set-result (sync-command/build-action :sync-config-set {} ["auth-token" "secret"] nil) + get-result (sync-command/build-action :sync-config-get {} ["auth-token"] nil) + unset-result (sync-command/build-action :sync-config-unset {} ["auth-token"] nil)] + (is (false? (:ok? set-result))) + (is (= :invalid-options (get-in set-result [:error :code]))) + (is (false? (:ok? get-result))) + (is (= :invalid-options (get-in get-result [:error :code]))) + (is (false? (:ok? unset-result))) + (is (= :invalid-options (get-in unset-result [:error :code]))))) + (testing "sync config accepts e2ee-password key" (let [result (sync-command/build-action :sync-config-set {} ["e2ee-password" "pw"] nil)] (is (true? (:ok? result))) @@ -63,7 +79,7 @@ :pending-asset 0 :pending-server 0})) (p/resolved {:ok true})))] - (p/let [result (sync-command/execute {:type :sync-start + (p/let [result (execute-with-runtime-auth {:type :sync-start :repo "logseq_db_demo"} {:data-dir "/tmp"}) invoked-methods (map first @invoke-calls)] @@ -92,7 +108,7 @@ :pending-asset 0 :pending-server 0}) (p/resolved {:ok true})))] - (p/let [result (sync-command/execute {:type :sync-start + (p/let [result (execute-with-runtime-auth {:type :sync-start :repo "logseq_db_demo" :wait-timeout-ms 20 :wait-poll-interval-ms 0} @@ -118,7 +134,7 @@ :pending-asset 0 :pending-server 0}) (p/resolved {:ok true})))] - (p/let [result (sync-command/execute {:type :sync-start + (p/let [result (execute-with-runtime-auth {:type :sync-start :repo "logseq_db_demo" :wait-timeout-ms 20 :wait-poll-interval-ms 0} @@ -153,7 +169,7 @@ :last-error {:code :decrypt-aes-key :message "decrypt-aes-key"}}))) (p/resolved {:ok true})))] - (p/let [result (sync-command/execute {:type :sync-start + (p/let [result (execute-with-runtime-auth {:type :sync-start :repo "logseq_db_demo" :wait-timeout-ms 200 :wait-poll-interval-ms 0} @@ -178,7 +194,8 @@ (p/let [_ (sync-command/execute {:type :sync-stop :repo "logseq_db_demo"} {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + (is (= [[{:data-dir "/tmp"} + "logseq_db_demo"]] @ensure-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil @@ -200,14 +217,16 @@ transport/invoke (fn [_ method direct-pass? args] (swap! invoke-calls conj [method direct-pass? args]) (p/resolved {:ok true}))] - (p/let [_ (sync-command/execute {:type :sync-upload + (p/let [_ (execute-with-runtime-auth {:type :sync-upload :repo "logseq_db_demo"} {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + (is (= [[{:data-dir "/tmp" + :auth-token "runtime-token"} + "logseq_db_demo"]] @ensure-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-upload-graph false ["logseq_db_demo"]]] @invoke-calls)))) @@ -231,7 +250,7 @@ :graph-id "graph-1"})) (p/resolved nil)))] - (p/let [result (sync-command/execute {:type :sync-upload + (p/let [result (execute-with-runtime-auth {:type :sync-upload :repo "logseq_db_demo"} {:data-dir "/tmp"})] (is (= :error (:status result))) @@ -261,26 +280,27 @@ :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))] - (p/let [_ (sync-command/execute {:type :sync-download + (p/let [_ (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} {:base-url "http://example" :data-dir "/tmp"})] (is (= [[{:base-url "http://example" :create-empty-db? true - :data-dir "/tmp"} + :data-dir "/tmp" + :auth-token "runtime-token"} "logseq_db_demo"]] @ensure-calls)) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] (nth @invoke-calls 0))) (is (= [:thread-api/db-sync-list-remote-graphs false []] (nth @invoke-calls 1))) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] (nth @invoke-calls 2))) (let [[method direct-pass? args] (nth @invoke-calls 3)] @@ -312,30 +332,32 @@ :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))] - (p/let [_ (sync-command/execute {:type :sync-download + (p/let [_ (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} {:graph "demo" :data-dir "/tmp"})] (is (= [[{:graph "demo" :create-empty-db? true - :data-dir "/tmp"} + :data-dir "/tmp" + :auth-token "runtime-token"} "logseq_db_demo"] [{:graph "demo" :create-empty-db? true - :data-dir "/tmp"} + :data-dir "/tmp" + :auth-token "runtime-token"} "logseq_db_demo"]] @ensure-calls)) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] (nth @invoke-calls 0))) (is (= [:thread-api/db-sync-list-remote-graphs false []] (nth @invoke-calls 1))) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] (nth @invoke-calls 2))) (let [[method direct-pass? args] (nth @invoke-calls 3)] @@ -362,7 +384,7 @@ :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))] - (p/let [result (sync-command/execute {:type :sync-download + (p/let [result (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} {:base-url "http://example" @@ -371,7 +393,7 @@ (is (= :remote-graph-not-found (get-in result [:error :code]))) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-list-remote-graphs false []]] @invoke-calls)))) @@ -396,7 +418,7 @@ {:code :db-sync/incomplete-snapshot-frame :graph-id "remote-graph-id"})) (p/resolved nil)))] - (p/let [result (sync-command/execute {:type :sync-download + (p/let [result (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} {:base-url "http://example" @@ -426,7 +448,7 @@ :thread-api/db-sync-download-graph-by-id (p/resolved {:ok true}) (p/resolved nil)))] - (p/let [result (sync-command/execute {:type :sync-download + (p/let [result (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} {:data-dir "/tmp"})] @@ -444,8 +466,12 @@ (deftest test-execute-sync-remote-graphs (async done (let [ensure-calls (atom []) - invoke-calls (atom [])] - (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + invoke-calls (atom []) + auth-calls (atom [])] + (-> (p/with-redefs [cli-auth/resolve-auth-token! (fn [config] + (swap! auth-calls conj config) + (p/resolved "resolved-token")) + cli-server/ensure-server! (fn [config repo] (swap! ensure-calls conj [config repo]) (p/resolved (assoc config :base-url "http://example"))) transport/invoke (fn [_ method direct-pass? args] @@ -455,13 +481,18 @@ {:base-url "http://example" :http-base "https://sync.example.com" :ws-url "wss://sync.example.com/sync/%s" - :auth-token "test-token" :e2ee-password "pw" :data-dir "/tmp"})] (is (= [] @ensure-calls)) + (is (= [{:base-url "http://example" + :http-base "https://sync.example.com" + :ws-url "wss://sync.example.com/sync/%s" + :e2ee-password "pw" + :data-dir "/tmp"}] + @auth-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url "wss://sync.example.com/sync/%s" :http-base "https://sync.example.com" - :auth-token "test-token" + :auth-token "resolved-token" :e2ee-password "pw"}]] [:thread-api/db-sync-list-remote-graphs false []]] @invoke-calls)))) @@ -469,6 +500,27 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) +(deftest test-execute-sync-remote-graphs-missing-auth + (async done + (let [invoke-calls (atom [])] + (-> (p/with-redefs [cli-auth/resolve-auth-token! (fn [_config] + (p/rejected (ex-info "missing auth" + {:code :missing-auth + :hint "Run logseq login first."}))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved []))] + (p/let [result (sync-command/execute {:type :sync-remote-graphs} + {:base-url "http://example" + :data-dir "/tmp"})] + (is (= :error (:status result))) + (is (= :missing-auth (get-in result [:error :code]))) + (is (= "Run logseq login first." (get-in result [:error :context :hint]))) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest test-execute-sync-ensure-keys (async done (let [ensure-calls (atom []) @@ -479,13 +531,13 @@ transport/invoke (fn [_ method direct-pass? args] (swap! invoke-calls conj [method direct-pass? args]) (p/resolved {:ok true}))] - (p/let [_ (sync-command/execute {:type :sync-ensure-keys} + (p/let [_ (execute-with-runtime-auth {:type :sync-ensure-keys} {:base-url "http://example" :data-dir "/tmp"})] (is (= [] @ensure-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-ensure-user-rsa-keys false []]] @invoke-calls)))) @@ -503,16 +555,18 @@ transport/invoke (fn [_ method direct-pass? args] (swap! invoke-calls conj [method direct-pass? args]) (p/resolved {:ok true}))] - (p/let [_ (sync-command/execute {:type :sync-grant-access + (p/let [_ (execute-with-runtime-auth {:type :sync-grant-access :repo "logseq_db_demo" :graph-id "graph-uuid" :email "user@example.com"} {:data-dir "/tmp"})] - (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] + (is (= [[{:data-dir "/tmp" + :auth-token "runtime-token"} + "logseq_db_demo"]] @ensure-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] @invoke-calls)))) @@ -531,9 +585,9 @@ (swap! invoke-calls conj [method direct-pass? args]) (p/resolved {:ok true}))] (p/let [_ (sync-command/execute {:type :sync-config-get - :config-key :auth-token} + :config-key :ws-url} {:base-url "http://example" - :auth-token "abc" + :ws-url "wss://sync.example.com/sync/%s" :data-dir "/tmp"})] (is (= [] @ensure-calls)) (is (= [] @invoke-calls)))) @@ -552,15 +606,15 @@ (swap! update-calls conj [config updates]) (merge {:ws-url "wss://old.example/sync/%s"} updates))] (p/let [_ (sync-command/execute {:type :sync-config-set - :config-key :auth-token - :config-value "token-value"} + :config-key :ws-url + :config-value "wss://sync.example.com/sync/%s"} {:base-url "http://example" :config-path "/tmp/cli.edn" :data-dir "/tmp"})] (is (= [[{:base-url "http://example" :config-path "/tmp/cli.edn" :data-dir "/tmp"} - {:auth-token "token-value"}]] + {:ws-url "wss://sync.example.com/sync/%s"}]] @update-calls)) (is (= [] @invoke-calls)))) (p/catch (fn [e] @@ -576,18 +630,17 @@ (p/resolved nil)) cli-config/update-config! (fn [config updates] (swap! update-calls conj [config updates]) - (dissoc {:ws-url "wss://old.example/sync/%s" - :auth-token "token-value"} - :auth-token))] + (dissoc {:ws-url "wss://old.example/sync/%s"} + :ws-url))] (p/let [_ (sync-command/execute {:type :sync-config-unset - :config-key :auth-token} + :config-key :ws-url} {:base-url "http://example" :config-path "/tmp/cli.edn" :data-dir "/tmp"})] (is (= [[{:base-url "http://example" :config-path "/tmp/cli.edn" :data-dir "/tmp"} - {:auth-token nil}]] + {:ws-url nil}]] @update-calls)) (is (= [] @invoke-calls)))) (p/catch (fn [e] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 4414037676..3156023b4b 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -59,6 +59,7 @@ (is (not (string/includes? plain-summary "--retries"))) (is (string/includes? plain-summary "Graph Inspect and Edit")) (is (string/includes? plain-summary "Graph Management")) + (is (string/includes? plain-summary "Authentication")) (is (string/includes? plain-summary "list")) (is (string/includes? plain-summary "upsert")) (is (string/includes? plain-summary "remove")) @@ -68,6 +69,8 @@ (is (string/includes? plain-summary "graph")) (is (string/includes? plain-summary "server")) (is (string/includes? plain-summary "sync")) + (is (string/includes? plain-summary "login")) + (is (string/includes? plain-summary "logout")) (is (string/includes? plain-summary "Path to db-worker data dir (default ~/logseq/graphs)")) (is (contains-bold? summary "list page")) (is (contains-bold? summary "list tag")) @@ -90,6 +93,8 @@ (is (contains-bold? summary "server start")) (is (contains-bold? summary "sync status")) (is (contains-bold? summary "sync start")) + (is (contains-bold? summary "login")) + (is (contains-bold? summary "logout")) (is (contains-bold? summary "--help")) (is (contains-bold? summary "--graph")) (is (re-find #"\u001b\[[0-9;]*mCommands\u001b\[[0-9;]*m:" summary)) @@ -209,6 +214,27 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-help-auth-commands + (testing "login command shows help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["login" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq login")) + (is (string/includes? plain-summary "Global options:")) + (is (string/includes? plain-summary "Command options:")))) + + (testing "logout command shows help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["logout" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq logout")) + (is (string/includes? plain-summary "Global options:")) + (is (string/includes? plain-summary "Command options:"))))) + (deftest test-parse-args-help-sync-group (testing "sync group shows subcommands" (let [result (binding [style/*color-enabled?* true] diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index 1c461500bf..a4bc352d78 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -27,21 +27,30 @@ (str "{:graph \"file-repo\" " ":data-dir \"file-data\" " ":timeout-ms 111 " - ":output-format :edn}")) + ":login-timeout-ms 444 " + ":logout-timeout-ms 555 " + ":output-format :edn " + ":auth-token \"file-secret\"}")) env {"LOGSEQ_CLI_GRAPH" "env-repo" "LOGSEQ_CLI_DATA_DIR" "env-data" "LOGSEQ_CLI_TIMEOUT_MS" "222" + "LOGSEQ_CLI_LOGIN_TIMEOUT_MS" "666" + "LOGSEQ_CLI_LOGOUT_TIMEOUT_MS" "777" "LOGSEQ_CLI_OUTPUT" "json"} opts {:config-path cfg-path :graph "cli-repo" :data-dir "cli-data" :timeout-ms 333 + :login-timeout-ms 888 + :logout-timeout-ms 999 :output-format :human} result (with-env env #(config/resolve-config opts))] (is (= cfg-path (:config-path result))) (is (= "cli-repo" (:graph result))) (is (= "cli-data" (:data-dir result))) (is (= 333 (:timeout-ms result))) + (is (= 888 (:login-timeout-ms result))) + (is (= 999 (:logout-timeout-ms result))) (is (nil? (:auth-token result))) (is (nil? (:retries result))) (is (= :human (:output-format result))))) @@ -82,7 +91,10 @@ (let [result (config/resolve-config {}) expected-config-path (node-path/join (.homedir os) "logseq" "cli.edn")] (is (= expected-config-path (:config-path result))) - (is (= "~/logseq/graphs" (:data-dir result))))) + (is (= "~/logseq/graphs" (:data-dir result))) + (is (= 10000 (:timeout-ms result))) + (is (= 300000 (:login-timeout-ms result))) + (is (= 120000 (:logout-timeout-ms result))))) (deftest test-update-config (let [dir (node-helper/create-tmp-dir "cli") @@ -96,7 +108,7 @@ (deftest test-update-config-strips-removed-options (let [dir (node-helper/create-tmp-dir "cli") cfg-path (node-path/join dir "cli.edn") - _ (fs/writeFileSync cfg-path "{:graph \"old\"}") + _ (fs/writeFileSync cfg-path "{:graph \"old\" :auth-token \"legacy-secret\"}") _ (config/update-config! {:config-path cfg-path} {:graph "new" :auth-token "secret" @@ -104,7 +116,7 @@ contents (.toString (fs/readFileSync cfg-path) "utf8") parsed (reader/read-string contents)] (is (= "new" (:graph parsed))) - (is (= "secret" (:auth-token parsed))) + (is (not (contains? parsed :auth-token))) (is (not (contains? parsed :retries))))) (deftest test-update-config-removes-nil-values diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index ca46565244..134af11084 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -341,18 +341,6 @@ (is (string/includes? result "Sync download")) (is (string/includes? result "demo-graph"))))) -(deftest test-human-output-sync-config-get-token-redaction - (testing "sync config get auth-token redacts value in human output" - (let [token "super-secret-token-value" - result (format/format-result {:status :ok - :command :sync-config-get - :data {:key :auth-token - :value token}} - {:output-format nil})] - (is (string/includes? result "auth-token")) - (is (string/includes? result "[REDACTED]")) - (is (not (string/includes? result token)))))) - (deftest test-human-output-sync-config-get-e2ee-password-redaction (testing "sync config get e2ee-password redacts value in human output" (let [password "super-secret-password" @@ -365,6 +353,46 @@ (is (string/includes? result "[REDACTED]")) (is (not (string/includes? result password)))))) +(deftest test-human-output-auth-commands + (testing "login human output reports auth path and user metadata without tokens" + (let [token "secret-token-value" + result (format/format-result {:status :ok + :command :login + :data {:auth-path "/tmp/auth.json" + :email "user@example.com" + :sub "user-123" + :authorize-url "https://example.com/oauth2/authorize?..." + :id-token token}} + {:output-format nil})] + (is (string/includes? result "Login successful")) + (is (string/includes? result "user@example.com")) + (is (string/includes? result "/tmp/auth.json")) + (is (not (string/includes? result token))))) + + (testing "logout human output reports whether auth was removed" + (let [result (format/format-result {:status :ok + :command :logout + :data {:auth-path "/tmp/auth.json" + :deleted? true + :opened? true + :logout-completed? true}} + {:output-format nil})] + (is (string/includes? result "Logged out")) + (is (string/includes? result "/tmp/auth.json")) + (is (string/includes? result "Cognito logout: completed")))) + + (testing "logout human output is still successful when auth file is absent" + (let [result (format/format-result {:status :ok + :command :logout + :data {:auth-path "/tmp/auth.json" + :deleted? false + :opened? true + :logout-completed? true}} + {:output-format nil})] + (is (string/includes? result "Already logged out")) + (is (string/includes? result "/tmp/auth.json")) + (is (string/includes? result "Cognito logout: completed"))))) + (deftest test-human-output-graph-info (testing "graph info includes key metadata lines and kv section" (let [result (format/format-result {:status :ok diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index fca33a55dd..c9e552e61d 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -7,6 +7,7 @@ [clojure.string :as string] [frontend.test.node-helper :as node-helper] [frontend.worker.db-worker-node-lock :as db-lock] + [logseq.cli.auth :as cli-auth] [logseq.cli.command.core :as command-core] [logseq.cli.command.show :as show-command] [logseq.cli.config :as cli-config] @@ -185,6 +186,136 @@ [payload] (first (get-in payload [:data :result]))) +(defn- sample-auth + ([] + (sample-auth {})) + ([overrides] + (merge {:provider "cognito" + :id-token "id-token-1" + :access-token "access-token-1" + :refresh-token "refresh-token-1" + :expires-at (+ (js/Date.now) 3600000) + :sub "user-123" + :email "user@example.com" + :updated-at 1735686000000} + overrides))) + +(deftest test-cli-login-integration + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-login-data") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + auth-path (node-path/join (node-helper/create-tmp-dir "cli-auth") "auth.json") + open-calls (atom []) + auth-data (sample-auth)] + (fs/writeFileSync cfg-path "{:output-format :json}") + (let [promise + (p/with-redefs [cli-auth/default-auth-path (fn [] auth-path) + cli-auth/open-browser! (fn [authorize-url] + (swap! open-calls conj authorize-url) + (let [parsed (js/URL. authorize-url) + redirect-uri (.get (.-searchParams parsed) "redirect_uri") + state (.get (.-searchParams parsed) "state")] + (-> (js/fetch (str redirect-uri "?code=integration-code&state=" state)) + (p/then (fn [_] + {:opened? true}))))) + cli-auth/exchange-code-for-auth! (fn [_opts payload] + (is (= "integration-code" (:code payload))) + (p/resolved auth-data))] + (p/let [result (run-cli ["login"] data-dir cfg-path) + payload (parse-json-output-safe result "login") + stored (cli-auth/read-auth-file {:auth-path auth-path})] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (= auth-path (get-in payload [:data :auth-path]))) + (is (= "user@example.com" (get-in payload [:data :email]))) + (is (= 1 (count @open-calls))) + (is (= auth-data stored)) + (is (fs/existsSync auth-path))))] + (-> promise + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (done)))))))) + +(deftest test-cli-logout-integration + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-logout-data") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + auth-path (node-path/join (node-helper/create-tmp-dir "cli-auth") "auth.json") + open-calls (atom [])] + (fs/writeFileSync cfg-path "{:output-format :json}") + (cli-auth/write-auth-file! {:auth-path auth-path} (sample-auth)) + (let [promise + (p/with-redefs [cli-auth/default-auth-path (fn [] auth-path) + cli-auth/open-browser! (fn [url] + (swap! open-calls conj url) + (let [parsed (js/URL. url) + logout-uri (.get (.-searchParams parsed) "logout_uri")] + (-> (js/fetch logout-uri) + (p/then (fn [_] + {:opened? true})))))] + (p/let [result (run-cli ["logout"] data-dir cfg-path) + payload (parse-json-output-safe result "logout")] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (= 1 (count @open-calls))) + (is (= auth-path (get-in payload [:data :auth-path]))) + (is (= true (get-in payload [:data :deleted?]))) + (is (= true (get-in payload [:data :opened?]))) + (is (= true (get-in payload [:data :logout-completed?]))) + (is (not (fs/existsSync auth-path)))))] + (-> promise + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (done)))))))) + +(deftest test-cli-sync-remote-graphs-refreshes-auth-file-and-injects-runtime-token + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-sync-auth") + cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + auth-path (node-path/join (node-helper/create-tmp-dir "cli-auth") "auth.json") + invoke-calls (atom []) + expired-auth (sample-auth {:id-token "expired-token" + :access-token "expired-access-token" + :expires-at 0}) + refreshed-auth (sample-auth {:id-token "fresh-token" + :access-token "fresh-access-token" + :expires-at (+ (js/Date.now) 7200000) + :updated-at 1735689600000})] + (fs/writeFileSync cfg-path "{:output-format :json}") + (cli-auth/write-auth-file! {:auth-path auth-path} expired-auth) + (let [promise + (p/with-redefs [cli-auth/default-auth-path (fn [] auth-path) + cli-auth/refresh-auth! (fn [_opts _auth-data] + (p/resolved refreshed-auth)) + cli-server/list-graphs (fn [_config] + ["demo"]) + cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) + :thread-api/db-sync-list-remote-graphs + (p/resolved []) + (p/resolved nil)))] + (p/let [result (run-cli ["sync" "remote-graphs"] data-dir cfg-path) + payload (parse-json-output-safe result "sync remote-graphs") + stored (cli-auth/read-auth-file {:auth-path auth-path})] + (is (= 0 (:exit-code result))) + (is (= "ok" (:status payload))) + (is (= :thread-api/set-db-sync-config (ffirst @invoke-calls))) + (is (= "fresh-token" (get-in (nth @invoke-calls 0) [2 0 :auth-token]))) + (is (= "fresh-token" (:id-token stored))) + (is (= "fresh-access-token" (:access-token stored)))))] + (-> promise + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (done)))))))) + (deftest test-cli-sync-download-and-start-readiness-with-mocked-sync (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-cli") @@ -199,7 +330,9 @@ _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) [download-result start-result] - (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/with-redefs [cli-auth/resolve-auth-token! (fn [_config] + (p/resolved "runtime-token")) + cli-server/ensure-server! (fn [config _repo] (p/resolved (assoc config :base-url "http://example"))) transport/invoke (fn [_ method _direct-pass? args] (swap! invoke-calls conj [method args]) @@ -268,7 +401,9 @@ _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) upload-result (p/with-redefs - [cli-server/ensure-server! (fn [config _repo] + [cli-auth/resolve-auth-token! (fn [_config] + (p/resolved "runtime-token")) + cli-server/ensure-server! (fn [config _repo] (p/resolved (assoc config :base-url "http://example"))) transport/invoke (fn [_ method _direct-pass? args] (swap! invoke-calls conj [method args]) @@ -287,7 +422,7 @@ (is (= "created-graph-id" (get-in upload-payload [:data :graph-id]))) (is (= [[:thread-api/set-db-sync-config [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-upload-graph ["logseq_db_sync-upload-graph"]]] @invoke-calls))) @@ -308,7 +443,9 @@ _ (is (= 0 (:exit-code create-result))) _ (is (= "ok" (:status create-payload))) [upload-result info-result] - (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/with-redefs [cli-auth/resolve-auth-token! (fn [_config] + (p/resolved "runtime-token")) + cli-server/ensure-server! (fn [config _repo] (p/resolved (assoc config :base-url "http://example"))) transport/invoke (fn [_ method _direct-pass? args] (swap! invoke-calls conj [method args]) diff --git a/src/test/logseq/common/cognito_config_test.cljs b/src/test/logseq/common/cognito_config_test.cljs new file mode 100644 index 0000000000..cea2e91fe1 --- /dev/null +++ b/src/test/logseq/common/cognito_config_test.cljs @@ -0,0 +1,15 @@ +(ns logseq.common.cognito-config-test + (:require [cljs.test :refer [deftest is]] + [frontend.config :as config] + [logseq.common.cognito-config :as cognito-config] + ["fs" :as fs])) + +(deftest test-shared-cognito-config-matches-frontend-config + (is (= config/LOGIN-URL cognito-config/LOGIN-URL)) + (is (= config/COGNITO-CLIENT-ID cognito-config/COGNITO-CLIENT-ID)) + (is (= config/OAUTH-DOMAIN cognito-config/OAUTH-DOMAIN))) + +(deftest test-logseq-cli-build-enables-prod-file-sync-by-default + (let [shadow-config (.toString (fs/readFileSync "shadow-cljs.edn") "utf8")] + (is (re-find #"(?s):logseq-cli\s+\{:target :node-script.*?logseq\.common\.cognito-config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env \[\"ENABLE_FILE_SYNC_PRODUCTION\" :as :bool :default true\]" + shadow-config)))) From b45a3a8b80da5964604349ddc3d91cb23065a72e Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 09:51:34 -0400 Subject: [PATCH 120/375] fix: common lint --- deps/common/.carve/config.edn | 1 + 1 file changed, 1 insertion(+) diff --git a/deps/common/.carve/config.edn b/deps/common/.carve/config.edn index 954cbd2144..6889804745 100644 --- a/deps/common/.carve/config.edn +++ b/deps/common/.carve/config.edn @@ -8,6 +8,7 @@ logseq.common.util.date-time logseq.common.date logseq.common.util.macro + logseq.common.cognito-config logseq.common.config logseq.common.defkeywords] :report {:format :ignore}} From c6a6156188d841a58596c09be78f74ab6c601323 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 10:35:00 -0400 Subject: [PATCH 121/375] fix: CLI integration tests failing in CI Also split out integration tests to separate step and command as they have their own setup --- .github/workflows/build.yml | 9 ++++++++- docs/cli/logseq-cli.md | 2 +- package.json | 3 ++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index c2bc61f315..ee69421e93 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -81,9 +81,16 @@ jobs: - name: Run ClojureScript query tests against basic query type run: DB_QUERY_TYPE=basic node static/tests.js -r frontend.db.query-dsl-test - - name: Run ClojureScript tests + - name: Run ClojureScript unit tests run: yarn cljs:run-test + - name: Setup CLI integration test + # logseq-cli.js needed just for test-cli-query-human-output-pipes-to-show + run: clojure -M:cljs compile logseq-cli && yarn db-worker-node:release:bundle + + - name: Run ClojureScript integration tests + run: yarn cljs:run-integration-test + lint: runs-on: ubuntu-22.04 diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index b900b63274..8e842e8c6f 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -5,7 +5,7 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to ## Build the CLI ```bash -clojure -M:cljs compile logseq-cli db-worker-node +clojure -M:cljs compile logseq-cli yarn db-worker-node:release:bundle ``` diff --git a/package.json b/package.json index 296d217187..c254804363 100644 --- a/package.json +++ b/package.json @@ -103,7 +103,8 @@ "cljs:release-app": "clojure -M:cljs release app db-worker db-worker-node inference-worker", "cljs:release-publishing": "clojure -M:cljs release app publishing", "cljs:test": "clojure -M:test compile test", - "cljs:run-test": "node static/tests.js -r '^(?!logseq.db-sync.).*' -e fix-me", + "cljs:run-test": "node static/tests.js -r '^(?!(?:logseq.db-sync.|logseq.cli.integration-test)).*' -e fix-me", + "cljs:run-integration-test": "node static/tests.js -r '^(logseq.cli.integration-test).*'", "cljs:test-no-worker": "clojure -M:test compile test-no-worker", "cljs:run-test-no-worker": "node static/tests-no-worker.js", "db-worker-node:compile": "clojure -M:cljs compile db-worker-node", From 9c2b45de911aff900790a72f28772503bc9749b2 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 10:56:37 -0400 Subject: [PATCH 122/375] fix: cli doctor tests Unit test shouldn't depend on db-worker-node assets. Removed one test as it tests usage of :script-path which is already done --- src/main/logseq/cli/command/doctor.cljs | 2 +- src/test/logseq/cli/command/doctor_test.cljs | 30 +++++++------------- 2 files changed, 12 insertions(+), 20 deletions(-) diff --git a/src/main/logseq/cli/command/doctor.cljs b/src/main/logseq/cli/command/doctor.cljs index 0f3adb8f55..b41bee879e 100644 --- a/src/main/logseq/cli/command/doctor.cljs +++ b/src/main/logseq/cli/command/doctor.cljs @@ -32,7 +32,7 @@ :data {:status :error :checks checks}}) -(defn- check-db-worker-script +(defn check-db-worker-script [action] (let [path (or (:script-path action) (cli-server/db-worker-runtime-script-path))] diff --git a/src/test/logseq/cli/command/doctor_test.cljs b/src/test/logseq/cli/command/doctor_test.cljs index 37b27984a5..df7b33b596 100644 --- a/src/test/logseq/cli/command/doctor_test.cljs +++ b/src/test/logseq/cli/command/doctor_test.cljs @@ -4,7 +4,8 @@ [logseq.cli.commands :as commands] [logseq.cli.data-dir :as data-dir] [logseq.cli.server :as cli-server] - [promesa.core :as p])) + [promesa.core :as p] + [logseq.cli.command.doctor :as doctor-command])) (deftest test-execute-doctor-script-missing (async done @@ -99,28 +100,19 @@ (deftest test-execute-doctor-default-script-checks-packaged-runtime-target (async done (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") - cli-server/list-servers (fn [_] (p/resolved []))] + cli-server/list-servers (fn [_] (p/resolved [])) + doctor-command/check-db-worker-script + (fn [_] + {:ok? true + :check {:id :db-worker-script + :status :ok + :path "/dist/db-worker-node.js" + :message (str "Found readable file: " "/dist/db-worker-node.js")}})] (p/let [result (commands/execute {:type :doctor} {:data-dir "/tmp/logseq-doctor"}) checked-path (get-in result [:data :checks 0 :path])] (is (= :ok (:status result))) - (is (= (cli-server/db-worker-script-path) checked-path)) (is (string/ends-with? checked-path "/dist/db-worker-node.js")))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) - (p/finally done)))) - -(deftest test-execute-doctor-explicit-script-path-checks-static-runtime-target - (async done - (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") - cli-server/list-servers (fn [_] (p/resolved []))] - (p/let [result (commands/execute {:type :doctor - :script-path (cli-server/db-worker-dev-script-path)} - {:data-dir "/tmp/logseq-doctor"}) - checked-path (get-in result [:data :checks 0 :path])] - (is (= :ok (:status result))) - (is (= (cli-server/db-worker-dev-script-path) checked-path)) - (is (string/ends-with? checked-path "/static/db-worker-node.js")))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally done)))) + (p/finally done)))) \ No newline at end of file From fab20b1081a3ecddc1132add0d28c2470b78e84f Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 12:03:59 -0400 Subject: [PATCH 123/375] fix: Unlinked graphs in graph list --- src/main/frontend/worker/platform/node.cljs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index 0f53316c96..285f77b5f8 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -1,7 +1,7 @@ (ns frontend.worker.platform.node "Node.js platform adapter for db-worker." - (:require ["node:sqlite" :as node-sqlite] - ["fs/promises" :as fs] + (:require ["fs/promises" :as fs] + ["node:sqlite" :as node-sqlite] ["os" :as os] ["path" :as node-path] ["ws" :as ws] @@ -10,6 +10,7 @@ [frontend.worker.db-worker-node-lock :as db-lock] [goog.object :as gobj] [lambdaisland.glogi :as log] + [logseq.common.config :as common-config] [promesa.core :as p])) (defn- resolve-database-sync-ctor @@ -72,6 +73,7 @@ (db-lock/decode-canonical-graph-dir-key (.-name dirent))) db-dirs)] (->> graph-names + (remove #(= % common-config/unlinked-graphs-dir)) (filter some?) (vec))))) From 6d2be5a87eeee671a5b8cf8a31154e728b55dd34 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 12:22:59 -0400 Subject: [PATCH 124/375] chore: bump to latest nbb-logseq bump to match latest datascript changes in this branch --- deps/cli/package.json | 2 +- deps/cli/yarn.lock | 6 +++--- deps/common/package.json | 2 +- deps/common/yarn.lock | 6 +++--- deps/db/package.json | 2 +- deps/db/yarn.lock | 6 +++--- deps/graph-parser/package.json | 2 +- deps/graph-parser/yarn.lock | 6 +++--- deps/outliner/package.json | 2 +- deps/outliner/yarn.lock | 6 +++--- deps/publishing/package.json | 2 +- deps/publishing/yarn.lock | 6 +++--- scripts/package.json | 2 +- scripts/yarn.lock | 6 +++--- 14 files changed, 28 insertions(+), 28 deletions(-) diff --git a/deps/cli/package.json b/deps/cli/package.json index 05615b9ca0..a408822def 100644 --- a/deps/cli/package.json +++ b/deps/cli/package.json @@ -10,7 +10,7 @@ }, "license": "MIT", "dependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31", + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32", "@modelcontextprotocol/sdk": "^1.17.5", "better-sqlite3": "~11.10.0", "fastify": "5.3.2", diff --git a/deps/cli/yarn.lock b/deps/cli/yarn.lock index 30fe2bf9ed..8899d4d157 100644 --- a/deps/cli/yarn.lock +++ b/deps/cli/yarn.lock @@ -43,9 +43,9 @@ "@fastify/forwarded" "^3.0.0" ipaddr.js "^2.1.0" -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/deps/common/package.json b/deps/common/package.json index b4a433c952..4d66890410 100644 --- a/deps/common/package.json +++ b/deps/common/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31" + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32" }, "scripts": { "test": "yarn nbb-logseq -cp test -m nextjournal.test-runner" diff --git a/deps/common/yarn.lock b/deps/common/yarn.lock index 5d5610005c..838aeb6e9c 100644 --- a/deps/common/yarn.lock +++ b/deps/common/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/deps/db/package.json b/deps/db/package.json index 2683b101a3..ab73583db8 100644 --- a/deps/db/package.json +++ b/deps/db/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31", + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32", "fs-extra": "^11.3.0" }, "dependencies": { diff --git a/deps/db/yarn.lock b/deps/db/yarn.lock index 9a5698efd3..226131458a 100644 --- a/deps/db/yarn.lock +++ b/deps/db/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/deps/graph-parser/package.json b/deps/graph-parser/package.json index 95e91d2934..7181a8a3ff 100644 --- a/deps/graph-parser/package.json +++ b/deps/graph-parser/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31", + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32", "better-sqlite3": "11.10.0" }, "dependencies": { diff --git a/deps/graph-parser/yarn.lock b/deps/graph-parser/yarn.lock index a21ee96d23..d0cb820485 100644 --- a/deps/graph-parser/yarn.lock +++ b/deps/graph-parser/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/deps/outliner/package.json b/deps/outliner/package.json index 3b468a0ed1..4d4ce77779 100644 --- a/deps/outliner/package.json +++ b/deps/outliner/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31" + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32" }, "dependencies": { "better-sqlite3": "11.10.0", diff --git a/deps/outliner/yarn.lock b/deps/outliner/yarn.lock index 3fff63e2e8..7bf27489da 100644 --- a/deps/outliner/yarn.lock +++ b/deps/outliner/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/deps/publishing/package.json b/deps/publishing/package.json index 4690c436a1..59ab3a1f32 100644 --- a/deps/publishing/package.json +++ b/deps/publishing/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31", + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32", "mldoc": "^1.5.9" }, "dependencies": { diff --git a/deps/publishing/yarn.lock b/deps/publishing/yarn.lock index 9ac029e25f..43fd54cccf 100644 --- a/deps/publishing/yarn.lock +++ b/deps/publishing/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" diff --git a/scripts/package.json b/scripts/package.json index d30efa66b7..648923b6ef 100644 --- a/scripts/package.json +++ b/scripts/package.json @@ -3,7 +3,7 @@ "version": "1.0.0", "private": true, "devDependencies": { - "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v31" + "@logseq/nbb-logseq": "github:logseq/nbb-logseq#feat-db-v32" }, "dependencies": { "better-sqlite3": "11.10.0", diff --git a/scripts/yarn.lock b/scripts/yarn.lock index 9ac029e25f..43fd54cccf 100644 --- a/scripts/yarn.lock +++ b/scripts/yarn.lock @@ -2,9 +2,9 @@ # yarn lockfile v1 -"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v31": - version "1.2.173-feat-db-v31" - resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/d5b76a675f484dbfb5fbff8235aec6ff84b2f980" +"@logseq/nbb-logseq@github:logseq/nbb-logseq#feat-db-v32": + version "1.2.173-feat-db-v32" + resolved "https://codeload.github.com/logseq/nbb-logseq/tar.gz/325d79eade096ce8f70fbe91e10fb3d8e41db345" dependencies: import-meta-resolve "^4.1.0" From 7032e6ba37f4f59d18b87d885256bc0ed14cf2d3 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 15:11:52 -0400 Subject: [PATCH 125/375] refactor: centralize default graphs dir spread across cli and electron Having different definitions doesn't allow for configurability and can lead to bugs. Also addressed sqlite-cli having one-off definition --- bb.edn | 2 +- deps/cli/src/logseq/cli/common/graph.cljs | 4 +--- deps/common/src/logseq/common/config.cljs | 1 + deps/common/src/logseq/common/graph.cljs | 13 +++++++++++++ deps/db/src/logseq/db/common/sqlite_cli.cljs | 5 ++--- src/main/frontend/worker/db_worker_node.cljs | 3 ++- src/main/frontend/worker/db_worker_node_lock.cljs | 5 +++-- src/main/frontend/worker/platform/node.cljs | 3 ++- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/config.cljs | 5 +++-- src/main/logseq/cli/data_dir.cljs | 13 ++----------- src/main/logseq/cli/server.cljs | 14 ++++---------- 12 files changed, 35 insertions(+), 35 deletions(-) diff --git a/bb.edn b/bb.edn index 3bea2155f3..03118dc0a9 100644 --- a/bb.edn +++ b/bb.edn @@ -70,7 +70,7 @@ dev:cli {:doc "Run CLI with current deps/db code. Commands with JS deps are not usable e.g. mcp-server" :task (apply shell {:dir "deps/db"} - "yarn nbb-logseq -cp src:../cli/src:../graph-parser/src:../outliner/src -m logseq.cli" *command-line-args*)} + "yarn -s nbb-logseq -cp src:../cli/src:../graph-parser/src:../outliner/src -m logseq.cli" *command-line-args*)} dev:query {:doc "Query a DB graph's datascript db" :requires ([babashka.fs :as fs]) diff --git a/deps/cli/src/logseq/cli/common/graph.cljs b/deps/cli/src/logseq/cli/common/graph.cljs index d8f636cf09..4a310199f5 100644 --- a/deps/cli/src/logseq/cli/common/graph.cljs +++ b/deps/cli/src/logseq/cli/common/graph.cljs @@ -1,8 +1,6 @@ (ns ^:node-only logseq.cli.common.graph "Graph related fns shared between CLI and electron" (:require ["fs-extra" :as fs-extra] - ["os" :as os] - ["path" :as node-path] [clojure.string :as string] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph])) @@ -17,7 +15,7 @@ (defn get-db-graphs-dir "Directory where DB graphs are stored" [] - (node-path/join (os/homedir) "logseq" "graphs")) + (common-graph/expand-home (common-graph/get-default-graphs-dir))) (defn get-db-based-graphs [] diff --git a/deps/common/src/logseq/common/config.cljs b/deps/common/src/logseq/common/config.cljs index da4fe27bd6..126cd38499 100644 --- a/deps/common/src/logseq/common/config.cljs +++ b/deps/common/src/logseq/common/config.cljs @@ -52,6 +52,7 @@ name'))] (str db-version-prefix stripped)))) +(defonce default-graphs-dir "~/logseq/graphs") (defonce local-assets-dir "assets") (defonce unlinked-graphs-dir "Unlinked graphs") diff --git a/deps/common/src/logseq/common/graph.cljs b/deps/common/src/logseq/common/graph.cljs index b255ac3b59..487e99a4c2 100644 --- a/deps/common/src/logseq/common/graph.cljs +++ b/deps/common/src/logseq/common/graph.cljs @@ -1,8 +1,10 @@ (ns ^:node-only logseq.common.graph "This ns provides common fns for a graph directory and only runs in a node environment" (:require ["fs" :as fs] + ["os" :as os] ["path" :as node-path] [clojure.string :as string] + [logseq.common.config :as common-config] [logseq.common.path :as path])) (def ^:private win32? @@ -97,3 +99,14 @@ Rules: (->> (readdir graph-dir) (remove (partial ignored-path? graph-dir)) (filter #(contains? allowed-formats (get-ext %))))) + +(defn get-default-graphs-dir + [] + common-config/default-graphs-dir) + +(defn expand-home + "Expands path if it starts with '~'" + [path] + (if (and (seq path) (string/starts-with? path "~")) + (node-path/join (os/homedir) (subs path 1)) + path)) diff --git a/deps/db/src/logseq/db/common/sqlite_cli.cljs b/deps/db/src/logseq/db/common/sqlite_cli.cljs index 776d5431fe..44fbf5b6c2 100644 --- a/deps/db/src/logseq/db/common/sqlite_cli.cljs +++ b/deps/db/src/logseq/db/common/sqlite_cli.cljs @@ -1,11 +1,11 @@ (ns ^:node-only logseq.db.common.sqlite-cli "Primary ns to interact with DB files with node.js based CLIs" (:require ["better-sqlite3" :as sqlite3] - ["os" :as os] ["path" :as node-path] [cljs-bean.core :as bean] [clojure.string :as string] [datascript.storage :refer [IStorage]] + [logseq.common.graph :as common-graph] [logseq.db.common.sqlite :as common-sqlite] [logseq.db.frontend.schema :as db-schema] [logseq.db.sqlite.util :as sqlite-util])) @@ -100,5 +100,4 @@ ;; $ORIGINAL_PWD used by bb tasks to correct current dir (node-path/join (or js/process.env.ORIGINAL_PWD ".") %))] ((juxt node-path/dirname node-path/basename) (resolve-path' graph-dir-or-path))) - ;; TODO: Reuse with get-db-graphs-dir when there is a db ns that is usable by electron i.e. no better-sqlite3 - [(node-path/join (os/homedir) "logseq" "graphs") graph-dir-or-path]))) + [(common-graph/expand-home (common-graph/get-default-graphs-dir)) graph-dir-or-path]))) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 47add71381..6c5b789cbe 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -10,6 +10,7 @@ [frontend.worker.state :as worker-state] [lambdaisland.glogi :as log] [logseq.cli.style :as style] + [logseq.common.config :as common-config] [logseq.cli.data-dir :as data-dir] [logseq.db :as ldb] [promesa.core :as p])) @@ -282,7 +283,7 @@ (defn- show-help! [] (println (str (style/bold "db-worker-node") " " (style/bold "options") ":")) - (println (str " " (style/bold "--data-dir") " (default ~/logseq/graphs)")) + (println (str " " (style/bold "--data-dir") " (default " common-config/default-graphs-dir ")")) (println (str " " (style/bold "--repo") " (required)")) (println (str " " (style/bold "--create-empty-db") " (start with empty initial datoms)")) (println (str " " (style/bold "--log-level") " (default info)")) diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index cf68a4f7e7..613c19f7de 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -4,10 +4,11 @@ ["os" :as os] ["path" :as node-path] [clojure.string :as string] - [frontend.worker.graph-dir :as graph-dir] [frontend.worker-common.util :as worker-util] + [frontend.worker.graph-dir :as graph-dir] [lambdaisland.glogi :as log] [logseq.common.config :as common-config] + [logseq.common.graph :as common-graph] [promesa.core :as p])) (defn- expand-home @@ -18,7 +19,7 @@ (defn resolve-data-dir [data-dir] - (expand-home (or data-dir "~/logseq/graphs"))) + (expand-home (or data-dir common-graph/get-default-graphs-dir))) (defn repo->graph-dir-key [repo] diff --git a/src/main/frontend/worker/platform/node.cljs b/src/main/frontend/worker/platform/node.cljs index 285f77b5f8..31159c5040 100644 --- a/src/main/frontend/worker/platform/node.cljs +++ b/src/main/frontend/worker/platform/node.cljs @@ -11,6 +11,7 @@ [goog.object :as gobj] [lambdaisland.glogi :as log] [logseq.common.config :as common-config] + [logseq.common.graph :as common-graph] [promesa.core :as p])) (defn- resolve-database-sync-ctor @@ -299,7 +300,7 @@ (defn node-platform [{:keys [data-dir event-fn write-guard-fn owner-source]}] - (let [data-dir (expand-home (or data-dir "~/logseq/graphs")) + (let [data-dir (expand-home (or data-dir (common-graph/get-default-graphs-dir))) owner-source (db-lock/normalize-owner-source owner-source) kv (kv-store data-dir)] (p/do! diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index c0134a6807..520c3a70b3 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -15,7 +15,7 @@ :alias :c} :graph {:desc "Graph name" :alias :g} - :data-dir {:desc "Path to db-worker data dir (default ~/logseq/graphs)"} + :data-dir {:desc (str "Path to db-worker data dir (default " common-config/default-graphs-dir ")")} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} :output {:desc "Output format (human, json, edn). Default: human" diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 4874c7d6c6..97865d5fb3 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -5,7 +5,8 @@ [goog.object :as gobj] ["fs" :as fs] ["os" :as os] - ["path" :as node-path])) + ["path" :as node-path] + [logseq.common.graph :as common-graph])) (defn- parse-int [value] @@ -98,7 +99,7 @@ :login-timeout-ms 300000 :logout-timeout-ms 120000 :output-format nil - :data-dir "~/logseq/graphs" + :data-dir (common-graph/get-default-graphs-dir) :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) diff --git a/src/main/logseq/cli/data_dir.cljs b/src/main/logseq/cli/data_dir.cljs index ccaaf0f385..96aa22fb69 100644 --- a/src/main/logseq/cli/data_dir.cljs +++ b/src/main/logseq/cli/data_dir.cljs @@ -1,21 +1,12 @@ (ns logseq.cli.data-dir "Data-dir validation and normalization for the CLI and db-worker-node." (:require ["fs" :as fs] - ["os" :as os] ["path" :as node-path] - [clojure.string :as string])) - -(def ^:private default-data-dir "~/logseq/graphs") - -(defn- expand-home - [path] - (if (and (seq path) (string/starts-with? path "~")) - (node-path/join (.homedir os) (subs path 1)) - path)) + [logseq.common.graph :as common-graph])) (defn normalize-data-dir [path] - (node-path/resolve (expand-home (or path default-data-dir)))) + (node-path/resolve (common-graph/expand-home (or path (common-graph/get-default-graphs-dir))))) (defn ensure-data-dir! [path] diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index f5fb089003..3eedc50ba8 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -1,24 +1,18 @@ (ns logseq.cli.server "db-worker-node lifecycle orchestration for logseq." (:require ["fs" :as fs] - ["os" :as os] ["path" :as node-path] [clojure.string :as string] [frontend.worker.db-worker-node-lock :as db-lock] - [logseq.common.config :as common-config] - [logseq.db-worker.daemon :as daemon] [lambdaisland.glogi :as log] + [logseq.common.config :as common-config] + [logseq.common.graph :as common-graph] + [logseq.db-worker.daemon :as daemon] [promesa.core :as p])) -(defn- expand-home - [path] - (if (string/starts-with? path "~") - (node-path/join (.homedir os) (subs path 1)) - path)) - (defn resolve-data-dir [config] - (expand-home (or (:data-dir config) "~/logseq/graphs"))) + (common-graph/expand-home (or (:data-dir config) (common-graph/get-default-graphs-dir)))) (defn- repo-dir [data-dir repo] From a081d01c6ddc352c00cafa2b80ac92a548034fff Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 15:16:26 -0400 Subject: [PATCH 126/375] enhance: allow graphs dir to change with $LOGSEQ_GRAPHS_DIR Affects electron and cli thanks to previous commit. Requested at least twice by #db-feedback posts: https://discord.com/channels/725182569297215569/1421133499477524490 and https://discord.com/channels/725182569297215569/1353859592634892400/1353859592634892400 --- deps/common/src/logseq/common/graph.cljs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/deps/common/src/logseq/common/graph.cljs b/deps/common/src/logseq/common/graph.cljs index 487e99a4c2..1cd38ae736 100644 --- a/deps/common/src/logseq/common/graph.cljs +++ b/deps/common/src/logseq/common/graph.cljs @@ -101,8 +101,9 @@ Rules: (filter #(contains? allowed-formats (get-ext %))))) (defn get-default-graphs-dir + "Get default dir for storing graphs by first looking in env var." [] - common-config/default-graphs-dir) + (or js/process.env.LOGSEQ_GRAPHS_DIR common-config/default-graphs-dir)) (defn expand-home "Expands path if it starts with '~'" From 83b7cc8e3ee5b55ad5f570be19acf962f1e9f8af Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Mon, 9 Mar 2026 15:55:22 -0400 Subject: [PATCH 127/375] fix: resolve-data-dir - caught by test Also fix lint and regression on noisy db-worker-node-test --- deps/common/.carve/ignore | 4 ++++ src/main/frontend/worker/db_worker_node_lock.cljs | 2 +- src/test/frontend/worker/db_worker_node_test.cljs | 8 ++++---- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/deps/common/.carve/ignore b/deps/common/.carve/ignore index e70a067333..2d0fac603b 100644 --- a/deps/common/.carve/ignore +++ b/deps/common/.carve/ignore @@ -3,6 +3,10 @@ logseq.common.graph/get-files ;; API fn logseq.common.graph/read-directories ;; API fn +logseq.common.graph/get-default-graphs-dir +;; API fn +logseq.common.graph/expand-home +;; API fn logseq.common.authorization/verify-jwt ;; Profile utils diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index 613c19f7de..e04a617ab3 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -19,7 +19,7 @@ (defn resolve-data-dir [data-dir] - (expand-home (or data-dir common-graph/get-default-graphs-dir))) + (expand-home (or data-dir (common-graph/get-default-graphs-dir)))) (defn repo->graph-dir-key [repo] diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index 0f6de7c617..f7117456d5 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -501,8 +501,8 @@ data-dir (node-helper/create-tmp-dir "db-worker-sync-status") repo (str "logseq_db_sync_status_" (subs (str (random-uuid)) 0 8))] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:host host :port port :stop! stop!}) {:keys [status body]} (invoke-raw host port "thread-api/db-sync-status" []) parsed (js->clj (js/JSON.parse body) :keywordize-keys true) @@ -532,8 +532,8 @@ data-dir (node-helper/create-tmp-dir "db-worker-sync-start") repo (str "logseq_db_sync_start_" (subs (str (random-uuid)) 0 8))] (-> (p/let [{:keys [host port stop!]} - (db-worker-node/start-daemon! {:data-dir data-dir - :repo repo}) + (start-daemon! {:data-dir data-dir + :repo repo}) _ (reset! daemon {:host host :port port :stop! stop!}) _ (invoke host port "thread-api/create-or-open-db" [repo {}]) _ (invoke host port "thread-api/set-db-sync-config" From 64d48b213c3915ef86d8b5ed57c7230d050631ec Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 10 Mar 2026 10:27:24 -0400 Subject: [PATCH 128/375] fix: 'Remove graph' command in electron deletes graph instead of moving to Unlinked graphs --- src/main/frontend/db/persist.cljs | 8 +++++--- src/test/frontend/db/persist_test.cljs | 22 ++++++++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/src/main/frontend/db/persist.cljs b/src/main/frontend/db/persist.cljs index 7767f5d2ca..b599ed3611 100644 --- a/src/main/frontend/db/persist.cljs +++ b/src/main/frontend/db/persist.cljs @@ -33,6 +33,8 @@ (defn delete-graph! [graph] - (p/let [_ (persist-db/ (p/with-redefs [util/electron? (constantly true) + persist-db/ Date: Tue, 10 Mar 2026 11:21:00 -0400 Subject: [PATCH 129/375] enhance: remove graph command behaves like electron remove Moves removed graph to 'Unlinked graphs' --- src/electron/electron/db.cljs | 17 +---------------- src/electron/electron/handler.cljs | 3 ++- src/main/logseq/cli/command/graph.cljs | 21 +++++++++++++++++---- src/main/logseq/cli/commands.cljs | 1 + src/main/logseq/cli/common.cljs | 25 +++++++++++++++++++++++++ src/main/logseq/cli/server.cljs | 2 +- src/test/logseq/cli/common_test.cljs | 25 +++++++++++++++++++++++++ 7 files changed, 72 insertions(+), 22 deletions(-) create mode 100644 src/main/logseq/cli/common.cljs create mode 100644 src/test/logseq/cli/common_test.cljs diff --git a/src/electron/electron/db.cljs b/src/electron/electron/db.cljs index 934d338584..78cc1e1884 100644 --- a/src/electron/electron/db.cljs +++ b/src/electron/electron/db.cljs @@ -4,7 +4,6 @@ ["path" :as node-path] [electron.backup-file :as backup-file] [logseq.cli.common.graph :as cli-common-graph] - [logseq.common.config :as common-config] [logseq.db.common.sqlite :as common-sqlite])) (defn ensure-graphs-dir! @@ -39,18 +38,4 @@ {:backups-dir backups-path :truncate-daily? true :keep-versions 12})) - (fs/writeFileSync db-path data))) - -(defn unlink-graph! - [repo] - (let [db-name (common-sqlite/sanitize-db-name repo) - path (node-path/join (cli-common-graph/get-db-graphs-dir) db-name) - unlinked (node-path/join (cli-common-graph/get-db-graphs-dir) common-config/unlinked-graphs-dir) - new-path (node-path/join unlinked db-name) - new-path-exists? (fs/existsSync new-path) - new-path' (if new-path-exists? - (node-path/join unlinked (str db-name "-" (random-uuid))) - new-path)] - (when (fs/existsSync path) - (fs/ensureDirSync unlinked) - (fs/moveSync path new-path')))) + (fs/writeFileSync db-path data))) \ No newline at end of file diff --git a/src/electron/electron/handler.cljs b/src/electron/electron/handler.cljs index 3453f88f8c..5f5f9e0ef8 100644 --- a/src/electron/electron/handler.cljs +++ b/src/electron/electron/handler.cljs @@ -29,6 +29,7 @@ [electron.window :as win] [electron.graph-switch-flow :as graph-switch-flow] [logseq.cli.common.graph :as cli-common-graph] + [logseq.cli.common :as cli-common] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] [logseq.db.sqlite.util :as sqlite-util] @@ -224,7 +225,7 @@ (defmethod handle :deleteGraph [_window [_ graph]] (when-let [repo (canonical-repo graph)] - (db/unlink-graph! repo))) + (cli-common/unlink-graph! repo))) ;; DB related IPCs start diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 3aefe9dc6c..78a9f7527c 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -3,6 +3,7 @@ (:require [cljs.pprint :as pprint] [clojure.string :as string] [logseq.cli.command.core :as core] + [logseq.cli.common :as cli-common] [logseq.cli.config :as cli-config] [logseq.cli.server :as cli-server] [logseq.cli.transport :as transport] @@ -91,11 +92,8 @@ (if-not (seq graph) (missing-graph-error) {:ok? true - :action {:type :invoke + :action {:type :graph-remove :command :graph-remove - :method :thread-api/unsafe-unlink-db - :direct-pass? false - :args [repo] :repo repo :graph (core/repo->graph repo)}}) @@ -196,6 +194,21 @@ :else {:status :ok :data {:result result}})))) +(defn execute-graph-remove + [action config] + (-> (p/let [stop-result (cli-server/stop-server! config (:repo action)) + _ (when-not (or (:ok? stop-result) + (= :server-not-found (get-in stop-result [:error :code]))) + (throw (ex-info (get-in stop-result [:error :message] "failed to stop server") + {:code (get-in stop-result [:error :code])}))) + unlinked-dir (cli-common/unlink-graph! (:repo action))] + (if unlinked-dir + {:status :ok + :data {:result nil}} + {:status :error + :error {:code :graph-not-removed + :message "unable to remove graph"}})))) + (defn execute-graph-switch [action config] (-> (p/let [graphs (cli-server/list-graphs config) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 21b178250e..e4f29e79f3 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -443,6 +443,7 @@ (case (:type action) :graph-list (graph-command/execute-graph-list action config) :invoke (graph-command/execute-invoke action config) + :graph-remove (graph-command/execute-graph-remove action config) :graph-switch (graph-command/execute-graph-switch action config) :graph-info (graph-command/execute-graph-info action config) :graph-export (graph-command/execute-graph-export action config) diff --git a/src/main/logseq/cli/common.cljs b/src/main/logseq/cli/common.cljs new file mode 100644 index 0000000000..a1809fc337 --- /dev/null +++ b/src/main/logseq/cli/common.cljs @@ -0,0 +1,25 @@ +(ns logseq.cli.common + "Common fns between CLI and electron" + (:require ["fs-extra" :as fs] + ["path" :as node-path] + [logseq.common.config :as common-config] + [logseq.common.graph :as common-graph] + [logseq.db.common.sqlite :as common-sqlite])) + +(defn unlink-graph! + "Unlinks the given repo by moving it to the 'Unlinked graphs' dir. + Returns path of unlinked dir if move is successful or nil if not" + [repo] + (let [db-name (common-sqlite/sanitize-db-name repo) + graphs-dir (common-graph/expand-home (common-graph/get-default-graphs-dir)) + path (node-path/join graphs-dir db-name) + unlinked (node-path/join graphs-dir common-config/unlinked-graphs-dir) + new-path (node-path/join unlinked db-name) + new-path-exists? (fs/existsSync new-path) + new-path' (if new-path-exists? + (node-path/join unlinked (str db-name "-" (random-uuid))) + new-path)] + (when (fs/existsSync path) + (fs/ensureDirSync unlinked) + (fs/moveSync path new-path') + new-path'))) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 3eedc50ba8..94a6b9fc8a 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -1,5 +1,5 @@ (ns logseq.cli.server - "db-worker-node lifecycle orchestration for logseq." + "db-worker-node lifecycle orchestration for logseq. Used by CLI and electron" (:require ["fs" :as fs] ["path" :as node-path] [clojure.string :as string] diff --git a/src/test/logseq/cli/common_test.cljs b/src/test/logseq/cli/common_test.cljs new file mode 100644 index 0000000000..b9616c46d5 --- /dev/null +++ b/src/test/logseq/cli/common_test.cljs @@ -0,0 +1,25 @@ +(ns logseq.cli.common-test + (:require ["fs-extra" :as fs] + ["path" :as node-path] + [cljs.test :refer [deftest is]] + [frontend.test.node-helper :as node-helper] + [logseq.cli.common :as cli-common] + [logseq.common.graph :as common-graph] + [logseq.common.config :as common-config])) + +(deftest unlink-graph-moves-to-unlinked-dir + (let [graphs-dir (node-helper/create-tmp-dir "unlink-graph") + graph-name "test-graph" + repo (str common-config/db-version-prefix graph-name) + graph-path (node-path/join graphs-dir graph-name) + unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir graph-name)] + (fs/mkdirSync graph-path #js {:recursive true}) + (fs/writeFileSync (node-path/join graph-path "db.sqlite") "test-data") + (with-redefs [common-graph/get-default-graphs-dir (fn [] graphs-dir)] + (cli-common/unlink-graph! repo) + (is (not (fs/existsSync graph-path)) + "Original graph directory should no longer exist") + (is (fs/existsSync unlinked-path) + "Graph directory should be moved to Unlinked graphs") + (is (fs/existsSync (node-path/join unlinked-path "db.sqlite")) + "Graph contents should be preserved after move")))) \ No newline at end of file From 81356c7e32266b06165b8d3171c3ae0cecdea55f Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 10 Mar 2026 13:18:55 -0400 Subject: [PATCH 130/375] fix: watched logseq-cli hangs `npx shadow-cljs watch logseq-cli` hangs. CLIs don't need hot-reload --- shadow-cljs.edn | 1 + 1 file changed, 1 insertion(+) diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 289a2e2cff..60af2cab22 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -99,6 +99,7 @@ :logseq-cli {:target :node-script :output-to "static/logseq-cli.js" :main logseq.cli.main/main + :devtools {:enabled false} :build-hooks [(shadow.hooks/logseq-cli-metadata-hook "--long --always --dirty")] :closure-defines {logseq.common.cognito-config/ENABLE-FILE-SYNC-PRODUCTION #shadow/env ["ENABLE_FILE_SYNC_PRODUCTION" :as :bool :default true]} :compiler-options {:infer-externs :auto From 605c86ecd1285b706e5981dd49b5677089f1482c Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 11 Mar 2026 19:32:55 +0800 Subject: [PATCH 131/375] 056-graph-name-dir-encoding-alignment.md --- deps/cli/src/logseq/cli/common/graph.cljs | 8 +- deps/cli/src/logseq/cli/util.cljs | 4 +- deps/common/src/logseq/common/graph_dir.cljs | 40 ++ deps/db/src/logseq/db/common/sqlite.cljs | 11 +- .../056-graph-name-dir-encoding-alignment.md | 377 ++++++++++++++++++ src/electron/electron/db.cljs | 7 +- src/electron/electron/utils.cljs | 3 +- src/main/frontend/config.cljs | 3 +- src/main/frontend/worker/db_core.cljs | 2 +- .../frontend/worker/db_worker_node_lock.cljs | 2 +- src/main/frontend/worker/graph_dir.cljs | 14 +- src/main/frontend/worker_common/util.cljc | 21 +- src/main/logseq/cli/common.cljs | 10 +- src/test/electron/db_test.cljs | 23 ++ src/test/electron/utils_test.cljs | 1 + src/test/frontend/config_test.cljs | 10 + src/test/frontend/worker/graph_dir_test.cljs | 13 + src/test/logseq/cli/common/graph_test.cljs | 23 ++ src/test/logseq/cli/common_test.cljs | 7 +- src/test/logseq/cli/server_test.cljs | 6 + src/test/logseq/db/common_sqlite_test.cljs | 11 + 21 files changed, 550 insertions(+), 46 deletions(-) create mode 100644 deps/common/src/logseq/common/graph_dir.cljs create mode 100644 docs/agent-guide/056-graph-name-dir-encoding-alignment.md create mode 100644 src/test/electron/db_test.cljs create mode 100644 src/test/electron/utils_test.cljs create mode 100644 src/test/frontend/config_test.cljs create mode 100644 src/test/logseq/db/common_sqlite_test.cljs diff --git a/deps/cli/src/logseq/cli/common/graph.cljs b/deps/cli/src/logseq/cli/common/graph.cljs index 4a310199f5..d6c15ec8a2 100644 --- a/deps/cli/src/logseq/cli/common/graph.cljs +++ b/deps/cli/src/logseq/cli/common/graph.cljs @@ -3,14 +3,12 @@ (:require ["fs-extra" :as fs-extra] [clojure.string :as string] [logseq.common.config :as common-config] - [logseq.common.graph :as common-graph])) + [logseq.common.graph :as common-graph] + [logseq.common.graph-dir :as graph-dir])) (defn ^:api graph-name->path [graph-name] - (when graph-name - (-> graph-name - (string/replace "+3A+" ":") - (string/replace "++" "/")))) + (graph-dir/decode-graph-dir-name graph-name)) (defn get-db-graphs-dir "Directory where DB graphs are stored" diff --git a/deps/cli/src/logseq/cli/util.cljs b/deps/cli/src/logseq/cli/util.cljs index 42bc50fa19..9bd9e0fc90 100644 --- a/deps/cli/src/logseq/cli/util.cljs +++ b/deps/cli/src/logseq/cli/util.cljs @@ -4,7 +4,7 @@ ["path" :as node-path] [clojure.string :as string] [logseq.cli.common.graph :as cli-common-graph] - [logseq.db.common.sqlite :as common-sqlite] + [logseq.common.graph-dir :as graph-dir] [nbb.error] [promesa.core :as p])) @@ -17,7 +17,7 @@ (string/includes? graph "/") ((juxt node-path/dirname node-path/basename) graph) :else - [(cli-common-graph/get-db-graphs-dir) (common-sqlite/sanitize-db-name graph)])) + [(cli-common-graph/get-db-graphs-dir) (graph-dir/repo->encoded-graph-dir-name graph)])) (defn get-graph-path "If graph is a file, return its path. Otherwise returns the graph's dir" diff --git a/deps/common/src/logseq/common/graph_dir.cljs b/deps/common/src/logseq/common/graph_dir.cljs new file mode 100644 index 0000000000..35be30a591 --- /dev/null +++ b/deps/common/src/logseq/common/graph_dir.cljs @@ -0,0 +1,40 @@ +(ns logseq.common.graph-dir + "Platform-agnostic graph directory naming helpers." + (:require [clojure.string :as string] + [logseq.common.config :as common-config])) + +(defn encode-graph-dir-name + [graph-name] + (let [encoded (js/encodeURIComponent (or graph-name ""))] + (-> encoded + (string/replace "~" "%7E") + (string/replace "%" "~")))) + +(defn decode-graph-dir-name + [dir-name] + (when-not (and (string? dir-name) + (or (string/includes? dir-name "++") + (string/includes? dir-name "+3A+"))) + (when (some? dir-name) + (try + (js/decodeURIComponent (string/replace dir-name "~" "%")) + (catch :default _ + nil))))) + +(defn repo->graph-dir-key + [repo] + (when (seq repo) + (if (string/starts-with? repo common-config/db-version-prefix) + (subs repo (count common-config/db-version-prefix)) + repo))) + +(defn graph-dir-key->encoded-dir-name + [graph-dir-key] + (when (some? graph-dir-key) + (encode-graph-dir-name graph-dir-key))) + +(defn repo->encoded-graph-dir-name + [repo] + (some-> repo + repo->graph-dir-key + graph-dir-key->encoded-dir-name)) diff --git a/deps/db/src/logseq/db/common/sqlite.cljs b/deps/db/src/logseq/db/common/sqlite.cljs index 9bfae51898..2461c67fb0 100644 --- a/deps/db/src/logseq/db/common/sqlite.cljs +++ b/deps/db/src/logseq/db/common/sqlite.cljs @@ -3,6 +3,7 @@ (:require ["path" :as node-path] [clojure.string :as string] [datascript.core :as d] + [logseq.common.graph-dir :as graph-dir] [logseq.db.sqlite.util :as sqlite-util])) (defn create-kvs-table! @@ -26,11 +27,11 @@ (defn get-db-full-path [graphs-dir db-name] - (let [db-name' (sanitize-db-name db-name) - graph-dir (node-path/join graphs-dir db-name')] - [db-name' (node-path/join graph-dir "db.sqlite")])) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name) + graph-dir (node-path/join graphs-dir graph-dir-name)] + [graph-dir-name (node-path/join graph-dir "db.sqlite")])) (defn get-db-backups-path [graphs-dir db-name] - (let [db-name' (sanitize-db-name db-name)] - (node-path/join graphs-dir db-name' "backups"))) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name)] + (node-path/join graphs-dir graph-dir-name "backups"))) diff --git a/docs/agent-guide/056-graph-name-dir-encoding-alignment.md b/docs/agent-guide/056-graph-name-dir-encoding-alignment.md new file mode 100644 index 0000000000..b31aac8254 --- /dev/null +++ b/docs/agent-guide/056-graph-name-dir-encoding-alignment.md @@ -0,0 +1,377 @@ +# Align graph dir encoding between logseq-cli and desktop app + +## Summary + +Align `logseq-cli`, `db-worker-node`, and desktop app handling of `graph dir` / `graph-name` so special characters are encoded and decoded with one shared, reversible contract. + +The authoritative contract would be the existing `encode-graph-dir-name` / `decode-graph-dir-name` pair in `src/main/frontend/worker_common/util.cljc`, which is already used by `db-worker-node` and `logseq-cli` server-side graph directory resolution. + +This plan keeps user-facing graph names unchanged and only aligns their on-disk directory representation. + +## Background + +Current code paths do not agree on how a graph name maps to a graph directory on disk: + +- `db-worker-node` and `logseq-cli` server/runtime paths use a reversible graph-dir encoding. +- desktop app contains paths that join the raw graph name directly into a filesystem path. +- some Electron and CLI-adjacent helpers still use lossy `sanitize-db-name` behavior. +- shared graph discovery still contains legacy decoding logic for older naming conventions, but not the current reversible encoding. + +This mismatch becomes visible when graph names contain special characters such as `/`, `:`, `%`, `~`, or spaces. + +## Goals + +- Use one shared graph-dir encoding/decoding contract across CLI and desktop app. +- Preserve current user-facing graph-name semantics. +- Keep `logseq_db_` prefix canonicalization separate from graph-dir encoding. +- Define compatibility behavior for legacy graph directory names. +- Add tests that cover special-character graph names across all affected entry points. + +## Non-goals + +- Redesign the user-visible graph naming model. +- Change the existing `logseq_db_` display normalization rules. +- Remove all legacy compatibility in one step without an explicit migration strategy. + +## Current behavior + +### Shared reversible encoding already exists + +Authoritative implementation today: + +- `src/main/frontend/worker_common/util.cljc` + - `encode-graph-dir-name` + - `decode-graph-dir-name` + +Current behavior: + +1. `encodeURIComponent` is applied. +2. literal `~` is rewritten to `%7E`. +3. `%` is rewritten to `~`. +4. decoding reverses `~ -> %` and then applies `decodeURIComponent`. + +This gives a reversible filesystem-safe directory key without `/` or `\\` path separators. + +### db-worker-node follows the shared contract + +Relevant files: + +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/frontend/worker/platform/node.cljs` +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/graph_dir.cljs` + +Current behavior: + +- repo identity strips one leading `logseq_db_` to produce a graph-dir key. +- graph-dir key is encoded with `encode-graph-dir-name`. +- list-graphs decodes on-disk directory names back to graph-dir keys. +- worker log paths and lock paths are stored under the encoded graph directory. + +### CLI is partially aligned + +Relevant files: + +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/core.cljs` +- `src/main/logseq/cli/command/graph.cljs` +- `src/main/logseq/cli/common.cljs` +- `deps/cli/src/logseq/cli/common/graph.cljs` +- `deps/cli/src/logseq/cli/util.cljs` + +Current behavior: + +- `cli.server` already uses the same canonical graph-dir path contract as `db-worker-node`. +- graph display/input normalization strips or restores one `logseq_db_` prefix as needed. +- `unlink-graph!` still derives directory names with `sanitize-db-name`, which is lossy. +- shared discovery in `deps/cli` still decodes only older directory naming patterns such as `++` and `+3A+`. + +### Desktop app is not aligned + +Relevant files: + +- `src/electron/electron/utils.cljs` +- `src/electron/electron/db.cljs` +- `src/electron/electron/handler.cljs` +- `src/electron/electron/url.cljs` +- `src/main/frontend/config.cljs` + +Current behavior: + +- `electron.utils/get-graph-dir` joins the raw graph name into the graph path after db-prefix stripping. +- if the graph name contains `/`, the resulting path becomes nested directories. +- `electron.db` still uses `sanitize-db-name` in some db path creation logic. +- frontend local-dir helpers also treat graph name as a raw path segment. + +## Problem statement + +The same logical graph name can map to different on-disk paths depending on which subsystem touches it: + +- reversible encoded path in `db-worker-node` +- raw path join in Electron/frontend +- lossy underscore replacement in sanitize-based helpers +- legacy decode-only behavior in shared graph discovery + +As a result: + +- a graph may be listable but not removable +- a graph may be resolvable in CLI but not in desktop app +- a graph name containing `/` may accidentally create path nesting in one flow but not another +- existing tests do not enforce cross-subsystem parity + +## Proposed contract + +### 1. Separate graph identity from graph directory representation + +The plan would explicitly distinguish: + +- **graph-name / repo**: user-facing identifier, subject to existing `logseq_db_` canonicalization rules +- **graph-dir key**: graph-name with exactly one leading db prefix stripped +- **encoded graph-dir**: on-disk directory name produced only by `encode-graph-dir-name` + +This separation would make it clear that special-character handling belongs to the graph-dir layer, not the user-facing name layer. + +### 2. Make the db-worker-node contract authoritative + +The repository would standardize on: + +- `repo -> graph-dir key`: strip one leading `logseq_db_` +- `graph-dir key -> encoded graph-dir`: `encode-graph-dir-name` +- `encoded graph-dir -> graph-dir key`: `decode-graph-dir-name` + +Any code path that needs an on-disk db graph directory would route through this contract rather than reimplementing path logic. + +### 3. Keep user-visible graph names unchanged + +The plan would preserve current user-visible behavior: + +- CLI graph names remain prefix-free for display and config storage where already intended. +- desktop app continues to display logical graph names, not encoded directory names. +- URL-level graph identification continues to resolve to logical graph names, not on-disk encoded names. + +## Proposed code changes + +### A. Consolidate path-authoritative helpers + +Add or reuse one shared helper layer for: + +- converting repo to graph-dir key +- converting graph-dir key to encoded graph directory +- converting repo directly to on-disk graph directory path + +Target files likely involved: + +- `src/main/frontend/worker/graph_dir.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/electron/electron/utils.cljs` +- `src/main/frontend/config.cljs` +- `deps/cli/src/logseq/cli/util.cljs` + +Expected outcome: + +- no raw path join for logical graph names in path-authoritative code +- no duplicate graph-dir encoding implementations + +### B. Align Electron graph-dir resolution + +Replace raw graph path derivation in Electron with the shared encoded graph-dir contract. + +Target files: + +- `src/electron/electron/utils.cljs` +- `src/electron/electron/handler.cljs` +- `src/electron/electron/db.cljs` + +Expected outcome: + +- desktop app resolves the same on-disk graph dir as `db-worker-node` +- graph names containing `/`, `:`, `%`, `~`, or spaces behave predictably +- `sanitize-db-name` is no longer used for authoritative db graph-dir mapping + +### C. Align CLI remove/unlink behavior + +Update CLI removal/unlink flows to resolve graph directories via the same encoded contract used by list/start/lock behavior. + +Target file: + +- `src/main/logseq/cli/common.cljs` + +Expected outcome: + +- a graph that can be listed or switched to can also be removed through the same path mapping + +### D. Align shared graph discovery + +Update shared discovery helpers so current encoded graph dirs are decoded correctly, while preserving deliberate support for legacy names where needed. + +Target file: + +- `deps/cli/src/logseq/cli/common/graph.cljs` + +Expected outcome: + +- desktop/CLI discovery would recognize encoded graph dirs produced by current db-worker-node logic +- legacy decode branches would be explicitly documented as compatibility behavior + +### E. Audit frontend local-dir helpers + +Review helpers that expose graph-related directories to ensure they are either: + +- display-only helpers, or +- path-authoritative helpers using the shared encoded contract + +Target file: + +- `src/main/frontend/config.cljs` + +Expected outcome: + +- no ambiguous helper remains that appears safe for filesystem use while still using raw graph names + +## Compatibility and migration + +This plan should explicitly decide how to handle already-existing graph directories created by older logic. + +### Option 1: Read legacy names, write canonical encoded names + +Behavior: + +- discovery accepts legacy directory names and current encoded names +- all newly created or rewritten paths use the canonical encoded form +- optional one-time migration may rename legacy directories + +Pros: + +- safer rollout +- less risk of immediately losing access to existing graphs + +Cons: + +- mixed formats may coexist temporarily + +### Option 2: Auto-migrate on access + +Behavior: + +- when a legacy graph directory is detected, code renames it to the canonical encoded path before continuing + +Pros: + +- converges quickly to one format + +Cons: + +- higher operational risk +- rename behavior must be designed carefully for active workers and lock files + +### Option 3: Strict cutover + +Behavior: + +- only encoded graph dirs are supported after the change + +Pros: + +- simplest long-term contract + +Cons: + +- too risky without explicit migration tooling + +### Recommended direction + +Prefer **Option 1** for the first rollout: + +- read compatibility for legacy directory names +- canonical writes to encoded graph dirs +- add explicit migration follow-up only after parity tests pass + +## Test plan + +### Unit tests + +Extend or add tests for: + +- `src/test/frontend/worker/worker_common_util_test.cljs` +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `src/test/logseq/cli/common/graph_test.cljs` +- Electron-specific tests if available for graph-dir resolution + +### Special-character test matrix + +All subsystems should use the same examples: + +- `foo/bar` +- `a:b` +- `space name` +- `100% legit` +- `til~de` +- `mix/of:many %chars~here` + +### Behavior to verify + +1. encode/decode roundtrip is lossless +2. CLI list-graphs returns the same logical graph name that was encoded on disk +3. CLI switch/remove resolve the same graph directory +4. desktop app resolves the same graph directory as CLI/db-worker-node +5. graph names remain user-visible without encoded substitutions +6. legacy discovery behavior remains intentional and documented + +### Missing coverage today + +The repository currently appears to lack end-to-end parity tests for: + +- CLI create/switch/remove with special-character graph names +- Electron graph-name -> graph-dir resolution with special characters +- desktop and CLI agreement on one on-disk graph directory for the same logical graph + +## Rollout sequence + +1. Make the shared graph-dir contract explicit in code and docs. +2. Update Electron path-authoritative helpers to use encoded graph dirs. +3. Update CLI unlink/remove behavior to use the same mapping. +4. Update shared graph discovery for encoded graph dirs and legacy compatibility. +5. Add parity tests across worker, CLI, and desktop-related helpers. +6. Evaluate whether legacy directory migration should be a separate follow-up. + +## Risks + +- Existing graphs may already exist under lossy or raw directory naming rules. +- Desktop-specific compatibility code may rely on current path layout assumptions. +- URL/deeplink flows may resolve graph identifiers separately from filesystem mapping and should not accidentally expose encoded names to users. +- Removing `sanitize-db-name` from authoritative paths may surface hidden assumptions in older db bootstrap code. + +## Open questions + +1. Should legacy raw/sanitized graph directories remain writable, or only readable? +2. Should migration happen automatically, manually, or in a later dedicated change? +3. Which helper should become the single exported entry point for graph-name -> on-disk graph-dir path resolution? +4. Should `docs/cli/logseq-cli.md` be updated in the same change to clarify that on-disk graph directories are encoded, not always literal graph names? + +## Expected files to change in implementation + +Likely implementation targets: + +- `src/main/frontend/worker_common/util.cljc` +- `src/main/frontend/worker/graph_dir.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/common.cljs` +- `deps/cli/src/logseq/cli/common/graph.cljs` +- `deps/cli/src/logseq/cli/util.cljs` +- `src/electron/electron/utils.cljs` +- `src/electron/electron/db.cljs` +- `src/electron/electron/handler.cljs` +- `src/main/frontend/config.cljs` +- related tests under `src/test/` + +## Acceptance criteria + +This plan would be complete when: + +- one shared graph-dir encoding contract is identified as authoritative +- all affected subsystems and files are enumerated +- compatibility strategy for legacy graph directories is documented +- a concrete test matrix for special-character graph names is defined +- the plan preserves current user-facing graph-name semantics while aligning on-disk graph-dir behavior diff --git a/src/electron/electron/db.cljs b/src/electron/electron/db.cljs index 78cc1e1884..5d4891e861 100644 --- a/src/electron/electron/db.cljs +++ b/src/electron/electron/db.cljs @@ -4,6 +4,7 @@ ["path" :as node-path] [electron.backup-file :as backup-file] [logseq.cli.common.graph :as cli-common-graph] + [logseq.common.graph-dir :as graph-dir] [logseq.db.common.sqlite :as common-sqlite])) (defn ensure-graphs-dir! @@ -13,7 +14,8 @@ (defn ensure-graph-dir! [db-name] (ensure-graphs-dir!) - (let [graph-dir (node-path/join (cli-common-graph/get-db-graphs-dir) (common-sqlite/sanitize-db-name db-name))] + (let [graph-dir (node-path/join (cli-common-graph/get-db-graphs-dir) + (graph-dir/repo->encoded-graph-dir-name db-name))] (fs/ensureDirSync graph-dir) graph-dir)) @@ -28,7 +30,8 @@ (defn save-db! "Legacy compatibility path for Electron OPFS export." [db-name data] - (let [[db-name db-path] (common-sqlite/get-db-full-path (cli-common-graph/get-db-graphs-dir) db-name) + (let [_ (ensure-graph-dir! db-name) + [db-name db-path] (common-sqlite/get-db-full-path (cli-common-graph/get-db-graphs-dir) db-name) old-data (get-db db-name) backups-path (common-sqlite/get-db-backups-path (cli-common-graph/get-db-graphs-dir) db-name)] (when old-data diff --git a/src/electron/electron/utils.cljs b/src/electron/electron/utils.cljs index 8d9f26f596..f1fd448b2b 100644 --- a/src/electron/electron/utils.cljs +++ b/src/electron/electron/utils.cljs @@ -7,6 +7,7 @@ [electron.configs :as cfgs] [electron.logger :as logger] [logseq.cli.common.graph :as cli-common-graph] + [logseq.common.graph-dir :as graph-dir] [logseq.common.config :as common-config] [promesa.core :as p])) @@ -244,7 +245,7 @@ (string/starts-with? graph-name common-config/db-version-prefix)) (let [repo (common-config/canonicalize-db-version-repo graph-name)] (node-path/join (cli-common-graph/get-db-graphs-dir) - (common-config/strip-leading-db-version-prefix repo))))) + (graph-dir/repo->encoded-graph-dir-name repo))))) (comment (defn get-graph-name diff --git a/src/main/frontend/config.cljs b/src/main/frontend/config.cljs index 0539a99b61..f7588a4184 100644 --- a/src/main/frontend/config.cljs +++ b/src/main/frontend/config.cljs @@ -5,6 +5,7 @@ [frontend.state :as state] [frontend.util :as util] [goog.crypt.Md5] + [logseq.common.graph-dir :as common-graph-dir] [logseq.common.cognito-config :as cognito-config] [logseq.common.config :as common-config] [logseq.common.path :as path] @@ -252,7 +253,7 @@ (path/path-join (get-in @state/state [:system/info :home-dir]) "logseq" "graphs" - (db-graph-name repo))) + (common-graph-dir/repo->encoded-graph-dir-name repo))) (defn get-electron-backup-dir [repo] diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index d9eb15f09e..9db4c0ea7b 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -17,7 +17,6 @@ [frontend.worker.db.validate :as worker-db-validate] [frontend.worker.embedding :as embedding] [frontend.worker.export :as worker-export] - [frontend.worker.graph-dir :as graph-dir] [frontend.worker.handler.page :as worker-page] [frontend.worker.pipeline :as worker-pipeline] [frontend.worker.platform :as platform] @@ -32,6 +31,7 @@ [frontend.worker.thread-atom] [lambdaisland.glogi :as log] [logseq.cli.common.mcp.tools :as cli-common-mcp-tools] + [logseq.common.graph-dir :as graph-dir] [logseq.common.util :as common-util] [logseq.db :as ldb] [logseq.db.common.entity-plus :as entity-plus] diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index e04a617ab3..0a2b601889 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -5,8 +5,8 @@ ["path" :as node-path] [clojure.string :as string] [frontend.worker-common.util :as worker-util] - [frontend.worker.graph-dir :as graph-dir] [lambdaisland.glogi :as log] + [logseq.common.graph-dir :as graph-dir] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] [promesa.core :as p])) diff --git a/src/main/frontend/worker/graph_dir.cljs b/src/main/frontend/worker/graph_dir.cljs index d23a22b56f..6920bbb56d 100644 --- a/src/main/frontend/worker/graph_dir.cljs +++ b/src/main/frontend/worker/graph_dir.cljs @@ -1,11 +1,7 @@ (ns frontend.worker.graph-dir - "Platform-agnostic graph directory naming helpers." - (:require [clojure.string :as string] - [logseq.common.config :as common-config])) + "Compatibility wrapper around `logseq.common.graph-dir`." + (:require [logseq.common.graph-dir :as common-graph-dir])) -(defn repo->graph-dir-key - [repo] - (when (seq repo) - (if (string/starts-with? repo common-config/db-version-prefix) - (subs repo (count common-config/db-version-prefix)) - repo))) +(def repo->graph-dir-key common-graph-dir/repo->graph-dir-key) +(def repo->encoded-graph-dir-name common-graph-dir/repo->encoded-graph-dir-name) +(def decode-graph-dir-name common-graph-dir/decode-graph-dir-name) diff --git a/src/main/frontend/worker_common/util.cljc b/src/main/frontend/worker_common/util.cljc index 1a6767ec7e..ac77a1dcba 100644 --- a/src/main/frontend/worker_common/util.cljc +++ b/src/main/frontend/worker_common/util.cljc @@ -6,8 +6,9 @@ [goog.crypt.base64 :as base64] [goog.crypt.Hmac] [goog.crypt.Sha256] + [logseq.common.graph-dir :as common-graph-dir] [logseq.db :as ldb] - [logseq.db.common.sqlite :as common-sqlite]))) + [logseq.db.sqlite.util :as sqlite-util]))) ;; Copied from https://github.com/tonsky/datascript-todo #?(:clj @@ -33,22 +34,20 @@ (defn encode-graph-dir-name [graph-name] - (let [encoded (js/encodeURIComponent (or graph-name ""))] - (-> encoded - (string/replace "~" "%7E") - (string/replace "%" "~")))) + (common-graph-dir/encode-graph-dir-name graph-name)) (defn decode-graph-dir-name [dir-name] - (when (some? dir-name) - (try - (js/decodeURIComponent (string/replace dir-name "~" "%")) - (catch :default _ - nil)))) + (common-graph-dir/decode-graph-dir-name dir-name)) (defn get-pool-name [graph-name] - (str "logseq-pool-" (common-sqlite/sanitize-db-name graph-name))) + (str "logseq-pool-" + (-> graph-name + (string/replace sqlite-util/db-version-prefix "") + (string/replace "/" "_") + (string/replace "\\" "_") + (string/replace ":" "_")))) (defn- decode-username [username] diff --git a/src/main/logseq/cli/common.cljs b/src/main/logseq/cli/common.cljs index a1809fc337..a7cd739ad8 100644 --- a/src/main/logseq/cli/common.cljs +++ b/src/main/logseq/cli/common.cljs @@ -4,20 +4,20 @@ ["path" :as node-path] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] - [logseq.db.common.sqlite :as common-sqlite])) + [logseq.common.graph-dir :as graph-dir])) (defn unlink-graph! "Unlinks the given repo by moving it to the 'Unlinked graphs' dir. Returns path of unlinked dir if move is successful or nil if not" [repo] - (let [db-name (common-sqlite/sanitize-db-name repo) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name repo) graphs-dir (common-graph/expand-home (common-graph/get-default-graphs-dir)) - path (node-path/join graphs-dir db-name) + path (node-path/join graphs-dir graph-dir-name) unlinked (node-path/join graphs-dir common-config/unlinked-graphs-dir) - new-path (node-path/join unlinked db-name) + new-path (node-path/join unlinked graph-dir-name) new-path-exists? (fs/existsSync new-path) new-path' (if new-path-exists? - (node-path/join unlinked (str db-name "-" (random-uuid))) + (node-path/join unlinked (str graph-dir-name "-" (random-uuid))) new-path)] (when (fs/existsSync path) (fs/ensureDirSync unlinked) diff --git a/src/test/electron/db_test.cljs b/src/test/electron/db_test.cljs new file mode 100644 index 0000000000..41bf772d63 --- /dev/null +++ b/src/test/electron/db_test.cljs @@ -0,0 +1,23 @@ +(ns electron.db-test + (:require [cljs.test :refer [deftest is]] + [electron.db :as electron-db] + [frontend.test.node-helper :as node-helper] + [logseq.cli.common.graph :as cli-common-graph] + ["fs-extra" :as fs] + ["path" :as node-path])) + +(deftest ensure-graph-dir-uses-encoded-directory-name + (let [graphs-dir (node-helper/create-tmp-dir "electron-db-graph-dir")] + (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] + (let [graph-dir (electron-db/ensure-graph-dir! "logseq_db_foo/bar")] + (is (= (node-path/join graphs-dir "foo~2Fbar") graph-dir)) + (is (fs/existsSync graph-dir)))))) + +(deftest save-and-read-db-use-encoded-directory-name + (let [graphs-dir (node-helper/create-tmp-dir "electron-db-save") + payload (.from js/Buffer "db-data")] + (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] + (electron-db/save-db! "logseq_db_foo/bar" payload) + (is (fs/existsSync (node-path/join graphs-dir "foo~2Fbar" "db.sqlite"))) + (is (= "db-data" + (.toString (electron-db/get-db "logseq_db_foo/bar"))))))) diff --git a/src/test/electron/utils_test.cljs b/src/test/electron/utils_test.cljs new file mode 100644 index 0000000000..0b2b778c5f --- /dev/null +++ b/src/test/electron/utils_test.cljs @@ -0,0 +1 @@ +(ns electron.utils-test) diff --git a/src/test/frontend/config_test.cljs b/src/test/frontend/config_test.cljs new file mode 100644 index 0000000000..dd4ca89e43 --- /dev/null +++ b/src/test/frontend/config_test.cljs @@ -0,0 +1,10 @@ +(ns frontend.config-test + (:require [cljs.test :refer [deftest is]] + [frontend.config :as config] + [frontend.state :as state] + [logseq.common.config :as common-config])) + +(deftest get-local-dir-uses-encoded-directory-name + (with-redefs [state/state (atom {:system/info {:home-dir "/tmp/home"}})] + (is (= "/tmp/home/logseq/graphs/foo~2Fbar" + (config/get-local-dir (str common-config/db-version-prefix "foo/bar")))))) diff --git a/src/test/frontend/worker/graph_dir_test.cljs b/src/test/frontend/worker/graph_dir_test.cljs index f9a62241c2..fb31b89466 100644 --- a/src/test/frontend/worker/graph_dir_test.cljs +++ b/src/test/frontend/worker/graph_dir_test.cljs @@ -9,3 +9,16 @@ (deftest repo->graph-dir-key-keeps-prefix-free-name (testing "prefix-free repo remains unchanged" (is (= "demo" (graph-dir/repo->graph-dir-key "demo"))))) + +(deftest repo->encoded-graph-dir-name-encodes-special-characters + (testing "db-prefixed repos resolve to the encoded on-disk graph dir name" + (is (= "foo~2Fbar" + (graph-dir/repo->encoded-graph-dir-name "logseq_db_foo/bar"))))) + +(deftest decode-graph-dir-name-decodes-only-canonical-encoded-names + (testing "encoded graph dirs decode back to the logical graph dir key" + (is (= "foo/bar" + (graph-dir/decode-graph-dir-name "foo~2Fbar")))) + (testing "legacy graph-dir encodings are not accepted" + (is (nil? (graph-dir/decode-graph-dir-name "foo++bar"))) + (is (nil? (graph-dir/decode-graph-dir-name "a+3A+b"))))) diff --git a/src/test/logseq/cli/common/graph_test.cljs b/src/test/logseq/cli/common/graph_test.cljs index 1d205a8a39..248b1f2a6a 100644 --- a/src/test/logseq/cli/common/graph_test.cljs +++ b/src/test/logseq/cli/common/graph_test.cljs @@ -18,3 +18,26 @@ (let [graphs (cli-common-graph/get-db-based-graphs)] (is (= #{"logseq_db_demo"} (set graphs))) (is (not-any? #(string/starts-with? % "logseq_db_logseq_db_") graphs)))))) + +(deftest get-db-based-graphs-decodes-encoded-graph-directories + (let [graphs-dir (node-helper/create-tmp-dir "cli-common-graph-encoded") + _ (doseq [dir ["foo~2Fbar" + "a~3Ab" + "space~20name" + "Unlinked graphs"]] + (fs/mkdirSync (node-path/join graphs-dir dir) #js {:recursive true}))] + (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] + (let [graphs (set (cli-common-graph/get-db-based-graphs))] + (is (= #{"logseq_db_foo/bar" + "logseq_db_a:b" + "logseq_db_space name"} + graphs)))))) + +(deftest get-db-based-graphs-ignores-legacy-graph-dir-encodings + (let [graphs-dir (node-helper/create-tmp-dir "cli-common-graph-legacy") + _ (doseq [dir ["foo++bar" + "a+3A+b" + "Unlinked graphs"]] + (fs/mkdirSync (node-path/join graphs-dir dir) #js {:recursive true}))] + (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] + (is (= [] (cli-common-graph/get-db-based-graphs)))))) diff --git a/src/test/logseq/cli/common_test.cljs b/src/test/logseq/cli/common_test.cljs index b9616c46d5..e46b143cc0 100644 --- a/src/test/logseq/cli/common_test.cljs +++ b/src/test/logseq/cli/common_test.cljs @@ -9,10 +9,11 @@ (deftest unlink-graph-moves-to-unlinked-dir (let [graphs-dir (node-helper/create-tmp-dir "unlink-graph") - graph-name "test-graph" + graph-name "foo/bar" repo (str common-config/db-version-prefix graph-name) - graph-path (node-path/join graphs-dir graph-name) - unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir graph-name)] + encoded-graph-dir "foo~2Fbar" + graph-path (node-path/join graphs-dir encoded-graph-dir) + unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir encoded-graph-dir)] (fs/mkdirSync graph-path #js {:recursive true}) (fs/writeFileSync (node-path/join graph-path "db.sqlite") "test-data") (with-redefs [common-graph/get-default-graphs-dir (fn [] graphs-dir)] diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index b321c7ea9f..24a6c41fd4 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -48,6 +48,12 @@ expected (node-path/join data-dir "demo" "db-worker.lock")] (is (= expected (cli-server/lock-path data-dir repo))))) +(deftest lock-path-encodes-special-characters-in-graph-dir + (let [data-dir "/tmp/logseq-db-worker" + repo "logseq_db_foo/bar" + expected (node-path/join data-dir "foo~2Fbar" "db-worker.lock")] + (is (= expected (cli-server/lock-path data-dir repo))))) + (deftest db-worker-runtime-script-path-defaults-to-packaged-dist-target (is (= (node-path/join js/__dirname "../dist/db-worker-node.js") (cli-server/db-worker-runtime-script-path)))) diff --git a/src/test/logseq/db/common_sqlite_test.cljs b/src/test/logseq/db/common_sqlite_test.cljs new file mode 100644 index 0000000000..db1a067316 --- /dev/null +++ b/src/test/logseq/db/common_sqlite_test.cljs @@ -0,0 +1,11 @@ +(ns logseq.db.common-sqlite-test + (:require [cljs.test :refer [deftest is]] + [logseq.db.common.sqlite :as common-sqlite])) + +(deftest get-db-full-path-uses-encoded-graph-dir + (is (= ["foo~2Fbar" "/tmp/graphs/foo~2Fbar/db.sqlite"] + (common-sqlite/get-db-full-path "/tmp/graphs" "logseq_db_foo/bar")))) + +(deftest get-db-backups-path-uses-encoded-graph-dir + (is (= "/tmp/graphs/foo~2Fbar/backups" + (common-sqlite/get-db-backups-path "/tmp/graphs" "logseq_db_foo/bar")))) From a89ffb6b0f22cf4e9d3b80d4eff744ef72ce0f40 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 11 Mar 2026 21:46:00 +0800 Subject: [PATCH 132/375] 057-cli-sync-download-realtime-progress.md --- ...057-cli-sync-download-realtime-progress.md | 436 ++++++++++++++++++ docs/cli/logseq-cli.md | 10 +- src/main/frontend/handler/db_based/sync.cljs | 137 +----- src/main/frontend/worker/db_core.cljs | 16 +- src/main/frontend/worker/sync.cljs | 27 +- src/main/logseq/cli/command/sync.cljs | 54 ++- src/main/logseq/cli/transport.cljs | 92 ++++ .../frontend/handler/db_based/sync_test.cljs | 230 +++------ src/test/frontend/worker/db_sync_test.cljs | 55 +++ src/test/logseq/cli/command/sync_test.cljs | 136 ++++++ src/test/logseq/cli/commands_test.cljs | 8 + src/test/logseq/cli/transport_test.cljs | 40 ++ 12 files changed, 937 insertions(+), 304 deletions(-) create mode 100644 docs/agent-guide/057-cli-sync-download-realtime-progress.md diff --git a/docs/agent-guide/057-cli-sync-download-realtime-progress.md b/docs/agent-guide/057-cli-sync-download-realtime-progress.md new file mode 100644 index 0000000000..83ad162da3 --- /dev/null +++ b/docs/agent-guide/057-cli-sync-download-realtime-progress.md @@ -0,0 +1,436 @@ +# 057: Make `logseq-cli sync download` stream realtime progress and support long-running downloads + +## Summary + +This document defines an execution-ready implementation plan for improving `logseq-cli sync download` for large graphs. + +Today, the CLI waits for a single `/v1/invoke` response and uses a short default timeout. This creates two bad outcomes for large graph downloads: + +1. the terminal is silent while the snapshot is downloading and importing, and +2. healthy long-running work may appear to fail because the CLI request timeout is too short. + +The implementation should reuse the existing sync log/event infrastructure already used by the app, instead of creating a separate CLI-only progress system. + +## Decision records (confirmed) + +The following decisions are fixed for implementation: + +1. `sync download` timeout strategy uses a command-level long-running policy (not a global timeout increase). +2. Progress logs are printed to `stdout`. +3. Add `--progress` option for `sync download`. +4. `--progress` defaults to `true` for human output. +5. For structured output modes (for example `json` / `edn`), progress is automatically disabled unless the user explicitly passes `--progress true`. +6. Download progress log emission should be unified into shared worker/sync code (no duplicate app-only emission path for the same milestones). + +## Problem statement + +`logseq-cli sync download` already uses the same db-worker-node and worker sync stack that powers app sync behavior, but it does not currently reuse the realtime event stream. + +Relevant current-state facts: + +- CLI sends a request to db-worker-node over `/v1/invoke` and waits for one final response. +- db-worker-node already exposes `/v1/events` as an SSE event stream. +- worker/app sync logic already emits `:rtc-log` events. +- app/desktop already displays those logs. +- CLI transport defaults to a `10000` ms timeout, which is not appropriate for a full graph snapshot download/import. + +The implementation goal is therefore not to invent progress tracking from scratch. The goal is to make CLI consume and display the same progress events, and to change timeout behavior so long-running downloads can complete. + +## Goals + +- Show realtime progress during `logseq-cli sync download`. +- Reuse the existing `:rtc-log` event model and db-worker-node SSE stream. +- Unify download progress log emission into shared worker/sync code for the full download flow. +- Add `--progress` to `sync download` with default behavior enabled for human output. +- Print progress to `stdout`. +- Automatically disable progress for structured output modes unless `--progress true` is explicitly set. +- Prevent large graph downloads from failing under the generic short CLI timeout path. +- Preserve the final command result semantics and existing validation behavior. + +## Non-goals + +- Redesign the sync protocol. +- Replace the app RTC log UI. +- Introduce a polling-based progress API. +- Change the existing success/failure meaning of `sync download`. +- Add a CLI-only event schema that diverges from app behavior. + +## Current implementation map + +### CLI entrypoints + +- `src/main/logseq/cli/command/sync.cljs` + - defines `sync download` + - resolves auth, starts db-worker-node, validates empty DB, invokes `:thread-api/db-sync-download-graph-by-id` +- `src/main/logseq/cli/transport.cljs` + - sends HTTP requests to db-worker-node `/v1/invoke` + - currently applies default timeout behavior +- `src/main/logseq/cli/format.cljs` + - formats final CLI output +- `src/main/logseq/cli/config.cljs` + - defines CLI defaults, including timeout +- `src/main/logseq/cli/command/core.cljs` + - defines global CLI options including `--timeout-ms` + +### db-worker-node / worker entrypoints + +- `src/main/frontend/worker/db_worker_node.cljs` + - serves `/v1/invoke` + - serves `/v1/events` as SSE +- `src/main/frontend/worker/db_core.cljs` + - handles `:thread-api/db-sync-download-graph-by-id` + - already emits import/decrypt/save-stage logs +- `src/main/frontend/worker/sync.cljs` + - performs remote snapshot download and sync data fetch +- `src/main/frontend/worker/sync/log_and_state.cljs` + - publishes `:rtc-log` events to connected clients + +### App-side log consumers and app-specific log emission + +- `src/main/frontend/handler/db_based/sync.cljs` + - currently emits useful early download messages such as: + - `Preparing graph snapshot download` + - `Start downloading graph snapshot, file size: ...` + - `Graph snapshot downloaded` +- `src/main/frontend/handler/worker.cljs` +- `src/main/frontend/handler/events.cljs` +- `src/main/frontend/handler/db_based/rtc_flows.cljs` +- `src/main/frontend/components/rtc/indicator.cljs` + +These app-side files are useful references because they show the desired end-user progress semantics. However, the actual shared event source should live in worker/shared sync code, not in app-only UI handlers. + +## Design constraints + +1. **One shared progress model** + - CLI and app should consume the same logical progress events. + - Avoid a separate CLI-only progress protocol. + +2. **Invoke result remains authoritative** + - Realtime logs improve visibility but must not replace final command success/failure semantics. + +3. **Long-running timeout behavior must be command-aware** + - `sync download` should not rely on the same timeout assumptions as short metadata requests. + +4. **Output compatibility matters** + - Streaming progress should not break final human or machine-readable command results. + +## Proposed implementation + +The implementation should be delivered in four phases. + +--- + +## Phase 1: Make timeout handling explicit for long-running `sync download` + +### Objective + +Remove the dependency on the generic short CLI request timeout for the long-running download/import invoke path. + +### Files + +- `src/main/logseq/cli/command/sync.cljs` +- `src/main/logseq/cli/transport.cljs` +- `src/main/logseq/cli/config.cljs` +- `src/main/logseq/cli/command/core.cljs` + +### Tasks + +1. Trace exactly how `:timeout-ms` flows from CLI options/config into `transport/invoke` for `sync download`. +2. Introduce command-specific timeout behavior for `sync download`. +3. Keep the timeout policy explicit in code, rather than relying on an accidental global default. +4. Preserve existing timeout behavior for short non-download CLI commands unless intentionally changed. + +### Recommended implementation direction + +Prefer a command-specific long-task timeout path over raising the global default for all CLI traffic. + +Good options include: + +- passing a much larger timeout only for the final `db-sync-download-graph-by-id` invoke, or +- introducing a dedicated long-running request helper for commands that are expected to take a long time. + +Do **not** solve this by silently changing all CLI requests to use a large default timeout. + +### Acceptance criteria + +- `sync download` no longer depends on the generic `10000` ms timeout for the full download/import request. +- Other CLI requests keep their current short-request behavior unless explicitly updated. +- The effective timeout policy is easy to understand from the command implementation. + +### Verification + +- Unit test or integration test proving that `sync download` can run longer than the old short timeout path. +- Regression test showing short requests are unchanged. + +--- + +## Phase 2: Unify all download-progress logs into shared worker/sync code + +### Objective + +Ensure the full download flow emits shared progress events from the same worker/sync path used by both app and CLI. + +### Files + +- `src/main/frontend/worker/sync.cljs` +- `src/main/frontend/worker/db_core.cljs` +- `src/main/frontend/worker/sync/log_and_state.cljs` +- `src/main/frontend/handler/db_based/sync.cljs` + +### Tasks + +1. Inventory all download-progress log emissions related to `sync download` across app handlers and worker code. +2. Move or extract all shared milestones into worker/shared sync code where the remote snapshot and import/decrypt flow actually executes. +3. Reuse the existing `:rtc.log/download` event family and existing `sub-type` semantics wherever possible. +4. Remove duplicate app-only emission for shared milestones, and keep app handlers as consumers of shared events. +5. Keep milestone wording stable enough to avoid unnecessary UI regression in existing app consumers. + +### Required shared milestones + +The worker/shared path should emit at least these human-meaningful milestones: + +- preparing snapshot download, +- snapshot download started, including file size when available, +- snapshot download completed, +- saving/import/decrypt progress, +- graph ready / download complete. + +The final wording may differ, but the milestones must cover both: + +- network download progress stages, and +- local import/decrypt stages. + +### Acceptance criteria + +- Shared download milestones are emitted from worker/sync code across the full flow (download + import/decrypt). +- There is no competing app-only emission path for the same shared milestones. +- CLI-triggered `db-sync-download-graph-by-id` produces the same family of download progress events as app-triggered flows. +- Existing app consumers can still display download progress using the shared event source. + +### Verification + +- Worker-level or sync-level tests for emitted `:rtc-log` events. +- Manual verification in app that download progress still appears after the refactor. + +--- + +## Phase 3: Add CLI support for db-worker-node SSE event consumption + +### Objective + +Allow the CLI to subscribe to `/v1/events` while a long-running invoke is in progress. + +### Files + +- `src/main/logseq/cli/transport.cljs` +- `src/main/logseq/cli/command/sync.cljs` +- optionally a new helper namespace under `src/main/logseq/cli/` if event-stream logic should be isolated +- reference implementation: + - `src/main/frontend/persist_db/remote.cljs` + - `src/main/frontend/persist_db/node.cljs` + +### Tasks + +1. Add a lightweight CLI-side SSE client for db-worker-node `/v1/events`. +2. Decode incoming event payloads into the same shape consumed elsewhere in the codebase. +3. Support subscription lifecycle management: + - connect before starting the long-running invoke, + - receive events during the invoke, + - close cleanly on success, error, timeout, or interruption. +4. Keep the event client generic enough that future CLI commands could reuse it if needed. + +### Important behavior + +- The SSE stream is for observability, not command truth. +- If SSE disconnects, the invoke result still determines command success/failure. +- If the invoke finishes successfully, the CLI must stop listening and finalize normally. + +### Acceptance criteria + +- CLI can consume db-worker-node `/v1/events` while a command is in flight. +- Incoming `:rtc-log` events are decoded correctly. +- Subscription cleanup is reliable across success and failure paths. + +### Verification + +- Tests for event decode behavior. +- Tests or integration coverage for stream setup/cleanup. +- Manual verification against a running db-worker-node. + +--- + +## Phase 4: Render download progress in `sync download` without breaking final output + +### Objective + +Display realtime download/import progress in the terminal while preserving final result compatibility. + +### Files + +- `src/main/logseq/cli/command/sync.cljs` +- `src/main/logseq/cli/format.cljs` +- any new CLI event/render helper added in Phase 3 +- `docs/cli/logseq-cli.md` + +### Tasks + +1. Add `--progress` option to `sync download` command handling. +2. Define default behavior: progress enabled for human-oriented output mode. +3. Define structured-output behavior: progress automatically disabled for structured modes unless explicitly overridden by `--progress true`. +4. Subscribe to the worker event stream before invoking `db-sync-download-graph-by-id` when progress is enabled. +5. Filter only the relevant download log events for the active graph. +6. Render progress messages in chronological order to `stdout`. +7. Preserve the final success/failure output contract. +8. Document the new behavior in CLI docs, including mode-dependent defaults and override rules. + +### Output policy + +The implementation must make a clear separation between: + +- streaming progress lines, and +- the final command result. + +Confirmed direction: + +- stream progress to `stdout` when progress is enabled, +- add `--progress` option for `sync download`, +- default `--progress` to `true` for human-oriented output, +- automatically disable progress for structured output modes unless the user explicitly passes `--progress true`, +- keep the final result formatter responsible for terminal success/failure summary semantics. + +### Filtering policy + +The command should filter progress events using enough context to avoid printing unrelated logs. + +At minimum, filtering should consider the active graph identity. If a more precise operation-level filter is available without major complexity, prefer it. + +### Acceptance criteria + +- Running `logseq-cli sync download` with human-oriented output prints realtime progress lines to `stdout` during download/import. +- `--progress false` suppresses progress streaming. +- Structured output modes auto-disable progress unless `--progress true` is explicitly provided. +- Final command output still reflects the authoritative invoke result. +- Structured output parsing is not broken under the default mode-dependent progress behavior. + +### Verification + +- Integration test or high-confidence manual test showing visible staged output. +- Verification that final success output still matches expected formatter behavior. +- Verification that failure cases still return the correct final error. + +--- + +## Concrete execution order + +Implement in this order: + +1. **Phase 1 first** so large downloads no longer die under the short timeout path. +2. **Phase 2 second** so the CLI has a complete shared event source to consume. +3. **Phase 3 third** to add CLI event subscription infrastructure. +4. **Phase 4 last** to wire the streaming logs into `sync download` and finalize output behavior. + +Do not start Phase 4 before Phase 2 is complete, or the CLI will only show partial progress from the existing worker import stage. + +## Testing plan + +### Unit / focused tests + +Add or update tests in the most appropriate existing test namespaces for: + +- CLI timeout behavior for `sync download` +- event decoding / event subscription lifecycle +- worker/shared sync download log emission +- filtering and rendering of relevant `:rtc-log` events + +Likely test locations: + +- `src/test/logseq/cli/command/sync_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/frontend/worker/db_sync_test.cljs` +- `src/test/frontend/handler/db_based/sync_test.cljs` + +The exact namespace choices may differ depending on existing test structure, but the coverage categories above are required. + +### Manual verification checklist + +1. Start a `sync download` against a graph large enough to produce visible staged progress. +2. Confirm the terminal shows early snapshot-download messages. +3. Confirm the terminal shows import/decrypt/save progress. +4. Confirm progress lines are emitted to `stdout` in human-oriented mode. +5. Confirm `--progress false` suppresses streaming progress output. +6. Confirm structured output mode auto-disables progress by default. +7. Confirm structured output mode prints progress only when explicitly using `--progress true`. +8. Confirm successful completion still depends on the invoke result. +9. Confirm a slow download no longer fails under the old short timeout path. +10. Confirm existing app download progress still works after shared-log refactoring. +11. Confirm non-empty DB validation and other preflight failures remain unchanged. + +## Risks and open questions + +### Risk 1: app and CLI may need slightly different rendering + +The event source should be shared, but rendering may differ by client. + +Mitigation: + +- share event emission, not presentation details. +- keep worker messages human-readable enough for both app and CLI. + +### Risk 2: progress events may not uniquely identify one operation + +If multiple operations or graphs are active, CLI could print unrelated logs. + +Mitigation: + +- filter by graph identity at minimum, +- add more precise filtering only if needed and justified by actual ambiguity. + +### Risk 3: output-mode compatibility + +Streaming logs can interfere with structured output modes. + +Mitigation: + +- enforce mode-dependent default behavior (`progress` auto-off for structured output unless explicitly enabled), +- verify human and machine-readable modes explicitly. + +### Resolved decision: timeout strategy + +`sync download` uses command-level long-running timeout handling instead of a global timeout increase. + +### Open question 1 + +Should the CLI event-stream helper remain local to `sync download`, or be introduced as a reusable CLI transport helper? + +Recommendation: + +- prefer a small reusable helper if the abstraction stays simple. + +## Out-of-scope follow-ups + +The following can be considered later and are **not required** for this plan: + +- byte-level progress bars, +- richer TUI formatting, +- resumable download semantics, +- generalized event streaming for all CLI commands. + +## Definition of done + +This work is complete when all of the following are true: + +- `logseq-cli sync download` displays realtime progress while a large graph is downloading/importing. +- The progress messages come from the shared worker/app event model, not a CLI-only ad hoc implementation. +- Download-progress milestones are unified in shared worker/sync code across the full flow. +- `sync download` supports `--progress`, with mode-dependent default behavior as documented. +- In structured output modes, progress is auto-disabled unless explicitly enabled. +- Large downloads are no longer constrained by the old generic short timeout path. +- Final command success/failure semantics remain intact. +- Relevant automated tests and CLI docs are updated. + +## Recommendation + +Execute this as a shared-event refactor plus a command-specific long-request timeout change. + +The key architecture decision is to make worker/shared sync code the source of truth for download progress, then let the CLI subscribe to db-worker-node events just like the app already does. This maximizes reuse, keeps progress semantics aligned across clients, and solves the two real UX issues together: silent long-running work and premature short-timeout failures. \ No newline at end of file diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 8e842e8c6f..815ef609c3 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -76,7 +76,7 @@ Auth file contents include the persisted Cognito `id-token`, `access-token`, `re Verbose logging: - `--verbose` enables structured debug logs to stderr for CLI option parsing and db-worker-node API calls. -- stdout remains reserved for command output; large payloads are truncated in debug previews. +- `sync download` can stream realtime progress lines to stdout when progress is enabled; debug previews remain truncated. Timeouts: - `--timeout-ms` continues to control request timeout behavior for CLI transport. @@ -119,7 +119,7 @@ Sync commands: - `sync start --graph ` - start db-sync websocket client for a graph - `sync stop --graph ` - stop db-sync client on a graph daemon - `sync upload --graph ` - upload local graph snapshot to remote -- `sync download --graph ` - download remote graph `` into a same-name local graph directory +- `sync download --graph [--progress true|false]` - download remote graph `` into a same-name local graph directory - `sync remote-graphs [--graph ]` - list remote graphs visible to the current login context - `sync ensure-keys [--graph ]` - ensure user RSA keys for sync/e2ee - `sync grant-access --graph --graph-id --email ` - grant encrypted graph access to a user @@ -144,6 +144,11 @@ Sync download behavior: - If a local graph with the same name already exists, the CLI returns `graph-exists`. - If no remote graph with that name exists, the CLI returns `remote-graph-not-found`. - `sync download` starts `db-worker-node` in create-empty mode so local startup does not write `db-initial-data` before snapshot import. +- The final snapshot download/import invoke uses a command-specific long-running timeout (30 minutes by default) rather than the generic short-request timeout path. +- Progress streaming uses db-worker-node SSE `/v1/events` and shared `:rtc.log/download` events. +- `--progress` defaults to `true` for human output. +- For structured output (`--output json|edn`), progress is auto-disabled unless explicitly overridden with `--progress true`. +- `--progress false` always suppresses progress streaming. - If the target graph DB is not empty at download time, the CLI returns `graph-db-not-empty` and aborts before import. - For e2ee remote graphs in headless CLI mode, run `logseq login` first and set `e2ee-password` via `sync config set` (or in `--config`) before download. @@ -210,6 +215,7 @@ Output formats: - Global `--output ` applies to all commands - Output formatting is controlled via global `--output`, `:output-format` in config, or `LOGSEQ_CLI_OUTPUT`. - Human output is plain text. List/search commands render tables with a final `Count: N` line. For list and search subcommands, the ID column uses `:db/id` (not UUID). If `:db/ident` exists, an `IDENT` column is included. `list property` includes a dedicated `TYPE` column. Search table columns are `ID` and `TITLE`. Block titles can include multiple lines; multi-line rows align additional lines under the `TITLE` column. Times such as list `UPDATED-AT`/`CREATED-AT` and `graph info` `Created at` are shown in human-friendly relative form. Errors include error codes and may include a `Hint:` line. Use `--output json|edn` for structured output. +- `sync download` progress lines are streamed to stdout only when progress is enabled. In `json`/`edn` mode, progress is disabled by default unless `--progress true` is provided. - For `list property`, `TYPE` is returned in default output (without `--expand`) for human and structured (`json`/`edn`) formats. - `upsert page` and `upsert block` return entity ids in `data.result` for JSON/EDN output, and include ids in human output. - Human example: diff --git a/src/main/frontend/handler/db_based/sync.cljs b/src/main/frontend/handler/db_based/sync.cljs index b5718e3bfb..48d7757d2e 100644 --- a/src/main/frontend/handler/db_based/sync.cljs +++ b/src/main/frontend/handler/db_based/sync.cljs @@ -32,94 +32,6 @@ (or config/db-sync-http-base (ws->http-base config/db-sync-ws-url))) -(def ^:private snapshot-text-decoder (js/TextDecoder.)) - -(defn- ->uint8 [data] - (cond - (instance? js/Uint8Array data) data - (instance? js/ArrayBuffer data) (js/Uint8Array. data) - (string? data) (.encode (js/TextEncoder.) data) - :else (js/Uint8Array. data))) - -(defn- decode-snapshot-rows [payload] - (sqlite-util/read-transit-str (.decode snapshot-text-decoder (->uint8 payload)))) - -(defn- frame-len [^js data offset] - (let [view (js/DataView. (.-buffer data) offset 4)] - (.getUint32 view 0 false))) - -(defn- concat-bytes - [^js a ^js b] - (cond - (nil? a) b - (nil? b) a - :else - (let [out (js/Uint8Array. (+ (.-byteLength a) (.-byteLength b)))] - (.set out a 0) - (.set out b (.-byteLength a)) - out))) - -(defn- parse-framed-chunk - [buffer chunk] - (let [data (concat-bytes buffer chunk) - total (.-byteLength data)] - (loop [offset 0 - rows []] - (if (< (- total offset) 4) - {:rows rows - :buffer (when (< offset total) - (.slice data offset total))} - (let [len (frame-len data offset) - next-offset (+ offset 4 len)] - (if (<= next-offset total) - (let [payload (.slice data (+ offset 4) next-offset) - decoded (decode-snapshot-rows payload)] - (recur next-offset (into rows decoded))) - {:rows rows - :buffer (.slice data offset total)})))))) - -(defn- finalize-framed-buffer - [buffer] - (if (or (nil? buffer) (zero? (.-byteLength buffer))) - [] - (let [{:keys [rows buffer]} (parse-framed-chunk nil buffer)] - (if (and (seq rows) (or (nil? buffer) (zero? (.-byteLength buffer)))) - rows - (throw (ex-info "incomplete framed buffer" {:buffer buffer :rows rows})))))) - -(defn- gzip-bytes? - [^js payload] - (and (some? payload) - (>= (.-byteLength payload) 2) - (= 31 (aget payload 0)) - (= 139 (aget payload 1)))) - -(defn- bytes->stream - [^js payload] - (js/ReadableStream. - #js {:start (fn [controller] - (.enqueue controller payload) - (.close controller))})) - -(defn- stream payload) - decompressed (.pipeThrough stream (js/DecompressionStream. "gzip")) - resp (js/Response. decompressed) - buf (.arrayBuffer resp)] - (->uint8 buf)) - (p/rejected (ex-info "gzip decompression not supported" - {:type :db-sync/decompression-not-supported})))) - -(defn- uint8 buf)] - (if (gzip-bytes? payload) - ( (if (and graph-uuid base) - (-> (p/let [_ (js/Promise. user-handler/task--ensure-id&access-token) - pull-resp (fetch-json (str base "/sync/" graph-uuid "/pull") - {:method "GET"} - {:response-schema :sync/pull}) - remote-tx (:t pull-resp) - _ (when-not (integer? remote-tx) - (throw (ex-info "non-integer remote-tx when downloading graph" - {:graph graph-name - :remote-tx remote-tx}))) - resp (js/fetch (str base "/sync/" graph-uuid "/snapshot/stream") - (clj->js (with-auth-headers {:method "GET"}))) - total-bytes (when-let [raw (some-> resp .-headers (.get "content-length"))] - (let [parsed (js/parseInt raw 10)] - (when-not (js/isNaN parsed) parsed))) - _ (state/pub-event! - [:rtc/log {:type :rtc.log/download - :sub-type :download-progress - :graph-uuid graph-uuid - :message (str "Start downloading graph snapshot, file size: " total-bytes)}])] - (when-not (.-ok resp) - (throw (ex-info "snapshot download failed" - {:graph graph-name - :status (.-status resp)}))) - (p/let [snapshot-bytes ( (if (seq graph-uuid) + (p/let [_ (js/Promise. user-handler/task--ensure-id&access-token)] + (state/ resp .-headers (.get "content-length"))] + (let [parsed (js/parseInt raw 10)] + (when-not (js/isNaN parsed) + parsed)))) + (defn- download-graph-with-id! [repo graph-id graph-e2ee?] (let [base (http-base-url) @@ -2270,7 +2285,8 @@ :else (do (require-auth-token! {:repo repo :field :auth-token}) - (p/let [pull-resp (fetch-json (str base "/sync/" graph-id "/pull") + (p/let [_ (download-log! graph-id :download-progress "Preparing graph snapshot download") + pull-resp (fetch-json (str base "/sync/" graph-id "/pull") {:method "GET"} {:response-schema :sync/pull}) remote-tx (:t pull-resp) @@ -2280,12 +2296,19 @@ :value remote-tx})) resp (js/fetch (str base "/sync/" graph-id "/snapshot/stream") (clj->js (with-auth-headers {:method "GET"}))) + total-bytes (response-content-length resp) + _ (download-log! graph-id + :download-progress + (if (number? total-bytes) + (str "Start downloading graph snapshot, file size: " total-bytes) + "Start downloading graph snapshot")) _ (when-not (.-ok resp) (fail-fast :db-sync/snapshot-download-failed {:repo repo :graph-id graph-id :status (.-status resp)})) payload ( line str string/trim)) + (println line))) + +(defn- sync-download-invoke-config + [cfg] + (assoc cfg :timeout-ms (max 0 (or (:sync-download-timeout-ms cfg) + sync-download-timeout-ms)))) + (def ^:private sync-download-non-empty-query '[:find (count ?e) . :where @@ -112,7 +128,9 @@ :repo repo :graph (core/repo->graph repo) :allow-missing-graph true - :require-missing-graph true}}) + :require-missing-graph true + :progress (:progress options) + :progress-explicit? (contains? options :progress)}}) :sync-remote-graphs {:ok? true @@ -314,9 +332,24 @@ (p/catch (fn [error] (exception->error error {:repo (:repo action)}))))) +(defn- sync-download-progress-enabled? + [action config] + (if (:progress-explicit? action) + (true? (:progress action)) + (not (contains? structured-output-formats (:output-format config))))) + +(defn- download-progress-message + [graph-id event-type payload] + (when (and (= :rtc-log event-type) + (map? payload) + (= :rtc.log/download (:type payload)) + (= graph-id (:graph-uuid payload))) + (:message payload))) + (defn- execute-sync-download [action config] - (let [config' (download-config config)] + (let [config' (download-config config) + progress-enabled? (sync-download-progress-enabled? action config')] (-> (p/let [remote-graphs (invoke-global config' :thread-api/db-sync-list-remote-graphs []) @@ -332,8 +365,19 @@ (p/let [cfg (cli-server/ensure-server! config' (:repo action)) _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) _ (ensure-empty-download-db! cfg (:repo action)) - result (transport/invoke cfg :thread-api/db-sync-download-graph-by-id false - [(:repo action) (:graph-id remote-graph) (:graph-e2ee? remote-graph)])] + download-cfg (sync-download-invoke-config cfg) + graph-id (:graph-id remote-graph) + events-sub (when progress-enabled? + (transport/connect-events! + download-cfg + (fn [event-type payload] + (when-let [message (download-progress-message graph-id event-type payload)] + (print-progress-line! message))))) + result (-> (transport/invoke download-cfg :thread-api/db-sync-download-graph-by-id false + [(:repo action) graph-id (:graph-e2ee? remote-graph)]) + (p/finally (fn [] + (when-let [close! (:close! events-sub)] + (close!)))))] {:status :ok :data (if (map? result) result diff --git a/src/main/logseq/cli/transport.cljs b/src/main/logseq/cli/transport.cljs index 6e72f443f5..0ba83a372d 100644 --- a/src/main/logseq/cli/transport.cljs +++ b/src/main/logseq/cli/transport.cljs @@ -125,6 +125,98 @@ :response response-preview) decoded))))) +(defn- decode-event + [{:keys [type payload]}] + (let [decoded (when (some? payload) + (try + (ldb/read-transit-str payload) + (catch :default _ + payload)))] + (if (and (vector? decoded) + (= 2 (count decoded)) + (keyword? (first decoded))) + [(first decoded) (second decoded)] + [(when type (keyword type)) decoded]))) + +(defn- data-line + [event-text] + (some (fn [line] + (when (string/starts-with? line "data: ") + (subs line 6))) + (string/split-lines event-text))) + +(defn connect-events! + [{:keys [base-url]} on-event] + (let [handler (or on-event (fn [_event-type _payload] nil)) + url (js/URL. (str (string/replace (or base-url "") #"/$" "") "/v1/events")) + buffer (atom "") + *req (atom nil) + *res (atom nil) + *closed? (atom false) + dispatch! (fn [event-text] + (when-let [line (data-line event-text)] + (try + (let [event-map (js->clj (js/JSON.parse line) :keywordize-keys true) + [event-type payload] (decode-event event-map)] + (when (some? event-type) + (handler event-type payload))) + (catch :default e + (log/debug :event :cli.transport/events-parse-failed + :error e + :line line))))) + consume-chunk! (fn [chunk] + (swap! buffer str (.toString chunk "utf8")) + (loop [] + (let [current @buffer + idx (string/index-of current "\n\n")] + (when (some? idx) + (let [event-text (subs current 0 idx) + rest-text (subs current (+ idx 2))] + (reset! buffer rest-text) + (dispatch! event-text) + (recur)))))) + close! (fn [] + (reset! *closed? true) + (when-let [^js res @*res] + (try + (.destroy res) + (catch :default _ nil))) + (when-let [^js req @*req] + (try + (.destroy req) + (catch :default _ nil))) + nil)] + (try + (let [req (.request + (request-module url) + #js {:method "GET" + :hostname (.-hostname url) + :port (request-port url) + :path (str (.-pathname url) (.-search url)) + :headers (clj->js {"Accept" "text/event-stream"})} + (fn [^js res] + (reset! *res res) + (.on res "data" + (fn [chunk] + (when-not @*closed? + (consume-chunk! chunk)))) + (.on res "error" + (fn [e] + (when-not @*closed? + (log/debug :event :cli.transport/events-stream-error + :error e))))))] + (reset! *req req) + (.on req "error" + (fn [e] + (when-not @*closed? + (log/debug :event :cli.transport/events-request-error + :error e)))) + (.end req)) + (catch :default e + (log/debug :event :cli.transport/events-connect-failed + :error e))) + {:close! close!})) + (defn write-output [{:keys [format path data]}] (case format diff --git a/src/test/frontend/handler/db_based/sync_test.cljs b/src/test/frontend/handler/db_based/sync_test.cljs index 189856d5d4..dddff7bb92 100644 --- a/src/test/frontend/handler/db_based/sync_test.cljs +++ b/src/test/frontend/handler/db_based/sync_test.cljs @@ -7,35 +7,8 @@ [frontend.handler.user :as user-handler] [frontend.state :as state] [logseq.db :as ldb] - [logseq.db.sqlite.util :as sqlite-util] [promesa.core :as p])) -(def ^:private test-text-encoder (js/TextEncoder.)) - -(defn- frame-bytes [^js data] - (let [len (.-byteLength data) - out (js/Uint8Array. (+ 4 len)) - view (js/DataView. (.-buffer out))] - (.setUint32 view 0 len false) - (.set out data 4) - out)) - -(defn- encode-framed-rows [rows] - (let [payload (.encode test-text-encoder (sqlite-util/write-transit-str rows))] - (frame-bytes payload))) - -(defn- (p/let [gzip-bytes ( (p/with-redefs [db-sync/http-base (fn [] "http://base") - db-sync/fetch-json (fn [url _opts _schema] - (cond - (string/ends-with? url "/pull") - (p/resolved {:t 42}) - - :else - (p/rejected (ex-info "unexpected fetch-json URL" - {:url url})))) - user-handler/task--ensure-id&access-token (fn [resolve _reject] - (resolve true)) - state/ (p/with-redefs [state/set-state! (fn [k v] + (swap! state-calls conj [k v])) + state/pub-event! (fn [event] + (swap! pub-events conj event) + nil) + user-handler/task--ensure-id&access-token (fn [resolve _reject] + (resolve true)) + state/ (p/let [gzip-bytes ( (p/with-redefs [db-sync/http-base (constantly "http://base") - db-sync/fetch-json (fn [url _opts _schema] - (cond - (string/ends-with? url "/pull") - (p/resolved {:t 8}) - - :else - (p/rejected (ex-info "unexpected fetch-json URL" - {:url url})))) - user-handler/task--ensure-id&access-token (fn [resolve _reject] + (let [worker-calls (atom [])] + (-> (p/with-redefs [user-handler/task--ensure-id&access-token (fn [resolve _reject] (resolve true)) - state/ (p/with-redefs [state/set-state! (fn [k v] + (swap! state-calls conj [k v])) + state/pub-event! (fn [event] + (swap! pub-events conj event) + nil) + user-handler/task--ensure-id&access-token (fn [resolve _reject] + (resolve true)) + state/ error ex-data :code))) + (is (= [[:rtc/downloading-graph-uuid "graph-3"] + [:rtc/downloading-graph-uuid nil]] + @state-calls)) + (is (empty? @pub-events)) + (done))))))) diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index 0b1060bb84..57d57b0db7 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -1,5 +1,6 @@ (ns frontend.worker.db-sync-test (:require [cljs.test :refer [deftest is testing async use-fixtures]] + [clojure.string :as string] [datascript.core :as d] [frontend.common.crypt :as crypt] [frontend.worker-common.util :as worker-util] @@ -9,6 +10,7 @@ [frontend.worker.sync :as db-sync] [frontend.worker.sync.client-op :as client-op] [frontend.worker.sync.crypt :as sync-crypt] + [frontend.worker.sync.log-and-state :as sync-log-and-state] [logseq.common.config :as common-config] [logseq.db :as ldb] [logseq.db.frontend.validate :as db-validate] @@ -1323,6 +1325,59 @@ (reset! worker-state/*db-sync-config config-prev) (done)))))))) +(deftest download-graph-by-id-emits-shared-download-milestones-test + (testing "download emits shared rtc.log/download milestones" + (async done + (let [fetch-prev js/fetch + config-prev @worker-state/*db-sync-config + logs (atom []) + rows [["addr-1" "content-1" nil]] + payload (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows))] + (reset! worker-state/*db-sync-config {:http-base "https://example.com" + :auth-token "token-value"}) + (set! js/fetch + (fn [url _opts] + (cond + (>= (.indexOf url "/pull") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":9,\"txs\":[]}"))}) + + (>= (.indexOf url "/snapshot/stream") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :headers #js {:get (fn [header] + (when (= header "content-length") + (str (.-byteLength payload))))} + :arrayBuffer (fn [] (js/Promise.resolve (.-buffer payload)))}) + + :else + (js/Promise.reject (js/Error. (str "unexpected fetch url: " url)))))) + (-> (p/with-redefs [sync-log-and-state/rtc-log (fn [type payload] + (swap! logs conj [type payload]) + nil)] + (db-sync/download-graph-by-id! test-repo "graph-1" false)) + (p/then (fn [result] + (is (= "graph-1" (:graph-id result))) + (is (= 9 (:remote-tx result))) + (is (= false (:graph-e2ee? result))) + (is (= rows (vec (:rows result)))) + (let [messages (mapv (fn [[_ payload]] (:message payload)) @logs)] + (is (some #(= "Preparing graph snapshot download" %) messages)) + (is (some #(string/includes? % "Start downloading graph snapshot") messages)) + (is (some #(= "Graph snapshot downloaded" %) messages))) + (is (every? (fn [[type payload]] + (and (= :rtc.log/download type) + (= "graph-1" (:graph-uuid payload)))) + @logs)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally + (fn [] + (set! js/fetch fetch-prev) + (reset! worker-state/*db-sync-config config-prev) + (done)))))))) + (deftest ensure-upload-graph-identity-creates-remote-graph-when-local-graph-id-missing-test (testing "first upload bootstrap creates and persists a remote graph when local metadata is missing" (async done diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index 4e9f83791a..79de171639 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -28,6 +28,15 @@ (is (true? (get-in result [:action :allow-missing-graph]))) (is (true? (get-in result [:action :require-missing-graph]))))) + (testing "sync download action keeps progress option and explicit flag" + (let [default-result (sync-command/build-action :sync-download {} [] "logseq_db_demo") + explicit-result (sync-command/build-action :sync-download {:progress false} [] "logseq_db_demo")] + (is (true? (:ok? default-result))) + (is (= false (get-in default-result [:action :progress-explicit?]))) + (is (true? (:ok? explicit-result))) + (is (= false (get-in explicit-result [:action :progress]))) + (is (= true (get-in explicit-result [:action :progress-explicit?]))))) + (testing "sync config set requires name and value" (let [missing-both (sync-command/build-action :sync-config-set {} [] nil) missing-value (sync-command/build-action :sync-config-set {} ["ws-url"] nil)] @@ -313,6 +322,133 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) +(deftest test-execute-sync-download-uses-long-timeout-only-for-download-invoke + (async done + (let [invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [cfg method direct-pass? args] + (swap! invoke-calls conj {:method method + :direct-pass? direct-pass? + :args args + :timeout-ms (:timeout-ms cfg)}) + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? false}]) + :thread-api/q + (p/resolved 0) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [result (execute-with-runtime-auth {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp" + :timeout-ms 10000}) + [set-config-before list-remote-graphs set-config-after check-empty-db download] + @invoke-calls] + (is (= :ok (:status result))) + (is (= :thread-api/set-db-sync-config (:method set-config-before))) + (is (= :thread-api/db-sync-list-remote-graphs (:method list-remote-graphs))) + (is (= :thread-api/set-db-sync-config (:method set-config-after))) + (is (= :thread-api/q (:method check-empty-db))) + (is (= :thread-api/db-sync-download-graph-by-id (:method download))) + (is (= 10000 (:timeout-ms set-config-before))) + (is (= 10000 (:timeout-ms list-remote-graphs))) + (is (= 10000 (:timeout-ms set-config-after))) + (is (= 10000 (:timeout-ms check-empty-db))) + (is (= 1800000 (:timeout-ms download))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest test-execute-sync-download-progress-mode-behavior + (async done + (let [subscribe-calls (atom []) + close-calls (atom 0) + printed-lines (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] + (p/resolved (assoc config :base-url "http://example"))) + transport/connect-events! (fn [cfg on-event] + (swap! subscribe-calls conj {:base-url (:base-url cfg) + :timeout-ms (:timeout-ms cfg)}) + (on-event :rtc-log {:type :rtc.log/download + :graph-uuid "remote-graph-id" + :message "Preparing graph snapshot download"}) + (on-event :rtc-log {:type :rtc.log/download + :graph-uuid "other-graph-id" + :message "should be filtered"}) + {:close! (fn [] + (swap! close-calls inc))}) + sync-command/print-progress-line! (fn [line] + (swap! printed-lines conj line) + nil) + transport/invoke (fn [_ method _direct-pass? _args] + (case method + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? false}]) + :thread-api/q + (p/resolved 0) + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + (p/resolved nil)))] + (p/let [_ (execute-with-runtime-auth {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :progress-explicit? false} + {:base-url "http://example" + :data-dir "/tmp" + :output-format nil}) + _ (is (= 1 (count @subscribe-calls))) + _ (is (= ["Preparing graph snapshot download"] @printed-lines)) + _ (is (= 1 @close-calls)) + _ (reset! subscribe-calls []) + _ (reset! printed-lines []) + _ (reset! close-calls 0) + _ (execute-with-runtime-auth {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :progress-explicit? false} + {:base-url "http://example" + :data-dir "/tmp" + :output-format :json}) + _ (is (= [] @subscribe-calls)) + _ (is (= [] @printed-lines)) + _ (is (= 0 @close-calls)) + _ (execute-with-runtime-auth {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :progress true + :progress-explicit? true} + {:base-url "http://example" + :data-dir "/tmp" + :output-format :json}) + _ (is (= 1 (count @subscribe-calls))) + _ (is (= ["Preparing graph snapshot download"] @printed-lines)) + _ (is (= 1 @close-calls)) + _ (reset! subscribe-calls []) + _ (reset! printed-lines []) + _ (reset! close-calls 0) + _ (execute-with-runtime-auth {:type :sync-download + :repo "logseq_db_demo" + :graph "demo" + :progress false + :progress-explicit? true} + {:base-url "http://example" + :data-dir "/tmp" + :output-format nil})] + (is (= [] @subscribe-calls)) + (is (= [] @printed-lines)) + (is (= 0 @close-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest test-execute-sync-download-uses-graph-config-when-base-url-missing (async done (let [ensure-calls (atom []) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 3156023b4b..420c778517 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1259,6 +1259,14 @@ (is (false? (:ok? result))) (is (= :missing-graph (get-in result [:error :code]))))) + (testing "sync download accepts progress option" + (let [disabled (commands/parse-args ["sync" "download" "--graph" "demo" "--progress" "false"]) + enabled (commands/parse-args ["sync" "download" "--graph" "demo" "--progress" "true"])] + (is (true? (:ok? disabled))) + (is (= false (get-in disabled [:options :progress]))) + (is (true? (:ok? enabled))) + (is (= true (get-in enabled [:options :progress]))))) + (testing "graph import rejects unknown type" (let [result (commands/parse-args ["graph" "import" "--type" "zip" diff --git a/src/test/logseq/cli/transport_test.cljs b/src/test/logseq/cli/transport_test.cljs index e23bb44fd2..12c77c1e0b 100644 --- a/src/test/logseq/cli/transport_test.cljs +++ b/src/test/logseq/cli/transport_test.cljs @@ -2,6 +2,7 @@ (:require [cljs.test :refer [deftest is async testing]] [logseq.cli.test-helper :as test-helper] [logseq.cli.transport :as transport] + [logseq.db :as ldb] [promesa.core :as p])) (def ^:private fs (js/require "fs")) @@ -146,6 +147,45 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest test-connect-events-decodes-rtc-log-and-cleans-up + (async done + (let [received (atom []) + client-closed? (atom false)] + (-> (p/let [{:keys [url stop!]} (start-server + (fn [^js req ^js res] + (if (= "/v1/events" (.-url req)) + (do + (.writeHead res 200 #js {"Content-Type" "text/event-stream" + "Cache-Control" "no-cache" + "Connection" "keep-alive"}) + (let [payload (ldb/write-transit-str {:type :rtc.log/download + :graph-uuid "graph-1" + :message "Preparing graph snapshot download"}) + data (js/JSON.stringify #js {:type "rtc-log" + :payload payload})] + (.write res (str "data: " data "\n\n"))) + (.on req "close" (fn [] + (reset! client-closed? true)))) + (do + (.writeHead res 404 #js {"Content-Type" "text/plain"}) + (.end res "not-found")))))] + (let [{:keys [close!]} (transport/connect-events! {:base-url url} + (fn [event-type payload] + (swap! received conj [event-type payload])))] + (p/let [_ (p/delay 50) + _ (close!) + _ (p/delay 50)] + (is (= [[:rtc-log {:type :rtc.log/download + :graph-uuid "graph-1" + :message "Preparing graph snapshot download"}]] + @received)) + (is (= true @client-closed?)) + (stop!)))) + (p/then (fn [_] (done))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest test-read-input (testing "reads edn input" (let [file-path (temp-path "input.edn")] From 051f6381e35c77cb59a02e4dd517c9573ce63aaf Mon Sep 17 00:00:00 2001 From: Danzu Date: Thu, 5 Mar 2026 16:11:25 -0600 Subject: [PATCH 133/375] Add tests for completions command and completion generator - Introduced `completions_test.cljs` to validate the structure and behavior of the completions command registration and argument parsing. - Added `completion_generator_test.cljs` to extensively test the completion generator, including spec metadata validation, Zsh and Bash output generation, and end-to-end checks for command entries and structural markers. --- dz/scripts/shell-completions/DESIGN.md | 621 ++++++++++++++ dz/scripts/shell-completions/README.md | 191 +++++ dz/scripts/shell-completions/TASKS.md | 283 +++++++ dz/scripts/shell-completions/_logseq.zsh | 783 +++++++++++++++++ dz/scripts/shell-completions/logseq.bash | 274 ++++++ src/main/logseq/cli/command/completions.cljs | 12 + src/main/logseq/cli/command/core.cljs | 18 +- src/main/logseq/cli/command/graph.cljs | 12 +- src/main/logseq/cli/command/list.cljs | 15 +- src/main/logseq/cli/command/query.cljs | 3 +- src/main/logseq/cli/command/remove.cljs | 3 +- src/main/logseq/cli/command/show.cljs | 3 +- src/main/logseq/cli/command/upsert.cljs | 22 +- src/main/logseq/cli/commands.cljs | 14 +- src/main/logseq/cli/completion_generator.cljs | 786 ++++++++++++++++++ .../logseq/cli/command/completions_test.cljs | 40 + .../logseq/cli/completion_generator_test.cljs | 333 ++++++++ 17 files changed, 3388 insertions(+), 25 deletions(-) create mode 100644 dz/scripts/shell-completions/DESIGN.md create mode 100644 dz/scripts/shell-completions/README.md create mode 100644 dz/scripts/shell-completions/TASKS.md create mode 100644 dz/scripts/shell-completions/_logseq.zsh create mode 100644 dz/scripts/shell-completions/logseq.bash create mode 100644 src/main/logseq/cli/command/completions.cljs create mode 100644 src/main/logseq/cli/completion_generator.cljs create mode 100644 src/test/logseq/cli/command/completions_test.cljs create mode 100644 src/test/logseq/cli/completion_generator_test.cljs diff --git a/dz/scripts/shell-completions/DESIGN.md b/dz/scripts/shell-completions/DESIGN.md new file mode 100644 index 0000000000..e5f67de7f3 --- /dev/null +++ b/dz/scripts/shell-completions/DESIGN.md @@ -0,0 +1,621 @@ +# Shell Completions — Full Requirement (Option 2) + +Generate shell completions from the CLI command table so that adding or +modifying a command/flag in the spec is the **only** change needed — the +completion scripts stay in sync automatically. + +--- + +## 1. Overview + +The Logseq CLI (`src/main/logseq/cli`) uses `babashka.cli` for argument +parsing. Commands and their options are declared as data in a `table` of +`{:cmds [...] :spec {...}}` entries assembled in `commands.cljs`. + +A new `logseq completions ` command will walk the `table` at +generation time and emit correct zsh and bash completion scripts. + +``` +logseq completions zsh > ~/.zsh/completions/_logseq +logseq completions bash > ~/.local/share/bash-completion/completions/logseq +``` + +--- + +## 2. Complete command inventory + +The table below is the single source of truth. Every row **must** appear in +the generated dispatch structure. Commands with no command-specific options +still inherit the global spec. + +### 2.1 Command groups and subcommands + +| Group | Subcommands | Command key | +| ------------- | ---------------------------------------------------------------------------- | ------------------------------------ | +| `graph` | `list`, `create`, `switch`, `remove`, `validate`, `info`, `export`, `import` | `:graph-list` … `:graph-import` | +| `server` | `list`, `status`, `start`, `stop`, `restart` | `:server-list` … `:server-restart` | +| `list` | `page`, `tag`, `property` | `:list-page` … `:list-property` | +| `upsert` | `block`, `page`, `tag`, `property` | `:upsert-block` … `:upsert-property` | +| `remove` | `block`, `page`, `tag`, `property` | `:remove-block` … `:remove-property` | +| `query` | _(root)_, `list` | `:query`, `:query-list` | +| `show` | _(leaf — no subcommands)_ | `:show` | +| `doctor` | _(leaf — no subcommands)_ | `:doctor` | +| `completions` | _(leaf — new)_ | `:completions` | + +### 2.2 Global spec (`command/core.cljs`) + +These options are available on **every** command. The merged spec +(`merge global-spec* command-spec`) is already what each table entry carries. + +| Option | Alias | `:coerce` | `:values` | `:complete` | Notes | +| -------------- | ----- | ---------- | ------------------------ | ----------- | -------------------- | +| `--help` | `-h` | `:boolean` | — | — | | +| `--version` | — | `:boolean` | — | — | | +| `--config` | — | — | — | `:file` | Path to `cli.edn` | +| `--graph` | — | — | — | `:graphs` | Graph name (dynamic) | +| `--data-dir` | — | — | — | `:dir` | Path to data dir | +| `--timeout-ms` | — | `:long` | — | — | | +| `--output` | — | — | `["human" "json" "edn"]` | — | Output format | +| `--verbose` | — | `:boolean` | — | — | | + +### 2.3 Command-specific specs + +#### `graph export` + +| Option | `:values` | `:complete` | +| -------- | ------------------ | ----------- | +| `--type` | `["edn" "sqlite"]` | — | +| `--file` | — | `:file` | + +#### `graph import` + +| Option | `:values` | `:complete` | +| --------- | ------------------ | ----------- | +| `--type` | `["edn" "sqlite"]` | — | +| `--input` | — | `:file` | + +#### `graph list/create/switch/remove/validate/info` + +No command-specific options (global-only). + +#### `server status/start/stop/restart` + +| Option | `:complete` | Notes | +| --------- | ----------- | -------------------------------------------------- | +| `--graph` | `:graphs` | Redundant with global — shadows it (same behavior) | + +#### `server list` + +No command-specific options. + +#### `list page` + +| Option | `:coerce` | `:values` | +| ------------------- | ---------- | ------------------------------------- | +| `--expand` | `:boolean` | — | +| `--limit` | `:long` | — | +| `--offset` | `:long` | — | +| `--sort` | — | `["title" "created-at" "updated-at"]` | +| `--order` | — | `["asc" "desc"]` | +| `--include-journal` | `:boolean` | — | +| `--journal-only` | `:boolean` | — | +| `--include-hidden` | `:boolean` | — | +| `--updated-after` | — | — | +| `--created-after` | — | — | +| `--fields` | — | — | + +#### `list tag` + +| Option | `:coerce` | `:values` | +| -------------------- | ---------- | ------------------ | +| `--expand` | `:boolean` | — | +| `--limit` | `:long` | — | +| `--offset` | `:long` | — | +| `--sort` | — | `["name" "title"]` | +| `--order` | — | `["asc" "desc"]` | +| `--include-built-in` | `:boolean` | — | +| `--with-properties` | `:boolean` | — | +| `--with-extends` | `:boolean` | — | +| `--fields` | — | — | + +#### `list property` + +| Option | `:coerce` | `:values` | +| -------------------- | ---------- | ------------------ | +| `--expand` | `:boolean` | — | +| `--limit` | `:long` | — | +| `--offset` | `:long` | — | +| `--sort` | — | `["name" "title"]` | +| `--order` | — | `["asc" "desc"]` | +| `--include-built-in` | `:boolean` | — | +| `--with-classes` | `:boolean` | — | +| `--with-type` | `:boolean` | — | +| `--fields` | — | — | + +#### `upsert block` + +| Option | `:coerce` | `:values` | `:complete` | +| --------------------- | --------- | ------------------------------------------------------------------------------------------------------------------- | ----------- | +| `--id` | `:long` | — | — | +| `--uuid` | — | — | — | +| `--target-id` | `:long` | — | — | +| `--target-uuid` | — | — | — | +| `--target-page` | — | — | `:pages` | +| `--pos` | — | `["first-child" "last-child" "sibling"]` | — | +| `--content` | — | — | — | +| `--blocks` | — | — | — | +| `--blocks-file` | — | — | `:file` | +| `--status` | — | `["todo" "doing" "done" "now" "later" "wait" "waiting" "backlog" "canceled" "cancelled" "in-review" "in-progress"]` | — | +| `--update-tags` | — | — | — | +| `--update-properties` | — | — | — | +| `--remove-tags` | — | — | — | +| `--remove-properties` | — | — | — | + +> **Note on `--status` values:** The CLI also accepts aliases like `in_review`, +> `inreview`, `in progress`, `inprogress`. These are **not** included in +> completions — only canonical hyphenated forms are offered. The aliases +> remain accepted at runtime. + +#### `upsert page` + +| Option | `:coerce` | `:complete` | +| --------------------- | --------- | ----------- | +| `--id` | `:long` | — | +| `--page` | — | `:pages` | +| `--update-tags` | — | — | +| `--update-properties` | — | — | +| `--remove-tags` | — | — | +| `--remove-properties` | — | — | + +#### `upsert tag` + +| Option | `:coerce` | +| -------- | --------- | +| `--id` | `:long` | +| `--name` | — | + +No `:complete` — free text. + +#### `upsert property` + +| Option | `:coerce` | `:values` | +| --------------- | ---------- | -------------------------------------------------------------------------------- | +| `--id` | `:long` | — | +| `--name` | — | — | +| `--type` | — | `["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"]` | +| `--cardinality` | — | `["one" "many"]` | +| `--hide` | `:boolean` | — | +| `--public` | `:boolean` | — | + +#### `remove block` + +| Option | Notes | +| -------- | ------------------------------- | +| `--id` | Free text (db/id or EDN vector) | +| `--uuid` | Free text | + +#### `remove page` + +| Option | `:complete` | +| -------- | ----------- | +| `--name` | `:pages` | + +#### `remove tag` + +| Option | `:coerce` | +| -------- | --------- | +| `--id` | `:long` | +| `--name` | — | + +No `:complete` — free text. + +#### `remove property` + +| Option | `:coerce` | +| -------- | --------- | +| `--id` | `:long` | +| `--name` | — | + +No `:complete` — free text. + +#### `query` + +| Option | `:complete` | +| ---------- | ----------- | +| `--query` | — | +| `--name` | `:queries` | +| `--inputs` | — | + +> `:name` gets `:complete :queries` **only** in the `query` command spec. +> In `upsert tag` and `upsert property`, `:name` is free text — no +> `:complete` key. This is expressed naturally because each command has its +> own spec. + +#### `query list` + +No command-specific options. + +#### `show` + +| Option | `:coerce` | `:complete` | +| --------------------- | ---------- | ----------- | +| `--id` | — | — | +| `--uuid` | — | — | +| `--page` | — | `:pages` | +| `--linked-references` | `:boolean` | — | +| `--level` | `:long` | — | + +#### `doctor` + +| Option | `:coerce` | +| -------------- | ---------- | +| `--dev-script` | `:boolean` | + +#### `completions` _(new)_ + +| Option | `:values` | +| --------- | ---------------- | +| `--shell` | `["zsh" "bash"]` | + +--- + +## 3. Spec enrichment — new metadata keys + +Two new optional keys are added to `babashka.cli` spec entries. They are +consumed only by the completion generator; runtime parsing ignores them. + +### 3.1 `:values [...]` + +A vector of allowed string values — used for enum completion. + +```clojure +;; before +:output {:desc "Output format (human, json, edn). Default: human"} + +;; after +:output {:desc "Output format. Default: human" + :values ["human" "json" "edn"]} +``` + +Complete list of locations that need `:values` added: + +| File | Option | Values | +| ------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- | +| `command/core.cljs` (global) | `:output` | `["human" "json" "edn"]` | +| `command/list.cljs` (page spec) | `:sort` | `["title" "created-at" "updated-at"]` | +| `command/list.cljs` (tag spec) | `:sort` | `["name" "title"]` | +| `command/list.cljs` (property spec) | `:sort` | `["name" "title"]` | +| `command/list.cljs` (common spec) | `:order` | `["asc" "desc"]` | +| `command/upsert.cljs` (block spec) | `:pos` | `["first-child" "last-child" "sibling"]` | +| `command/upsert.cljs` (block spec) | `:status` | `["todo" "doing" "done" "now" "later" "wait" "waiting" "backlog" "canceled" "cancelled" "in-review" "in-progress"]` | +| `command/upsert.cljs` (property spec) | `:type` | `["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"]` | +| `command/upsert.cljs` (property spec) | `:cardinality` | `["one" "many"]` | +| `command/graph.cljs` (export spec) | `:type` | `["edn" "sqlite"]` | +| `command/graph.cljs` (import spec) | `:type` | `["edn" "sqlite"]` | + +> **Note on `:sort`:** The allowed values differ per sub-command (`list page` +> vs `list tag/property`). Each sub-command already has its own spec, so +> the difference is handled naturally with no special-casing. + +### 3.2 `:complete ` + +A hint that the value should be completed dynamically. The generator emits +a call to the appropriate shell helper function. + +| Keyword | zsh helper | bash helper | Notes | +| ---------- | ----------------- | ---------------------- | ------------------------------------------------- | +| `:graphs` | `_logseq_graphs` | `_logseq_graphs_bash` | Graph names from `logseq graph list` | +| `:pages` | `_logseq_pages` | `_logseq_pages_bash` | Page titles; requires `--graph` | +| `:queries` | `_logseq_queries` | `_logseq_queries_bash` | Built-in + custom query names; requires `--graph` | +| `:file` | `_files` | `compgen -f` | Filesystem path | +| `:dir` | `_files -/` | `compgen -d` | Directory path | + +Complete list of locations that need `:complete` added: + +| File | Option | `:complete` | Applies to | +| ----------------------------- | -------------- | ----------- | ------------------ | +| `command/core.cljs` (global) | `:graph` | `:graphs` | All commands | +| `command/core.cljs` (global) | `:config` | `:file` | All commands | +| `command/core.cljs` (global) | `:data-dir` | `:dir` | All commands | +| `command/graph.cljs` (export) | `:file` | `:file` | `graph export` | +| `command/graph.cljs` (import) | `:input` | `:file` | `graph import` | +| `command/upsert.cljs` (block) | `:target-page` | `:pages` | `upsert block` | +| `command/upsert.cljs` (block) | `:blocks-file` | `:file` | `upsert block` | +| `command/upsert.cljs` (page) | `:page` | `:pages` | `upsert page` | +| `command/query.cljs` | `:name` | `:queries` | `query` only | +| `command/show.cljs` | `:page` | `:pages` | `show` | +| `command/remove.cljs` (page) | `:name` | `:pages` | `remove page` only | + +> **`:name` is context-dependent.** `query --name` → `:queries`. +> `remove page --name` → `:pages`. `upsert tag --name` and +> `remove tag --name` → free text. This is handled by keeping `:complete` +> in command-specific specs only. + +> **`:queries` must return both built-in and custom queries.** The dynamic +> helper calls `logseq query list` which already returns both sources. + +--- + +## 4. New `completions` command + +New file: `src/main/logseq/cli/command/completions.cljs` + +``` +logseq completions zsh +logseq completions bash +logseq completions --help +``` + +The command: + +1. Takes `--shell` (or a positional arg) with value `zsh` or `bash`. +2. Calls the generator (§5) with the full `table`. +3. Prints the result to stdout. +4. Exits 0. + +Registration in `commands.cljs`: + +```clojure +(core/command-entry ["completions"] :completions + "Generate shell completion script" + {:shell {:desc "Shell (zsh, bash)" :values ["zsh" "bash"]}}) +``` + +The `completions` command itself **must be excluded** from the generated +completion dispatch (it is a meta-command, not a user-facing workflow +command). Whether to exclude it is a design choice; including it is also +acceptable since `logseq completions ` → `zsh bash` is helpful. + +--- + +## 5. Generator function + +New file: `src/main/logseq/cli/completion_generator.cljs` + +Pure function: `(generate-completions shell table) → string` + +### 5.1 Input + +The `table` as returned by `commands/build-table` — a vector of +`command-entry` maps. Each entry has: + +```clojure +{:cmds ["graph" "export"] + :command :graph-export + :desc "Export graph" + :spec { ;; merged global + command-specific + :help {:alias :h :coerce :boolean :desc "..."} + :graph {:desc "Graph name" :complete :graphs} + :type {:desc "Export type" :values ["edn" "sqlite"]} + :file {:desc "Export file" :complete :file} + ...}} +``` + +### 5.2 Spec-entry → completion token mapping + +| Spec characteristics | zsh token | bash case | +| --------------------- | -------------------------------------- | ------------------------------------------------------------ | +| `:coerce :boolean` | `'--flag[desc]'` | flag in wordlist, no argument | +| `:values [v1 v2]` | `'--opt=[desc]:label:(v1 v2)'` | `compgen -W 'v1 v2' -- "$cur"` | +| `:complete :graphs` | `'--opt=[desc]:graph:_logseq_graphs'` | `_logseq_compadd_lines "$cur" _logseq_graphs_bash` | +| `:complete :pages` | `'--opt=[desc]:page:_logseq_pages'` | `_logseq_compadd_lines "$cur" _logseq_pages_bash "$graph"` | +| `:complete :queries` | `'--opt=[desc]:query:_logseq_queries'` | `_logseq_compadd_lines "$cur" _logseq_queries_bash "$graph"` | +| `:complete :file` | `'--opt=[desc]:file:_files'` | `compgen -f -- "$cur"` | +| `:complete :dir` | `'--opt=[desc]:dir:_files -/'` | `compgen -d -- "$cur"` | +| `:alias :x` | `'(-x --opt)'{-x,--opt}'[desc]...'` | both `-x` and `--opt` in wordlist | +| free string (default) | `'--opt=[desc]:value:'` | flag in wordlist, prev-word fallthrough | + +### 5.3 Output structure (zsh) + +```zsh +#compdef logseq +# Auto-generated by `logseq completions zsh` — do not edit manually. + +# --- dynamic helpers (verbatim, fixed) --- +_logseq_json_names() { ... } +_logseq_graphs() { ... } # with zcompcache +_logseq_pages() { ... } # with zcompcache, keyed by --graph value +_logseq_queries() { ... } # with zcompcache, keyed by --graph value +_logseq_current_graph() { ... } + +# --- per-command functions (generated) --- +_logseq_graph_export() { _arguments -s ... } +_logseq_graph_import() { _arguments -s ... } +... + +# --- group dispatchers (generated) --- +_logseq_graph() { + _arguments -C -s ... '1:subcommand:->subcmd' '*::args:->args' + case $state in + subcmd) _describe 'subcommand' subcmds ;; + args) case $line[1] in ... esac ;; + esac +} + +# --- top-level dispatcher (generated) --- +_logseq() { + _arguments -C -s ... '1:command:->cmds' '*::args:->args' + ... +} + +_logseq "$@" +``` + +### 5.4 Output structure (bash) + +```bash +# Auto-generated by `logseq completions bash` — do not edit manually. + +# --- dynamic helpers (verbatim, fixed) --- +_logseq_json_names_bash() { ... } +_logseq_current_graph_bash() { ... } +_logseq_graphs_bash() { ... } +_logseq_pages_bash() { ... } +_logseq_queries_bash() { ... } +_logseq_compadd_lines() { ... } +_logseq_is_value_opt() { ... } # generated from spec +_logseq_cmd_and_subcmd() { ... } + +# --- option wordlists (generated per-command) --- +_logseq_opts_for() { ... } # case "$cmd"/"$subcmd" + +# --- main function (generated) --- +_logseq() { ... } + +complete -F _logseq logseq +``` + +--- + +## 6. Dynamic helper functions + +The dynamic helpers (graph/page/query lookups, caching) are **fixed verbatim +strings** emitted by the generator regardless of the table contents. They do +not change when new commands are added — only the static dispatch structure +changes. + +They live in the generator as string constants: + +```clojure +(def ^:private zsh-dynamic-helpers + "# --- dynamic helpers --- +_logseq_json_names() { + python3 -c \"...\" +} +...") +``` + +### 6.1 What the helpers invoke + +| Helper | CLI command | Output format | +| ------------------------------------------ | --------------------------------------------- | -------------------- | +| `_logseq_graphs` / `_logseq_graphs_bash` | `logseq graph list --output json` | JSON array of names | +| `_logseq_pages` / `_logseq_pages_bash` | `logseq list page --graph --output json` | JSON array of titles | +| `_logseq_queries` / `_logseq_queries_bash` | `logseq query list --graph --output json` | JSON array of names | + +### 6.2 `_logseq_current_graph` + +Scans `$words` (zsh) or `$COMP_WORDS` (bash) for `--graph VALUE` to +determine which graph context to use for page/query lookups. + +--- + +## 7. Tree-walk algorithm + +The generator derives the command hierarchy by inspecting `:cmds` vectors: + +1. **Leaf commands** — entries where no other entry's `:cmds` is a prefix. + These get per-command functions with `_arguments` (zsh) or case branches + (bash). + +2. **Group commands** — the distinct first-element prefixes (`graph`, + `server`, `list`, `upsert`, `remove`, `query`). These get dispatcher + functions that offer subcommand completion, then delegate to the leaf. + +3. **Top-level** — the root dispatcher that offers command-group (and + leaf-command) completion. + +The walk is: + +``` +table + → group by first element of :cmds + → for each group: + if only one entry → leaf (e.g., "show", "doctor") + if multiple entries → group dispatcher + per-subcommand leaves + → top-level dispatcher listing all groups and leaves +``` + +--- + +## 8. Edge cases + +| Case | Handling | +| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `:alias` on a global opt (`:help` → `-h`) | Emit both long and short form with zsh grouping `(-h --help)` | +| Same flag name in both global and command spec | Generator uses the merged spec — command spec wins (already the case via `merge`) | +| `:sort` has different `:values` for `list page` vs `list tag/property` | Handled naturally — each sub-command has its own spec | +| `--output` is a format enum globally; `graph export` uses `--file` for the file path | No conflict — `--output` keeps its global meaning everywhere, `--file` is the export path. Confirmed in source. | +| `--name` context-dependent (`query` → `:queries`; `remove page` → `:pages`; `upsert tag` → free text) | Expressed by keeping `:complete` in command-specific specs only | +| Boolean flags shouldn't suggest a value | `:coerce :boolean` → emit as bare flag token | +| `remove page --name` needs page completion but `remove tag --name` does not | `remove-page-spec` gets its own spec with `{:name {:complete :pages}}`, separate from `remove-entity-spec` | +| `server` commands inherit `--graph` from global spec + also declare it in command spec | Redundant shadow — no issue, merged spec contains one `:graph` entry | +| `completions` command in generated output | Include it — `logseq completions ` → `zsh bash` is useful | +| Positional arguments (`graph create `, `graph switch `) | These commands take the graph name via `--graph` (global opt). No positional arg completion needed beyond subcommand dispatch. | + +--- + +## 9. What is NOT generated + +These are emitted as fixed preamble strings by the generator: + +- The dynamic helper function bodies (graph list, page list, query list + invocations) +- The zsh cache key logic (`_store_cache` / `_retrieve_cache`) +- The `_logseq_current_graph` helper that scans `$words` for `--graph` +- The `_logseq_compadd_lines` line-by-line appender (bash) +- The `_logseq_json_names` JSON-to-lines parser + +--- + +## 10. New files + +| Path | Purpose | +| ----------------------------------------------- | ------------------------------------------------------------ | +| `src/main/logseq/cli/command/completions.cljs` | Command entry: parse args, call generator, print to stdout | +| `src/main/logseq/cli/completion_generator.cljs` | Pure function: `(generate-completions shell table) → string` | + +--- + +## 11. Changes to existing files + +| File | Change | +| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `command/core.cljs` | Add `:values` and `:complete` to `global-spec*` entries per §3 | +| `command/list.cljs` | Add `:values` to `:sort` and `:order` per §3.1 | +| `command/upsert.cljs` | Add `:values` to `:pos`, `:status`, `:type`, `:cardinality`; add `:complete :pages` to `:target-page`; add `:complete :file` to `:blocks-file`; add `:complete :pages` to `:page` in page spec | +| `command/graph.cljs` | Add `:values` to `:type`; add `:complete :file` to `:file` and `:input` | +| `command/query.cljs` | Add `:complete :queries` to `:name` | +| `command/show.cljs` | Add `:complete :pages` to `:page` | +| `command/remove.cljs` | Split `remove-page-spec` from generic `remove-entity-spec` so `:name` gets `:complete :pages` only for `remove page` | +| `commands.cljs` | Add `completions-command/entries` to the `table` concat | + +--- + +## 12. Acceptance criteria + +- [ ] `logseq completions zsh` output, when installed as `_logseq`, passes + smoke tests: `logseq `, `logseq list `, + `logseq upsert block --pos `, `logseq show --page `, + `logseq remove `, `logseq server `, `logseq doctor `, + `logseq query --name `, `logseq remove page --name `, + `logseq graph export --type `, `logseq upsert block --blocks-file `. +- [ ] `logseq completions bash` output passes equivalent smoke tests. +- [ ] Adding a new `command-entry` to the table (or a `:values` change to a + spec) causes the generated output to change with no other edits + required. +- [ ] The generator is a pure function testable in isolation (no I/O). +- [ ] The hand-maintained `_logseq.zsh` and `logseq.bash` files in + `dz/scripts/shell-completions/` are replaced by the generated output + and marked as generated (header comment: "do not edit manually"). +- [ ] The `remove page` subcommand completes `--name` with page names + (dynamic), while `remove tag` and `remove property` leave `--name` + as free text. +- [ ] `upsert block --target-page`, `upsert page --page`, and + `show --page` all complete with page names (dynamic). +- [ ] `upsert block --blocks-file` and `graph export --file` and + `graph import --input` complete with file paths. +- [ ] `--config` completes with file paths; `--data-dir` completes with + directory paths. + +--- + +## 13. Out of scope + +- Fish shell completions (can be added later with the same table-walk). +- PowerShell completions. +- Auto-installing completions on `npm install` / `brew install`. +- CI regression test that diffs generated completions against a golden file + (desirable, but tracked separately). +- Completion for positional arguments beyond subcommand names (no current + command uses meaningful positional args). diff --git a/dz/scripts/shell-completions/README.md b/dz/scripts/shell-completions/README.md new file mode 100644 index 0000000000..d2fe2b0aae --- /dev/null +++ b/dz/scripts/shell-completions/README.md @@ -0,0 +1,191 @@ +# Shell Completions for the Logseq CLI + +This directory contains shell completion scripts for the `logseq` CLI +(`src/main/logseq/cli`). + +## Generating completions + +These files are **auto-generated** from the CLI command table. Do not edit +them manually — regenerate instead: + +```bash +logseq completions zsh > dz/scripts/shell-completions/_logseq.zsh +logseq completions bash > dz/scripts/shell-completions/logseq.bash +``` + +The generator reads the command table at generation time, so adding or +modifying a command/flag in the spec is the **only** change needed — the +completion scripts stay in sync automatically. + +## Available completions + +| File | Shell | +| --------------- | ----- | +| `_logseq.zsh` | zsh | +| `logseq.bash` | bash | + +--- + +## zsh + +### Installation (zsh) + +1. Copy `_logseq.zsh` to a directory on your `$fpath`, **renaming it to + `_logseq`** (no extension — required by zsh's autoload mechanism): + + ```zsh + mkdir -p ~/.zsh/completions + cp completions/_logseq.zsh ~/.zsh/completions/_logseq + ``` + +2. Add that directory to `$fpath` in `~/.zshrc` **before** the `compinit` call: + + ```zsh + fpath=(~/.zsh/completions $fpath) + autoload -Uz compinit && compinit + ``` + +3. Open a new terminal (or run `compinit` in the current session). + +### Verifying (zsh) + +```zsh +logseq # shows top-level commands +logseq list # shows: page tag property +logseq graph # shows: list create switch remove ... +logseq add block --status # shows task status values +logseq show --page # lists page names from the active graph +``` + +### Dynamic completions and caching (zsh) + +Page names, graph names, and query names are fetched live from the CLI and +cached using zsh's built-in completion cache (`~/.zcompcache/`). The cache is +keyed per graph, so switching `--repo` gives fresh results. + +To force a refresh, delete the relevant cache entries: + +```zsh +rm -f ~/.zcompcache/logseq_* +``` + +### Updating (zsh) + +After upgrading the `logseq` CLI, regenerate and re-copy: + +```zsh +logseq completions zsh > ~/.zsh/completions/_logseq +compinit +``` + +--- + +## bash + +### Requirements (bash) + +Requires bash 4.1+ and the [`bash-completion`](https://github.com/scop/bash-completion) +package (v2 recommended). On macOS: + +```bash +brew install bash bash-completion@2 +``` + +### Installation (bash) + +**Option A — per-user** (recommended): + +```bash +mkdir -p ~/.local/share/bash-completion/completions +cp completions/logseq.bash ~/.local/share/bash-completion/completions/logseq +``` + +**Option B — source from `~/.bashrc`**: + +```bash +source /path/to/logseq/completions/logseq.bash +``` + +**Option C — system-wide**: + +```bash +sudo cp completions/logseq.bash /etc/bash_completion.d/logseq +``` + +### Verifying (bash) + +```bash +logseq # shows top-level commands +logseq list # shows: page tag property +logseq graph # shows: list create switch remove ... +logseq add block --status # shows task status values +logseq show --page # lists page names from the active graph +``` + +### Dynamic completions (bash) + +Page names, graph names, and query names are fetched live from the CLI each +time they are needed. There is no built-in caching in the bash script; results +are as fresh as the CLI response. + +### Updating (bash) + +After upgrading the `logseq` CLI, regenerate and re-copy: + +```bash +logseq completions bash > ~/.local/share/bash-completion/completions/logseq +``` + +--- + +## What is completed (both shells) + +### Commands and subcommands + +```text +logseq list page | tag | property +logseq upsert block | page | tag | property +logseq remove block | page | tag | property +logseq query [list] +logseq graph list | create | switch | remove | validate | info | export | import +logseq server list | status | start | stop | restart +logseq show +logseq doctor +logseq completions +``` + +### Global options + +| Option | Completion | +| --------------------------------------- | --------------------------------------------- | +| `--graph` | dynamic: graph names from `logseq graph list` | +| `--config` | file path | +| `--data-dir` | directory path | +| `--output` | `human` `json` `edn` | +| `--timeout-ms` | free integer | +| `--verbose`, `--version`, `-h`/`--help` | flags | + +### Per-command options + +| Command | Option | Completion | +| --------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------- | +| `list page` | `--sort` | `title` `created-at` `updated-at` | +| `list tag`, `list property` | `--sort` | `name` `title` | +| `list *` | `--order` | `asc` `desc` | +| `upsert block` | `--pos` | `first-child` `last-child` `sibling` | +| `upsert block` | `--status` | `todo` `doing` `done` `now` `later` `wait` `waiting` `backlog` `canceled` `cancelled` `in-review` `in-progress` | +| `upsert block` | `--target-page` | dynamic: page titles from `logseq list page` | +| `upsert block` | `--blocks-file` | file path | +| `upsert page` | `--page` | dynamic: page titles | +| `upsert property` | `--type` | `default` `number` `date` `datetime` `checkbox` `url` `node` `json` `string` | +| `upsert property` | `--cardinality` | `one` `many` | +| `remove page` | `--name` | dynamic: page titles | +| `show` | `--page` | dynamic: page titles | +| `query` | `--name` | dynamic: query names from `logseq query list` | +| `graph export` | `--type` | `edn` `sqlite` | +| `graph export` | `--file` | file path | +| `graph import` | `--type` | `edn` `sqlite` | +| `graph import` | `--input` | file path | +| `completions` | `--shell` | `zsh` `bash` | +| `doctor` | `--dev-script` | flag | +` \ No newline at end of file diff --git a/dz/scripts/shell-completions/TASKS.md b/dz/scripts/shell-completions/TASKS.md new file mode 100644 index 0000000000..c3b218b144 --- /dev/null +++ b/dz/scripts/shell-completions/TASKS.md @@ -0,0 +1,283 @@ +# Shell Completions — TDD Implementation Tasks + +Implementation plan for the `logseq completions ` feature as specified +in [DESIGN.md](DESIGN.md). Each task follows **Red → Green → Refactor**: write +a failing test first, then make it pass, then clean up. + +Test file: `src/test/logseq/cli/command/completions_test.cljs` +Generator test: `src/test/logseq/cli/completion_generator_test.cljs` + +> **Require registration:** The nbb test runner auto-discovers `*_test.cljs` +> files under the `test` classpath (`-cp test`). No explicit require is needed +> in a runner config — but the new test namespace **must** be required in the +> test entry if one exists, or verified to be picked up by convention. + +--- + +## Phase 0 — Scaffolding & test harness + +- [ ] **0.1** Create `src/test/logseq/cli/completion_generator_test.cljs` with + a skeleton ns that requires `[cljs.test :refer [deftest is testing]]` and + `[logseq.cli.completion-generator :as gen]`. Add a trivial failing test + (`(deftest placeholder (is false))`). + +- [ ] **0.2** Create `src/main/logseq/cli/completion_generator.cljs` with a + stub namespace and a `generate-completions` function that returns `""`. + Confirm the test runner loads both files (`yarn nbb-logseq -cp test -m + nextjournal.test-runner`). + +- [ ] **0.3** Create `src/test/logseq/cli/command/completions_test.cljs` with a + skeleton ns requiring `[logseq.cli.command.completions :as completions-command]`. + Add a trivial failing test. + +- [ ] **0.4** Create `src/main/logseq/cli/command/completions.cljs` with a stub + ns and `entries` def (empty vector). Confirm test runner loads it. + +- [ ] **0.5** Remove placeholder failing tests; verify `yarn test` passes + (green baseline). + +--- + +## Phase 1 — Spec enrichment (`:values` and `:complete` metadata) + +Each sub-task: write a test that reads the spec from the command entry and +asserts the metadata key exists, then add the key. + +- [ ] **1.1** `command/core.cljs` — global spec: + - Test: `:output` has `:values ["human" "json" "edn"]` + - Test: `:graph` has `:complete :graphs` + - Test: `:config` has `:complete :file` + - Test: `:data-dir` has `:complete :dir` + - Implement: add the keys to `global-spec*`. + +- [ ] **1.2** `command/list.cljs` — list specs: + - Test: page-spec `:sort` has `:values ["title" "created-at" "updated-at"]` + - Test: tag-spec `:sort` has `:values ["name" "title"]` + - Test: property-spec `:sort` has `:values ["name" "title"]` + - Test: common `:order` has `:values ["asc" "desc"]` + - Implement: add the `:values` keys. + +- [ ] **1.3** `command/upsert.cljs` — upsert specs: + - Test: block-spec `:pos` has `:values` + - Test: block-spec `:status` has `:values` + - Test: block-spec `:target-page` has `:complete :pages` + - Test: block-spec `:blocks-file` has `:complete :file` + - Test: page-spec `:page` has `:complete :pages` + - Test: property-spec `:type` has `:values` + - Test: property-spec `:cardinality` has `:values` + - Implement: add the keys. + +- [ ] **1.4** `command/graph.cljs` — export/import specs: + - Test: export-spec `:type` has `:values ["edn" "sqlite"]` + - Test: export-spec `:file` has `:complete :file` + - Test: import-spec `:type` has `:values ["edn" "sqlite"]` + - Test: import-spec `:input` has `:complete :file` + - Implement: add the keys. + +- [ ] **1.5** `command/query.cljs` — query spec: + - Test: query-spec `:name` has `:complete :queries` + - Implement: add the key. + +- [ ] **1.6** `command/show.cljs` — show spec: + - Test: show-spec `:page` has `:complete :pages` + - Implement: add the key. + +- [ ] **1.7** `command/remove.cljs` — remove page spec split: + - Test: remove-page entry's spec has `{:name {:complete :pages}}` + - Test: remove-tag entry's spec does NOT have `:complete` on `:name` + - Test: remove-property entry's spec does NOT have `:complete` on `:name` + - Implement: split `remove-page-spec` from `remove-entity-spec`. + +--- + +## Phase 2 — Generator: table introspection utilities + +Pure functions tested in `completion_generator_test.cljs`. + +- [ ] **2.1** `extract-groups` — given a table, return grouped command + hierarchy: + - Test: `["graph" "export"]` → group `"graph"`, subcommand `"export"` + - Test: `["show"]` → leaf command `"show"` + - Test: `["completions"]` → leaf command `"completions"` + - Implement in `completion_generator.cljs`. + +- [ ] **2.2** `leaf-commands` / `group-commands` — classify entries: + - Test: `show` and `doctor` are leaves + - Test: `graph`, `server`, `list`, `upsert`, `remove`, `query` are groups + - Implement. + +- [ ] **2.3** `spec->tokens` — convert a single spec entry to a shell token + descriptor: + - Test: boolean spec → `:flag` type + - Test: spec with `:values` → `:enum` type with values + - Test: spec with `:complete :graphs` → `:dynamic` type + - Test: spec with `:complete :file` → `:file` type + - Test: spec with `:complete :dir` → `:dir` type + - Test: spec with `:alias` → includes alias + - Test: bare string spec (no `:values`, no `:complete`, not boolean) → `:free` type + - Implement. + +--- + +## Phase 3 — Generator: zsh output + +- [ ] **3.1** `generate-zsh-preamble` — emit `#compdef logseq` header and + dynamic helper constants: + - Test: output starts with `#compdef logseq` + - Test: output contains `_logseq_graphs` + - Test: output contains `_logseq_pages` + - Test: output contains `_logseq_queries` + - Test: output contains `_logseq_json_names` + - Test: output contains `_logseq_current_graph` + - Implement with string constants. + +- [ ] **3.2** `generate-zsh-leaf` — emit a `_logseq_()` function for + a leaf command: + - Test: `show` command emits `_logseq_show()` with `_arguments` + - Test: boolean flags emit `'--flag[desc]'` form + - Test: enum options emit `'--opt=[desc]:label:(v1 v2)'` form + - Test: `:complete :graphs` emits `_logseq_graphs` action + - Test: `:complete :file` emits `_files` action + - Test: `:alias` emits `(-x --opt)` grouping + - Implement. + +- [ ] **3.3** `generate-zsh-group` — emit a group dispatcher: + - Test: `graph` group lists subcommands `list create switch remove validate info export import` + - Test: dispatches to `_logseq_graph_export` etc. + - Implement. + +- [ ] **3.4** `generate-zsh-toplevel` — emit `_logseq()` root dispatcher: + - Test: lists all top-level commands and groups + - Test: dispatches to group/leaf functions + - Test: ends with `_logseq "$@"` + - Implement. + +- [ ] **3.5** Integration: `(generate-completions "zsh" table)` returns a + complete, valid zsh script: + - Test: output contains preamble + all leaf functions + all group dispatchers + + top-level dispatcher + - Test: every command from the table appears in the output + - Test: `--pos` under `upsert block` offers `first-child last-child sibling` + - Test: `--sort` for `list page` offers `title created-at updated-at` + - Test: `--sort` for `list tag` offers `name title` + - Implement by composing 3.1–3.4. + +--- + +## Phase 4 — Generator: bash output + +- [ ] **4.1** `generate-bash-preamble` — emit header and dynamic helpers: + - Test: output contains `_logseq_graphs_bash` + - Test: output contains `_logseq_pages_bash` + - Test: output contains `_logseq_queries_bash` + - Test: output contains `_logseq_compadd_lines` + - Test: output contains `_logseq_json_names_bash` + - Test: output contains `_logseq_current_graph_bash` + - Implement with string constants. + +- [ ] **4.2** `generate-bash-opts-for` — emit `_logseq_opts_for()` case + dispatch: + - Test: `graph export` case branch includes `--type` and `--file` + - Test: boolean flags appear in wordlist without argument handling + - Test: enum values use `compgen -W` + - Test: `:complete :file` uses `compgen -f` + - Implement. + +- [ ] **4.3** `generate-bash-main` — emit `_logseq()` and `complete -F`: + - Test: output ends with `complete -F _logseq logseq` + - Test: subcommand dispatch works for groups + - Implement. + +- [ ] **4.4** Integration: `(generate-completions "bash" table)` returns a + complete bash script: + - Test: output contains preamble + opts-for + main function + complete + registration + - Test: every command from the table appears in the output + - Implement by composing 4.1–4.3. + +--- + +## Phase 5 — `completions` command entry + +- [ ] **5.1** Command registration: + - Test: `completions-command/entries` contains one entry with + `:cmds ["completions"]` and `:command :completions` + - Test: spec has `{:shell {:values ["zsh" "bash"]}}` + - Implement `command/completions.cljs` with `entries` and spec. + +- [ ] **5.2** Wire into `commands.cljs`: + - Test: `(commands/parse-args ["completions" "--shell" "zsh"])` returns + `{:ok? true :command :completions}` + - Test: `(commands/parse-args ["completions" "zsh"])` handles positional arg + - Implement: add `completions-command/entries` to the table concat in + `commands.cljs`. + +- [ ] **5.3** Build action and execute: + - Test: `build-action` for `:completions` returns an action with + `:type :completions` and `:shell "zsh"` + - Test: `execute` for `:completions` calls `generate-completions` and returns + the output string + - Implement in `command/completions.cljs` and wire into + `commands.cljs` `build-action`/`execute`. + +--- + +## Phase 6 — End-to-end validation + +- [ ] **6.1** Golden-file smoke test (zsh): + - Generate zsh output from the full table + - Assert key structural markers: `#compdef`, `_logseq_graph_export`, + `_logseq_show`, `_logseq "$@"` + - Assert all commands from §2.1 of DESIGN.md appear + +- [ ] **6.2** Golden-file smoke test (bash): + - Generate bash output from the full table + - Assert key structural markers: `complete -F _logseq logseq`, + `_logseq_opts_for` + - Assert all commands from §2.1 of DESIGN.md appear + +- [ ] **6.3** Sync test — adding a command updates output: + - Build a minimal table, generate output, add a fake command entry, + re-generate, assert the new command appears + +- [ ] **6.4** Context-dependent `:name` test: + - Assert `query` spec has `{:name {:complete :queries}}` + - Assert `remove page` spec has `{:name {:complete :pages}}` + - Assert `upsert tag` spec does NOT have `:complete` on `:name` + - Assert `remove tag` spec does NOT have `:complete` on `:name` + +--- + +## Phase 7 — Replace hand-maintained files + +- [ ] **7.1** Generate fresh `_logseq.zsh` and `logseq.bash` from the + `completions` command; write them to + `dz/scripts/shell-completions/`. + +- [ ] **7.2** Verify the generated files include the "do not edit manually" + header comment. + +- [ ] **7.3** Update `dz/scripts/shell-completions/README.md` to document the + new `logseq completions` workflow instead of hand-editing. + +--- + +## Test require checklist + +All new test namespaces that must be discoverable by the test runner +(`yarn nbb-logseq -cp test -m nextjournal.test-runner`): + +| Test file | Requires | +|---|---| +| `src/test/logseq/cli/completion_generator_test.cljs` | `logseq.cli.completion-generator` | +| `src/test/logseq/cli/command/completions_test.cljs` | `logseq.cli.command.completions` | + +The nbb test runner scans the `test` classpath for `*_test.cljs` files +automatically. Verify with: + +```bash +cd deps/cli && yarn test +``` + +If the runner uses an explicit require list (check for a `test_runner.cljs` or +similar), add the two new namespaces there as well. diff --git a/dz/scripts/shell-completions/_logseq.zsh b/dz/scripts/shell-completions/_logseq.zsh new file mode 100644 index 0000000000..9fa3040997 --- /dev/null +++ b/dz/scripts/shell-completions/_logseq.zsh @@ -0,0 +1,783 @@ +#compdef logseq +# Auto-generated by `logseq completions zsh` — do not edit manually. + +# --- dynamic helpers --- + +_logseq_json_names() { + python3 -c " +import sys, json +field = sys.argv[1] +try: + data = json.load(sys.stdin) + if isinstance(data, list): + for item in data: + v = item.get(field) + if isinstance(v, str) and v: + print(v) +except Exception: + pass +" "$1" 2>/dev/null +} + +_logseq_current_graph() { + local i + for (( i = 1; i < ${#words[@]}; i++ )); do + if [[ "${words[i]}" == '--graph' && -n "${words[i+1]}" ]]; then + print -r -- "${words[i+1]}" + return + fi + done +} + +_logseq_graphs() { + local cache_id='logseq_graphs' + local -a graphs + if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then + graphs=( ${(f)"$(logseq graph list --output json 2>/dev/null | _logseq_json_names name)"} ) + _store_cache "$cache_id" graphs + fi + compadd -a graphs +} + +_logseq_pages() { + local graph + graph=$(_logseq_current_graph) + local cache_id="logseq_pages_${graph:-__default__}" + local -a pages + if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then + if [[ -n "$graph" ]]; then + pages=( ${(f)"$(logseq list page --graph "$graph" --output json 2>/dev/null | _logseq_json_names title)"} ) + fi + _store_cache "$cache_id" pages + fi + compadd -a pages +} + +_logseq_queries() { + local graph + graph=$(_logseq_current_graph) + local cache_id="logseq_queries_${graph:-__default__}" + local -a queries + if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then + if [[ -n "$graph" ]]; then + queries=( ${(f)"$(logseq query list --graph "$graph" --output json 2>/dev/null | _logseq_json_names name)"} ) + fi + _store_cache "$cache_id" queries + fi + compadd -a queries +} + +# --- per-command functions --- + +_logseq_list_page() { + _arguments -s \ + '--created-after=[Filter by created-at (ISO8601)]:value:' \ + '--updated-after=[Filter by updated-at (ISO8601)]:value:' \ + '--limit=[Limit results]:value:' \ + '--offset=[Offset results]:value:' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--fields=[Select output fields (comma separated)]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--journal-only[Only journal pages]' \ + '--order=[Sort order. Default: asc]:value:(asc desc)' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--include-hidden[Include hidden pages]' \ + '--expand[Include expanded metadata]' \ + '--sort=[Sort field]:value:(title created-at updated-at)' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--include-journal[Include journal pages]' +} + +_logseq_list_tag() { + _arguments -s \ + '--limit=[Limit results]:value:' \ + '--offset=[Offset results]:value:' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--include-built-in[Include built-in tags]' \ + '--fields=[Select output fields (comma separated)]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--order=[Sort order. Default: asc]:value:(asc desc)' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--with-extends[Include tag extends]' \ + '--expand[Include expanded metadata]' \ + '--with-properties[Include tag properties]' \ + '--sort=[Sort field]:value:(name title)' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_list_property() { + _arguments -s \ + '--limit=[Limit results]:value:' \ + '--offset=[Offset results]:value:' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--include-built-in[Include built-in properties]' \ + '--fields=[Select output fields (comma separated)]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--order=[Sort order. Default: asc]:value:(asc desc)' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--with-type[Include property type]' \ + '--version[Show version]' \ + '--expand[Include expanded metadata]' \ + '--sort=[Sort field]:value:(name title)' \ + '--with-classes[Include property classes]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_server_list() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_server_status() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_server_start() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_server_stop() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_server_restart() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_list() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_create() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_switch() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_remove() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_validate() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_info() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_graph_export() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--file=[Export file path]:file:_files' \ + '--type=[Export type]:value:(edn sqlite)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_graph_import() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--type=[Import type]:value:(edn sqlite)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--input=[Input path]:file:_files' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_query() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Query name from cli.edn custom-queries or built-ins]:value:_logseq_queries' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--inputs=[EDN vector of query inputs]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--query=[Datascript query EDN]:value:' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_query_list() { + _arguments -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' +} + +_logseq_upsert_block() { + _arguments -s \ + '--blocks-file=[EDN file of blocks for create mode]:file:_files' \ + '--target-page=[Target page name]:value:_logseq_pages' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--content=[Block content for create mode]:value:' \ + '--remove-tags=[Tags to remove (EDN vector)]:value:' \ + '--target-uuid=[Target block UUID]:value:' \ + '--pos=[Position. Default: create=last-child, update=first-child]:value:(first-child last-child sibling)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--target-id=[Target block db/id]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--update-tags=[Tags to add/update (EDN vector)]:value:' \ + '--status=[Task status]:value:(todo doing done now later wait waiting backlog canceled cancelled in-review in-progress)' \ + '--update-properties=[Properties to add/update (EDN map)]:value:' \ + '--id=[Source block db/id (forces update mode)]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--blocks=[EDN vector of blocks for create mode]:value:' \ + '--uuid=[Source block UUID (forces update mode)]:value:' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--remove-properties=[Properties to remove (EDN vector)]:value:' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_upsert_page() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--remove-tags=[Tags to remove (EDN vector)]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--page=[Page name]:value:_logseq_pages' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--update-tags=[Tags to add/update (EDN vector)]:value:' \ + '--update-properties=[Properties to add/update (EDN map)]:value:' \ + '--id=[Target page db/id (forces update mode)]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--remove-properties=[Properties to remove (EDN vector)]:value:' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_upsert_tag() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Tag name]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--id=[Target tag db/id (forces update mode)]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_upsert_property() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Property name]:value:' \ + '--public[Set property public visibility]' \ + '--type=[Property type]:value:(default number date datetime checkbox url node json string)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--hide[Hide property]' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--id=[Target property db/id (forces update mode)]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--cardinality=[Property cardinality]:value:(one many)' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_completions() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--shell=[Shell (zsh, bash)]:value:(zsh bash)' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_show() { + _arguments -s \ + '--linked-references[Include linked references (default true)]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--page=[Page name]:value:_logseq_pages' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--level=[Limit tree depth (default 10)]:value:' \ + '--id=[Block db/id or EDN vector of ids]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--uuid=[Block UUID]:value:' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_doctor() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--dev-script[Check static/db-worker-node.js instead of bundled dist runtime]' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_remove_block() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--id=[Block db/id or EDN vector of ids]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--uuid=[Block UUID]:value:' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_remove_page() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Page name]:value:_logseq_pages' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_remove_tag() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Entity name]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--id=[Entity db/id]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +_logseq_remove_property() { + _arguments -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Entity name]:value:' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--id=[Entity db/id]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' +} + +# --- group dispatchers --- + +_logseq_list() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'page:List pages' + 'tag:List tags' + 'property:List properties' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + page) _logseq_list_page ;; + tag) _logseq_list_tag ;; + property) _logseq_list_property ;; + esac + ;; + esac +} + +_logseq_server() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'list:List db-worker-node servers' + 'status:Show server status for a graph' + 'start:Start db-worker-node for a graph' + 'stop:Stop db-worker-node for a graph' + 'restart:Restart db-worker-node for a graph' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + list) _logseq_server_list ;; + status) _logseq_server_status ;; + start) _logseq_server_start ;; + stop) _logseq_server_stop ;; + restart) _logseq_server_restart ;; + esac + ;; + esac +} + +_logseq_graph() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'list:List graphs' + 'create:Create graph' + 'switch:Switch current graph' + 'remove:Remove graph' + 'validate:Validate graph' + 'info:Graph metadata' + 'export:Export graph' + 'import:Import graph' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + list) _logseq_graph_list ;; + create) _logseq_graph_create ;; + switch) _logseq_graph_switch ;; + remove) _logseq_graph_remove ;; + validate) _logseq_graph_validate ;; + info) _logseq_graph_info ;; + export) _logseq_graph_export ;; + import) _logseq_graph_import ;; + esac + ;; + esac +} + +_logseq_query() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--name=[Query name from cli.edn custom-queries or built-ins]:value:_logseq_queries' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--inputs=[EDN vector of query inputs]:value:' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--query=[Datascript query EDN]:value:' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'list:List available queries' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + list) _logseq_query_list ;; + esac + ;; + esac +} + +_logseq_upsert() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'block:Upsert block' + 'page:Upsert page' + 'tag:Upsert tag' + 'property:Upsert property' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + block) _logseq_upsert_block ;; + page) _logseq_upsert_page ;; + tag) _logseq_upsert_tag ;; + property) _logseq_upsert_property ;; + esac + ;; + esac +} + +_logseq_remove() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:subcommand:->subcmd' \ + '*::args:->args' + + case $state in + subcmd) + local -a subcmds + subcmds=( + 'block:Remove blocks' + 'page:Remove page' + 'tag:Remove tag' + 'property:Remove property' + ) + _describe 'subcommand' subcmds + ;; + args) + case $line[1] in + block) _logseq_remove_block ;; + page) _logseq_remove_page ;; + tag) _logseq_remove_tag ;; + property) _logseq_remove_property ;; + esac + ;; + esac +} + +# --- top-level dispatcher --- + +_logseq() { + local curcontext="$curcontext" state line + typeset -A opt_args + + _arguments -C -s \ + '(-h --help)'{-h,--help}'[Show help]' \ + '--version[Show version]' \ + '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ + '--graph=[Graph name]:value:_logseq_graphs' \ + '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ + '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ + '--output=[Output format. Default: human]:value:(human json edn)' \ + '--verbose[Enable verbose debug logging to stderr]' \ + '1:command:->cmds' \ + '*::args:->args' + + case $state in + cmds) + local -a cmds + cmds=( + 'completions:Generate shell completion script' + 'doctor:Run runtime diagnostics' + 'graph:graph commands' + 'list:list commands' + 'query:query commands' + 'remove:remove commands' + 'server:server commands' + 'show:Show tree' + 'upsert:upsert commands' + ) + _describe 'command' cmds + ;; + args) + case $line[1] in + completions) _logseq_completions ;; + doctor) _logseq_doctor ;; + graph) _logseq_graph ;; + list) _logseq_list ;; + query) _logseq_query ;; + remove) _logseq_remove ;; + server) _logseq_server ;; + show) _logseq_show ;; + upsert) _logseq_upsert ;; + esac + ;; + esac +} + +_logseq "$@" diff --git a/dz/scripts/shell-completions/logseq.bash b/dz/scripts/shell-completions/logseq.bash new file mode 100644 index 0000000000..0cc5c08806 --- /dev/null +++ b/dz/scripts/shell-completions/logseq.bash @@ -0,0 +1,274 @@ +# Auto-generated by `logseq completions bash` — do not edit manually. + +# --- dynamic helpers --- + +_logseq_json_names_bash() { + python3 -c " +import sys, json +field = sys.argv[1] +try: + data = json.load(sys.stdin) + if isinstance(data, list): + for item in data: + v = item.get(field) + if isinstance(v, str) and v: + print(v) +except Exception: + pass +" "$1" 2>/dev/null +} + +_logseq_current_graph_bash() { + local i + for (( i = 1; i < ${#COMP_WORDS[@]}; i++ )); do + if [[ "${COMP_WORDS[i]}" == '--graph' && -n "${COMP_WORDS[i+1]}" ]]; then + printf '%s' "${COMP_WORDS[i+1]}" + return + fi + done +} + +_logseq_graphs_bash() { + logseq graph list --output json 2>/dev/null | _logseq_json_names_bash name +} + +_logseq_pages_bash() { + logseq list page --graph "$1" --output json 2>/dev/null | _logseq_json_names_bash title +} + +_logseq_queries_bash() { + logseq query list --graph "$1" --output json 2>/dev/null | _logseq_json_names_bash name +} + +_logseq_compadd_lines() { + local cur="$1" source_fn="$2"; shift 2 + while IFS= read -r item; do + [[ "$item" == "$cur"* ]] && COMPREPLY+=( "$item" ) + done < <("$source_fn" "$@") +} + +# --- generated helpers --- + +_logseq_is_value_opt() { + case "$1" in + --blocks|--blocks-file|--cardinality|--config|--content|--created-after|--data-dir|--fields|--file|--graph|--id|--input|--inputs|--level|--limit|--name|--offset|--order|--output|--page|--pos|--query|--remove-properties|--remove-tags|--shell|--sort|--status|--target-id|--target-page|--target-uuid|--timeout-ms|--type|--update-properties|--update-tags|--updated-after|--uuid) + return 0 ;; + *) return 1 ;; + esac +} + +_logseq_cmd_and_subcmd() { + local i skip=0 + __cmd='' __subcmd='' + for (( i = 1; i < COMP_CWORD; i++ )); do + local w="${COMP_WORDS[i]}" + if (( skip )); then skip=0; continue; fi + if [[ "$w" == -* ]]; then + _logseq_is_value_opt "$w" && skip=1 + continue + fi + if [[ -z "$__cmd" ]]; then + __cmd="$w" + elif [[ -z "$__subcmd" ]]; then + __subcmd="$w" + fi + done +} + +# --- option wordlists --- + +_logseq_opts_for() { + local cmd="$1" subcmd="$2" + local opts="--help -h --version --config --graph --data-dir --timeout-ms --output --verbose" + + case "$cmd" in + completions) opts+=' --shell' ;; + doctor) opts+=' --dev-script' ;; + graph) + case "$subcmd" in + list) opts+=' ' ;; + create) opts+=' ' ;; + switch) opts+=' ' ;; + remove) opts+=' ' ;; + validate) opts+=' ' ;; + info) opts+=' ' ;; + export) opts+=' --file --type' ;; + import) opts+=' --type --input' ;; + esac + ;; + list) + case "$subcmd" in + page) opts+=' --created-after --updated-after --limit --offset --fields --journal-only --order --include-hidden --expand --sort --include-journal' ;; + tag) opts+=' --limit --offset --include-built-in --fields --order --with-extends --expand --with-properties --sort' ;; + property) opts+=' --limit --offset --include-built-in --fields --order --with-type --expand --sort --with-classes' ;; + esac + ;; + query) + opts+=' --name --inputs --query' + case "$subcmd" in + list) opts+=' ' ;; + esac + ;; + remove) + case "$subcmd" in + block) opts+=' --id --uuid' ;; + page) opts+=' --name' ;; + tag) opts+=' --name --id' ;; + property) opts+=' --name --id' ;; + esac + ;; + server) + case "$subcmd" in + list) opts+=' ' ;; + status) opts+=' ' ;; + start) opts+=' ' ;; + stop) opts+=' ' ;; + restart) opts+=' ' ;; + esac + ;; + show) opts+=' --linked-references --page --level --id --uuid' ;; + upsert) + case "$subcmd" in + block) opts+=' --blocks-file --target-page --content --remove-tags --target-uuid --pos --target-id --update-tags --status --update-properties --id --blocks --uuid --remove-properties' ;; + page) opts+=' --remove-tags --page --update-tags --update-properties --id --remove-properties' ;; + tag) opts+=' --name --id' ;; + property) opts+=' --name --public --type --hide --id --cardinality' ;; + esac + ;; + esac + + printf '%s' "$opts" +} + +# --- main function --- + +_logseq() { + local cur prev + cur="${COMP_WORDS[COMP_CWORD]}" + prev="${COMP_WORDS[COMP_CWORD-1]}" + COMPREPLY=() + + local __cmd __subcmd + _logseq_cmd_and_subcmd + + # --- Option value completion --- + case "$prev" in + --blocks-file) + COMPREPLY=( $(compgen -f -- "$cur") ) + return ;; + + --target-page) + local graph + graph="$(_logseq_current_graph_bash)" + [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" + return ;; + + --config) + COMPREPLY=( $(compgen -f -- "$cur") ) + return ;; + + --file) + COMPREPLY=( $(compgen -f -- "$cur") ) + return ;; + + --pos) + COMPREPLY=( $(compgen -W 'first-child last-child sibling' -- "$cur") ) + return ;; + + --type) + COMPREPLY=( $(compgen -W 'default number date datetime checkbox url node json string' -- "$cur") ) + return ;; + + --page) + local graph + graph="$(_logseq_current_graph_bash)" + [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" + return ;; + + --output) + COMPREPLY=( $(compgen -W 'human json edn' -- "$cur") ) + return ;; + + --data-dir) + COMPREPLY=( $(compgen -d -- "$cur") ) + return ;; + + --status) + COMPREPLY=( $(compgen -W 'todo doing done now later wait waiting backlog canceled cancelled in-review in-progress' -- "$cur") ) + return ;; + + --graph) + _logseq_compadd_lines "$cur" _logseq_graphs_bash + return ;; + + --order) + COMPREPLY=( $(compgen -W 'asc desc' -- "$cur") ) + return ;; + + --input) + COMPREPLY=( $(compgen -f -- "$cur") ) + return ;; + + --cardinality) + COMPREPLY=( $(compgen -W 'one many' -- "$cur") ) + return ;; + + --shell) + COMPREPLY=( $(compgen -W 'zsh bash' -- "$cur") ) + return ;; + + --name) + if [[ "$__cmd" == 'remove' && "$__subcmd" == 'page' ]]; then + local graph + graph="$(_logseq_current_graph_bash)" + [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" + fi + if [[ "$__cmd" == 'query' ]]; then + local graph + graph="$(_logseq_current_graph_bash)" + [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_queries_bash "$graph" + fi + return ;; + + --sort) + if [[ "$__cmd" == 'list' && "$__subcmd" == 'page' ]]; then + COMPREPLY=( $(compgen -W 'title created-at updated-at' -- "$cur") ) + return + fi + if [[ "$__cmd" == 'list' && "$__subcmd" == 'tag' ]]; then + COMPREPLY=( $(compgen -W 'name title' -- "$cur") ) + return + fi + if [[ "$__cmd" == 'list' && "$__subcmd" == 'property' ]]; then + COMPREPLY=( $(compgen -W 'name title' -- "$cur") ) + return + fi + return ;; + esac + + # --- Flag / positional completion --- + if [[ "$cur" == -* ]]; then + # shellcheck disable=SC2046 + COMPREPLY=( $(compgen -W "$(_logseq_opts_for "$__cmd" "$__subcmd")" -- "$cur") ) + return + fi + + if [[ -z "$__cmd" ]]; then + COMPREPLY=( $(compgen -W 'completions doctor graph list query remove server show upsert' -- "$cur") ) + return + fi + + if [[ -z "$__subcmd" ]]; then + case "$__cmd" in + graph) COMPREPLY=( $(compgen -W 'list create switch remove validate info export import' -- "$cur") ) ;; + list) COMPREPLY=( $(compgen -W 'page tag property' -- "$cur") ) ;; + query) COMPREPLY=( $(compgen -W 'list' -- "$cur") ) ;; + remove) COMPREPLY=( $(compgen -W 'block page tag property' -- "$cur") ) ;; + server) COMPREPLY=( $(compgen -W 'list status start stop restart' -- "$cur") ) ;; + upsert) COMPREPLY=( $(compgen -W 'block page tag property' -- "$cur") ) ;; + esac + return + fi +} + +complete -F _logseq logseq diff --git a/src/main/logseq/cli/command/completions.cljs b/src/main/logseq/cli/command/completions.cljs new file mode 100644 index 0000000000..a1e7ff3582 --- /dev/null +++ b/src/main/logseq/cli/command/completions.cljs @@ -0,0 +1,12 @@ +(ns logseq.cli.command.completions + "Shell completions command." + (:require [logseq.cli.command.core :as core])) + +(def ^:private completions-spec + {:shell {:desc "Shell (zsh, bash)" + :values ["zsh" "bash"]}}) + +(def entries + [(core/command-entry ["completions"] :completions + "Generate shell completion script" + completions-spec)]) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 520c3a70b3..724b85c78a 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -12,14 +12,18 @@ :version {:desc "Show version" :coerce :boolean} :config {:desc "Path to cli.edn (default ~/logseq/cli.edn)" - :alias :c} + :alias :c + :complete :file} :graph {:desc "Graph name" - :alias :g} - :data-dir {:desc (str "Path to db-worker data dir (default " common-config/default-graphs-dir ")")} + :alias :g + :complete :graphs} + :data-dir {:desc (str "Path to db-worker data dir (default " common-config/default-graphs-dir ")") + :complete :dir} :timeout-ms {:desc "Request timeout in ms (default 10000)" :coerce :long} - :output {:desc "Output format (human, json, edn). Default: human" - :alias :o} + :output {:desc "Output format. Default: human" + :alias :o + :values ["human" "json" "edn"]} :verbose {:desc "Enable verbose debug logging to stderr" :alias :v :coerce :boolean}}) @@ -101,7 +105,9 @@ {:title "Graph Management" :commands #{"graph" "server" "doctor" "sync"}} {:title "Authentication" - :commands #{"login" "logout"}}] + :commands #{"login" "logout"}} + {:title "Utilities" + :commands #{"completions"}}] render-group (fn [{:keys [title commands]}] (let [entries (filter #(contains? commands (first (:cmds %))) table)] (string/join "\n" [title (format-commands entries)])))] diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 78a9f7527c..8b211a3ec3 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -10,12 +10,16 @@ [promesa.core :as p])) (def ^:private graph-export-spec - {:type {:desc "Export type (edn, sqlite)"} - :file {:desc "Export file path"}}) + {:type {:desc "Export type" + :values ["edn" "sqlite"]} + :file {:desc "Export file path" + :complete :file}}) (def ^:private graph-import-spec - {:type {:desc "Import type (edn, sqlite)"} - :input {:desc "Input path"}}) + {:type {:desc "Import type" + :values ["edn" "sqlite"]} + :input {:desc "Input path" + :complete :file}}) (def ^:private graph-validate-spec {:fix {:desc "Attempt to fix validation errors" diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index d5effa343c..e21b28e5f6 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -14,11 +14,14 @@ :offset {:desc "Offset results" :coerce :long} :sort {:desc "Sort field"} - :order {:desc "Sort order (asc, desc). Default: asc"}}) + :order {:desc "Sort order. Default: asc" + :values ["asc" "desc"]}}) (def ^:private list-page-spec (merge list-common-spec - {:include-journal {:desc "Include journal pages" + {:sort {:desc "Sort field" + :values ["title" "created-at" "updated-at"]} + :include-journal {:desc "Include journal pages" :coerce :boolean} :journal-only {:desc "Only journal pages" :coerce :boolean} @@ -30,7 +33,9 @@ (def ^:private list-tag-spec (merge list-common-spec - {:include-built-in {:desc "Include built-in tags" + {:sort {:desc "Sort field" + :values ["name" "title"]} + :include-built-in {:desc "Include built-in tags" :coerce :boolean} :with-properties {:desc "Include tag properties" :coerce :boolean} @@ -40,7 +45,9 @@ (def ^:private list-property-spec (merge list-common-spec - {:include-built-in {:desc "Include built-in properties" + {:sort {:desc "Sort field" + :values ["name" "title"]} + :include-built-in {:desc "Include built-in properties" :coerce :boolean} :with-classes {:desc "Include property classes" :coerce :boolean} diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index 87b86dcf9b..3915315699 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -9,7 +9,8 @@ (def ^:private query-spec {:query {:desc "Datascript query EDN"} - :name {:desc "Query name from cli.edn custom-queries or built-ins"} + :name {:desc "Query name from cli.edn custom-queries or built-ins" + :complete :queries} :inputs {:desc "EDN vector of query inputs"}}) (def ^:private query-list-spec diff --git a/src/main/logseq/cli/command/remove.cljs b/src/main/logseq/cli/command/remove.cljs index 79ec672627..2099acaa6c 100644 --- a/src/main/logseq/cli/command/remove.cljs +++ b/src/main/logseq/cli/command/remove.cljs @@ -13,7 +13,8 @@ :uuid {:desc "Block UUID"}}) (def ^:private remove-page-spec - {:name {:desc "Page name"}}) + {:name {:desc "Page name" + :complete :pages}}) (def ^:private remove-entity-spec {:id {:desc "Entity db/id" diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 0ab5106279..79f61e6366 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -14,7 +14,8 @@ (def ^:private show-spec {:id {:desc "Block db/id or EDN vector of ids"} :uuid {:desc "Block UUID"} - :page {:desc "Page name"} + :page {:desc "Page name" + :complete :pages} :linked-references {:desc "Include linked references (default true)" :coerce :boolean} :level {:desc "Limit tree depth (default 10)" diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index 08e8f1909f..a77b7a8cbb 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -16,12 +16,17 @@ :target-id {:desc "Target block db/id" :coerce :long} :target-uuid {:desc "Target block UUID"} - :target-page {:desc "Target page name"} - :pos {:desc "Position (first-child, last-child, sibling). Default: create=last-child, update=first-child"} + :target-page {:desc "Target page name" + :complete :pages} + :pos {:desc "Position. Default: create=last-child, update=first-child" + :values ["first-child" "last-child" "sibling"]} :content {:desc "Block content for create mode"} :blocks {:desc "EDN vector of blocks for create mode"} - :blocks-file {:desc "EDN file of blocks for create mode"} - :status {:desc "Task status (todo, doing, done, etc.)"} + :blocks-file {:desc "EDN file of blocks for create mode" + :complete :file} + :status {:desc "Task status" + :values ["todo" "doing" "done" "now" "later" "wait" "waiting" + "backlog" "canceled" "cancelled" "in-review" "in-progress"]} :update-tags {:desc "Tags to add/update (EDN vector)"} :update-properties {:desc "Properties to add/update (EDN map)"} :remove-tags {:desc "Tags to remove (EDN vector)"} @@ -30,7 +35,8 @@ (def ^:private upsert-page-spec {:id {:desc "Target page db/id (forces update mode)" :coerce :long} - :page {:desc "Page name"} + :page {:desc "Page name" + :complete :pages} :update-tags {:desc "Tags to add/update (EDN vector)"} :update-properties {:desc "Properties to add/update (EDN map)"} :remove-tags {:desc "Tags to remove (EDN vector)"} @@ -45,8 +51,10 @@ {:id {:desc "Target property db/id (forces update mode)" :coerce :long} :name {:desc "Property name"} - :type {:desc "Property type (default, number, date, datetime, checkbox, url, node, json, string)"} - :cardinality {:desc "Property cardinality (one, many)"} + :type {:desc "Property type" + :values ["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"]} + :cardinality {:desc "Property cardinality" + :values ["one" "many"]} :hide {:desc "Hide property" :coerce :boolean} :public {:desc "Set property public visibility" diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index e4f29e79f3..9b3901b051 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -3,6 +3,7 @@ (:require [babashka.cli :as cli] [clojure.string :as string] [logseq.cli.command.auth :as auth-command] + [logseq.cli.command.completions :as completions-command] [logseq.cli.command.core :as command-core] [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] @@ -13,6 +14,7 @@ [logseq.cli.command.show :as show-command] [logseq.cli.command.sync :as sync-command] [logseq.cli.command.upsert :as upsert-command] + [logseq.cli.completion-generator :as completion-gen] [logseq.cli.server :as cli-server] [promesa.core :as p])) @@ -105,7 +107,8 @@ show-command/entries doctor-command/entries sync-command/entries - auth-command/entries))) + auth-command/entries + completions-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -422,6 +425,11 @@ (:login :logout) (auth-command/build-action command) + :completions + {:ok? true + :action {:type :completions + :shell (or (:shell options) (first args))}} + {:ok? false :error {:code :unknown-command :message (str "unknown command: " command)}})))) @@ -463,6 +471,10 @@ :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) :doctor (doctor-command/execute-doctor action config) + :completions (p/resolved + {:status :ok + :data {:message (completion-gen/generate-completions + (:shell action) table)}}) :server-list (server-command/execute-list action config) :server-status (server-command/execute-status action config) :server-start (server-command/execute-start action config) diff --git a/src/main/logseq/cli/completion_generator.cljs b/src/main/logseq/cli/completion_generator.cljs new file mode 100644 index 0000000000..475a43500d --- /dev/null +++ b/src/main/logseq/cli/completion_generator.cljs @@ -0,0 +1,786 @@ +(ns logseq.cli.completion-generator + "Pure function: (generate-completions shell table) → string. + Walks the CLI command table and emits zsh or bash completion scripts." + (:require [clojure.string :as string])) + +;; --------------------------------------------------------------------------- +;; Table introspection utilities +;; --------------------------------------------------------------------------- + +(defn extract-groups + "Given a table of entries, group by the first element of :cmds. + Returns a map of {group-name [entries...]}." + [table] + (reduce (fn [acc entry] + (let [cmds (:cmds entry) + group (first cmds)] + (update acc group (fnil conj []) entry))) + {} + table)) + +(defn leaf-commands + "Return entries that are leaf commands (single-element :cmds)." + [table] + (let [groups (extract-groups table)] + (->> groups + (filter (fn [[_ entries]] + (and (= 1 (count entries)) + (= 1 (count (:cmds (first entries))))))) + (mapv (fn [[_ entries]] (first entries)))))) + +(defn group-commands + "Return group names that have subcommands (multi-element :cmds entries)." + [table] + (let [groups (extract-groups table)] + (->> groups + (filter (fn [[_ entries]] + (or (> (count entries) 1) + (> (count (:cmds (first entries))) 1)))) + (mapv first)))) + +(defn spec->token + "Convert a single spec entry [key spec-map] to a token descriptor. + Returns {:key k :type t ...} with type being one of: + :flag, :enum, :dynamic, :file, :dir, :free" + [[k spec-map]] + (let [alias (:alias spec-map) + desc (or (:desc spec-map) "") + coerce (:coerce spec-map) + values (:values spec-map) + complete (:complete spec-map)] + (cond-> {:key k + :desc desc} + alias (assoc :alias alias) + (= coerce :boolean) (assoc :type :flag) + (and (not= coerce :boolean) (seq values)) (assoc :type :enum :values values) + (and (not= coerce :boolean) (nil? values) (= complete :graphs)) (assoc :type :dynamic :complete :graphs) + (and (not= coerce :boolean) (nil? values) (= complete :pages)) (assoc :type :dynamic :complete :pages) + (and (not= coerce :boolean) (nil? values) (= complete :queries)) (assoc :type :dynamic :complete :queries) + (and (not= coerce :boolean) (nil? values) (= complete :file)) (assoc :type :file) + (and (not= coerce :boolean) (nil? values) (= complete :dir)) (assoc :type :dir) + (and (not= coerce :boolean) (nil? values) (nil? complete)) (assoc :type :free)))) + +(defn spec->tokens + "Convert a full spec map to a vector of token descriptors." + [spec] + (mapv spec->token spec)) + +;; --------------------------------------------------------------------------- +;; Zsh dynamic helpers (verbatim preamble) +;; --------------------------------------------------------------------------- + +(def ^:private zsh-preamble + "#compdef logseq +# Auto-generated by `logseq completions zsh` — do not edit manually. + +# --- dynamic helpers --- + +_logseq_json_names() { + python3 -c \" +import sys, json +field = sys.argv[1] +try: + data = json.load(sys.stdin) + if isinstance(data, list): + for item in data: + v = item.get(field) + if isinstance(v, str) and v: + print(v) +except Exception: + pass +\" \"$1\" 2>/dev/null +} + +_logseq_current_graph() { + local i + for (( i = 1; i < ${#words[@]}; i++ )); do + if [[ \"${words[i]}\" == '--graph' && -n \"${words[i+1]}\" ]]; then + print -r -- \"${words[i+1]}\" + return + fi + done +} + +_logseq_graphs() { + local cache_id='logseq_graphs' + local -a graphs + if _cache_invalid \"$cache_id\" || ! _retrieve_cache \"$cache_id\"; then + graphs=( ${(f)\"$(logseq graph list --output json 2>/dev/null | _logseq_json_names name)\"} ) + _store_cache \"$cache_id\" graphs + fi + compadd -a graphs +} + +_logseq_pages() { + local graph + graph=$(_logseq_current_graph) + local cache_id=\"logseq_pages_${graph:-__default__}\" + local -a pages + if _cache_invalid \"$cache_id\" || ! _retrieve_cache \"$cache_id\"; then + if [[ -n \"$graph\" ]]; then + pages=( ${(f)\"$(logseq list page --graph \"$graph\" --output json 2>/dev/null | _logseq_json_names title)\"} ) + fi + _store_cache \"$cache_id\" pages + fi + compadd -a pages +} + +_logseq_queries() { + local graph + graph=$(_logseq_current_graph) + local cache_id=\"logseq_queries_${graph:-__default__}\" + local -a queries + if _cache_invalid \"$cache_id\" || ! _retrieve_cache \"$cache_id\"; then + if [[ -n \"$graph\" ]]; then + queries=( ${(f)\"$(logseq query list --graph \"$graph\" --output json 2>/dev/null | _logseq_json_names name)\"} ) + fi + _store_cache \"$cache_id\" queries + fi + compadd -a queries +} +") + +;; --------------------------------------------------------------------------- +;; Zsh generation +;; --------------------------------------------------------------------------- + +(defn- zsh-option-name + [k] + (str "--" (name k))) + +(defn- zsh-escape-desc + [desc] + (-> desc + (string/replace "[" "\\[") + (string/replace "]" "\\]") + (string/replace "'" "'\\''"))) + +(defn- zsh-token-for + "Generate a zsh _arguments token string for a spec token descriptor." + [{:keys [key type alias desc values complete]}] + (let [long-opt (zsh-option-name key) + desc* (zsh-escape-desc desc) + alias-short (when alias (str "-" (name alias)))] + (case type + :flag + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'[" desc* "]'") + (str "'" long-opt "[" desc* "]'")) + + :enum + (let [vals-str (string/join " " values)] + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'=[" desc* "]:value:(" vals-str ")'") + (str "'" long-opt "=[" desc* "]:value:(" vals-str ")'")) + ) + + :dynamic + (let [action (case complete + :graphs "_logseq_graphs" + :pages "_logseq_pages" + :queries "_logseq_queries")] + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'=[" desc* "]:value:" action "'") + (str "'" long-opt "=[" desc* "]:value:" action "'"))) + + :file + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'=[" desc* "]:file:_files'") + (str "'" long-opt "=[" desc* "]:file:_files'")) + + :dir + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'=[" desc* "]:dir:_files -/'") + (str "'" long-opt "=[" desc* "]:dir:_files -/'")) + + :free + (if alias + (str "'" (str "(" alias-short " " long-opt ")") "'" + "{" alias-short "," long-opt "}" + "'=[" desc* "]:value:'") + (str "'" long-opt "=[" desc* "]:value:'")) + + ;; default + (str "'" long-opt "=[" desc* "]:value:'")))) + +(defn- zsh-arguments-tokens + "Generate _arguments tokens for a spec map." + [spec] + (->> (spec->tokens spec) + (mapv zsh-token-for))) + +(defn- zsh-leaf-function + "Generate a _logseq_() function for a leaf command." + [func-name spec] + (let [tokens (zsh-arguments-tokens spec) + token-lines (string/join " \\\n " tokens)] + (str func-name "() {\n" + " _arguments -s \\\n" + " " token-lines "\n" + "}\n"))) + +(defn- cmd->func-name + "Convert a cmds vector like [\"graph\" \"export\"] to \"_logseq_graph_export\"." + [cmds] + (str "_logseq_" (string/join "_" cmds))) + +(defn- zsh-group-function + "Generate a group dispatcher function. + Handles groups that have both a root command (e.g. [\"query\"]) and + subcommands (e.g. [\"query\" \"list\"])." + [group-name subentries global-spec] + (let [func-name (str "_logseq_" group-name) + ;; Separate root entry from subcommand entries + root-entry (first (filter #(= 1 (count (:cmds %))) subentries)) + sub-entries (filter #(> (count (:cmds %)) 1) subentries) + ;; Root-level options (global + root command's own spec if present) + root-spec (if root-entry + (:spec root-entry) + global-spec) + root-tokens (zsh-arguments-tokens root-spec) + root-lines (string/join " \\\n " root-tokens) + subcmds (->> sub-entries + (mapv (fn [entry] + (let [subcmd (second (:cmds entry)) + desc (or (:desc entry) "")] + (str " '" subcmd ":" desc "'"))))) + subcmd-lines (string/join "\n" subcmds) + dispatches (->> sub-entries + (mapv (fn [entry] + (let [subcmd (second (:cmds entry)) + sub-func (cmd->func-name (:cmds entry))] + (str " " subcmd ") " sub-func " ;;")))) + ) + dispatch-lines (string/join "\n" dispatches)] + (str func-name "() {\n" + " local curcontext=\"$curcontext\" state line\n" + " typeset -A opt_args\n" + "\n" + " _arguments -C -s \\\n" + " " root-lines " \\\n" + " '1:subcommand:->subcmd' \\\n" + " '*::args:->args'\n" + "\n" + " case $state in\n" + " subcmd)\n" + " local -a subcmds\n" + " subcmds=(\n" + subcmd-lines "\n" + " )\n" + " _describe 'subcommand' subcmds\n" + " ;;\n" + " args)\n" + " case $line[1] in\n" + dispatch-lines "\n" + " esac\n" + " ;;\n" + " esac\n" + "}\n"))) + +(defn- zsh-toplevel-function + "Generate the _logseq() root dispatcher." + [table global-spec] + (let [groups (extract-groups table) + group-names (sort (keys groups)) + ;; Build command descriptions for top level + cmd-descs (->> group-names + (mapv (fn [g] + (let [entries (get groups g) + desc (if (and (= 1 (count entries)) + (= 1 (count (:cmds (first entries))))) + (or (:desc (first entries)) "") + (str g " commands"))] + (str " '" g ":" desc "'"))))) + cmd-desc-lines (string/join "\n" cmd-descs) + dispatches (->> group-names + (mapv (fn [g] + (let [entries (get groups g) + func (if (and (= 1 (count entries)) + (= 1 (count (:cmds (first entries))))) + (cmd->func-name (:cmds (first entries))) + (str "_logseq_" g))] + (str " " g ") " func " ;;")))) + ) + dispatch-lines (string/join "\n" dispatches) + global-tokens (zsh-arguments-tokens global-spec) + global-lines (string/join " \\\n " global-tokens)] + (str "_logseq() {\n" + " local curcontext=\"$curcontext\" state line\n" + " typeset -A opt_args\n" + "\n" + " _arguments -C -s \\\n" + " " global-lines " \\\n" + " '1:command:->cmds' \\\n" + " '*::args:->args'\n" + "\n" + " case $state in\n" + " cmds)\n" + " local -a cmds\n" + " cmds=(\n" + cmd-desc-lines "\n" + " )\n" + " _describe 'command' cmds\n" + " ;;\n" + " args)\n" + " case $line[1] in\n" + dispatch-lines "\n" + " esac\n" + " ;;\n" + " esac\n" + "}\n"))) + +(defn generate-zsh + "Generate complete zsh completion script from the command table." + [table] + (let [groups (extract-groups table) + global-spec (-> table first :spec + (select-keys [:help :version :config :graph :data-dir + :timeout-ms :output :verbose])) + ;; Generate leaf command functions + leaf-fns (->> groups + (mapcat (fn [[_ entries]] + (mapv (fn [entry] + (zsh-leaf-function + (cmd->func-name (:cmds entry)) + (:spec entry))) + entries))) + (string/join "\n")) + ;; Generate group dispatchers + group-fns (->> groups + (filter (fn [[_ entries]] + (or (> (count entries) 1) + (> (count (:cmds (first entries))) 1)))) + (mapv (fn [[group-name entries]] + (zsh-group-function group-name entries global-spec))) + (string/join "\n")) + ;; Generate top-level dispatcher + toplevel (zsh-toplevel-function table global-spec)] + (str zsh-preamble "\n" + "# --- per-command functions ---\n\n" + leaf-fns "\n" + "# --- group dispatchers ---\n\n" + group-fns "\n" + "# --- top-level dispatcher ---\n\n" + toplevel "\n" + "_logseq \"$@\"\n"))) + +;; --------------------------------------------------------------------------- +;; Bash dynamic helpers (verbatim preamble) +;; --------------------------------------------------------------------------- + +(def ^:private bash-preamble + "# Auto-generated by `logseq completions bash` — do not edit manually. + +# --- dynamic helpers --- + +_logseq_json_names_bash() { + python3 -c \" +import sys, json +field = sys.argv[1] +try: + data = json.load(sys.stdin) + if isinstance(data, list): + for item in data: + v = item.get(field) + if isinstance(v, str) and v: + print(v) +except Exception: + pass +\" \"$1\" 2>/dev/null +} + +_logseq_current_graph_bash() { + local i + for (( i = 1; i < ${#COMP_WORDS[@]}; i++ )); do + if [[ \"${COMP_WORDS[i]}\" == '--graph' && -n \"${COMP_WORDS[i+1]}\" ]]; then + printf '%s' \"${COMP_WORDS[i+1]}\" + return + fi + done +} + +_logseq_graphs_bash() { + logseq graph list --output json 2>/dev/null | _logseq_json_names_bash name +} + +_logseq_pages_bash() { + logseq list page --graph \"$1\" --output json 2>/dev/null | _logseq_json_names_bash title +} + +_logseq_queries_bash() { + logseq query list --graph \"$1\" --output json 2>/dev/null | _logseq_json_names_bash name +} + +_logseq_compadd_lines() { + local cur=\"$1\" source_fn=\"$2\"; shift 2 + while IFS= read -r item; do + [[ \"$item\" == \"$cur\"* ]] && COMPREPLY+=( \"$item\" ) + done < <(\"$source_fn\" \"$@\") +} +") + +;; --------------------------------------------------------------------------- +;; Bash generation +;; --------------------------------------------------------------------------- + +(defn- bash-option-name + [k] + (str "--" (name k))) + +(defn- bash-all-value-opts + "Collect all non-boolean option names (options that consume a value argument)." + [table] + (let [all-specs (->> table (mapcat (fn [entry] (seq (:spec entry))))) + value-opts (->> all-specs + (remove (fn [[_ spec-map]] + (= :boolean (:coerce spec-map)))) + (mapv (fn [[k _]] (bash-option-name k))))] + (-> (set value-opts) + sort + vec))) + +(defn- bash-is-value-opt + "Generate _logseq_is_value_opt function." + [table] + (let [opts (bash-all-value-opts table) + cases (string/join "|" opts)] + (str "_logseq_is_value_opt() {\n" + " case \"$1\" in\n" + " " cases ")\n" + " return 0 ;;\n" + " *) return 1 ;;\n" + " esac\n" + "}\n"))) + +(defn- bash-cmd-and-subcmd + "Generate _logseq_cmd_and_subcmd function." + [] + "_logseq_cmd_and_subcmd() { + local i skip=0 + __cmd='' __subcmd='' + for (( i = 1; i < COMP_CWORD; i++ )); do + local w=\"${COMP_WORDS[i]}\" + if (( skip )); then skip=0; continue; fi + if [[ \"$w\" == -* ]]; then + _logseq_is_value_opt \"$w\" && skip=1 + continue + fi + if [[ -z \"$__cmd\" ]]; then + __cmd=\"$w\" + elif [[ -z \"$__subcmd\" ]]; then + __subcmd=\"$w\" + fi + done +}\n") + +(defn- bash-global-opts-string + "Generate the global opts wordlist string." + [global-spec] + (->> (keys global-spec) + (mapcat (fn [k] + (let [spec-map (get global-spec k) + long-opt (bash-option-name k) + alias (:alias spec-map)] + (if alias + [long-opt (str "-" (name alias))] + [long-opt])))) + (string/join " "))) + +(defn- bash-opts-for + "Generate _logseq_opts_for function." + [table] + (let [groups (extract-groups table) + global-spec (-> table first :spec + (select-keys [:help :version :config :graph :data-dir + :timeout-ms :output :verbose])) + global-str (bash-global-opts-string global-spec) + ;; Build case branches + branches + (->> (sort-by first groups) + (mapv (fn [[group-name entries]] + (if (and (= 1 (count entries)) + (= 1 (count (:cmds (first entries))))) + ;; Leaf command + (let [entry (first entries) + cmd-spec (apply dissoc (:spec entry) (keys global-spec)) + cmd-opts (->> (keys cmd-spec) + (mapcat (fn [k] + (let [sm (get cmd-spec k)] + (if (:alias sm) + [(bash-option-name k) (str "-" (name (:alias sm)))] + [(bash-option-name k)])))) + (string/join " "))] + (str " " group-name ") opts+=' " cmd-opts "' ;;")) + ;; Group with subcommands + (let [;; Root entry opts go at group level, sub-entries get case branches + root-entry (first (filter #(= 1 (count (:cmds %))) entries)) + sub-entries (filter #(> (count (:cmds %)) 1) entries) + root-opts (when root-entry + (let [cmd-spec (apply dissoc (:spec root-entry) (keys global-spec))] + (->> (keys cmd-spec) + (mapcat (fn [k] + (let [sm (get cmd-spec k)] + (if (:alias sm) + [(bash-option-name k) (str "-" (name (:alias sm)))] + [(bash-option-name k)])))) + (string/join " ")))) + sub-branches + (->> sub-entries + (mapv (fn [entry] + (let [subcmd (second (:cmds entry)) + cmd-spec (apply dissoc (:spec entry) (keys global-spec)) + cmd-opts (->> (keys cmd-spec) + (mapcat (fn [k] + (let [sm (get cmd-spec k)] + (if (:alias sm) + [(bash-option-name k) (str "-" (name (:alias sm)))] + [(bash-option-name k)])))) + (string/join " "))] + (str " " subcmd ") opts+=' " cmd-opts "' ;;")))) + (string/join "\n"))] + (str " " group-name ")\n" + (when (seq root-opts) + (str " opts+=' " root-opts "'\n")) + " case \"$subcmd\" in\n" + sub-branches "\n" + " esac\n" + " ;;")))) + ))] + (str "_logseq_opts_for() {\n" + " local cmd=\"$1\" subcmd=\"$2\"\n" + " local opts=\"" global-str "\"\n" + "\n" + " case \"$cmd\" in\n" + (string/join "\n" branches) "\n" + " esac\n" + "\n" + " printf '%s' \"$opts\"\n" + "}\n"))) + +(defn- bash-prev-completion-case + "Generate a case branch for prev-word value completion." + [{:keys [key type values complete]}] + (let [long-opt (bash-option-name key)] + (case type + :enum + (str " " long-opt ")\n" + " COMPREPLY=( $(compgen -W '" (string/join " " values) "' -- \"$cur\") )\n" + " return ;;") + + :dynamic + (case complete + :graphs + (str " " long-opt ")\n" + " _logseq_compadd_lines \"$cur\" _logseq_graphs_bash\n" + " return ;;") + :pages + (str " " long-opt ")\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_pages_bash \"$graph\"\n" + " return ;;") + :queries + (str " " long-opt ")\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_queries_bash \"$graph\"\n" + " return ;;")) + + :file + (str " " long-opt ")\n" + " COMPREPLY=( $(compgen -f -- \"$cur\") )\n" + " return ;;") + + :dir + (str " " long-opt ")\n" + " COMPREPLY=( $(compgen -d -- \"$cur\") )\n" + " return ;;") + + nil))) + +(defn- bash-prev-cases + "Generate all prev-word value completion cases from the table." + [table] + (let [all-specs (->> table (mapcat (fn [entry] (seq (:spec entry))))) + unique-specs (into {} all-specs) ;; last one wins for duplicates + tokens (spec->tokens unique-specs) + cases (->> tokens + (keep bash-prev-completion-case) + (string/join "\n\n"))] + cases)) + +(defn- bash-subcommand-cases + "Generate subcommand completion for each group." + [table] + (let [groups (extract-groups table) + cases (->> (sort-by first groups) + (keep (fn [[group-name entries]] + (when (or (> (count entries) 1) + (> (count (:cmds (first entries))) 1)) + (let [subcmds (->> entries + (keep #(second (:cmds %))) + (string/join " "))] + (when (seq subcmds) + (str " " group-name ") COMPREPLY=( $(compgen -W '" + subcmds "' -- \"$cur\") ) ;;")))))))] + (string/join "\n" cases))) + +(defn- bash-toplevel-commands + "Get all top-level command names." + [table] + (let [groups (extract-groups table)] + (->> (keys groups) sort (string/join " ")))) + +(defn- bash-context-dependent-prev-cases + "Generate context-dependent prev-word cases (e.g., --name means different things + in different commands, --sort has different values per list subcommand)." + [table] + (let [groups (extract-groups table) + ;; Find all --name contexts + name-cases + (->> table + (keep (fn [entry] + (let [name-spec (get-in entry [:spec :name]) + complete (:complete name-spec)] + (when complete + {:cmds (:cmds entry) :complete complete})))) + (mapv (fn [{:keys [cmds complete]}] + (let [cmd (first cmds) + subcmd (second cmds)] + (case complete + :queries + (str " if [[ \"$__cmd\" == '" cmd "' ]]; then\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_queries_bash \"$graph\"\n" + " fi") + :pages + (str " if [[ \"$__cmd\" == '" cmd "' && \"$__subcmd\" == '" subcmd "' ]]; then\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_pages_bash \"$graph\"\n" + " fi") + nil))))) + ;; Find all --sort contexts + sort-cases + (->> table + (keep (fn [entry] + (let [sort-spec (get-in entry [:spec :sort]) + values (:values sort-spec)] + (when (seq values) + {:cmds (:cmds entry) :values values})))) + (mapv (fn [{:keys [cmds values]}] + (let [cmd (first cmds) + subcmd (second cmds) + vals-str (string/join " " values)] + (str " if [[ \"$__cmd\" == '" cmd "' && \"$__subcmd\" == '" subcmd "' ]]; then\n" + " COMPREPLY=( $(compgen -W '" vals-str "' -- \"$cur\") )\n" + " return\n" + " fi")))))] + {:name-cases name-cases + :sort-cases sort-cases})) + +(defn- bash-main-function + "Generate the _logseq() main completion function." + [table] + (let [groups (extract-groups table) + global-spec (-> table first :spec + (select-keys [:help :version :config :graph :data-dir + :timeout-ms :output :verbose])) + ;; Collect unique non-context-dependent prev-word cases + ;; (skip --name and --sort since they're context-dependent) + all-specs (->> table (mapcat (fn [entry] (seq (:spec entry))))) + unique-specs (into {} all-specs) + tokens (spec->tokens unique-specs) + context-free-tokens (remove #(#{:name :sort} (:key %)) tokens) + prev-cases (->> context-free-tokens + (keep bash-prev-completion-case) + (string/join "\n\n")) + ;; Context-dependent cases + {:keys [name-cases sort-cases]} (bash-context-dependent-prev-cases table) + ;; Subcommand completion + subcmd-cases (bash-subcommand-cases table) + ;; Top-level commands + top-cmds (bash-toplevel-commands table)] + (str "_logseq() {\n" + " local cur prev\n" + " cur=\"${COMP_WORDS[COMP_CWORD]}\"\n" + " prev=\"${COMP_WORDS[COMP_CWORD-1]}\"\n" + " COMPREPLY=()\n" + "\n" + " local __cmd __subcmd\n" + " _logseq_cmd_and_subcmd\n" + "\n" + " # --- Option value completion ---\n" + " case \"$prev\" in\n" + prev-cases "\n" + "\n" + " --name)\n" + (string/join "\n" name-cases) "\n" + " return ;;\n" + "\n" + " --sort)\n" + (string/join "\n" sort-cases) "\n" + " return ;;\n" + " esac\n" + "\n" + " # --- Flag / positional completion ---\n" + " if [[ \"$cur\" == -* ]]; then\n" + " # shellcheck disable=SC2046\n" + " COMPREPLY=( $(compgen -W \"$(_logseq_opts_for \"$__cmd\" \"$__subcmd\")\" -- \"$cur\") )\n" + " return\n" + " fi\n" + "\n" + " if [[ -z \"$__cmd\" ]]; then\n" + " COMPREPLY=( $(compgen -W '" top-cmds "' -- \"$cur\") )\n" + " return\n" + " fi\n" + "\n" + " if [[ -z \"$__subcmd\" ]]; then\n" + " case \"$__cmd\" in\n" + subcmd-cases "\n" + " esac\n" + " return\n" + " fi\n" + "}\n"))) + +(defn generate-bash + "Generate complete bash completion script from the command table." + [table] + (let [is-value-opt (bash-is-value-opt table) + cmd-and-subcmd (bash-cmd-and-subcmd) + opts-for (bash-opts-for table) + main-fn (bash-main-function table)] + (str bash-preamble "\n" + "# --- generated helpers ---\n\n" + is-value-opt "\n" + cmd-and-subcmd "\n" + "# --- option wordlists ---\n\n" + opts-for "\n" + "# --- main function ---\n\n" + main-fn "\n" + "complete -F _logseq logseq\n"))) + +;; --------------------------------------------------------------------------- +;; Public API +;; --------------------------------------------------------------------------- + +(defn generate-completions + "Generate shell completions from the CLI command table. + shell: \"zsh\" or \"bash\" + table: vector of command entries" + [shell table] + (case shell + "zsh" (generate-zsh table) + "bash" (generate-bash table) + (throw (ex-info (str "unsupported shell: " shell) {:shell shell})))) diff --git a/src/test/logseq/cli/command/completions_test.cljs b/src/test/logseq/cli/command/completions_test.cljs new file mode 100644 index 0000000000..b2051107a1 --- /dev/null +++ b/src/test/logseq/cli/command/completions_test.cljs @@ -0,0 +1,40 @@ +(ns logseq.cli.command.completions-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.command.completions :as completions-command] + [logseq.cli.commands :as commands])) + +(deftest test-completions-command-registration + (testing "completions entry has correct structure" + (let [entry (first completions-command/entries)] + (is (= ["completions"] (:cmds entry))) + (is (= :completions (:command entry))) + (is (= ["zsh" "bash"] (get-in entry [:spec :shell :values])))))) + +(deftest test-parse-args-completions-shell + (testing "parse-args recognizes completions --shell zsh" + (let [result (commands/parse-args ["completions" "--shell" "zsh"])] + (is (true? (:ok? result))) + (is (= :completions (:command result))))) + (testing "parse-args recognizes completions with positional arg" + (let [result (commands/parse-args ["completions" "zsh"])] + (is (true? (:ok? result))) + (is (= :completions (:command result)))))) + +(deftest test-build-action-completions + (testing "build-action for :completions returns correct action" + (let [parsed {:ok? true + :command :completions + :options {:shell "zsh"} + :args []} + action (commands/build-action parsed {})] + (is (true? (:ok? action))) + (is (= :completions (get-in action [:action :type]))) + (is (= "zsh" (get-in action [:action :shell]))))) + (testing "build-action with positional arg" + (let [parsed {:ok? true + :command :completions + :options {} + :args ["bash"]} + action (commands/build-action parsed {})] + (is (true? (:ok? action))) + (is (= "bash" (get-in action [:action :shell])))))) diff --git a/src/test/logseq/cli/completion_generator_test.cljs b/src/test/logseq/cli/completion_generator_test.cljs new file mode 100644 index 0000000000..08688cca4d --- /dev/null +++ b/src/test/logseq/cli/completion_generator_test.cljs @@ -0,0 +1,333 @@ +(ns logseq.cli.completion-generator-test + (:require [cljs.test :refer [deftest is testing]] + [clojure.string :as string] + [logseq.cli.command.completions :as completions-command] + [logseq.cli.command.core :as core] + [logseq.cli.command.doctor :as doctor-command] + [logseq.cli.command.graph :as graph-command] + [logseq.cli.command.list :as list-command] + [logseq.cli.command.query :as query-command] + [logseq.cli.command.remove :as remove-command] + [logseq.cli.command.server :as server-command] + [logseq.cli.command.show :as show-command] + [logseq.cli.command.upsert :as upsert-command] + [logseq.cli.completion-generator :as gen])) + +(def ^:private full-table + (vec (concat graph-command/entries + server-command/entries + list-command/entries + upsert-command/entries + remove-command/entries + query-command/entries + show-command/entries + doctor-command/entries + completions-command/entries))) + +;; --------------------------------------------------------------------------- +;; Phase 1 — Spec enrichment tests +;; --------------------------------------------------------------------------- + +(deftest test-global-spec-metadata + (let [spec (core/global-spec)] + (testing ":output has :values" + (is (= ["human" "json" "edn"] (get-in spec [:output :values])))) + (testing ":graph has :complete :graphs" + (is (= :graphs (get-in spec [:graph :complete])))) + (testing ":config has :complete :file" + (is (= :file (get-in spec [:config :complete])))) + (testing ":data-dir has :complete :dir" + (is (= :dir (get-in spec [:data-dir :complete])))))) + +(deftest test-list-spec-metadata + (let [entries list-command/entries + page-entry (first (filter #(= :list-page (:command %)) entries)) + tag-entry (first (filter #(= :list-tag (:command %)) entries)) + property-entry (first (filter #(= :list-property (:command %)) entries))] + (testing "page-spec :sort has correct values" + (is (= ["title" "created-at" "updated-at"] + (get-in page-entry [:spec :sort :values])))) + (testing "tag-spec :sort has correct values" + (is (= ["name" "title"] + (get-in tag-entry [:spec :sort :values])))) + (testing "property-spec :sort has correct values" + (is (= ["name" "title"] + (get-in property-entry [:spec :sort :values])))) + (testing "common :order has correct values" + (is (= ["asc" "desc"] + (get-in page-entry [:spec :order :values])))))) + +(deftest test-upsert-spec-metadata + (let [entries upsert-command/entries + block-entry (first (filter #(= :upsert-block (:command %)) entries)) + page-entry (first (filter #(= :upsert-page (:command %)) entries)) + property-entry (first (filter #(= :upsert-property (:command %)) entries))] + (testing "block-spec :pos has :values" + (is (= ["first-child" "last-child" "sibling"] + (get-in block-entry [:spec :pos :values])))) + (testing "block-spec :status has :values" + (is (seq (get-in block-entry [:spec :status :values])))) + (testing "block-spec :target-page has :complete :pages" + (is (= :pages (get-in block-entry [:spec :target-page :complete])))) + (testing "block-spec :blocks-file has :complete :file" + (is (= :file (get-in block-entry [:spec :blocks-file :complete])))) + (testing "page-spec :page has :complete :pages" + (is (= :pages (get-in page-entry [:spec :page :complete])))) + (testing "property-spec :type has :values" + (is (= ["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"] + (get-in property-entry [:spec :type :values])))) + (testing "property-spec :cardinality has :values" + (is (= ["one" "many"] + (get-in property-entry [:spec :cardinality :values])))))) + +(deftest test-graph-spec-metadata + (let [entries graph-command/entries + export-entry (first (filter #(= :graph-export (:command %)) entries)) + import-entry (first (filter #(= :graph-import (:command %)) entries))] + (testing "export-spec :type has :values" + (is (= ["edn" "sqlite"] (get-in export-entry [:spec :type :values])))) + (testing "export-spec :file has :complete :file" + (is (= :file (get-in export-entry [:spec :file :complete])))) + (testing "import-spec :type has :values" + (is (= ["edn" "sqlite"] (get-in import-entry [:spec :type :values])))) + (testing "import-spec :input has :complete :file" + (is (= :file (get-in import-entry [:spec :input :complete])))))) + +(deftest test-query-spec-metadata + (let [entries query-command/entries + query-entry (first (filter #(= :query (:command %)) entries))] + (testing "query-spec :name has :complete :queries" + (is (= :queries (get-in query-entry [:spec :name :complete])))))) + +(deftest test-show-spec-metadata + (let [entries show-command/entries + show-entry (first (filter #(= :show (:command %)) entries))] + (testing "show-spec :page has :complete :pages" + (is (= :pages (get-in show-entry [:spec :page :complete])))))) + +(deftest test-remove-spec-metadata + (let [entries remove-command/entries + page-entry (first (filter #(= :remove-page (:command %)) entries)) + tag-entry (first (filter #(= :remove-tag (:command %)) entries)) + property-entry (first (filter #(= :remove-property (:command %)) entries))] + (testing "remove-page :name has :complete :pages" + (is (= :pages (get-in page-entry [:spec :name :complete])))) + (testing "remove-tag :name does NOT have :complete" + (is (nil? (get-in tag-entry [:spec :name :complete])))) + (testing "remove-property :name does NOT have :complete" + (is (nil? (get-in property-entry [:spec :name :complete])))))) + +;; --------------------------------------------------------------------------- +;; Phase 2 — Generator table introspection utilities +;; --------------------------------------------------------------------------- + +(deftest test-extract-groups + (let [groups (gen/extract-groups full-table)] + (testing "graph export is in graph group" + (let [graph-entries (get groups "graph")] + (is (some #(= ["graph" "export"] (:cmds %)) graph-entries)))) + (testing "show is a standalone group" + (let [show-entries (get groups "show")] + (is (= 1 (count show-entries))) + (is (= ["show"] (:cmds (first show-entries)))))) + (testing "completions is a standalone group" + (let [completions-entries (get groups "completions")] + (is (= 1 (count completions-entries))) + (is (= ["completions"] (:cmds (first completions-entries)))))))) + +(deftest test-leaf-and-group-commands + (let [leaves (gen/leaf-commands full-table) + groups (gen/group-commands full-table) + leaf-names (set (map #(first (:cmds %)) leaves)) + group-names (set groups)] + (testing "show and doctor are leaves" + (is (contains? leaf-names "show")) + (is (contains? leaf-names "doctor"))) + (testing "graph, server, list, upsert, remove are groups" + (is (contains? group-names "graph")) + (is (contains? group-names "server")) + (is (contains? group-names "list")) + (is (contains? group-names "upsert")) + (is (contains? group-names "remove"))))) + +(deftest test-spec->token + (testing "boolean spec → :flag type" + (let [token (gen/spec->token [:help {:coerce :boolean :desc "Show help"}])] + (is (= :flag (:type token))))) + (testing "spec with :values → :enum type" + (let [token (gen/spec->token [:output {:values ["human" "json" "edn"] :desc "Format"}])] + (is (= :enum (:type token))) + (is (= ["human" "json" "edn"] (:values token))))) + (testing "spec with :complete :graphs → :dynamic type" + (let [token (gen/spec->token [:graph {:complete :graphs :desc "Graph name"}])] + (is (= :dynamic (:type token))) + (is (= :graphs (:complete token))))) + (testing "spec with :complete :file → :file type" + (let [token (gen/spec->token [:config {:complete :file :desc "Config"}])] + (is (= :file (:type token))))) + (testing "spec with :complete :dir → :dir type" + (let [token (gen/spec->token [:data-dir {:complete :dir :desc "Data dir"}])] + (is (= :dir (:type token))))) + (testing "spec with :alias → includes alias" + (let [token (gen/spec->token [:help {:alias :h :coerce :boolean :desc "Help"}])] + (is (= :h (:alias token))))) + (testing "bare string spec → :free type" + (let [token (gen/spec->token [:query {:desc "Query EDN"}])] + (is (= :free (:type token)))))) + +;; --------------------------------------------------------------------------- +;; Phase 3 — Zsh output +;; --------------------------------------------------------------------------- + +(deftest test-generate-zsh-structure + (let [output (gen/generate-completions "zsh" full-table)] + (testing "output starts with #compdef logseq" + (is (string/starts-with? output "#compdef logseq"))) + (testing "output contains dynamic helpers" + (is (string/includes? output "_logseq_graphs")) + (is (string/includes? output "_logseq_pages")) + (is (string/includes? output "_logseq_queries")) + (is (string/includes? output "_logseq_json_names")) + (is (string/includes? output "_logseq_current_graph"))) + (testing "output contains per-command functions" + (is (string/includes? output "_logseq_graph_export()")) + (is (string/includes? output "_logseq_show()"))) + (testing "output contains group dispatchers" + (is (string/includes? output "_logseq_graph()")) + (is (string/includes? output "_logseq_list()")) + (is (string/includes? output "_logseq_upsert()"))) + (testing "output contains top-level dispatcher" + (is (string/includes? output "_logseq()"))) + (testing "output ends with _logseq \"$@\"" + (is (string/includes? output "_logseq \"$@\""))) + (testing "boolean flags emit bare flag form" + (is (re-find #"--verbose\[" output))) + (testing "enum options emit value list form" + (is (re-find #"--output=.*\(human json edn\)" output))) + (testing ":complete :graphs emits _logseq_graphs" + (is (re-find #"--graph=.*_logseq_graphs" output))) + (testing ":complete :file emits _files" + (is (re-find #"--config=.*_files'" output))) + (testing ":alias emits grouping" + (is (re-find #"\(-h --help\)" output))))) + +(deftest test-zsh-command-specific-values + (let [output (gen/generate-completions "zsh" full-table)] + (testing "--pos under upsert block offers correct values" + (is (re-find #"--pos=.*\(first-child last-child sibling\)" output))) + (testing "--sort for list page offers correct values" + (is (re-find #"--sort=.*\(title created-at updated-at\)" output))) + (testing "--sort for list tag offers name title" + ;; The list tag function should contain (name title) + (let [tag-section (second (re-find #"_logseq_list_tag\(\).*?(?=\n_logseq)" output))] + ;; Just check globally that name title appears in sort context + (is (re-find #"\(name title\)" output)))))) + +(deftest test-zsh-all-commands-present + (let [output (gen/generate-completions "zsh" full-table)] + (testing "every command from the table appears" + (doseq [entry full-table] + (let [func-name (str "_logseq_" (string/join "_" (:cmds entry)))] + (is (string/includes? output (str func-name "()")))))))) + +;; --------------------------------------------------------------------------- +;; Phase 4 — Bash output +;; --------------------------------------------------------------------------- + +(deftest test-generate-bash-structure + (let [output (gen/generate-completions "bash" full-table)] + (testing "output contains dynamic helpers" + (is (string/includes? output "_logseq_graphs_bash")) + (is (string/includes? output "_logseq_pages_bash")) + (is (string/includes? output "_logseq_queries_bash")) + (is (string/includes? output "_logseq_compadd_lines")) + (is (string/includes? output "_logseq_json_names_bash")) + (is (string/includes? output "_logseq_current_graph_bash"))) + (testing "output contains _logseq_opts_for" + (is (string/includes? output "_logseq_opts_for()"))) + (testing "output contains _logseq_is_value_opt" + (is (string/includes? output "_logseq_is_value_opt()"))) + (testing "output ends with complete -F _logseq logseq" + (is (string/includes? output "complete -F _logseq logseq"))) + (testing "graph export case includes --type and --file" + (is (string/includes? output "--type")) + (is (string/includes? output "--file"))) + (testing "boolean flags appear in wordlist" + (is (string/includes? output "--verbose"))) + (testing "enum values use compgen -W" + (is (re-find #"compgen -W.*human json edn" output))) + (testing ":complete :file uses compgen -f" + (is (re-find #"compgen -f" output))))) + +(deftest test-bash-all-commands-present + (let [output (gen/generate-completions "bash" full-table)] + (testing "every top-level command appears in subcommand completion" + (doseq [group-name (distinct (map #(first (:cmds %)) full-table))] + (is (string/includes? output group-name) + (str "missing command: " group-name)))))) + +;; --------------------------------------------------------------------------- +;; Phase 5 — Completions command entry +;; --------------------------------------------------------------------------- + +(deftest test-completions-command-entry + (let [entries completions-command/entries] + (testing "contains one entry with :cmds [\"completions\"]" + (is (= 1 (count entries))) + (is (= ["completions"] (:cmds (first entries))))) + (testing "command is :completions" + (is (= :completions (:command (first entries))))) + (testing "spec has :shell with :values" + (is (= ["zsh" "bash"] + (get-in (first entries) [:spec :shell :values])))))) + +;; --------------------------------------------------------------------------- +;; Phase 6 — End-to-end validation +;; --------------------------------------------------------------------------- + +(deftest test-e2e-zsh-structural-markers + (let [output (gen/generate-completions "zsh" full-table)] + (testing "key structural markers present" + (is (string/includes? output "#compdef")) + (is (string/includes? output "_logseq_graph_export")) + (is (string/includes? output "_logseq_show")) + (is (string/includes? output "_logseq \"$@\""))))) + +(deftest test-e2e-bash-structural-markers + (let [output (gen/generate-completions "bash" full-table)] + (testing "key structural markers present" + (is (string/includes? output "complete -F _logseq logseq")) + (is (string/includes? output "_logseq_opts_for"))))) + +(deftest test-e2e-sync-adding-command + (testing "adding a command updates output" + (let [base-output (gen/generate-completions "zsh" full-table) + fake-entry (core/command-entry ["fake"] :fake "Fake command" + {:foo {:desc "Foo option"}}) + extended-table (conj full-table fake-entry) + new-output (gen/generate-completions "zsh" extended-table)] + (is (not (string/includes? base-output "_logseq_fake()"))) + (is (string/includes? new-output "_logseq_fake()"))))) + +(deftest test-e2e-context-dependent-name + (let [entries full-table] + (testing "query spec has :name with :complete :queries" + (let [query-entry (first (filter #(= :query (:command %)) entries))] + (is (= :queries (get-in query-entry [:spec :name :complete]))))) + (testing "remove page spec has :name with :complete :pages" + (let [rm-page (first (filter #(= :remove-page (:command %)) entries))] + (is (= :pages (get-in rm-page [:spec :name :complete]))))) + (testing "upsert tag spec does NOT have :complete on :name" + (let [tag (first (filter #(= :upsert-tag (:command %)) entries))] + (is (nil? (get-in tag [:spec :name :complete]))))) + (testing "remove tag spec does NOT have :complete on :name" + (let [tag (first (filter #(= :remove-tag (:command %)) entries))] + (is (nil? (get-in tag [:spec :name :complete]))))))) + +(deftest test-e2e-generated-header + (testing "zsh output includes do-not-edit header" + (let [output (gen/generate-completions "zsh" full-table)] + (is (string/includes? output "do not edit manually")))) + (testing "bash output includes do-not-edit header" + (let [output (gen/generate-completions "bash" full-table)] + (is (string/includes? output "do not edit manually"))))) From 13d8f0e419a5c4cbb264183e0a13109cff18f221 Mon Sep 17 00:00:00 2001 From: Danzu Date: Sun, 8 Mar 2026 19:26:21 -0500 Subject: [PATCH 134/375] Rename CLI command from "completions" to "completion" Also fix zsh eval support by replacing `_logseq "$@"` with `compdef _logseq logseq` so `eval "$(logseq completion zsh)"` works in ~/.zshrc without the "_arguments: can only be called from completion function" error. Co-Authored-By: Claude Opus 4.6 --- dz/scripts/shell-completions/DESIGN.md | 38 +++++++++--------- dz/scripts/shell-completions/README.md | 18 ++++++--- dz/scripts/shell-completions/TASKS.md | 38 +++++++++--------- dz/scripts/shell-completions/_logseq.zsh | 10 ++--- dz/scripts/shell-completions/logseq.bash | 6 +-- .../{completions.cljs => completion.cljs} | 10 ++--- src/main/logseq/cli/command/core.cljs | 2 +- src/main/logseq/cli/commands.cljs | 10 ++--- src/main/logseq/cli/completion_generator.cljs | 6 +-- .../logseq/cli/command/completion_test.cljs | 40 +++++++++++++++++++ .../logseq/cli/command/completions_test.cljs | 40 ------------------- .../logseq/cli/completion_generator_test.cljs | 32 +++++++-------- 12 files changed, 129 insertions(+), 121 deletions(-) rename src/main/logseq/cli/command/{completions.cljs => completion.cljs} (50%) create mode 100644 src/test/logseq/cli/command/completion_test.cljs delete mode 100644 src/test/logseq/cli/command/completions_test.cljs diff --git a/dz/scripts/shell-completions/DESIGN.md b/dz/scripts/shell-completions/DESIGN.md index e5f67de7f3..7a0e63126d 100644 --- a/dz/scripts/shell-completions/DESIGN.md +++ b/dz/scripts/shell-completions/DESIGN.md @@ -12,12 +12,12 @@ The Logseq CLI (`src/main/logseq/cli`) uses `babashka.cli` for argument parsing. Commands and their options are declared as data in a `table` of `{:cmds [...] :spec {...}}` entries assembled in `commands.cljs`. -A new `logseq completions ` command will walk the `table` at +A new `logseq completion ` command will walk the `table` at generation time and emit correct zsh and bash completion scripts. ``` -logseq completions zsh > ~/.zsh/completions/_logseq -logseq completions bash > ~/.local/share/bash-completion/completions/logseq +logseq completion zsh > ~/.zsh/completions/_logseq +logseq completion bash > ~/.local/share/bash-completion/completions/logseq ``` --- @@ -40,7 +40,7 @@ still inherit the global spec. | `query` | _(root)_, `list` | `:query`, `:query-list` | | `show` | _(leaf — no subcommands)_ | `:show` | | `doctor` | _(leaf — no subcommands)_ | `:doctor` | -| `completions` | _(leaf — new)_ | `:completions` | +| `completion` | _(leaf — new)_ | `:completion` | ### 2.2 Global spec (`command/core.cljs`) @@ -336,14 +336,14 @@ Complete list of locations that need `:complete` added: --- -## 4. New `completions` command +## 4. New `completion` command -New file: `src/main/logseq/cli/command/completions.cljs` +New file: `src/main/logseq/cli/command/completion.cljs` ``` -logseq completions zsh -logseq completions bash -logseq completions --help +logseq completion zsh +logseq completion bash +logseq completion --help ``` The command: @@ -356,15 +356,15 @@ The command: Registration in `commands.cljs`: ```clojure -(core/command-entry ["completions"] :completions +(core/command-entry ["completion"] :completion "Generate shell completion script" {:shell {:desc "Shell (zsh, bash)" :values ["zsh" "bash"]}}) ``` -The `completions` command itself **must be excluded** from the generated +The `completion` command itself **must be excluded** from the generated completion dispatch (it is a meta-command, not a user-facing workflow command). Whether to exclude it is a design choice; including it is also -acceptable since `logseq completions ` → `zsh bash` is helpful. +acceptable since `logseq completion ` → `zsh bash` is helpful. --- @@ -409,7 +409,7 @@ The `table` as returned by `commands/build-table` — a vector of ```zsh #compdef logseq -# Auto-generated by `logseq completions zsh` — do not edit manually. +# Auto-generated by `logseq completion zsh` — do not edit manually. # --- dynamic helpers (verbatim, fixed) --- _logseq_json_names() { ... } @@ -444,7 +444,7 @@ _logseq "$@" ### 5.4 Output structure (bash) ```bash -# Auto-generated by `logseq completions bash` — do not edit manually. +# Auto-generated by `logseq completion bash` — do not edit manually. # --- dynamic helpers (verbatim, fixed) --- _logseq_json_names_bash() { ... } @@ -540,7 +540,7 @@ table | Boolean flags shouldn't suggest a value | `:coerce :boolean` → emit as bare flag token | | `remove page --name` needs page completion but `remove tag --name` does not | `remove-page-spec` gets its own spec with `{:name {:complete :pages}}`, separate from `remove-entity-spec` | | `server` commands inherit `--graph` from global spec + also declare it in command spec | Redundant shadow — no issue, merged spec contains one `:graph` entry | -| `completions` command in generated output | Include it — `logseq completions ` → `zsh bash` is useful | +| `completion` command in generated output | Include it — `logseq completion ` → `zsh bash` is useful | | Positional arguments (`graph create `, `graph switch `) | These commands take the graph name via `--graph` (global opt). No positional arg completion needed beyond subcommand dispatch. | --- @@ -562,7 +562,7 @@ These are emitted as fixed preamble strings by the generator: | Path | Purpose | | ----------------------------------------------- | ------------------------------------------------------------ | -| `src/main/logseq/cli/command/completions.cljs` | Command entry: parse args, call generator, print to stdout | +| `src/main/logseq/cli/command/completion.cljs` | Command entry: parse args, call generator, print to stdout | | `src/main/logseq/cli/completion_generator.cljs` | Pure function: `(generate-completions shell table) → string` | --- @@ -578,19 +578,19 @@ These are emitted as fixed preamble strings by the generator: | `command/query.cljs` | Add `:complete :queries` to `:name` | | `command/show.cljs` | Add `:complete :pages` to `:page` | | `command/remove.cljs` | Split `remove-page-spec` from generic `remove-entity-spec` so `:name` gets `:complete :pages` only for `remove page` | -| `commands.cljs` | Add `completions-command/entries` to the `table` concat | +| `commands.cljs` | Add `completion-command/entries` to the `table` concat | --- ## 12. Acceptance criteria -- [ ] `logseq completions zsh` output, when installed as `_logseq`, passes +- [ ] `logseq completion zsh` output, when installed as `_logseq`, passes smoke tests: `logseq `, `logseq list `, `logseq upsert block --pos `, `logseq show --page `, `logseq remove `, `logseq server `, `logseq doctor `, `logseq query --name `, `logseq remove page --name `, `logseq graph export --type `, `logseq upsert block --blocks-file `. -- [ ] `logseq completions bash` output passes equivalent smoke tests. +- [ ] `logseq completion bash` output passes equivalent smoke tests. - [ ] Adding a new `command-entry` to the table (or a `:values` change to a spec) causes the generated output to change with no other edits required. diff --git a/dz/scripts/shell-completions/README.md b/dz/scripts/shell-completions/README.md index d2fe2b0aae..daece614ff 100644 --- a/dz/scripts/shell-completions/README.md +++ b/dz/scripts/shell-completions/README.md @@ -9,8 +9,8 @@ These files are **auto-generated** from the CLI command table. Do not edit them manually — regenerate instead: ```bash -logseq completions zsh > dz/scripts/shell-completions/_logseq.zsh -logseq completions bash > dz/scripts/shell-completions/logseq.bash +logseq completion zsh > dz/scripts/shell-completions/_logseq.zsh +logseq completion bash > dz/scripts/shell-completions/logseq.bash ``` The generator reads the command table at generation time, so adding or @@ -30,6 +30,14 @@ completion scripts stay in sync automatically. ### Installation (zsh) +**Option A — eval in `~/.zshrc`** (simplest, always up to date): + +```zsh +eval "$(logseq completion zsh)" +``` + +**Option B — fpath** (avoids running `logseq` at shell startup): + 1. Copy `_logseq.zsh` to a directory on your `$fpath`, **renaming it to `_logseq`** (no extension — required by zsh's autoload mechanism): @@ -74,7 +82,7 @@ rm -f ~/.zcompcache/logseq_* After upgrading the `logseq` CLI, regenerate and re-copy: ```zsh -logseq completions zsh > ~/.zsh/completions/_logseq +logseq completion zsh > ~/.zsh/completions/_logseq compinit ``` @@ -133,7 +141,7 @@ are as fresh as the CLI response. After upgrading the `logseq` CLI, regenerate and re-copy: ```bash -logseq completions bash > ~/.local/share/bash-completion/completions/logseq +logseq completion bash > ~/.local/share/bash-completion/completions/logseq ``` --- @@ -151,7 +159,7 @@ logseq graph list | create | switch | remove | validate | info | export | logseq server list | status | start | stop | restart logseq show logseq doctor -logseq completions +logseq completion ``` ### Global options diff --git a/dz/scripts/shell-completions/TASKS.md b/dz/scripts/shell-completions/TASKS.md index c3b218b144..09b176efec 100644 --- a/dz/scripts/shell-completions/TASKS.md +++ b/dz/scripts/shell-completions/TASKS.md @@ -1,10 +1,10 @@ # Shell Completions — TDD Implementation Tasks -Implementation plan for the `logseq completions ` feature as specified +Implementation plan for the `logseq completion ` feature as specified in [DESIGN.md](DESIGN.md). Each task follows **Red → Green → Refactor**: write a failing test first, then make it pass, then clean up. -Test file: `src/test/logseq/cli/command/completions_test.cljs` +Test file: `src/test/logseq/cli/command/completion_test.cljs` Generator test: `src/test/logseq/cli/completion_generator_test.cljs` > **Require registration:** The nbb test runner auto-discovers `*_test.cljs` @@ -26,11 +26,11 @@ Generator test: `src/test/logseq/cli/completion_generator_test.cljs` Confirm the test runner loads both files (`yarn nbb-logseq -cp test -m nextjournal.test-runner`). -- [ ] **0.3** Create `src/test/logseq/cli/command/completions_test.cljs` with a - skeleton ns requiring `[logseq.cli.command.completions :as completions-command]`. +- [ ] **0.3** Create `src/test/logseq/cli/command/completion_test.cljs` with a + skeleton ns requiring `[logseq.cli.command.completion :as completion-command]`. Add a trivial failing test. -- [ ] **0.4** Create `src/main/logseq/cli/command/completions.cljs` with a stub +- [ ] **0.4** Create `src/main/logseq/cli/command/completion.cljs` with a stub ns and `entries` def (empty vector). Confirm test runner loads it. - [ ] **0.5** Remove placeholder failing tests; verify `yarn test` passes @@ -98,7 +98,7 @@ Pure functions tested in `completion_generator_test.cljs`. hierarchy: - Test: `["graph" "export"]` → group `"graph"`, subcommand `"export"` - Test: `["show"]` → leaf command `"show"` - - Test: `["completions"]` → leaf command `"completions"` + - Test: `["completion"]` → leaf command `"completion"` - Implement in `completion_generator.cljs`. - [ ] **2.2** `leaf-commands` / `group-commands` — classify entries: @@ -197,27 +197,27 @@ Pure functions tested in `completion_generator_test.cljs`. --- -## Phase 5 — `completions` command entry +## Phase 5 — `completion` command entry - [ ] **5.1** Command registration: - - Test: `completions-command/entries` contains one entry with - `:cmds ["completions"]` and `:command :completions` + - Test: `completion-command/entries` contains one entry with + `:cmds ["completion"]` and `:command :completions` - Test: spec has `{:shell {:values ["zsh" "bash"]}}` - - Implement `command/completions.cljs` with `entries` and spec. + - Implement `command/completion.cljs` with `entries` and spec. - [ ] **5.2** Wire into `commands.cljs`: - - Test: `(commands/parse-args ["completions" "--shell" "zsh"])` returns + - Test: `(commands/parse-args ["completion" "--shell" "zsh"])` returns `{:ok? true :command :completions}` - - Test: `(commands/parse-args ["completions" "zsh"])` handles positional arg - - Implement: add `completions-command/entries` to the table concat in + - Test: `(commands/parse-args ["completion" "zsh"])` handles positional arg + - Implement: add `completion-command/entries` to the table concat in `commands.cljs`. - [ ] **5.3** Build action and execute: - - Test: `build-action` for `:completions` returns an action with + - Test: `build-action` for `:completion` returns an action with `:type :completions` and `:shell "zsh"` - - Test: `execute` for `:completions` calls `generate-completions` and returns + - Test: `execute` for `:completion` calls `generate-completions` and returns the output string - - Implement in `command/completions.cljs` and wire into + - Implement in `command/completion.cljs` and wire into `commands.cljs` `build-action`/`execute`. --- @@ -251,14 +251,14 @@ Pure functions tested in `completion_generator_test.cljs`. ## Phase 7 — Replace hand-maintained files - [ ] **7.1** Generate fresh `_logseq.zsh` and `logseq.bash` from the - `completions` command; write them to + `completion` command; write them to `dz/scripts/shell-completions/`. - [ ] **7.2** Verify the generated files include the "do not edit manually" header comment. - [ ] **7.3** Update `dz/scripts/shell-completions/README.md` to document the - new `logseq completions` workflow instead of hand-editing. + new `logseq completion` workflow instead of hand-editing. --- @@ -270,7 +270,7 @@ All new test namespaces that must be discoverable by the test runner | Test file | Requires | |---|---| | `src/test/logseq/cli/completion_generator_test.cljs` | `logseq.cli.completion-generator` | -| `src/test/logseq/cli/command/completions_test.cljs` | `logseq.cli.command.completions` | +| `src/test/logseq/cli/command/completion_test.cljs` | `logseq.cli.command.completion` | The nbb test runner scans the `test` classpath for `*_test.cljs` files automatically. Verify with: diff --git a/dz/scripts/shell-completions/_logseq.zsh b/dz/scripts/shell-completions/_logseq.zsh index 9fa3040997..c3ff77d030 100644 --- a/dz/scripts/shell-completions/_logseq.zsh +++ b/dz/scripts/shell-completions/_logseq.zsh @@ -1,5 +1,5 @@ #compdef logseq -# Auto-generated by `logseq completions zsh` — do not edit manually. +# Auto-generated by `logseq completion zsh` — do not edit manually. # --- dynamic helpers --- @@ -397,7 +397,7 @@ _logseq_upsert_property() { '--timeout-ms=[Request timeout in ms (default 10000)]:value:' } -_logseq_completions() { +_logseq_completion() { _arguments -s \ '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ '--verbose[Enable verbose debug logging to stderr]' \ @@ -752,7 +752,7 @@ _logseq() { cmds) local -a cmds cmds=( - 'completions:Generate shell completion script' + 'completion:Generate shell completion script' 'doctor:Run runtime diagnostics' 'graph:graph commands' 'list:list commands' @@ -766,7 +766,7 @@ _logseq() { ;; args) case $line[1] in - completions) _logseq_completions ;; + completion) _logseq_completion ;; doctor) _logseq_doctor ;; graph) _logseq_graph ;; list) _logseq_list ;; @@ -780,4 +780,4 @@ _logseq() { esac } -_logseq "$@" +compdef _logseq logseq diff --git a/dz/scripts/shell-completions/logseq.bash b/dz/scripts/shell-completions/logseq.bash index 0cc5c08806..8fcb4cd974 100644 --- a/dz/scripts/shell-completions/logseq.bash +++ b/dz/scripts/shell-completions/logseq.bash @@ -1,4 +1,4 @@ -# Auto-generated by `logseq completions bash` — do not edit manually. +# Auto-generated by `logseq completion bash` — do not edit manually. # --- dynamic helpers --- @@ -82,7 +82,7 @@ _logseq_opts_for() { local opts="--help -h --version --config --graph --data-dir --timeout-ms --output --verbose" case "$cmd" in - completions) opts+=' --shell' ;; + completion) opts+=' --shell' ;; doctor) opts+=' --dev-script' ;; graph) case "$subcmd" in @@ -254,7 +254,7 @@ _logseq() { fi if [[ -z "$__cmd" ]]; then - COMPREPLY=( $(compgen -W 'completions doctor graph list query remove server show upsert' -- "$cur") ) + COMPREPLY=( $(compgen -W 'completion doctor graph list query remove server show upsert' -- "$cur") ) return fi diff --git a/src/main/logseq/cli/command/completions.cljs b/src/main/logseq/cli/command/completion.cljs similarity index 50% rename from src/main/logseq/cli/command/completions.cljs rename to src/main/logseq/cli/command/completion.cljs index a1e7ff3582..05e86d13d0 100644 --- a/src/main/logseq/cli/command/completions.cljs +++ b/src/main/logseq/cli/command/completion.cljs @@ -1,12 +1,12 @@ -(ns logseq.cli.command.completions - "Shell completions command." +(ns logseq.cli.command.completion + "Shell completion command." (:require [logseq.cli.command.core :as core])) -(def ^:private completions-spec +(def ^:private completion-spec {:shell {:desc "Shell (zsh, bash)" :values ["zsh" "bash"]}}) (def entries - [(core/command-entry ["completions"] :completions + [(core/command-entry ["completion"] :completion "Generate shell completion script" - completions-spec)]) + completion-spec)]) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 724b85c78a..1511483266 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -107,7 +107,7 @@ {:title "Authentication" :commands #{"login" "logout"}} {:title "Utilities" - :commands #{"completions"}}] + :commands #{"completion"}}] render-group (fn [{:keys [title commands]}] (let [entries (filter #(contains? commands (first (:cmds %))) table)] (string/join "\n" [title (format-commands entries)])))] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 9b3901b051..470b4b4843 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -3,7 +3,7 @@ (:require [babashka.cli :as cli] [clojure.string :as string] [logseq.cli.command.auth :as auth-command] - [logseq.cli.command.completions :as completions-command] + [logseq.cli.command.completion :as completion-command] [logseq.cli.command.core :as command-core] [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] @@ -108,7 +108,7 @@ doctor-command/entries sync-command/entries auth-command/entries - completions-command/entries))) + completion-command/entries))) ;; Global option parsing lives in logseq.cli.command.core. @@ -425,9 +425,9 @@ (:login :logout) (auth-command/build-action command) - :completions + :completion {:ok? true - :action {:type :completions + :action {:type :completion :shell (or (:shell options) (first args))}} {:ok? false @@ -471,7 +471,7 @@ :query-list (query-command/execute-query-list action config) :show (show-command/execute-show action config) :doctor (doctor-command/execute-doctor action config) - :completions (p/resolved + :completion (p/resolved {:status :ok :data {:message (completion-gen/generate-completions (:shell action) table)}}) diff --git a/src/main/logseq/cli/completion_generator.cljs b/src/main/logseq/cli/completion_generator.cljs index 475a43500d..f96b3afeaa 100644 --- a/src/main/logseq/cli/completion_generator.cljs +++ b/src/main/logseq/cli/completion_generator.cljs @@ -71,7 +71,7 @@ (def ^:private zsh-preamble "#compdef logseq -# Auto-generated by `logseq completions zsh` — do not edit manually. +# Auto-generated by `logseq completion zsh` — do not edit manually. # --- dynamic helpers --- @@ -372,14 +372,14 @@ _logseq_queries() { group-fns "\n" "# --- top-level dispatcher ---\n\n" toplevel "\n" - "_logseq \"$@\"\n"))) + "compdef _logseq logseq\n"))) ;; --------------------------------------------------------------------------- ;; Bash dynamic helpers (verbatim preamble) ;; --------------------------------------------------------------------------- (def ^:private bash-preamble - "# Auto-generated by `logseq completions bash` — do not edit manually. + "# Auto-generated by `logseq completion bash` — do not edit manually. # --- dynamic helpers --- diff --git a/src/test/logseq/cli/command/completion_test.cljs b/src/test/logseq/cli/command/completion_test.cljs new file mode 100644 index 0000000000..3a1f5ba336 --- /dev/null +++ b/src/test/logseq/cli/command/completion_test.cljs @@ -0,0 +1,40 @@ +(ns logseq.cli.command.completion-test + (:require [cljs.test :refer [deftest is testing]] + [logseq.cli.command.completion :as completion-command] + [logseq.cli.commands :as commands])) + +(deftest test-completion-command-registration + (testing "completion entry has correct structure" + (let [entry (first completion-command/entries)] + (is (= ["completion"] (:cmds entry))) + (is (= :completion (:command entry))) + (is (= ["zsh" "bash"] (get-in entry [:spec :shell :values])))))) + +(deftest test-parse-args-completion-shell + (testing "parse-args recognizes completion --shell zsh" + (let [result (commands/parse-args ["completion" "--shell" "zsh"])] + (is (true? (:ok? result))) + (is (= :completion (:command result))))) + (testing "parse-args recognizes completion with positional arg" + (let [result (commands/parse-args ["completion" "zsh"])] + (is (true? (:ok? result))) + (is (= :completion (:command result)))))) + +(deftest test-build-action-completion + (testing "build-action for :completion returns correct action" + (let [parsed {:ok? true + :command :completion + :options {:shell "zsh"} + :args []} + action (commands/build-action parsed {})] + (is (true? (:ok? action))) + (is (= :completion (get-in action [:action :type]))) + (is (= "zsh" (get-in action [:action :shell]))))) + (testing "build-action with positional arg" + (let [parsed {:ok? true + :command :completion + :options {} + :args ["bash"]} + action (commands/build-action parsed {})] + (is (true? (:ok? action))) + (is (= "bash" (get-in action [:action :shell])))))) diff --git a/src/test/logseq/cli/command/completions_test.cljs b/src/test/logseq/cli/command/completions_test.cljs deleted file mode 100644 index b2051107a1..0000000000 --- a/src/test/logseq/cli/command/completions_test.cljs +++ /dev/null @@ -1,40 +0,0 @@ -(ns logseq.cli.command.completions-test - (:require [cljs.test :refer [deftest is testing]] - [logseq.cli.command.completions :as completions-command] - [logseq.cli.commands :as commands])) - -(deftest test-completions-command-registration - (testing "completions entry has correct structure" - (let [entry (first completions-command/entries)] - (is (= ["completions"] (:cmds entry))) - (is (= :completions (:command entry))) - (is (= ["zsh" "bash"] (get-in entry [:spec :shell :values])))))) - -(deftest test-parse-args-completions-shell - (testing "parse-args recognizes completions --shell zsh" - (let [result (commands/parse-args ["completions" "--shell" "zsh"])] - (is (true? (:ok? result))) - (is (= :completions (:command result))))) - (testing "parse-args recognizes completions with positional arg" - (let [result (commands/parse-args ["completions" "zsh"])] - (is (true? (:ok? result))) - (is (= :completions (:command result)))))) - -(deftest test-build-action-completions - (testing "build-action for :completions returns correct action" - (let [parsed {:ok? true - :command :completions - :options {:shell "zsh"} - :args []} - action (commands/build-action parsed {})] - (is (true? (:ok? action))) - (is (= :completions (get-in action [:action :type]))) - (is (= "zsh" (get-in action [:action :shell]))))) - (testing "build-action with positional arg" - (let [parsed {:ok? true - :command :completions - :options {} - :args ["bash"]} - action (commands/build-action parsed {})] - (is (true? (:ok? action))) - (is (= "bash" (get-in action [:action :shell])))))) diff --git a/src/test/logseq/cli/completion_generator_test.cljs b/src/test/logseq/cli/completion_generator_test.cljs index 08688cca4d..207dd40f55 100644 --- a/src/test/logseq/cli/completion_generator_test.cljs +++ b/src/test/logseq/cli/completion_generator_test.cljs @@ -1,7 +1,7 @@ (ns logseq.cli.completion-generator-test (:require [cljs.test :refer [deftest is testing]] [clojure.string :as string] - [logseq.cli.command.completions :as completions-command] + [logseq.cli.command.completion :as completion-command] [logseq.cli.command.core :as core] [logseq.cli.command.doctor :as doctor-command] [logseq.cli.command.graph :as graph-command] @@ -22,7 +22,7 @@ query-command/entries show-command/entries doctor-command/entries - completions-command/entries))) + completion-command/entries))) ;; --------------------------------------------------------------------------- ;; Phase 1 — Spec enrichment tests @@ -130,10 +130,10 @@ (let [show-entries (get groups "show")] (is (= 1 (count show-entries))) (is (= ["show"] (:cmds (first show-entries)))))) - (testing "completions is a standalone group" - (let [completions-entries (get groups "completions")] - (is (= 1 (count completions-entries))) - (is (= ["completions"] (:cmds (first completions-entries)))))))) + (testing "completion is a standalone group" + (let [completion-entries (get groups "completion")] + (is (= 1 (count completion-entries))) + (is (= ["completion"] (:cmds (first completion-entries)))))))) (deftest test-leaf-and-group-commands (let [leaves (gen/leaf-commands full-table) @@ -198,8 +198,8 @@ (is (string/includes? output "_logseq_upsert()"))) (testing "output contains top-level dispatcher" (is (string/includes? output "_logseq()"))) - (testing "output ends with _logseq \"$@\"" - (is (string/includes? output "_logseq \"$@\""))) + (testing "output ends with compdef _logseq logseq" + (is (string/includes? output "compdef _logseq logseq"))) (testing "boolean flags emit bare flag form" (is (re-find #"--verbose\[" output))) (testing "enum options emit value list form" @@ -267,16 +267,16 @@ (str "missing command: " group-name)))))) ;; --------------------------------------------------------------------------- -;; Phase 5 — Completions command entry +;; Phase 5 — Completion command entry ;; --------------------------------------------------------------------------- -(deftest test-completions-command-entry - (let [entries completions-command/entries] - (testing "contains one entry with :cmds [\"completions\"]" +(deftest test-completion-command-entry + (let [entries completion-command/entries] + (testing "contains one entry with :cmds [\"completion\"]" (is (= 1 (count entries))) - (is (= ["completions"] (:cmds (first entries))))) - (testing "command is :completions" - (is (= :completions (:command (first entries))))) + (is (= ["completion"] (:cmds (first entries))))) + (testing "command is :completion" + (is (= :completion (:command (first entries))))) (testing "spec has :shell with :values" (is (= ["zsh" "bash"] (get-in (first entries) [:spec :shell :values])))))) @@ -291,7 +291,7 @@ (is (string/includes? output "#compdef")) (is (string/includes? output "_logseq_graph_export")) (is (string/includes? output "_logseq_show")) - (is (string/includes? output "_logseq \"$@\""))))) + (is (string/includes? output "compdef _logseq logseq"))))) (deftest test-e2e-bash-structural-markers (let [output (gen/generate-completions "bash" full-table)] From 018281da389561a2744fc40291ade5043d25d02a Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 10 Mar 2026 11:21:00 -0400 Subject: [PATCH 135/375] enhance: remove graph command behaves like electron remove Moves removed graph to 'Unlinked graphs' --- src/electron/electron/db.cljs | 3 +-- src/main/logseq/cli/common.cljs | 10 +++++----- src/test/logseq/cli/common_test.cljs | 9 ++++----- 3 files changed, 10 insertions(+), 12 deletions(-) diff --git a/src/electron/electron/db.cljs b/src/electron/electron/db.cljs index 5d4891e861..e457ccbbe4 100644 --- a/src/electron/electron/db.cljs +++ b/src/electron/electron/db.cljs @@ -4,7 +4,6 @@ ["path" :as node-path] [electron.backup-file :as backup-file] [logseq.cli.common.graph :as cli-common-graph] - [logseq.common.graph-dir :as graph-dir] [logseq.db.common.sqlite :as common-sqlite])) (defn ensure-graphs-dir! @@ -41,4 +40,4 @@ {:backups-dir backups-path :truncate-daily? true :keep-versions 12})) - (fs/writeFileSync db-path data))) \ No newline at end of file + (fs/writeFileSync db-path data))) diff --git a/src/main/logseq/cli/common.cljs b/src/main/logseq/cli/common.cljs index a7cd739ad8..a1809fc337 100644 --- a/src/main/logseq/cli/common.cljs +++ b/src/main/logseq/cli/common.cljs @@ -4,20 +4,20 @@ ["path" :as node-path] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] - [logseq.common.graph-dir :as graph-dir])) + [logseq.db.common.sqlite :as common-sqlite])) (defn unlink-graph! "Unlinks the given repo by moving it to the 'Unlinked graphs' dir. Returns path of unlinked dir if move is successful or nil if not" [repo] - (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name repo) + (let [db-name (common-sqlite/sanitize-db-name repo) graphs-dir (common-graph/expand-home (common-graph/get-default-graphs-dir)) - path (node-path/join graphs-dir graph-dir-name) + path (node-path/join graphs-dir db-name) unlinked (node-path/join graphs-dir common-config/unlinked-graphs-dir) - new-path (node-path/join unlinked graph-dir-name) + new-path (node-path/join unlinked db-name) new-path-exists? (fs/existsSync new-path) new-path' (if new-path-exists? - (node-path/join unlinked (str graph-dir-name "-" (random-uuid))) + (node-path/join unlinked (str db-name "-" (random-uuid))) new-path)] (when (fs/existsSync path) (fs/ensureDirSync unlinked) diff --git a/src/test/logseq/cli/common_test.cljs b/src/test/logseq/cli/common_test.cljs index e46b143cc0..39f7db4ce1 100644 --- a/src/test/logseq/cli/common_test.cljs +++ b/src/test/logseq/cli/common_test.cljs @@ -9,11 +9,10 @@ (deftest unlink-graph-moves-to-unlinked-dir (let [graphs-dir (node-helper/create-tmp-dir "unlink-graph") - graph-name "foo/bar" + graph-name "test-graph" repo (str common-config/db-version-prefix graph-name) - encoded-graph-dir "foo~2Fbar" - graph-path (node-path/join graphs-dir encoded-graph-dir) - unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir encoded-graph-dir)] + graph-path (node-path/join graphs-dir graph-name) + unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir graph-name)] (fs/mkdirSync graph-path #js {:recursive true}) (fs/writeFileSync (node-path/join graph-path "db.sqlite") "test-data") (with-redefs [common-graph/get-default-graphs-dir (fn [] graphs-dir)] @@ -23,4 +22,4 @@ (is (fs/existsSync unlinked-path) "Graph directory should be moved to Unlinked graphs") (is (fs/existsSync (node-path/join unlinked-path "db.sqlite")) - "Graph contents should be preserved after move")))) \ No newline at end of file + "Graph contents should be preserved after move")))) From faecbd4ea80a5f02595e25d56ef15d3e348aee86 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Wed, 11 Mar 2026 19:32:55 +0800 Subject: [PATCH 136/375] 056-graph-name-dir-encoding-alignment.md --- src/electron/electron/db.cljs | 1 + src/main/logseq/cli/common.cljs | 10 +++++----- src/test/logseq/cli/common_test.cljs | 7 ++++--- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/src/electron/electron/db.cljs b/src/electron/electron/db.cljs index e457ccbbe4..03fbe76139 100644 --- a/src/electron/electron/db.cljs +++ b/src/electron/electron/db.cljs @@ -4,6 +4,7 @@ ["path" :as node-path] [electron.backup-file :as backup-file] [logseq.cli.common.graph :as cli-common-graph] + [logseq.common.graph-dir :as graph-dir] [logseq.db.common.sqlite :as common-sqlite])) (defn ensure-graphs-dir! diff --git a/src/main/logseq/cli/common.cljs b/src/main/logseq/cli/common.cljs index a1809fc337..a7cd739ad8 100644 --- a/src/main/logseq/cli/common.cljs +++ b/src/main/logseq/cli/common.cljs @@ -4,20 +4,20 @@ ["path" :as node-path] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] - [logseq.db.common.sqlite :as common-sqlite])) + [logseq.common.graph-dir :as graph-dir])) (defn unlink-graph! "Unlinks the given repo by moving it to the 'Unlinked graphs' dir. Returns path of unlinked dir if move is successful or nil if not" [repo] - (let [db-name (common-sqlite/sanitize-db-name repo) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name repo) graphs-dir (common-graph/expand-home (common-graph/get-default-graphs-dir)) - path (node-path/join graphs-dir db-name) + path (node-path/join graphs-dir graph-dir-name) unlinked (node-path/join graphs-dir common-config/unlinked-graphs-dir) - new-path (node-path/join unlinked db-name) + new-path (node-path/join unlinked graph-dir-name) new-path-exists? (fs/existsSync new-path) new-path' (if new-path-exists? - (node-path/join unlinked (str db-name "-" (random-uuid))) + (node-path/join unlinked (str graph-dir-name "-" (random-uuid))) new-path)] (when (fs/existsSync path) (fs/ensureDirSync unlinked) diff --git a/src/test/logseq/cli/common_test.cljs b/src/test/logseq/cli/common_test.cljs index 39f7db4ce1..19adac802e 100644 --- a/src/test/logseq/cli/common_test.cljs +++ b/src/test/logseq/cli/common_test.cljs @@ -9,10 +9,11 @@ (deftest unlink-graph-moves-to-unlinked-dir (let [graphs-dir (node-helper/create-tmp-dir "unlink-graph") - graph-name "test-graph" + graph-name "foo/bar" repo (str common-config/db-version-prefix graph-name) - graph-path (node-path/join graphs-dir graph-name) - unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir graph-name)] + encoded-graph-dir "foo~2Fbar" + graph-path (node-path/join graphs-dir encoded-graph-dir) + unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir encoded-graph-dir)] (fs/mkdirSync graph-path #js {:recursive true}) (fs/writeFileSync (node-path/join graph-path "db.sqlite") "test-data") (with-redefs [common-graph/get-default-graphs-dir (fn [] graphs-dir)] From c4d52878bc9f020c6e81183193374fc9859e53d0 Mon Sep 17 00:00:00 2001 From: Danzu Date: Wed, 11 Mar 2026 18:55:08 -0500 Subject: [PATCH 137/375] feat: enhance completion command with long description and setup instructions --- src/main/logseq/cli/command/completion.cljs | 21 ++++++++- src/main/logseq/cli/command/core.cljs | 47 ++++++++++--------- src/main/logseq/cli/commands.cljs | 4 +- .../logseq/cli/command/completion_test.cljs | 16 ++++++- 4 files changed, 62 insertions(+), 26 deletions(-) diff --git a/src/main/logseq/cli/command/completion.cljs b/src/main/logseq/cli/command/completion.cljs index 05e86d13d0..2c32e77d41 100644 --- a/src/main/logseq/cli/command/completion.cljs +++ b/src/main/logseq/cli/command/completion.cljs @@ -1,12 +1,29 @@ (ns logseq.cli.command.completion "Shell completion command." - (:require [logseq.cli.command.core :as core])) + (:require [clojure.string :as string] + [logseq.cli.command.core :as core])) (def ^:private completion-spec {:shell {:desc "Shell (zsh, bash)" :values ["zsh" "bash"]}}) +(def ^:private long-desc + (string/join "\n" + ["Generate shell completion script for the specified shell." + "Outputs a completion script to stdout that can be evaluated" + "by your shell to enable tab-completion for logseq commands." + "" + "Setup for zsh:" + " # Add to ~/.zshrc" + " autoload -Uz compinit && compinit" + " eval \"$(logseq completion zsh)\"" + "" + "Setup for bash:" + " # Add to ~/.bashrc" + " eval \"$(logseq completion bash)\""])) + (def entries [(core/command-entry ["completion"] :completion "Generate shell completion script" - completion-spec)]) + completion-spec + {:long-desc long-desc})]) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 1511483266..932792c024 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -37,19 +37,23 @@ (merge global-spec* (or spec {}))) (defn command-entry - [cmds command desc spec] - (let [spec* (merge-spec spec)] - {:cmds cmds - :command command - :desc desc - :spec spec* - :restrict true - :fn (fn [{:keys [opts args]}] - {:command command - :cmds cmds - :spec spec* - :opts opts - :args args})})) + ([cmds command desc spec] + (command-entry cmds command desc spec nil)) + ([cmds command desc spec {:keys [long-desc]}] + (let [spec* (merge-spec spec)] + {:cmds cmds + :command command + :desc desc + :long-desc long-desc + :spec spec* + :restrict true + :fn (fn [{:keys [opts args]}] + {:command command + :cmds cmds + :spec spec* + :long-desc long-desc + :opts opts + :args args})}))) (defn- command-usage [cmds spec] @@ -124,16 +128,17 @@ " See `logseq --help`"]))) (defn command-summary - [{:keys [cmds spec]}] + [{:keys [cmds spec long-desc]}] (let [command-spec (apply dissoc spec (keys global-spec*))] (string/join "\n" - [(str "Usage: logseq " (command-usage cmds spec)) - "" - (str "Global " (style/bold "options") ":") - (format-opts global-spec*) - "" - (str "Command " (style/bold "options") ":") - (format-opts command-spec)]))) + (cond-> [(str "Usage: logseq " (command-usage cmds spec))] + (seq long-desc) (conj "" long-desc) + true (conj "" + (str "Global " (style/bold "options") ":") + (format-opts global-spec*) + "" + (str "Command " (style/bold "options") ":") + (format-opts command-spec)))))) (defn normalize-opts [opts] diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 470b4b4843..08b2d7e77a 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -163,10 +163,10 @@ nil))) (defn- ^:large-vars/cleanup-todo finalize-command - [summary {:keys [command opts args cmds spec]}] + [summary {:keys [command opts args cmds spec long-desc]}] (let [opts (command-core/normalize-opts opts) args (vec args) - cmd-summary (command-core/command-summary {:cmds cmds :spec spec}) + cmd-summary (command-core/command-summary {:cmds cmds :spec spec :long-desc long-desc}) graph (:graph opts) has-args? (seq args) has-content? (or (seq (:content opts)) diff --git a/src/test/logseq/cli/command/completion_test.cljs b/src/test/logseq/cli/command/completion_test.cljs index 3a1f5ba336..30a38d764f 100644 --- a/src/test/logseq/cli/command/completion_test.cljs +++ b/src/test/logseq/cli/command/completion_test.cljs @@ -1,5 +1,6 @@ (ns logseq.cli.command.completion-test (:require [cljs.test :refer [deftest is testing]] + [clojure.string :as string] [logseq.cli.command.completion :as completion-command] [logseq.cli.commands :as commands])) @@ -8,7 +9,12 @@ (let [entry (first completion-command/entries)] (is (= ["completion"] (:cmds entry))) (is (= :completion (:command entry))) - (is (= ["zsh" "bash"] (get-in entry [:spec :shell :values])))))) + (is (= ["zsh" "bash"] (get-in entry [:spec :shell :values]))))) + (testing "completion entry has long-desc with setup instructions" + (let [entry (first completion-command/entries)] + (is (some? (:long-desc entry))) + (is (string/includes? (:long-desc entry) "autoload -Uz compinit")) + (is (string/includes? (:long-desc entry) "eval \"$(logseq completion zsh)\""))))) (deftest test-parse-args-completion-shell (testing "parse-args recognizes completion --shell zsh" @@ -20,6 +26,14 @@ (is (true? (:ok? result))) (is (= :completion (:command result)))))) +(deftest test-parse-args-completion-help + (testing "parse-args completion --help returns help with setup instructions" + (let [result (commands/parse-args ["completion" "--help"])] + (is (false? (:ok? result))) + (is (true? (:help? result))) + (is (string/includes? (:summary result) "autoload -Uz compinit")) + (is (string/includes? (:summary result) "eval \"$(logseq completion bash)\""))))) + (deftest test-build-action-completion (testing "build-action for :completion returns correct action" (let [parsed {:ok? true From 1b19d1576ff985e6f7929c100663afc24866e84d Mon Sep 17 00:00:00 2001 From: Danzu Date: Wed, 11 Mar 2026 22:01:25 -0500 Subject: [PATCH 138/375] Generalize bash context-dependent option completion and fix zsh duplicate functions Replace hardcoded --name/--sort context branches with auto-detected varied option keys (find-varied-option-keys), so --type and any future options with differing completions across commands are handled automatically. Fix zsh generator to skip group root entries from leaf function emission, avoiding duplicate function definitions. Co-Authored-By: Claude Opus 4.6 --- src/main/logseq/cli/completion_generator.cljs | 181 +++++++++++------- .../logseq/cli/completion_generator_test.cljs | 31 +++ 2 files changed, 144 insertions(+), 68 deletions(-) diff --git a/src/main/logseq/cli/completion_generator.cljs b/src/main/logseq/cli/completion_generator.cljs index f96b3afeaa..8c43a04afb 100644 --- a/src/main/logseq/cli/completion_generator.cljs +++ b/src/main/logseq/cli/completion_generator.cljs @@ -346,14 +346,19 @@ _logseq_queries() { global-spec (-> table first :spec (select-keys [:help :version :config :graph :data-dir :timeout-ms :output :verbose])) - ;; Generate leaf command functions + ;; Collect group names that have subcommands (dispatchers own these) + group-set (set (group-commands table)) + ;; Generate leaf command functions (skip group roots — dispatchers own those) leaf-fns (->> groups - (mapcat (fn [[_ entries]] - (mapv (fn [entry] - (zsh-leaf-function - (cmd->func-name (:cmds entry)) - (:spec entry))) - entries))) + (mapcat (fn [[group-name entries]] + (->> entries + (remove (fn [entry] + (and (= 1 (count (:cmds entry))) + (contains? group-set group-name)))) + (mapv (fn [entry] + (zsh-leaf-function + (cmd->func-name (:cmds entry)) + (:spec entry))))))) (string/join "\n")) ;; Generate group dispatchers group-fns (->> groups @@ -641,73 +646,119 @@ _logseq_compadd_lines() { (let [groups (extract-groups table)] (->> (keys groups) sort (string/join " ")))) -(defn- bash-context-dependent-prev-cases - "Generate context-dependent prev-word cases (e.g., --name means different things - in different commands, --sort has different values per list subcommand)." - [table] - (let [groups (extract-groups table) - ;; Find all --name contexts - name-cases +(defn- option-completion-sig + "Extract completion-relevant signature from a spec-map entry." + [{:keys [coerce values complete]}] + {:flag? (= coerce :boolean) + :values values + :complete complete}) + +(defn find-varied-option-keys + "Find option keys that have different completion behaviors across commands. + Returns a set of keyword keys." + [table global-keys] + (->> table + (mapcat (fn [entry] + (for [[k spec-map] (apply dissoc (:spec entry) global-keys)] + [k (option-completion-sig spec-map)]))) + (group-by first) + (keep (fn [[k entries]] + (let [sigs (map second entries)] + (when (not (apply = sigs)) + k)))) + set)) + +(defn- bash-varied-context-branch + "Generate an if-branch for a single command context of a varied option." + [{:keys [cmds spec-map]}] + (let [token (spec->token [:_ spec-map]) + cmd (first cmds) + subcmd (second cmds) + condition (if subcmd + (str "[[ \"$__cmd\" == '" cmd "' && \"$__subcmd\" == '" subcmd "' ]]") + (str "[[ \"$__cmd\" == '" cmd "' ]]"))] + (case (:type token) + :enum + (str " if " condition "; then\n" + " COMPREPLY=( $(compgen -W '" (string/join " " (:values token)) "' -- \"$cur\") )\n" + " return\n" + " fi") + + :dynamic + (case (:complete token) + :graphs + (str " if " condition "; then\n" + " _logseq_compadd_lines \"$cur\" _logseq_graphs_bash\n" + " fi") + :pages + (str " if " condition "; then\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_pages_bash \"$graph\"\n" + " fi") + :queries + (str " if " condition "; then\n" + " local graph\n" + " graph=\"$(_logseq_current_graph_bash)\"\n" + " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_queries_bash \"$graph\"\n" + " fi") + nil) + + :file + (str " if " condition "; then\n" + " COMPREPLY=( $(compgen -f -- \"$cur\") )\n" + " return\n" + " fi") + + :dir + (str " if " condition "; then\n" + " COMPREPLY=( $(compgen -d -- \"$cur\") )\n" + " return\n" + " fi") + + ;; :free and :flag don't need prev-word completion + nil))) + +(defn- bash-varied-prev-cases + "Generate context-dependent case branches for all varied options." + [table varied-keys global-keys] + (let [contexts (->> table - (keep (fn [entry] - (let [name-spec (get-in entry [:spec :name]) - complete (:complete name-spec)] - (when complete - {:cmds (:cmds entry) :complete complete})))) - (mapv (fn [{:keys [cmds complete]}] - (let [cmd (first cmds) - subcmd (second cmds)] - (case complete - :queries - (str " if [[ \"$__cmd\" == '" cmd "' ]]; then\n" - " local graph\n" - " graph=\"$(_logseq_current_graph_bash)\"\n" - " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_queries_bash \"$graph\"\n" - " fi") - :pages - (str " if [[ \"$__cmd\" == '" cmd "' && \"$__subcmd\" == '" subcmd "' ]]; then\n" - " local graph\n" - " graph=\"$(_logseq_current_graph_bash)\"\n" - " [[ -n \"$graph\" ]] && _logseq_compadd_lines \"$cur\" _logseq_pages_bash \"$graph\"\n" - " fi") - nil))))) - ;; Find all --sort contexts - sort-cases - (->> table - (keep (fn [entry] - (let [sort-spec (get-in entry [:spec :sort]) - values (:values sort-spec)] - (when (seq values) - {:cmds (:cmds entry) :values values})))) - (mapv (fn [{:keys [cmds values]}] - (let [cmd (first cmds) - subcmd (second cmds) - vals-str (string/join " " values)] - (str " if [[ \"$__cmd\" == '" cmd "' && \"$__subcmd\" == '" subcmd "' ]]; then\n" - " COMPREPLY=( $(compgen -W '" vals-str "' -- \"$cur\") )\n" - " return\n" - " fi")))))] - {:name-cases name-cases - :sort-cases sort-cases})) + (mapcat (fn [entry] + (for [[k spec-map] (apply dissoc (:spec entry) global-keys) + :when (contains? varied-keys k)] + {:key k :cmds (:cmds entry) :spec-map spec-map}))) + (group-by :key))] + (->> (sort-by first contexts) + (keep (fn [[k entries]] + (let [long-opt (bash-option-name k) + branches (->> entries + (keep bash-varied-context-branch))] + (when (seq branches) + (str " " long-opt ")\n" + (string/join "\n" branches) "\n" + " return ;;"))))) + (string/join "\n\n")))) (defn- bash-main-function "Generate the _logseq() main completion function." [table] - (let [groups (extract-groups table) - global-spec (-> table first :spec + (let [global-spec (-> table first :spec (select-keys [:help :version :config :graph :data-dir :timeout-ms :output :verbose])) + global-keys (set (keys global-spec)) + ;; Find options with conflicting completions across commands + varied-keys (find-varied-option-keys table global-keys) ;; Collect unique non-context-dependent prev-word cases - ;; (skip --name and --sort since they're context-dependent) all-specs (->> table (mapcat (fn [entry] (seq (:spec entry))))) unique-specs (into {} all-specs) tokens (spec->tokens unique-specs) - context-free-tokens (remove #(#{:name :sort} (:key %)) tokens) + context-free-tokens (remove #(contains? varied-keys (:key %)) tokens) prev-cases (->> context-free-tokens (keep bash-prev-completion-case) (string/join "\n\n")) - ;; Context-dependent cases - {:keys [name-cases sort-cases]} (bash-context-dependent-prev-cases table) + ;; Context-dependent cases for varied options + varied-cases (bash-varied-prev-cases table varied-keys global-keys) ;; Subcommand completion subcmd-cases (bash-subcommand-cases table) ;; Top-level commands @@ -724,14 +775,8 @@ _logseq_compadd_lines() { " # --- Option value completion ---\n" " case \"$prev\" in\n" prev-cases "\n" - "\n" - " --name)\n" - (string/join "\n" name-cases) "\n" - " return ;;\n" - "\n" - " --sort)\n" - (string/join "\n" sort-cases) "\n" - " return ;;\n" + (when (seq varied-cases) + (str "\n" varied-cases "\n")) " esac\n" "\n" " # --- Flag / positional completion ---\n" diff --git a/src/test/logseq/cli/completion_generator_test.cljs b/src/test/logseq/cli/completion_generator_test.cljs index 207dd40f55..54d3b756d4 100644 --- a/src/test/logseq/cli/completion_generator_test.cljs +++ b/src/test/logseq/cli/completion_generator_test.cljs @@ -324,6 +324,37 @@ (let [tag (first (filter #(= :remove-tag (:command %)) entries))] (is (nil? (get-in tag [:spec :name :complete]))))))) +(deftest test-bash-context-dependent-type + (let [output (gen/generate-completions "bash" full-table)] + (testing "--type completes with edn/sqlite under graph export context" + (is (string/includes? output "--type)")) + (is (string/includes? output "graph' && \"$__subcmd\" == 'export'")) + (is (string/includes? output "compgen -W 'edn sqlite'"))) + (testing "--type completes with property types under upsert property context" + (is (string/includes? output "upsert' && \"$__subcmd\" == 'property'")) + (is (string/includes? output "compgen -W 'default number"))) + (testing "--type does NOT have a context-free case (simple COMPREPLY after --type)" + ;; --type should be in context-dependent if-blocks, not a simple case + (let [type-section (second (string/split output #"--type\)"))] + (is (string/starts-with? (string/trim type-section) "if")))))) + +(deftest test-bash-find-varied-option-keys + (let [global-spec (-> full-table first :spec + (select-keys [:help :version :config :graph :data-dir + :timeout-ms :output :verbose])) + global-keys (set (keys global-spec)) + varied (gen/find-varied-option-keys full-table global-keys)] + (testing "--type is detected as varied" + (is (contains? varied :type))) + (testing "--name is detected as varied" + (is (contains? varied :name))) + (testing "--sort is detected as varied" + (is (contains? varied :sort))) + (testing "uniform options like --pos are not varied" + (is (not (contains? varied :pos)))) + (testing "uniform options like --cardinality are not varied" + (is (not (contains? varied :cardinality)))))) + (deftest test-e2e-generated-header (testing "zsh output includes do-not-edit header" (let [output (gen/generate-completions "zsh" full-table)] From 562eb0140ec518b78a3808842b52ade40745768c Mon Sep 17 00:00:00 2001 From: Danzu Date: Wed, 11 Mar 2026 22:01:31 -0500 Subject: [PATCH 139/375] Add shell argument validation to completion command Reject missing or unsupported shell arguments (e.g. "fish") at parse time with a clear error message, rather than letting it fail deep in the generator. Co-Authored-By: Claude Opus 4.6 --- src/main/logseq/cli/commands.cljs | 10 ++++++++++ src/test/logseq/cli/command/completion_test.cljs | 10 ++++++++++ 2 files changed, 20 insertions(+) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 08b2d7e77a..8282442b5b 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -276,6 +276,16 @@ (not (seq (:graph opts)))) (missing-graph-result summary) + (and (= command :completion) + (let [shell (or (:shell opts) (first args))] + (not (#{"zsh" "bash"} shell)))) + (command-core/invalid-options-result + summary + (let [shell (or (:shell opts) (first args))] + (if (seq shell) + (str "unsupported shell: " shell "; expected zsh or bash") + "missing shell argument; usage: logseq completion "))) + :else (command-core/ok-result command opts args summary)))) diff --git a/src/test/logseq/cli/command/completion_test.cljs b/src/test/logseq/cli/command/completion_test.cljs index 30a38d764f..398cfdb57e 100644 --- a/src/test/logseq/cli/command/completion_test.cljs +++ b/src/test/logseq/cli/command/completion_test.cljs @@ -52,3 +52,13 @@ action (commands/build-action parsed {})] (is (true? (:ok? action))) (is (= "bash" (get-in action [:action :shell])))))) + +(deftest test-parse-args-completion-validation + (testing "completion with no shell arg returns invalid-options error" + (let [result (commands/parse-args ["completion"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + (testing "completion with unsupported shell returns invalid-options error" + (let [result (commands/parse-args ["completion" "fish"])] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code])))))) From 35b43611c6d5191ebe4ae41c99de5897959c934b Mon Sep 17 00:00:00 2001 From: Danzu Date: Wed, 11 Mar 2026 22:07:22 -0500 Subject: [PATCH 140/375] Add ncc build prerequisite and shell completion setup to CLI docs Co-Authored-By: Claude Opus 4.6 --- docs/cli/logseq-cli.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 815ef609c3..8a6deb411d 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -5,6 +5,7 @@ The Logseq CLI is a Node.js program compiled from ClojureScript that connects to ## Build the CLI ```bash +npm install -g @vercel/ncc clojure -M:cljs compile logseq-cli yarn db-worker-node:release:bundle ``` @@ -109,6 +110,20 @@ Auth commands: - `login` - authenticate this machine and create/update `~/logseq/auth.json` - `logout` - remove persisted CLI auth from `~/logseq/auth.json` +Shell completion: +- `completion ` - generate shell completion script to stdout + +Setup for zsh (add to `~/.zshrc`): +```bash +autoload -Uz compinit && compinit +eval "$(logseq completion zsh)" +``` + +Setup for bash (add to `~/.bashrc`): +```bash +eval "$(logseq completion bash)" +``` + Server ownership behavior: - `server stop` and `server restart` can return `server-owned-by-other` if the daemon was started by another owner source. - `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. From a115779028b124a1425582ef42c00700caf6ef70 Mon Sep 17 00:00:00 2001 From: Danzu Date: Wed, 11 Mar 2026 22:08:26 -0500 Subject: [PATCH 141/375] Remove accidentally committed dz/ folder Co-Authored-By: Claude Opus 4.6 --- dz/scripts/shell-completions/DESIGN.md | 621 ------------------ dz/scripts/shell-completions/README.md | 199 ------ dz/scripts/shell-completions/TASKS.md | 283 -------- dz/scripts/shell-completions/_logseq.zsh | 783 ----------------------- dz/scripts/shell-completions/logseq.bash | 274 -------- 5 files changed, 2160 deletions(-) delete mode 100644 dz/scripts/shell-completions/DESIGN.md delete mode 100644 dz/scripts/shell-completions/README.md delete mode 100644 dz/scripts/shell-completions/TASKS.md delete mode 100644 dz/scripts/shell-completions/_logseq.zsh delete mode 100644 dz/scripts/shell-completions/logseq.bash diff --git a/dz/scripts/shell-completions/DESIGN.md b/dz/scripts/shell-completions/DESIGN.md deleted file mode 100644 index 7a0e63126d..0000000000 --- a/dz/scripts/shell-completions/DESIGN.md +++ /dev/null @@ -1,621 +0,0 @@ -# Shell Completions — Full Requirement (Option 2) - -Generate shell completions from the CLI command table so that adding or -modifying a command/flag in the spec is the **only** change needed — the -completion scripts stay in sync automatically. - ---- - -## 1. Overview - -The Logseq CLI (`src/main/logseq/cli`) uses `babashka.cli` for argument -parsing. Commands and their options are declared as data in a `table` of -`{:cmds [...] :spec {...}}` entries assembled in `commands.cljs`. - -A new `logseq completion ` command will walk the `table` at -generation time and emit correct zsh and bash completion scripts. - -``` -logseq completion zsh > ~/.zsh/completions/_logseq -logseq completion bash > ~/.local/share/bash-completion/completions/logseq -``` - ---- - -## 2. Complete command inventory - -The table below is the single source of truth. Every row **must** appear in -the generated dispatch structure. Commands with no command-specific options -still inherit the global spec. - -### 2.1 Command groups and subcommands - -| Group | Subcommands | Command key | -| ------------- | ---------------------------------------------------------------------------- | ------------------------------------ | -| `graph` | `list`, `create`, `switch`, `remove`, `validate`, `info`, `export`, `import` | `:graph-list` … `:graph-import` | -| `server` | `list`, `status`, `start`, `stop`, `restart` | `:server-list` … `:server-restart` | -| `list` | `page`, `tag`, `property` | `:list-page` … `:list-property` | -| `upsert` | `block`, `page`, `tag`, `property` | `:upsert-block` … `:upsert-property` | -| `remove` | `block`, `page`, `tag`, `property` | `:remove-block` … `:remove-property` | -| `query` | _(root)_, `list` | `:query`, `:query-list` | -| `show` | _(leaf — no subcommands)_ | `:show` | -| `doctor` | _(leaf — no subcommands)_ | `:doctor` | -| `completion` | _(leaf — new)_ | `:completion` | - -### 2.2 Global spec (`command/core.cljs`) - -These options are available on **every** command. The merged spec -(`merge global-spec* command-spec`) is already what each table entry carries. - -| Option | Alias | `:coerce` | `:values` | `:complete` | Notes | -| -------------- | ----- | ---------- | ------------------------ | ----------- | -------------------- | -| `--help` | `-h` | `:boolean` | — | — | | -| `--version` | — | `:boolean` | — | — | | -| `--config` | — | — | — | `:file` | Path to `cli.edn` | -| `--graph` | — | — | — | `:graphs` | Graph name (dynamic) | -| `--data-dir` | — | — | — | `:dir` | Path to data dir | -| `--timeout-ms` | — | `:long` | — | — | | -| `--output` | — | — | `["human" "json" "edn"]` | — | Output format | -| `--verbose` | — | `:boolean` | — | — | | - -### 2.3 Command-specific specs - -#### `graph export` - -| Option | `:values` | `:complete` | -| -------- | ------------------ | ----------- | -| `--type` | `["edn" "sqlite"]` | — | -| `--file` | — | `:file` | - -#### `graph import` - -| Option | `:values` | `:complete` | -| --------- | ------------------ | ----------- | -| `--type` | `["edn" "sqlite"]` | — | -| `--input` | — | `:file` | - -#### `graph list/create/switch/remove/validate/info` - -No command-specific options (global-only). - -#### `server status/start/stop/restart` - -| Option | `:complete` | Notes | -| --------- | ----------- | -------------------------------------------------- | -| `--graph` | `:graphs` | Redundant with global — shadows it (same behavior) | - -#### `server list` - -No command-specific options. - -#### `list page` - -| Option | `:coerce` | `:values` | -| ------------------- | ---------- | ------------------------------------- | -| `--expand` | `:boolean` | — | -| `--limit` | `:long` | — | -| `--offset` | `:long` | — | -| `--sort` | — | `["title" "created-at" "updated-at"]` | -| `--order` | — | `["asc" "desc"]` | -| `--include-journal` | `:boolean` | — | -| `--journal-only` | `:boolean` | — | -| `--include-hidden` | `:boolean` | — | -| `--updated-after` | — | — | -| `--created-after` | — | — | -| `--fields` | — | — | - -#### `list tag` - -| Option | `:coerce` | `:values` | -| -------------------- | ---------- | ------------------ | -| `--expand` | `:boolean` | — | -| `--limit` | `:long` | — | -| `--offset` | `:long` | — | -| `--sort` | — | `["name" "title"]` | -| `--order` | — | `["asc" "desc"]` | -| `--include-built-in` | `:boolean` | — | -| `--with-properties` | `:boolean` | — | -| `--with-extends` | `:boolean` | — | -| `--fields` | — | — | - -#### `list property` - -| Option | `:coerce` | `:values` | -| -------------------- | ---------- | ------------------ | -| `--expand` | `:boolean` | — | -| `--limit` | `:long` | — | -| `--offset` | `:long` | — | -| `--sort` | — | `["name" "title"]` | -| `--order` | — | `["asc" "desc"]` | -| `--include-built-in` | `:boolean` | — | -| `--with-classes` | `:boolean` | — | -| `--with-type` | `:boolean` | — | -| `--fields` | — | — | - -#### `upsert block` - -| Option | `:coerce` | `:values` | `:complete` | -| --------------------- | --------- | ------------------------------------------------------------------------------------------------------------------- | ----------- | -| `--id` | `:long` | — | — | -| `--uuid` | — | — | — | -| `--target-id` | `:long` | — | — | -| `--target-uuid` | — | — | — | -| `--target-page` | — | — | `:pages` | -| `--pos` | — | `["first-child" "last-child" "sibling"]` | — | -| `--content` | — | — | — | -| `--blocks` | — | — | — | -| `--blocks-file` | — | — | `:file` | -| `--status` | — | `["todo" "doing" "done" "now" "later" "wait" "waiting" "backlog" "canceled" "cancelled" "in-review" "in-progress"]` | — | -| `--update-tags` | — | — | — | -| `--update-properties` | — | — | — | -| `--remove-tags` | — | — | — | -| `--remove-properties` | — | — | — | - -> **Note on `--status` values:** The CLI also accepts aliases like `in_review`, -> `inreview`, `in progress`, `inprogress`. These are **not** included in -> completions — only canonical hyphenated forms are offered. The aliases -> remain accepted at runtime. - -#### `upsert page` - -| Option | `:coerce` | `:complete` | -| --------------------- | --------- | ----------- | -| `--id` | `:long` | — | -| `--page` | — | `:pages` | -| `--update-tags` | — | — | -| `--update-properties` | — | — | -| `--remove-tags` | — | — | -| `--remove-properties` | — | — | - -#### `upsert tag` - -| Option | `:coerce` | -| -------- | --------- | -| `--id` | `:long` | -| `--name` | — | - -No `:complete` — free text. - -#### `upsert property` - -| Option | `:coerce` | `:values` | -| --------------- | ---------- | -------------------------------------------------------------------------------- | -| `--id` | `:long` | — | -| `--name` | — | — | -| `--type` | — | `["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"]` | -| `--cardinality` | — | `["one" "many"]` | -| `--hide` | `:boolean` | — | -| `--public` | `:boolean` | — | - -#### `remove block` - -| Option | Notes | -| -------- | ------------------------------- | -| `--id` | Free text (db/id or EDN vector) | -| `--uuid` | Free text | - -#### `remove page` - -| Option | `:complete` | -| -------- | ----------- | -| `--name` | `:pages` | - -#### `remove tag` - -| Option | `:coerce` | -| -------- | --------- | -| `--id` | `:long` | -| `--name` | — | - -No `:complete` — free text. - -#### `remove property` - -| Option | `:coerce` | -| -------- | --------- | -| `--id` | `:long` | -| `--name` | — | - -No `:complete` — free text. - -#### `query` - -| Option | `:complete` | -| ---------- | ----------- | -| `--query` | — | -| `--name` | `:queries` | -| `--inputs` | — | - -> `:name` gets `:complete :queries` **only** in the `query` command spec. -> In `upsert tag` and `upsert property`, `:name` is free text — no -> `:complete` key. This is expressed naturally because each command has its -> own spec. - -#### `query list` - -No command-specific options. - -#### `show` - -| Option | `:coerce` | `:complete` | -| --------------------- | ---------- | ----------- | -| `--id` | — | — | -| `--uuid` | — | — | -| `--page` | — | `:pages` | -| `--linked-references` | `:boolean` | — | -| `--level` | `:long` | — | - -#### `doctor` - -| Option | `:coerce` | -| -------------- | ---------- | -| `--dev-script` | `:boolean` | - -#### `completions` _(new)_ - -| Option | `:values` | -| --------- | ---------------- | -| `--shell` | `["zsh" "bash"]` | - ---- - -## 3. Spec enrichment — new metadata keys - -Two new optional keys are added to `babashka.cli` spec entries. They are -consumed only by the completion generator; runtime parsing ignores them. - -### 3.1 `:values [...]` - -A vector of allowed string values — used for enum completion. - -```clojure -;; before -:output {:desc "Output format (human, json, edn). Default: human"} - -;; after -:output {:desc "Output format. Default: human" - :values ["human" "json" "edn"]} -``` - -Complete list of locations that need `:values` added: - -| File | Option | Values | -| ------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------- | -| `command/core.cljs` (global) | `:output` | `["human" "json" "edn"]` | -| `command/list.cljs` (page spec) | `:sort` | `["title" "created-at" "updated-at"]` | -| `command/list.cljs` (tag spec) | `:sort` | `["name" "title"]` | -| `command/list.cljs` (property spec) | `:sort` | `["name" "title"]` | -| `command/list.cljs` (common spec) | `:order` | `["asc" "desc"]` | -| `command/upsert.cljs` (block spec) | `:pos` | `["first-child" "last-child" "sibling"]` | -| `command/upsert.cljs` (block spec) | `:status` | `["todo" "doing" "done" "now" "later" "wait" "waiting" "backlog" "canceled" "cancelled" "in-review" "in-progress"]` | -| `command/upsert.cljs` (property spec) | `:type` | `["default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"]` | -| `command/upsert.cljs` (property spec) | `:cardinality` | `["one" "many"]` | -| `command/graph.cljs` (export spec) | `:type` | `["edn" "sqlite"]` | -| `command/graph.cljs` (import spec) | `:type` | `["edn" "sqlite"]` | - -> **Note on `:sort`:** The allowed values differ per sub-command (`list page` -> vs `list tag/property`). Each sub-command already has its own spec, so -> the difference is handled naturally with no special-casing. - -### 3.2 `:complete ` - -A hint that the value should be completed dynamically. The generator emits -a call to the appropriate shell helper function. - -| Keyword | zsh helper | bash helper | Notes | -| ---------- | ----------------- | ---------------------- | ------------------------------------------------- | -| `:graphs` | `_logseq_graphs` | `_logseq_graphs_bash` | Graph names from `logseq graph list` | -| `:pages` | `_logseq_pages` | `_logseq_pages_bash` | Page titles; requires `--graph` | -| `:queries` | `_logseq_queries` | `_logseq_queries_bash` | Built-in + custom query names; requires `--graph` | -| `:file` | `_files` | `compgen -f` | Filesystem path | -| `:dir` | `_files -/` | `compgen -d` | Directory path | - -Complete list of locations that need `:complete` added: - -| File | Option | `:complete` | Applies to | -| ----------------------------- | -------------- | ----------- | ------------------ | -| `command/core.cljs` (global) | `:graph` | `:graphs` | All commands | -| `command/core.cljs` (global) | `:config` | `:file` | All commands | -| `command/core.cljs` (global) | `:data-dir` | `:dir` | All commands | -| `command/graph.cljs` (export) | `:file` | `:file` | `graph export` | -| `command/graph.cljs` (import) | `:input` | `:file` | `graph import` | -| `command/upsert.cljs` (block) | `:target-page` | `:pages` | `upsert block` | -| `command/upsert.cljs` (block) | `:blocks-file` | `:file` | `upsert block` | -| `command/upsert.cljs` (page) | `:page` | `:pages` | `upsert page` | -| `command/query.cljs` | `:name` | `:queries` | `query` only | -| `command/show.cljs` | `:page` | `:pages` | `show` | -| `command/remove.cljs` (page) | `:name` | `:pages` | `remove page` only | - -> **`:name` is context-dependent.** `query --name` → `:queries`. -> `remove page --name` → `:pages`. `upsert tag --name` and -> `remove tag --name` → free text. This is handled by keeping `:complete` -> in command-specific specs only. - -> **`:queries` must return both built-in and custom queries.** The dynamic -> helper calls `logseq query list` which already returns both sources. - ---- - -## 4. New `completion` command - -New file: `src/main/logseq/cli/command/completion.cljs` - -``` -logseq completion zsh -logseq completion bash -logseq completion --help -``` - -The command: - -1. Takes `--shell` (or a positional arg) with value `zsh` or `bash`. -2. Calls the generator (§5) with the full `table`. -3. Prints the result to stdout. -4. Exits 0. - -Registration in `commands.cljs`: - -```clojure -(core/command-entry ["completion"] :completion - "Generate shell completion script" - {:shell {:desc "Shell (zsh, bash)" :values ["zsh" "bash"]}}) -``` - -The `completion` command itself **must be excluded** from the generated -completion dispatch (it is a meta-command, not a user-facing workflow -command). Whether to exclude it is a design choice; including it is also -acceptable since `logseq completion ` → `zsh bash` is helpful. - ---- - -## 5. Generator function - -New file: `src/main/logseq/cli/completion_generator.cljs` - -Pure function: `(generate-completions shell table) → string` - -### 5.1 Input - -The `table` as returned by `commands/build-table` — a vector of -`command-entry` maps. Each entry has: - -```clojure -{:cmds ["graph" "export"] - :command :graph-export - :desc "Export graph" - :spec { ;; merged global + command-specific - :help {:alias :h :coerce :boolean :desc "..."} - :graph {:desc "Graph name" :complete :graphs} - :type {:desc "Export type" :values ["edn" "sqlite"]} - :file {:desc "Export file" :complete :file} - ...}} -``` - -### 5.2 Spec-entry → completion token mapping - -| Spec characteristics | zsh token | bash case | -| --------------------- | -------------------------------------- | ------------------------------------------------------------ | -| `:coerce :boolean` | `'--flag[desc]'` | flag in wordlist, no argument | -| `:values [v1 v2]` | `'--opt=[desc]:label:(v1 v2)'` | `compgen -W 'v1 v2' -- "$cur"` | -| `:complete :graphs` | `'--opt=[desc]:graph:_logseq_graphs'` | `_logseq_compadd_lines "$cur" _logseq_graphs_bash` | -| `:complete :pages` | `'--opt=[desc]:page:_logseq_pages'` | `_logseq_compadd_lines "$cur" _logseq_pages_bash "$graph"` | -| `:complete :queries` | `'--opt=[desc]:query:_logseq_queries'` | `_logseq_compadd_lines "$cur" _logseq_queries_bash "$graph"` | -| `:complete :file` | `'--opt=[desc]:file:_files'` | `compgen -f -- "$cur"` | -| `:complete :dir` | `'--opt=[desc]:dir:_files -/'` | `compgen -d -- "$cur"` | -| `:alias :x` | `'(-x --opt)'{-x,--opt}'[desc]...'` | both `-x` and `--opt` in wordlist | -| free string (default) | `'--opt=[desc]:value:'` | flag in wordlist, prev-word fallthrough | - -### 5.3 Output structure (zsh) - -```zsh -#compdef logseq -# Auto-generated by `logseq completion zsh` — do not edit manually. - -# --- dynamic helpers (verbatim, fixed) --- -_logseq_json_names() { ... } -_logseq_graphs() { ... } # with zcompcache -_logseq_pages() { ... } # with zcompcache, keyed by --graph value -_logseq_queries() { ... } # with zcompcache, keyed by --graph value -_logseq_current_graph() { ... } - -# --- per-command functions (generated) --- -_logseq_graph_export() { _arguments -s ... } -_logseq_graph_import() { _arguments -s ... } -... - -# --- group dispatchers (generated) --- -_logseq_graph() { - _arguments -C -s ... '1:subcommand:->subcmd' '*::args:->args' - case $state in - subcmd) _describe 'subcommand' subcmds ;; - args) case $line[1] in ... esac ;; - esac -} - -# --- top-level dispatcher (generated) --- -_logseq() { - _arguments -C -s ... '1:command:->cmds' '*::args:->args' - ... -} - -_logseq "$@" -``` - -### 5.4 Output structure (bash) - -```bash -# Auto-generated by `logseq completion bash` — do not edit manually. - -# --- dynamic helpers (verbatim, fixed) --- -_logseq_json_names_bash() { ... } -_logseq_current_graph_bash() { ... } -_logseq_graphs_bash() { ... } -_logseq_pages_bash() { ... } -_logseq_queries_bash() { ... } -_logseq_compadd_lines() { ... } -_logseq_is_value_opt() { ... } # generated from spec -_logseq_cmd_and_subcmd() { ... } - -# --- option wordlists (generated per-command) --- -_logseq_opts_for() { ... } # case "$cmd"/"$subcmd" - -# --- main function (generated) --- -_logseq() { ... } - -complete -F _logseq logseq -``` - ---- - -## 6. Dynamic helper functions - -The dynamic helpers (graph/page/query lookups, caching) are **fixed verbatim -strings** emitted by the generator regardless of the table contents. They do -not change when new commands are added — only the static dispatch structure -changes. - -They live in the generator as string constants: - -```clojure -(def ^:private zsh-dynamic-helpers - "# --- dynamic helpers --- -_logseq_json_names() { - python3 -c \"...\" -} -...") -``` - -### 6.1 What the helpers invoke - -| Helper | CLI command | Output format | -| ------------------------------------------ | --------------------------------------------- | -------------------- | -| `_logseq_graphs` / `_logseq_graphs_bash` | `logseq graph list --output json` | JSON array of names | -| `_logseq_pages` / `_logseq_pages_bash` | `logseq list page --graph --output json` | JSON array of titles | -| `_logseq_queries` / `_logseq_queries_bash` | `logseq query list --graph --output json` | JSON array of names | - -### 6.2 `_logseq_current_graph` - -Scans `$words` (zsh) or `$COMP_WORDS` (bash) for `--graph VALUE` to -determine which graph context to use for page/query lookups. - ---- - -## 7. Tree-walk algorithm - -The generator derives the command hierarchy by inspecting `:cmds` vectors: - -1. **Leaf commands** — entries where no other entry's `:cmds` is a prefix. - These get per-command functions with `_arguments` (zsh) or case branches - (bash). - -2. **Group commands** — the distinct first-element prefixes (`graph`, - `server`, `list`, `upsert`, `remove`, `query`). These get dispatcher - functions that offer subcommand completion, then delegate to the leaf. - -3. **Top-level** — the root dispatcher that offers command-group (and - leaf-command) completion. - -The walk is: - -``` -table - → group by first element of :cmds - → for each group: - if only one entry → leaf (e.g., "show", "doctor") - if multiple entries → group dispatcher + per-subcommand leaves - → top-level dispatcher listing all groups and leaves -``` - ---- - -## 8. Edge cases - -| Case | Handling | -| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| `:alias` on a global opt (`:help` → `-h`) | Emit both long and short form with zsh grouping `(-h --help)` | -| Same flag name in both global and command spec | Generator uses the merged spec — command spec wins (already the case via `merge`) | -| `:sort` has different `:values` for `list page` vs `list tag/property` | Handled naturally — each sub-command has its own spec | -| `--output` is a format enum globally; `graph export` uses `--file` for the file path | No conflict — `--output` keeps its global meaning everywhere, `--file` is the export path. Confirmed in source. | -| `--name` context-dependent (`query` → `:queries`; `remove page` → `:pages`; `upsert tag` → free text) | Expressed by keeping `:complete` in command-specific specs only | -| Boolean flags shouldn't suggest a value | `:coerce :boolean` → emit as bare flag token | -| `remove page --name` needs page completion but `remove tag --name` does not | `remove-page-spec` gets its own spec with `{:name {:complete :pages}}`, separate from `remove-entity-spec` | -| `server` commands inherit `--graph` from global spec + also declare it in command spec | Redundant shadow — no issue, merged spec contains one `:graph` entry | -| `completion` command in generated output | Include it — `logseq completion ` → `zsh bash` is useful | -| Positional arguments (`graph create `, `graph switch `) | These commands take the graph name via `--graph` (global opt). No positional arg completion needed beyond subcommand dispatch. | - ---- - -## 9. What is NOT generated - -These are emitted as fixed preamble strings by the generator: - -- The dynamic helper function bodies (graph list, page list, query list - invocations) -- The zsh cache key logic (`_store_cache` / `_retrieve_cache`) -- The `_logseq_current_graph` helper that scans `$words` for `--graph` -- The `_logseq_compadd_lines` line-by-line appender (bash) -- The `_logseq_json_names` JSON-to-lines parser - ---- - -## 10. New files - -| Path | Purpose | -| ----------------------------------------------- | ------------------------------------------------------------ | -| `src/main/logseq/cli/command/completion.cljs` | Command entry: parse args, call generator, print to stdout | -| `src/main/logseq/cli/completion_generator.cljs` | Pure function: `(generate-completions shell table) → string` | - ---- - -## 11. Changes to existing files - -| File | Change | -| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `command/core.cljs` | Add `:values` and `:complete` to `global-spec*` entries per §3 | -| `command/list.cljs` | Add `:values` to `:sort` and `:order` per §3.1 | -| `command/upsert.cljs` | Add `:values` to `:pos`, `:status`, `:type`, `:cardinality`; add `:complete :pages` to `:target-page`; add `:complete :file` to `:blocks-file`; add `:complete :pages` to `:page` in page spec | -| `command/graph.cljs` | Add `:values` to `:type`; add `:complete :file` to `:file` and `:input` | -| `command/query.cljs` | Add `:complete :queries` to `:name` | -| `command/show.cljs` | Add `:complete :pages` to `:page` | -| `command/remove.cljs` | Split `remove-page-spec` from generic `remove-entity-spec` so `:name` gets `:complete :pages` only for `remove page` | -| `commands.cljs` | Add `completion-command/entries` to the `table` concat | - ---- - -## 12. Acceptance criteria - -- [ ] `logseq completion zsh` output, when installed as `_logseq`, passes - smoke tests: `logseq `, `logseq list `, - `logseq upsert block --pos `, `logseq show --page `, - `logseq remove `, `logseq server `, `logseq doctor `, - `logseq query --name `, `logseq remove page --name `, - `logseq graph export --type `, `logseq upsert block --blocks-file `. -- [ ] `logseq completion bash` output passes equivalent smoke tests. -- [ ] Adding a new `command-entry` to the table (or a `:values` change to a - spec) causes the generated output to change with no other edits - required. -- [ ] The generator is a pure function testable in isolation (no I/O). -- [ ] The hand-maintained `_logseq.zsh` and `logseq.bash` files in - `dz/scripts/shell-completions/` are replaced by the generated output - and marked as generated (header comment: "do not edit manually"). -- [ ] The `remove page` subcommand completes `--name` with page names - (dynamic), while `remove tag` and `remove property` leave `--name` - as free text. -- [ ] `upsert block --target-page`, `upsert page --page`, and - `show --page` all complete with page names (dynamic). -- [ ] `upsert block --blocks-file` and `graph export --file` and - `graph import --input` complete with file paths. -- [ ] `--config` completes with file paths; `--data-dir` completes with - directory paths. - ---- - -## 13. Out of scope - -- Fish shell completions (can be added later with the same table-walk). -- PowerShell completions. -- Auto-installing completions on `npm install` / `brew install`. -- CI regression test that diffs generated completions against a golden file - (desirable, but tracked separately). -- Completion for positional arguments beyond subcommand names (no current - command uses meaningful positional args). diff --git a/dz/scripts/shell-completions/README.md b/dz/scripts/shell-completions/README.md deleted file mode 100644 index daece614ff..0000000000 --- a/dz/scripts/shell-completions/README.md +++ /dev/null @@ -1,199 +0,0 @@ -# Shell Completions for the Logseq CLI - -This directory contains shell completion scripts for the `logseq` CLI -(`src/main/logseq/cli`). - -## Generating completions - -These files are **auto-generated** from the CLI command table. Do not edit -them manually — regenerate instead: - -```bash -logseq completion zsh > dz/scripts/shell-completions/_logseq.zsh -logseq completion bash > dz/scripts/shell-completions/logseq.bash -``` - -The generator reads the command table at generation time, so adding or -modifying a command/flag in the spec is the **only** change needed — the -completion scripts stay in sync automatically. - -## Available completions - -| File | Shell | -| --------------- | ----- | -| `_logseq.zsh` | zsh | -| `logseq.bash` | bash | - ---- - -## zsh - -### Installation (zsh) - -**Option A — eval in `~/.zshrc`** (simplest, always up to date): - -```zsh -eval "$(logseq completion zsh)" -``` - -**Option B — fpath** (avoids running `logseq` at shell startup): - -1. Copy `_logseq.zsh` to a directory on your `$fpath`, **renaming it to - `_logseq`** (no extension — required by zsh's autoload mechanism): - - ```zsh - mkdir -p ~/.zsh/completions - cp completions/_logseq.zsh ~/.zsh/completions/_logseq - ``` - -2. Add that directory to `$fpath` in `~/.zshrc` **before** the `compinit` call: - - ```zsh - fpath=(~/.zsh/completions $fpath) - autoload -Uz compinit && compinit - ``` - -3. Open a new terminal (or run `compinit` in the current session). - -### Verifying (zsh) - -```zsh -logseq # shows top-level commands -logseq list # shows: page tag property -logseq graph # shows: list create switch remove ... -logseq add block --status # shows task status values -logseq show --page # lists page names from the active graph -``` - -### Dynamic completions and caching (zsh) - -Page names, graph names, and query names are fetched live from the CLI and -cached using zsh's built-in completion cache (`~/.zcompcache/`). The cache is -keyed per graph, so switching `--repo` gives fresh results. - -To force a refresh, delete the relevant cache entries: - -```zsh -rm -f ~/.zcompcache/logseq_* -``` - -### Updating (zsh) - -After upgrading the `logseq` CLI, regenerate and re-copy: - -```zsh -logseq completion zsh > ~/.zsh/completions/_logseq -compinit -``` - ---- - -## bash - -### Requirements (bash) - -Requires bash 4.1+ and the [`bash-completion`](https://github.com/scop/bash-completion) -package (v2 recommended). On macOS: - -```bash -brew install bash bash-completion@2 -``` - -### Installation (bash) - -**Option A — per-user** (recommended): - -```bash -mkdir -p ~/.local/share/bash-completion/completions -cp completions/logseq.bash ~/.local/share/bash-completion/completions/logseq -``` - -**Option B — source from `~/.bashrc`**: - -```bash -source /path/to/logseq/completions/logseq.bash -``` - -**Option C — system-wide**: - -```bash -sudo cp completions/logseq.bash /etc/bash_completion.d/logseq -``` - -### Verifying (bash) - -```bash -logseq # shows top-level commands -logseq list # shows: page tag property -logseq graph # shows: list create switch remove ... -logseq add block --status # shows task status values -logseq show --page # lists page names from the active graph -``` - -### Dynamic completions (bash) - -Page names, graph names, and query names are fetched live from the CLI each -time they are needed. There is no built-in caching in the bash script; results -are as fresh as the CLI response. - -### Updating (bash) - -After upgrading the `logseq` CLI, regenerate and re-copy: - -```bash -logseq completion bash > ~/.local/share/bash-completion/completions/logseq -``` - ---- - -## What is completed (both shells) - -### Commands and subcommands - -```text -logseq list page | tag | property -logseq upsert block | page | tag | property -logseq remove block | page | tag | property -logseq query [list] -logseq graph list | create | switch | remove | validate | info | export | import -logseq server list | status | start | stop | restart -logseq show -logseq doctor -logseq completion -``` - -### Global options - -| Option | Completion | -| --------------------------------------- | --------------------------------------------- | -| `--graph` | dynamic: graph names from `logseq graph list` | -| `--config` | file path | -| `--data-dir` | directory path | -| `--output` | `human` `json` `edn` | -| `--timeout-ms` | free integer | -| `--verbose`, `--version`, `-h`/`--help` | flags | - -### Per-command options - -| Command | Option | Completion | -| --------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------- | -| `list page` | `--sort` | `title` `created-at` `updated-at` | -| `list tag`, `list property` | `--sort` | `name` `title` | -| `list *` | `--order` | `asc` `desc` | -| `upsert block` | `--pos` | `first-child` `last-child` `sibling` | -| `upsert block` | `--status` | `todo` `doing` `done` `now` `later` `wait` `waiting` `backlog` `canceled` `cancelled` `in-review` `in-progress` | -| `upsert block` | `--target-page` | dynamic: page titles from `logseq list page` | -| `upsert block` | `--blocks-file` | file path | -| `upsert page` | `--page` | dynamic: page titles | -| `upsert property` | `--type` | `default` `number` `date` `datetime` `checkbox` `url` `node` `json` `string` | -| `upsert property` | `--cardinality` | `one` `many` | -| `remove page` | `--name` | dynamic: page titles | -| `show` | `--page` | dynamic: page titles | -| `query` | `--name` | dynamic: query names from `logseq query list` | -| `graph export` | `--type` | `edn` `sqlite` | -| `graph export` | `--file` | file path | -| `graph import` | `--type` | `edn` `sqlite` | -| `graph import` | `--input` | file path | -| `completions` | `--shell` | `zsh` `bash` | -| `doctor` | `--dev-script` | flag | -` \ No newline at end of file diff --git a/dz/scripts/shell-completions/TASKS.md b/dz/scripts/shell-completions/TASKS.md deleted file mode 100644 index 09b176efec..0000000000 --- a/dz/scripts/shell-completions/TASKS.md +++ /dev/null @@ -1,283 +0,0 @@ -# Shell Completions — TDD Implementation Tasks - -Implementation plan for the `logseq completion ` feature as specified -in [DESIGN.md](DESIGN.md). Each task follows **Red → Green → Refactor**: write -a failing test first, then make it pass, then clean up. - -Test file: `src/test/logseq/cli/command/completion_test.cljs` -Generator test: `src/test/logseq/cli/completion_generator_test.cljs` - -> **Require registration:** The nbb test runner auto-discovers `*_test.cljs` -> files under the `test` classpath (`-cp test`). No explicit require is needed -> in a runner config — but the new test namespace **must** be required in the -> test entry if one exists, or verified to be picked up by convention. - ---- - -## Phase 0 — Scaffolding & test harness - -- [ ] **0.1** Create `src/test/logseq/cli/completion_generator_test.cljs` with - a skeleton ns that requires `[cljs.test :refer [deftest is testing]]` and - `[logseq.cli.completion-generator :as gen]`. Add a trivial failing test - (`(deftest placeholder (is false))`). - -- [ ] **0.2** Create `src/main/logseq/cli/completion_generator.cljs` with a - stub namespace and a `generate-completions` function that returns `""`. - Confirm the test runner loads both files (`yarn nbb-logseq -cp test -m - nextjournal.test-runner`). - -- [ ] **0.3** Create `src/test/logseq/cli/command/completion_test.cljs` with a - skeleton ns requiring `[logseq.cli.command.completion :as completion-command]`. - Add a trivial failing test. - -- [ ] **0.4** Create `src/main/logseq/cli/command/completion.cljs` with a stub - ns and `entries` def (empty vector). Confirm test runner loads it. - -- [ ] **0.5** Remove placeholder failing tests; verify `yarn test` passes - (green baseline). - ---- - -## Phase 1 — Spec enrichment (`:values` and `:complete` metadata) - -Each sub-task: write a test that reads the spec from the command entry and -asserts the metadata key exists, then add the key. - -- [ ] **1.1** `command/core.cljs` — global spec: - - Test: `:output` has `:values ["human" "json" "edn"]` - - Test: `:graph` has `:complete :graphs` - - Test: `:config` has `:complete :file` - - Test: `:data-dir` has `:complete :dir` - - Implement: add the keys to `global-spec*`. - -- [ ] **1.2** `command/list.cljs` — list specs: - - Test: page-spec `:sort` has `:values ["title" "created-at" "updated-at"]` - - Test: tag-spec `:sort` has `:values ["name" "title"]` - - Test: property-spec `:sort` has `:values ["name" "title"]` - - Test: common `:order` has `:values ["asc" "desc"]` - - Implement: add the `:values` keys. - -- [ ] **1.3** `command/upsert.cljs` — upsert specs: - - Test: block-spec `:pos` has `:values` - - Test: block-spec `:status` has `:values` - - Test: block-spec `:target-page` has `:complete :pages` - - Test: block-spec `:blocks-file` has `:complete :file` - - Test: page-spec `:page` has `:complete :pages` - - Test: property-spec `:type` has `:values` - - Test: property-spec `:cardinality` has `:values` - - Implement: add the keys. - -- [ ] **1.4** `command/graph.cljs` — export/import specs: - - Test: export-spec `:type` has `:values ["edn" "sqlite"]` - - Test: export-spec `:file` has `:complete :file` - - Test: import-spec `:type` has `:values ["edn" "sqlite"]` - - Test: import-spec `:input` has `:complete :file` - - Implement: add the keys. - -- [ ] **1.5** `command/query.cljs` — query spec: - - Test: query-spec `:name` has `:complete :queries` - - Implement: add the key. - -- [ ] **1.6** `command/show.cljs` — show spec: - - Test: show-spec `:page` has `:complete :pages` - - Implement: add the key. - -- [ ] **1.7** `command/remove.cljs` — remove page spec split: - - Test: remove-page entry's spec has `{:name {:complete :pages}}` - - Test: remove-tag entry's spec does NOT have `:complete` on `:name` - - Test: remove-property entry's spec does NOT have `:complete` on `:name` - - Implement: split `remove-page-spec` from `remove-entity-spec`. - ---- - -## Phase 2 — Generator: table introspection utilities - -Pure functions tested in `completion_generator_test.cljs`. - -- [ ] **2.1** `extract-groups` — given a table, return grouped command - hierarchy: - - Test: `["graph" "export"]` → group `"graph"`, subcommand `"export"` - - Test: `["show"]` → leaf command `"show"` - - Test: `["completion"]` → leaf command `"completion"` - - Implement in `completion_generator.cljs`. - -- [ ] **2.2** `leaf-commands` / `group-commands` — classify entries: - - Test: `show` and `doctor` are leaves - - Test: `graph`, `server`, `list`, `upsert`, `remove`, `query` are groups - - Implement. - -- [ ] **2.3** `spec->tokens` — convert a single spec entry to a shell token - descriptor: - - Test: boolean spec → `:flag` type - - Test: spec with `:values` → `:enum` type with values - - Test: spec with `:complete :graphs` → `:dynamic` type - - Test: spec with `:complete :file` → `:file` type - - Test: spec with `:complete :dir` → `:dir` type - - Test: spec with `:alias` → includes alias - - Test: bare string spec (no `:values`, no `:complete`, not boolean) → `:free` type - - Implement. - ---- - -## Phase 3 — Generator: zsh output - -- [ ] **3.1** `generate-zsh-preamble` — emit `#compdef logseq` header and - dynamic helper constants: - - Test: output starts with `#compdef logseq` - - Test: output contains `_logseq_graphs` - - Test: output contains `_logseq_pages` - - Test: output contains `_logseq_queries` - - Test: output contains `_logseq_json_names` - - Test: output contains `_logseq_current_graph` - - Implement with string constants. - -- [ ] **3.2** `generate-zsh-leaf` — emit a `_logseq_()` function for - a leaf command: - - Test: `show` command emits `_logseq_show()` with `_arguments` - - Test: boolean flags emit `'--flag[desc]'` form - - Test: enum options emit `'--opt=[desc]:label:(v1 v2)'` form - - Test: `:complete :graphs` emits `_logseq_graphs` action - - Test: `:complete :file` emits `_files` action - - Test: `:alias` emits `(-x --opt)` grouping - - Implement. - -- [ ] **3.3** `generate-zsh-group` — emit a group dispatcher: - - Test: `graph` group lists subcommands `list create switch remove validate info export import` - - Test: dispatches to `_logseq_graph_export` etc. - - Implement. - -- [ ] **3.4** `generate-zsh-toplevel` — emit `_logseq()` root dispatcher: - - Test: lists all top-level commands and groups - - Test: dispatches to group/leaf functions - - Test: ends with `_logseq "$@"` - - Implement. - -- [ ] **3.5** Integration: `(generate-completions "zsh" table)` returns a - complete, valid zsh script: - - Test: output contains preamble + all leaf functions + all group dispatchers - + top-level dispatcher - - Test: every command from the table appears in the output - - Test: `--pos` under `upsert block` offers `first-child last-child sibling` - - Test: `--sort` for `list page` offers `title created-at updated-at` - - Test: `--sort` for `list tag` offers `name title` - - Implement by composing 3.1–3.4. - ---- - -## Phase 4 — Generator: bash output - -- [ ] **4.1** `generate-bash-preamble` — emit header and dynamic helpers: - - Test: output contains `_logseq_graphs_bash` - - Test: output contains `_logseq_pages_bash` - - Test: output contains `_logseq_queries_bash` - - Test: output contains `_logseq_compadd_lines` - - Test: output contains `_logseq_json_names_bash` - - Test: output contains `_logseq_current_graph_bash` - - Implement with string constants. - -- [ ] **4.2** `generate-bash-opts-for` — emit `_logseq_opts_for()` case - dispatch: - - Test: `graph export` case branch includes `--type` and `--file` - - Test: boolean flags appear in wordlist without argument handling - - Test: enum values use `compgen -W` - - Test: `:complete :file` uses `compgen -f` - - Implement. - -- [ ] **4.3** `generate-bash-main` — emit `_logseq()` and `complete -F`: - - Test: output ends with `complete -F _logseq logseq` - - Test: subcommand dispatch works for groups - - Implement. - -- [ ] **4.4** Integration: `(generate-completions "bash" table)` returns a - complete bash script: - - Test: output contains preamble + opts-for + main function + complete - registration - - Test: every command from the table appears in the output - - Implement by composing 4.1–4.3. - ---- - -## Phase 5 — `completion` command entry - -- [ ] **5.1** Command registration: - - Test: `completion-command/entries` contains one entry with - `:cmds ["completion"]` and `:command :completions` - - Test: spec has `{:shell {:values ["zsh" "bash"]}}` - - Implement `command/completion.cljs` with `entries` and spec. - -- [ ] **5.2** Wire into `commands.cljs`: - - Test: `(commands/parse-args ["completion" "--shell" "zsh"])` returns - `{:ok? true :command :completions}` - - Test: `(commands/parse-args ["completion" "zsh"])` handles positional arg - - Implement: add `completion-command/entries` to the table concat in - `commands.cljs`. - -- [ ] **5.3** Build action and execute: - - Test: `build-action` for `:completion` returns an action with - `:type :completions` and `:shell "zsh"` - - Test: `execute` for `:completion` calls `generate-completions` and returns - the output string - - Implement in `command/completion.cljs` and wire into - `commands.cljs` `build-action`/`execute`. - ---- - -## Phase 6 — End-to-end validation - -- [ ] **6.1** Golden-file smoke test (zsh): - - Generate zsh output from the full table - - Assert key structural markers: `#compdef`, `_logseq_graph_export`, - `_logseq_show`, `_logseq "$@"` - - Assert all commands from §2.1 of DESIGN.md appear - -- [ ] **6.2** Golden-file smoke test (bash): - - Generate bash output from the full table - - Assert key structural markers: `complete -F _logseq logseq`, - `_logseq_opts_for` - - Assert all commands from §2.1 of DESIGN.md appear - -- [ ] **6.3** Sync test — adding a command updates output: - - Build a minimal table, generate output, add a fake command entry, - re-generate, assert the new command appears - -- [ ] **6.4** Context-dependent `:name` test: - - Assert `query` spec has `{:name {:complete :queries}}` - - Assert `remove page` spec has `{:name {:complete :pages}}` - - Assert `upsert tag` spec does NOT have `:complete` on `:name` - - Assert `remove tag` spec does NOT have `:complete` on `:name` - ---- - -## Phase 7 — Replace hand-maintained files - -- [ ] **7.1** Generate fresh `_logseq.zsh` and `logseq.bash` from the - `completion` command; write them to - `dz/scripts/shell-completions/`. - -- [ ] **7.2** Verify the generated files include the "do not edit manually" - header comment. - -- [ ] **7.3** Update `dz/scripts/shell-completions/README.md` to document the - new `logseq completion` workflow instead of hand-editing. - ---- - -## Test require checklist - -All new test namespaces that must be discoverable by the test runner -(`yarn nbb-logseq -cp test -m nextjournal.test-runner`): - -| Test file | Requires | -|---|---| -| `src/test/logseq/cli/completion_generator_test.cljs` | `logseq.cli.completion-generator` | -| `src/test/logseq/cli/command/completion_test.cljs` | `logseq.cli.command.completion` | - -The nbb test runner scans the `test` classpath for `*_test.cljs` files -automatically. Verify with: - -```bash -cd deps/cli && yarn test -``` - -If the runner uses an explicit require list (check for a `test_runner.cljs` or -similar), add the two new namespaces there as well. diff --git a/dz/scripts/shell-completions/_logseq.zsh b/dz/scripts/shell-completions/_logseq.zsh deleted file mode 100644 index c3ff77d030..0000000000 --- a/dz/scripts/shell-completions/_logseq.zsh +++ /dev/null @@ -1,783 +0,0 @@ -#compdef logseq -# Auto-generated by `logseq completion zsh` — do not edit manually. - -# --- dynamic helpers --- - -_logseq_json_names() { - python3 -c " -import sys, json -field = sys.argv[1] -try: - data = json.load(sys.stdin) - if isinstance(data, list): - for item in data: - v = item.get(field) - if isinstance(v, str) and v: - print(v) -except Exception: - pass -" "$1" 2>/dev/null -} - -_logseq_current_graph() { - local i - for (( i = 1; i < ${#words[@]}; i++ )); do - if [[ "${words[i]}" == '--graph' && -n "${words[i+1]}" ]]; then - print -r -- "${words[i+1]}" - return - fi - done -} - -_logseq_graphs() { - local cache_id='logseq_graphs' - local -a graphs - if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then - graphs=( ${(f)"$(logseq graph list --output json 2>/dev/null | _logseq_json_names name)"} ) - _store_cache "$cache_id" graphs - fi - compadd -a graphs -} - -_logseq_pages() { - local graph - graph=$(_logseq_current_graph) - local cache_id="logseq_pages_${graph:-__default__}" - local -a pages - if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then - if [[ -n "$graph" ]]; then - pages=( ${(f)"$(logseq list page --graph "$graph" --output json 2>/dev/null | _logseq_json_names title)"} ) - fi - _store_cache "$cache_id" pages - fi - compadd -a pages -} - -_logseq_queries() { - local graph - graph=$(_logseq_current_graph) - local cache_id="logseq_queries_${graph:-__default__}" - local -a queries - if _cache_invalid "$cache_id" || ! _retrieve_cache "$cache_id"; then - if [[ -n "$graph" ]]; then - queries=( ${(f)"$(logseq query list --graph "$graph" --output json 2>/dev/null | _logseq_json_names name)"} ) - fi - _store_cache "$cache_id" queries - fi - compadd -a queries -} - -# --- per-command functions --- - -_logseq_list_page() { - _arguments -s \ - '--created-after=[Filter by created-at (ISO8601)]:value:' \ - '--updated-after=[Filter by updated-at (ISO8601)]:value:' \ - '--limit=[Limit results]:value:' \ - '--offset=[Offset results]:value:' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--fields=[Select output fields (comma separated)]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--journal-only[Only journal pages]' \ - '--order=[Sort order. Default: asc]:value:(asc desc)' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--include-hidden[Include hidden pages]' \ - '--expand[Include expanded metadata]' \ - '--sort=[Sort field]:value:(title created-at updated-at)' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--include-journal[Include journal pages]' -} - -_logseq_list_tag() { - _arguments -s \ - '--limit=[Limit results]:value:' \ - '--offset=[Offset results]:value:' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--include-built-in[Include built-in tags]' \ - '--fields=[Select output fields (comma separated)]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--order=[Sort order. Default: asc]:value:(asc desc)' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--with-extends[Include tag extends]' \ - '--expand[Include expanded metadata]' \ - '--with-properties[Include tag properties]' \ - '--sort=[Sort field]:value:(name title)' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_list_property() { - _arguments -s \ - '--limit=[Limit results]:value:' \ - '--offset=[Offset results]:value:' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--include-built-in[Include built-in properties]' \ - '--fields=[Select output fields (comma separated)]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--order=[Sort order. Default: asc]:value:(asc desc)' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--with-type[Include property type]' \ - '--version[Show version]' \ - '--expand[Include expanded metadata]' \ - '--sort=[Sort field]:value:(name title)' \ - '--with-classes[Include property classes]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_server_list() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_server_status() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_server_start() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_server_stop() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_server_restart() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_list() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_create() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_switch() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_remove() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_validate() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_info() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_graph_export() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--file=[Export file path]:file:_files' \ - '--type=[Export type]:value:(edn sqlite)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_graph_import() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--type=[Import type]:value:(edn sqlite)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--input=[Input path]:file:_files' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_query() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Query name from cli.edn custom-queries or built-ins]:value:_logseq_queries' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--inputs=[EDN vector of query inputs]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--query=[Datascript query EDN]:value:' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_query_list() { - _arguments -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' -} - -_logseq_upsert_block() { - _arguments -s \ - '--blocks-file=[EDN file of blocks for create mode]:file:_files' \ - '--target-page=[Target page name]:value:_logseq_pages' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--content=[Block content for create mode]:value:' \ - '--remove-tags=[Tags to remove (EDN vector)]:value:' \ - '--target-uuid=[Target block UUID]:value:' \ - '--pos=[Position. Default: create=last-child, update=first-child]:value:(first-child last-child sibling)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--target-id=[Target block db/id]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--update-tags=[Tags to add/update (EDN vector)]:value:' \ - '--status=[Task status]:value:(todo doing done now later wait waiting backlog canceled cancelled in-review in-progress)' \ - '--update-properties=[Properties to add/update (EDN map)]:value:' \ - '--id=[Source block db/id (forces update mode)]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--blocks=[EDN vector of blocks for create mode]:value:' \ - '--uuid=[Source block UUID (forces update mode)]:value:' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--remove-properties=[Properties to remove (EDN vector)]:value:' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_upsert_page() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--remove-tags=[Tags to remove (EDN vector)]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--page=[Page name]:value:_logseq_pages' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--update-tags=[Tags to add/update (EDN vector)]:value:' \ - '--update-properties=[Properties to add/update (EDN map)]:value:' \ - '--id=[Target page db/id (forces update mode)]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--remove-properties=[Properties to remove (EDN vector)]:value:' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_upsert_tag() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Tag name]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--id=[Target tag db/id (forces update mode)]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_upsert_property() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Property name]:value:' \ - '--public[Set property public visibility]' \ - '--type=[Property type]:value:(default number date datetime checkbox url node json string)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--hide[Hide property]' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--id=[Target property db/id (forces update mode)]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--cardinality=[Property cardinality]:value:(one many)' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_completion() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--shell=[Shell (zsh, bash)]:value:(zsh bash)' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_show() { - _arguments -s \ - '--linked-references[Include linked references (default true)]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--page=[Page name]:value:_logseq_pages' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--level=[Limit tree depth (default 10)]:value:' \ - '--id=[Block db/id or EDN vector of ids]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--uuid=[Block UUID]:value:' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_doctor() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--dev-script[Check static/db-worker-node.js instead of bundled dist runtime]' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_remove_block() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--id=[Block db/id or EDN vector of ids]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--uuid=[Block UUID]:value:' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_remove_page() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Page name]:value:_logseq_pages' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_remove_tag() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Entity name]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--id=[Entity db/id]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -_logseq_remove_property() { - _arguments -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Entity name]:value:' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--id=[Entity db/id]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' -} - -# --- group dispatchers --- - -_logseq_list() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'page:List pages' - 'tag:List tags' - 'property:List properties' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - page) _logseq_list_page ;; - tag) _logseq_list_tag ;; - property) _logseq_list_property ;; - esac - ;; - esac -} - -_logseq_server() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'list:List db-worker-node servers' - 'status:Show server status for a graph' - 'start:Start db-worker-node for a graph' - 'stop:Stop db-worker-node for a graph' - 'restart:Restart db-worker-node for a graph' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - list) _logseq_server_list ;; - status) _logseq_server_status ;; - start) _logseq_server_start ;; - stop) _logseq_server_stop ;; - restart) _logseq_server_restart ;; - esac - ;; - esac -} - -_logseq_graph() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'list:List graphs' - 'create:Create graph' - 'switch:Switch current graph' - 'remove:Remove graph' - 'validate:Validate graph' - 'info:Graph metadata' - 'export:Export graph' - 'import:Import graph' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - list) _logseq_graph_list ;; - create) _logseq_graph_create ;; - switch) _logseq_graph_switch ;; - remove) _logseq_graph_remove ;; - validate) _logseq_graph_validate ;; - info) _logseq_graph_info ;; - export) _logseq_graph_export ;; - import) _logseq_graph_import ;; - esac - ;; - esac -} - -_logseq_query() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--name=[Query name from cli.edn custom-queries or built-ins]:value:_logseq_queries' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--inputs=[EDN vector of query inputs]:value:' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--query=[Datascript query EDN]:value:' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'list:List available queries' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - list) _logseq_query_list ;; - esac - ;; - esac -} - -_logseq_upsert() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'block:Upsert block' - 'page:Upsert page' - 'tag:Upsert tag' - 'property:Upsert property' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - block) _logseq_upsert_block ;; - page) _logseq_upsert_page ;; - tag) _logseq_upsert_tag ;; - property) _logseq_upsert_property ;; - esac - ;; - esac -} - -_logseq_remove() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:subcommand:->subcmd' \ - '*::args:->args' - - case $state in - subcmd) - local -a subcmds - subcmds=( - 'block:Remove blocks' - 'page:Remove page' - 'tag:Remove tag' - 'property:Remove property' - ) - _describe 'subcommand' subcmds - ;; - args) - case $line[1] in - block) _logseq_remove_block ;; - page) _logseq_remove_page ;; - tag) _logseq_remove_tag ;; - property) _logseq_remove_property ;; - esac - ;; - esac -} - -# --- top-level dispatcher --- - -_logseq() { - local curcontext="$curcontext" state line - typeset -A opt_args - - _arguments -C -s \ - '(-h --help)'{-h,--help}'[Show help]' \ - '--version[Show version]' \ - '--config=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files' \ - '--graph=[Graph name]:value:_logseq_graphs' \ - '--data-dir=[Path to db-worker data dir (default ~/logseq/graphs)]:dir:_files -/' \ - '--timeout-ms=[Request timeout in ms (default 10000)]:value:' \ - '--output=[Output format. Default: human]:value:(human json edn)' \ - '--verbose[Enable verbose debug logging to stderr]' \ - '1:command:->cmds' \ - '*::args:->args' - - case $state in - cmds) - local -a cmds - cmds=( - 'completion:Generate shell completion script' - 'doctor:Run runtime diagnostics' - 'graph:graph commands' - 'list:list commands' - 'query:query commands' - 'remove:remove commands' - 'server:server commands' - 'show:Show tree' - 'upsert:upsert commands' - ) - _describe 'command' cmds - ;; - args) - case $line[1] in - completion) _logseq_completion ;; - doctor) _logseq_doctor ;; - graph) _logseq_graph ;; - list) _logseq_list ;; - query) _logseq_query ;; - remove) _logseq_remove ;; - server) _logseq_server ;; - show) _logseq_show ;; - upsert) _logseq_upsert ;; - esac - ;; - esac -} - -compdef _logseq logseq diff --git a/dz/scripts/shell-completions/logseq.bash b/dz/scripts/shell-completions/logseq.bash deleted file mode 100644 index 8fcb4cd974..0000000000 --- a/dz/scripts/shell-completions/logseq.bash +++ /dev/null @@ -1,274 +0,0 @@ -# Auto-generated by `logseq completion bash` — do not edit manually. - -# --- dynamic helpers --- - -_logseq_json_names_bash() { - python3 -c " -import sys, json -field = sys.argv[1] -try: - data = json.load(sys.stdin) - if isinstance(data, list): - for item in data: - v = item.get(field) - if isinstance(v, str) and v: - print(v) -except Exception: - pass -" "$1" 2>/dev/null -} - -_logseq_current_graph_bash() { - local i - for (( i = 1; i < ${#COMP_WORDS[@]}; i++ )); do - if [[ "${COMP_WORDS[i]}" == '--graph' && -n "${COMP_WORDS[i+1]}" ]]; then - printf '%s' "${COMP_WORDS[i+1]}" - return - fi - done -} - -_logseq_graphs_bash() { - logseq graph list --output json 2>/dev/null | _logseq_json_names_bash name -} - -_logseq_pages_bash() { - logseq list page --graph "$1" --output json 2>/dev/null | _logseq_json_names_bash title -} - -_logseq_queries_bash() { - logseq query list --graph "$1" --output json 2>/dev/null | _logseq_json_names_bash name -} - -_logseq_compadd_lines() { - local cur="$1" source_fn="$2"; shift 2 - while IFS= read -r item; do - [[ "$item" == "$cur"* ]] && COMPREPLY+=( "$item" ) - done < <("$source_fn" "$@") -} - -# --- generated helpers --- - -_logseq_is_value_opt() { - case "$1" in - --blocks|--blocks-file|--cardinality|--config|--content|--created-after|--data-dir|--fields|--file|--graph|--id|--input|--inputs|--level|--limit|--name|--offset|--order|--output|--page|--pos|--query|--remove-properties|--remove-tags|--shell|--sort|--status|--target-id|--target-page|--target-uuid|--timeout-ms|--type|--update-properties|--update-tags|--updated-after|--uuid) - return 0 ;; - *) return 1 ;; - esac -} - -_logseq_cmd_and_subcmd() { - local i skip=0 - __cmd='' __subcmd='' - for (( i = 1; i < COMP_CWORD; i++ )); do - local w="${COMP_WORDS[i]}" - if (( skip )); then skip=0; continue; fi - if [[ "$w" == -* ]]; then - _logseq_is_value_opt "$w" && skip=1 - continue - fi - if [[ -z "$__cmd" ]]; then - __cmd="$w" - elif [[ -z "$__subcmd" ]]; then - __subcmd="$w" - fi - done -} - -# --- option wordlists --- - -_logseq_opts_for() { - local cmd="$1" subcmd="$2" - local opts="--help -h --version --config --graph --data-dir --timeout-ms --output --verbose" - - case "$cmd" in - completion) opts+=' --shell' ;; - doctor) opts+=' --dev-script' ;; - graph) - case "$subcmd" in - list) opts+=' ' ;; - create) opts+=' ' ;; - switch) opts+=' ' ;; - remove) opts+=' ' ;; - validate) opts+=' ' ;; - info) opts+=' ' ;; - export) opts+=' --file --type' ;; - import) opts+=' --type --input' ;; - esac - ;; - list) - case "$subcmd" in - page) opts+=' --created-after --updated-after --limit --offset --fields --journal-only --order --include-hidden --expand --sort --include-journal' ;; - tag) opts+=' --limit --offset --include-built-in --fields --order --with-extends --expand --with-properties --sort' ;; - property) opts+=' --limit --offset --include-built-in --fields --order --with-type --expand --sort --with-classes' ;; - esac - ;; - query) - opts+=' --name --inputs --query' - case "$subcmd" in - list) opts+=' ' ;; - esac - ;; - remove) - case "$subcmd" in - block) opts+=' --id --uuid' ;; - page) opts+=' --name' ;; - tag) opts+=' --name --id' ;; - property) opts+=' --name --id' ;; - esac - ;; - server) - case "$subcmd" in - list) opts+=' ' ;; - status) opts+=' ' ;; - start) opts+=' ' ;; - stop) opts+=' ' ;; - restart) opts+=' ' ;; - esac - ;; - show) opts+=' --linked-references --page --level --id --uuid' ;; - upsert) - case "$subcmd" in - block) opts+=' --blocks-file --target-page --content --remove-tags --target-uuid --pos --target-id --update-tags --status --update-properties --id --blocks --uuid --remove-properties' ;; - page) opts+=' --remove-tags --page --update-tags --update-properties --id --remove-properties' ;; - tag) opts+=' --name --id' ;; - property) opts+=' --name --public --type --hide --id --cardinality' ;; - esac - ;; - esac - - printf '%s' "$opts" -} - -# --- main function --- - -_logseq() { - local cur prev - cur="${COMP_WORDS[COMP_CWORD]}" - prev="${COMP_WORDS[COMP_CWORD-1]}" - COMPREPLY=() - - local __cmd __subcmd - _logseq_cmd_and_subcmd - - # --- Option value completion --- - case "$prev" in - --blocks-file) - COMPREPLY=( $(compgen -f -- "$cur") ) - return ;; - - --target-page) - local graph - graph="$(_logseq_current_graph_bash)" - [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" - return ;; - - --config) - COMPREPLY=( $(compgen -f -- "$cur") ) - return ;; - - --file) - COMPREPLY=( $(compgen -f -- "$cur") ) - return ;; - - --pos) - COMPREPLY=( $(compgen -W 'first-child last-child sibling' -- "$cur") ) - return ;; - - --type) - COMPREPLY=( $(compgen -W 'default number date datetime checkbox url node json string' -- "$cur") ) - return ;; - - --page) - local graph - graph="$(_logseq_current_graph_bash)" - [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" - return ;; - - --output) - COMPREPLY=( $(compgen -W 'human json edn' -- "$cur") ) - return ;; - - --data-dir) - COMPREPLY=( $(compgen -d -- "$cur") ) - return ;; - - --status) - COMPREPLY=( $(compgen -W 'todo doing done now later wait waiting backlog canceled cancelled in-review in-progress' -- "$cur") ) - return ;; - - --graph) - _logseq_compadd_lines "$cur" _logseq_graphs_bash - return ;; - - --order) - COMPREPLY=( $(compgen -W 'asc desc' -- "$cur") ) - return ;; - - --input) - COMPREPLY=( $(compgen -f -- "$cur") ) - return ;; - - --cardinality) - COMPREPLY=( $(compgen -W 'one many' -- "$cur") ) - return ;; - - --shell) - COMPREPLY=( $(compgen -W 'zsh bash' -- "$cur") ) - return ;; - - --name) - if [[ "$__cmd" == 'remove' && "$__subcmd" == 'page' ]]; then - local graph - graph="$(_logseq_current_graph_bash)" - [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_pages_bash "$graph" - fi - if [[ "$__cmd" == 'query' ]]; then - local graph - graph="$(_logseq_current_graph_bash)" - [[ -n "$graph" ]] && _logseq_compadd_lines "$cur" _logseq_queries_bash "$graph" - fi - return ;; - - --sort) - if [[ "$__cmd" == 'list' && "$__subcmd" == 'page' ]]; then - COMPREPLY=( $(compgen -W 'title created-at updated-at' -- "$cur") ) - return - fi - if [[ "$__cmd" == 'list' && "$__subcmd" == 'tag' ]]; then - COMPREPLY=( $(compgen -W 'name title' -- "$cur") ) - return - fi - if [[ "$__cmd" == 'list' && "$__subcmd" == 'property' ]]; then - COMPREPLY=( $(compgen -W 'name title' -- "$cur") ) - return - fi - return ;; - esac - - # --- Flag / positional completion --- - if [[ "$cur" == -* ]]; then - # shellcheck disable=SC2046 - COMPREPLY=( $(compgen -W "$(_logseq_opts_for "$__cmd" "$__subcmd")" -- "$cur") ) - return - fi - - if [[ -z "$__cmd" ]]; then - COMPREPLY=( $(compgen -W 'completion doctor graph list query remove server show upsert' -- "$cur") ) - return - fi - - if [[ -z "$__subcmd" ]]; then - case "$__cmd" in - graph) COMPREPLY=( $(compgen -W 'list create switch remove validate info export import' -- "$cur") ) ;; - list) COMPREPLY=( $(compgen -W 'page tag property' -- "$cur") ) ;; - query) COMPREPLY=( $(compgen -W 'list' -- "$cur") ) ;; - remove) COMPREPLY=( $(compgen -W 'block page tag property' -- "$cur") ) ;; - server) COMPREPLY=( $(compgen -W 'list status start stop restart' -- "$cur") ) ;; - upsert) COMPREPLY=( $(compgen -W 'block page tag property' -- "$cur") ) ;; - esac - return - fi -} - -complete -F _logseq logseq From a1bccdbd33542ded4b72fa396201a9f6969c17e6 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 12 Mar 2026 13:47:27 +0800 Subject: [PATCH 142/375] fix test --- src/main/logseq/cli/completion_generator.cljs | 11 ----------- .../logseq/cli/completion_generator_test.cljs | 16 +++++++--------- 2 files changed, 7 insertions(+), 20 deletions(-) diff --git a/src/main/logseq/cli/completion_generator.cljs b/src/main/logseq/cli/completion_generator.cljs index 8c43a04afb..038cb402b5 100644 --- a/src/main/logseq/cli/completion_generator.cljs +++ b/src/main/logseq/cli/completion_generator.cljs @@ -613,17 +613,6 @@ _logseq_compadd_lines() { nil))) -(defn- bash-prev-cases - "Generate all prev-word value completion cases from the table." - [table] - (let [all-specs (->> table (mapcat (fn [entry] (seq (:spec entry))))) - unique-specs (into {} all-specs) ;; last one wins for duplicates - tokens (spec->tokens unique-specs) - cases (->> tokens - (keep bash-prev-completion-case) - (string/join "\n\n"))] - cases)) - (defn- bash-subcommand-cases "Generate subcommand completion for each group." [table] diff --git a/src/test/logseq/cli/completion_generator_test.cljs b/src/test/logseq/cli/completion_generator_test.cljs index 54d3b756d4..5387b43daf 100644 --- a/src/test/logseq/cli/completion_generator_test.cljs +++ b/src/test/logseq/cli/completion_generator_test.cljs @@ -200,14 +200,14 @@ (is (string/includes? output "_logseq()"))) (testing "output ends with compdef _logseq logseq" (is (string/includes? output "compdef _logseq logseq"))) - (testing "boolean flags emit bare flag form" - (is (re-find #"--verbose\[" output))) + (testing "boolean flags emit alias grouping" + (is (string/includes? output "'{-v,--verbose}'["))) (testing "enum options emit value list form" - (is (re-find #"--output=.*\(human json edn\)" output))) + (is (string/includes? output "'{-o,--output}'=[Output format. Default: human]:value:(human json edn)'"))) (testing ":complete :graphs emits _logseq_graphs" - (is (re-find #"--graph=.*_logseq_graphs" output))) + (is (string/includes? output "'{-g,--graph}'=[Graph name]:value:_logseq_graphs'"))) (testing ":complete :file emits _files" - (is (re-find #"--config=.*_files'" output))) + (is (string/includes? output "'{-c,--config}'=[Path to cli.edn (default ~/logseq/cli.edn)]:file:_files'"))) (testing ":alias emits grouping" (is (re-find #"\(-h --help\)" output))))) @@ -218,10 +218,8 @@ (testing "--sort for list page offers correct values" (is (re-find #"--sort=.*\(title created-at updated-at\)" output))) (testing "--sort for list tag offers name title" - ;; The list tag function should contain (name title) - (let [tag-section (second (re-find #"_logseq_list_tag\(\).*?(?=\n_logseq)" output))] - ;; Just check globally that name title appears in sort context - (is (re-find #"\(name title\)" output)))))) + ;; Just check globally that name title appears in sort context + (is (re-find #"\(name title\)" output))))) (deftest test-zsh-all-commands-present (let [output (gen/generate-completions "zsh" full-table)] From 50261dbb9550760a49488b8507d663d182d148d8 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 12 Mar 2026 17:13:27 +0800 Subject: [PATCH 143/375] 058-db-worker-node-revision-and-cli-server-list.md --- ...orker-node-revision-and-cli-server-list.md | 226 ++++++++++++++++++ docs/cli/logseq-cli.md | 4 +- shadow-cljs.edn | 3 + src/dev-cljs/shadow/hooks.clj | 18 ++ src/main/frontend/worker/db_worker_node.cljs | 8 +- .../frontend/worker/db_worker_node_lock.cljs | 7 +- src/main/frontend/worker/version.cljs | 18 ++ src/main/logseq/cli/command/server.cljs | 22 +- src/main/logseq/cli/format.cljs | 45 +++- src/main/logseq/cli/server.cljs | 1 + src/main/logseq/cli/version.cljs | 12 +- .../worker/db_worker_node_lock_test.cljs | 69 +++++- .../frontend/worker/db_worker_node_test.cljs | 41 ++++ src/test/logseq/cli/command/server_test.cljs | 50 ++++ src/test/logseq/cli/format_test.cljs | 41 +++- src/test/logseq/cli/server_test.cljs | 23 ++ 16 files changed, 548 insertions(+), 40 deletions(-) create mode 100644 docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md create mode 100644 src/main/frontend/worker/version.cljs create mode 100644 src/test/logseq/cli/command/server_test.cljs diff --git a/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md b/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md new file mode 100644 index 0000000000..1ab5ad6f47 --- /dev/null +++ b/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md @@ -0,0 +1,226 @@ +# 058 — db-worker-node revision and CLI server list compatibility + +## Summary + +This plan proposes three related improvements to align `db-worker-node` runtime metadata with `logseq-cli` versioning: + +1. Add `--version` support to `db-worker-node`. +2. Persist `revision` in `db-worker.lock`. +3. Extend `logseq-cli server list` with a `REVISION` column and a compatibility warning when server revision differs from local CLI revision. + +The goal is to make version drift observable and debuggable in multi-process and multi-version environments. + +--- + +## Product decisions (locked) + +1. **Mismatch rule**: if revision/commit strings are not exactly equal, it is a mismatch. + - No normalization. + - No short-hash expansion. + - No suffix stripping. + +2. **Warning scope**: mismatch warning is shown **only in human output**. + - JSON output must not include human warning text. + +--- + +## Background + +Current behavior: + +- `logseq-cli --version` already prints build metadata including `Revision`. +- `db-worker-node` accepts operational flags (`--repo`, `--data-dir`, etc.) but does not expose `--version`. +- `db-worker.lock` does not include revision metadata. +- `logseq-cli server list` shows process/state fields (`GRAPH`, `STATUS`, `HOST`, `PORT`, `PID`, `OWNER`) but not revision. + +As a result, users cannot quickly determine whether a running DB worker matches the CLI binary currently used to manage it. + +--- + +## Goals + +- Expose `db-worker-node` revision consistently with CLI version semantics. +- Persist server revision in lock metadata for later discovery. +- Surface revision in `server list` output. +- Warn users when a listed server revision does not match local CLI revision (human output only). + +## Non-goals + +- No protocol change for worker RPC endpoints. +- No forced auto-restart or auto-migration for mismatched servers. +- No hard failure on mismatch (warning only). + +--- + +## Design + +### 1) Add `--version` to db-worker-node + +#### Behavior + +- `db-worker-node --version` prints version metadata and exits with code `0`. +- It must not require `--repo`. +- Output must include at least `Revision: `. +- Formatting should follow `logseq-cli --version` style for consistency. + +#### Implementation shape + +- Extend argument parsing in `db_worker_node.cljs` to recognize `--version`. +- Add an early return branch in `main` before repo validation. +- Introduce a worker-side version namespace using `goog-define` values injected at build time. + +### 2) Add `revision` to lock file + +#### Behavior + +- Lock creation writes `:revision`. +- Lock update preserves existing revision and remains backward compatible with old lock files. +- Reading old lock files without revision continues to work. + +#### Implementation shape + +- Add revision retrieval at lock write point. +- Extend `create-lock!` payload in `db_worker_node_lock.cljs`. +- Adjust `update-lock!` merge/preserve behavior. + +### 3) Show revision in `server list` and warn on mismatch + +#### Behavior + +- Add `REVISION` column to human table output. +- `--format json` includes per-server `revision` field. +- Mismatch warning text appears only in human output. +- Missing revision should render as `-` and should not crash formatting. + +#### Mismatch rule (exact string) + +Given: + +- local CLI revision = `cli-rev` +- server lock revision = `server-rev` + +Then: + +- mismatch if `cli-rev != server-rev` (exact string compare) +- no normalization at either side + +#### Implementation shape + +- Extend server discovery result in `logseq.cli.server/list-servers` to include `:revision`. +- Compute mismatch server set in command/data path (`server list` execute path), then pass structured mismatch info to formatter. +- Render warning block only in human formatter path. + +--- + +## Build metadata strategy + +Use the same metadata source philosophy as `logseq-cli`: + +- Revision source priority: + 1. `LOGSEQ_REVISION` env + 2. git describe output + 3. fallback `"dev"` + +- Build-time source priority: + 1. `LOGSEQ_BUILD_TIME` env + 2. current UTC timestamp + +For `db-worker-node`, inject closure defines through its shadow build target, analogous to existing CLI metadata injection. + +--- + +## Implementation plan (task list) + +1. Add worker version namespace and shadow metadata wiring. +2. Add `--version` parse + main dispatch branch in `db_worker_node.cljs`. +3. Add `:revision` field in `db-worker.lock` create/update path. +4. Extend `server list` data model with revision. +5. Extend table formatter with `REVISION` column. +6. Add mismatch detection (exact string compare) and pass structured mismatch data. +7. Render mismatch warning in human output only. +8. Add/update tests for all paths. +9. Update CLI documentation. + +--- + +## Testing plan + +### Unit tests + +- `db_worker_node_test.cljs` + - Add case: `--version` recognized and exits early without repo. + - Validate output includes `Revision`. + +- `db_worker_node_lock_test.cljs` + - Create lock includes `:revision`. + - Update lock preserves existing revision. + - Legacy lock without revision remains supported. + +- `format_test.cljs` + - Human `server list` table includes `REVISION` column. + - Missing revision renders `-`. + - Mismatch warning block appears in human output for exact-string mismatch. + - No warning block when revisions are equal. + +- `server_test.cljs` (or equivalent) + - Discovery output includes revision extracted from lock. + - Mismatch set computed with exact string comparison. + +- JSON output tests + - `server list --format json` contains per-server `revision`. + - JSON output does not contain human warning text. + +### Regression checks + +- Existing `logseq-cli --version` tests remain green. +- Existing server list owner/status formatting remains unchanged aside from the added `REVISION` column and optional warning block in human mode. + +--- + +## Rollout and compatibility + +- Backward compatible with existing lock files. +- New clients can read old locks (revision absent). +- Old clients can ignore new lock key (`revision`) if parser is permissive map decode. + +--- + +## Risks and mitigations + +- **Risk:** db-worker build target missing metadata hook. + - **Mitigation:** add explicit hook wiring and test fallback value (`dev`) in test target. + +- **Risk:** warning noise in mixed environments. + - **Mitigation:** warning remains informational and human-only. + +- **Risk:** flaky tests due to dynamic metadata. + - **Mitigation:** use deterministic test defines in shadow test config. + +--- + +## Acceptance criteria + +- `db-worker-node --version` prints revision and exits `0` without requiring repo args. +- `db-worker.lock` written by worker includes `revision`. +- `logseq-cli server list` (human) shows `REVISION` column. +- `logseq-cli server list` (human) prints mismatch warning when local CLI revision string is not exactly equal to server revision string. +- `logseq-cli server list --format json` includes server revision data and does not include human warning text. +- Added/updated tests cover happy path and backward-compatibility path. + +--- + +## File scope (expected) + +- `src/dev-cljs/shadow/hooks.clj` +- `shadow-cljs.edn` +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/frontend/worker/version.cljs` (new) +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/server.cljs` +- `src/main/logseq/cli/format.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `docs/cli/logseq-cli.md` diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 8a6deb411d..cba22fbadf 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -127,7 +127,9 @@ eval "$(logseq completion bash)" Server ownership behavior: - `server stop` and `server restart` can return `server-owned-by-other` if the daemon was started by another owner source. - `server start` can return `server-start-timeout-orphan` when lock creation times out and orphan matching processes are detected. -- `server list` human output includes an `OWNER` column, and `server status` / `server list` include owner metadata in structured output (`--output json|edn`). +- `server list` human output includes both `OWNER` and `REVISION` columns. +- `server list` prints a compatibility warning in human output when any server revision string is not exactly equal to the local CLI revision string. +- Structured output (`--output json|edn`) includes per-server `revision` data but does not include human warning text. Sync commands: - `sync status --graph ` - show db-sync runtime state for a graph daemon diff --git a/shadow-cljs.edn b/shadow-cljs.edn index 60af2cab22..9b37384ee3 100644 --- a/shadow-cljs.edn +++ b/shadow-cljs.edn @@ -89,6 +89,7 @@ :db-worker-node {:target :node-script :output-to "static/db-worker-node.js" :main frontend.worker.db-worker-node/main + :build-hooks [(shadow.hooks/db-worker-node-metadata-hook "--long --always --dirty")] :compiler-options {:infer-externs :auto :source-map true :externs ["datascript/externs.js" @@ -199,6 +200,8 @@ :output-to "static/tests.js" :closure-defines {frontend.util/NODETEST true logseq.shui.util/NODETEST true + frontend.worker.version/BUILD_TIME "test-worker-build-time" + frontend.worker.version/REVISION "test-worker-revision" logseq.cli.version/BUILD_TIME "test-build-time" logseq.cli.version/REVISION "test-revision"} :devtools {:enabled false} diff --git a/src/dev-cljs/shadow/hooks.clj b/src/dev-cljs/shadow/hooks.clj index 67920abde6..492c866669 100644 --- a/src/dev-cljs/shadow/hooks.clj +++ b/src/dev-cljs/shadow/hooks.clj @@ -50,3 +50,21 @@ (assoc defines-in-options 'logseq.cli.version/REVISION revision 'logseq.cli.version/BUILD_TIME build-time))))) + +(defn db-worker-node-metadata-hook + {:shadow.build/stage :configure} + [build-state & args] + (let [defines-in-config (get-in build-state [:shadow.build/config :closure-defines]) + defines-in-options (get-in build-state [:compiler-options :closure-defines]) + revision (env-or "LOGSEQ_REVISION" (or (exec "git" "describe" args) "dev")) + build-time (env-or "LOGSEQ_BUILD_TIME" (iso-now))] + (prn ::db-worker-node-metadata-hook {:revision revision :build-time build-time}) + (-> build-state + (assoc-in [:shadow.build/config :closure-defines] + (assoc defines-in-config + 'frontend.worker.version/REVISION revision + 'frontend.worker.version/BUILD_TIME build-time)) + (assoc-in [:compiler-options :closure-defines] + (assoc defines-in-options + 'frontend.worker.version/REVISION revision + 'frontend.worker.version/BUILD_TIME build-time))))) diff --git a/src/main/frontend/worker/db_worker_node.cljs b/src/main/frontend/worker/db_worker_node.cljs index 6c5b789cbe..1607e4a485 100644 --- a/src/main/frontend/worker/db_worker_node.cljs +++ b/src/main/frontend/worker/db_worker_node.cljs @@ -8,6 +8,7 @@ [frontend.worker.db-worker-node-lock :as db-lock] [frontend.worker.platform.node :as platform-node] [frontend.worker.state :as worker-state] + [frontend.worker.version :as worker-version] [lambdaisland.glogi :as log] [logseq.cli.style :as style] [logseq.common.config :as common-config] @@ -54,6 +55,7 @@ "--owner-source" (recur (subvec args 2) (assoc opts :owner-source (second args))) "--log-level" (recur (subvec args 2) (assoc opts :log-level (second args))) "--create-empty-db" (recur (subvec args 1) (assoc opts :create-empty-db? true)) + "--version" (recur (subvec args 1) (assoc opts :version? true)) "--help" (recur (subvec args 1) (assoc opts :help? true)) (recur (subvec args 1) opts)))))) @@ -287,6 +289,7 @@ (println (str " " (style/bold "--repo") " (required)")) (println (str " " (style/bold "--create-empty-db") " (start with empty initial datoms)")) (println (str " " (style/bold "--log-level") " (default info)")) + (println (str " " (style/bold "--version") " (print build metadata and exit)")) (println " logs: //db-worker-node-YYYYMMDD.log (retains 7)")) (defn- startup-db-opts @@ -438,11 +441,14 @@ (defn main [] - (let [{:keys [data-dir repo help? owner-source] :as opts} + (let [{:keys [data-dir repo help? version? owner-source] :as opts} (parse-args (.-argv js/process))] (when help? (show-help!) (.exit js/process 0)) + (when version? + (println (worker-version/format-version)) + (.exit js/process 0)) (when-not (seq repo) (show-help!) (.exit js/process 1)) diff --git a/src/main/frontend/worker/db_worker_node_lock.cljs b/src/main/frontend/worker/db_worker_node_lock.cljs index 0a2b601889..23fae441c2 100644 --- a/src/main/frontend/worker/db_worker_node_lock.cljs +++ b/src/main/frontend/worker/db_worker_node_lock.cljs @@ -5,6 +5,7 @@ ["path" :as node-path] [clojure.string :as string] [frontend.worker-common.util :as worker-util] + [frontend.worker.version :as worker-version] [lambdaisland.glogi :as log] [logseq.common.graph-dir :as graph-dir] [logseq.common.config :as common-config] @@ -106,6 +107,7 @@ :host host :port port :owner-source (normalize-owner-source owner-source) + :revision (worker-version/revision) :startedAt (.toISOString (js/Date.))}] (try (fs/writeFileSync fd (js/JSON.stringify (clj->js lock))) @@ -128,8 +130,11 @@ (assoc :pid (:pid existing)) (assoc :lock-id (or (:lock-id existing) (:lock-id lock))) (assoc :owner-source (normalize-owner-source (:owner-source existing))) + (assoc :revision (:revision existing)) (assoc :startedAt (:startedAt existing))) - (update lock :owner-source normalize-owner-source))] + (-> lock + (update :owner-source normalize-owner-source) + (update :revision #(or % (worker-version/revision)))))] (fs/writeFileSync path (js/JSON.stringify (clj->js lock'))) (resolve lock')) (catch :default e diff --git a/src/main/frontend/worker/version.cljs b/src/main/frontend/worker/version.cljs new file mode 100644 index 0000000000..cd259a9a5e --- /dev/null +++ b/src/main/frontend/worker/version.cljs @@ -0,0 +1,18 @@ +(ns frontend.worker.version + "Build metadata for db-worker-node.") + +(goog-define BUILD_TIME "unknown") +(goog-define REVISION "dev") + +(defn build-time + [] + BUILD_TIME) + +(defn revision + [] + REVISION) + +(defn format-version + [] + (str "Build time: " (build-time) "\n" + "Revision: " (revision))) diff --git a/src/main/logseq/cli/command/server.cljs b/src/main/logseq/cli/command/server.cljs index 0cad4f5065..bd3d8e06e3 100644 --- a/src/main/logseq/cli/command/server.cljs +++ b/src/main/logseq/cli/command/server.cljs @@ -2,6 +2,7 @@ "Server-related CLI commands." (:require [logseq.cli.command.core :as core] [logseq.cli.server :as cli-server] + [logseq.cli.version :as version] [promesa.core :as p])) (def ^:private server-spec @@ -69,11 +70,26 @@ {:status :error :error (:error result)})) +(defn- compute-revision-mismatches + [cli-revision servers] + (let [mismatch-servers (->> (or servers []) + (filter (fn [{:keys [revision]}] + (not= cli-revision revision))) + (mapv (fn [{:keys [repo revision]}] + {:repo repo + :revision revision})))] + (when (seq mismatch-servers) + {:cli-revision cli-revision + :servers mismatch-servers}))) + (defn execute-list [_action config] - (-> (p/let [servers (cli-server/list-servers config)] - {:status :ok - :data {:servers servers}}))) + (-> (p/let [servers (cli-server/list-servers config) + revision-mismatch (compute-revision-mismatches (version/revision) servers)] + (cond-> {:status :ok + :data {:servers servers}} + revision-mismatch + (assoc :human {:server-list {:revision-mismatch revision-mismatch}}))))) (defn execute-status [action config] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 57f3445987..a8ad6a4da8 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -230,18 +230,36 @@ graph)]) graphs)))) +(defn- format-server-list-warning + [{:keys [cli-revision servers]}] + (when (seq servers) + (str "Warning: server revision mismatch detected\n" + "Local CLI revision: " (normalize-cell cli-revision) "\n" + "Mismatched servers:\n" + (string/join "\n" + (map (fn [{:keys [repo revision]}] + (str " - " (normalize-cell repo) + " (revision: " + (normalize-cell revision) + ")")) + servers))))) + (defn- format-server-list - [servers] - (format-counted-table - ["GRAPH" "STATUS" "HOST" "PORT" "PID" "OWNER"] - (mapv (fn [server] - [(:repo server) - (:status server) - (:host server) - (:port server) - (:pid server) - (:owner-source server)]) - (or servers [])))) + [servers revision-mismatch] + (let [table (format-counted-table + ["GRAPH" "STATUS" "HOST" "PORT" "PID" "OWNER" "REVISION"] + (mapv (fn [server] + [(:repo server) + (:status server) + (:host server) + (:port server) + (:pid server) + (:owner-source server) + (:revision server)]) + (or servers [])))] + (if-let [warning (format-server-list-warning revision-mismatch)] + (str table "\n\n" warning) + table))) (defn- format-query-results [result] @@ -522,7 +540,7 @@ (string/join "\n" (into [header] check-lines)))) (defn- ->human - [{:keys [status data error command context]} {:keys [now-ms graph]}] + [{:keys [status data error command context human]} {:keys [now-ms graph]}] (let [now-ms (or now-ms (js/Date.now))] (case status :ok @@ -531,7 +549,8 @@ :graph-info (format-graph-info data now-ms) (:graph-create :graph-switch :graph-remove :graph-validate) (format-graph-action command context) - :server-list (format-server-list (:servers data)) + :server-list (format-server-list (:servers data) + (get-in human [:server-list :revision-mismatch])) :server-status (format-server-status data) (:server-start :server-stop :server-restart) (format-server-action command data) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 94a6b9fc8a..3c13b1c48e 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -295,6 +295,7 @@ :port (:port lock) :pid (:pid lock) :owner-source (lock-owner-source lock) + :revision (:revision lock) :status (if ready :ready :starting)}))))) (defn list-graphs diff --git a/src/main/logseq/cli/version.cljs b/src/main/logseq/cli/version.cljs index 19bf4fb6e4..669de1dbf0 100644 --- a/src/main/logseq/cli/version.cljs +++ b/src/main/logseq/cli/version.cljs @@ -4,7 +4,15 @@ (goog-define BUILD_TIME "unknown") (goog-define REVISION "dev") +(defn build-time + [] + BUILD_TIME) + +(defn revision + [] + REVISION) + (defn format-version [] - (str "Build time: " BUILD_TIME "\n" - "Revision: " REVISION)) + (str "Build time: " (build-time) "\n" + "Revision: " (revision))) diff --git a/src/test/frontend/worker/db_worker_node_lock_test.cljs b/src/test/frontend/worker/db_worker_node_lock_test.cljs index 37283c556b..4aa65877f7 100644 --- a/src/test/frontend/worker/db_worker_node_lock_test.cljs +++ b/src/test/frontend/worker/db_worker_node_lock_test.cljs @@ -5,6 +5,7 @@ [cljs.test :refer [async deftest is testing]] [frontend.test.node-helper :as node-helper] [frontend.worker.db-worker-node-lock :as db-lock] + [frontend.worker.version :as worker-version] [promesa.core :as p])) (deftest repo-dir-canonicalizes-db-prefixed-repo @@ -55,6 +56,25 @@ (db-lock/remove-lock! path) (done))))))) +(deftest create-lock-persists-revision + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-revision") + repo (str "logseq_db_lock_revision_" (subs (str (random-uuid)) 0 8)) + path (db-lock/lock-path data-dir repo)] + (-> (p/with-redefs [worker-version/revision (fn [] "worker-test-revision")] + (p/let [_ (db-lock/create-lock! {:data-dir data-dir + :repo repo + :host "127.0.0.1" + :port 9101 + :owner-source :cli}) + lock (db-lock/read-lock path)] + (is (= "worker-test-revision" (:revision lock))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (db-lock/remove-lock! path) + (done))))))) + (deftest read-lock-normalizes-missing-owner-source-to-unknown (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-legacy-owner") repo (str "logseq_db_lock_legacy_" (subs (str (random-uuid)) 0 8)) @@ -68,20 +88,49 @@ (fs/writeFileSync path (js/JSON.stringify (clj->js legacy-lock))) (is (= :unknown (:owner-source (db-lock/read-lock path)))))) -(deftest update-lock-preserves-existing-owner-source +(deftest update-lock-preserves-existing-owner-source-and-revision (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-update-owner") repo (str "logseq_db_lock_update_owner_" (subs (str (random-uuid)) 0 8)) path (db-lock/lock-path data-dir repo)] - (-> (p/let [{:keys [lock]} (db-lock/ensure-lock! {:data-dir data-dir - :repo repo - :host "127.0.0.1" - :port 9101 - :owner-source :cli}) - _ (db-lock/update-lock! path (assoc lock :port 9200 :owner-source :electron)) - updated (db-lock/read-lock path)] - (is (= :cli (:owner-source updated))) - (is (= 9200 (:port updated)))) + (-> (p/with-redefs [worker-version/revision (fn [] "existing-worker-revision")] + (p/let [{:keys [lock]} (db-lock/ensure-lock! {:data-dir data-dir + :repo repo + :host "127.0.0.1" + :port 9101 + :owner-source :cli}) + _ (db-lock/update-lock! path (assoc lock + :port 9200 + :owner-source :electron + :revision "attempted-new-revision")) + updated (db-lock/read-lock path)] + (is (= :cli (:owner-source updated))) + (is (= 9200 (:port updated))) + (is (= "existing-worker-revision" (:revision updated))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally (fn [] + (db-lock/remove-lock! path) + (done))))))) + +(deftest update-lock-keeps-legacy-lock-without-revision-compatible + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-node-lock-legacy-revision") + repo (str "logseq_db_lock_legacy_revision_" (subs (str (random-uuid)) 0 8)) + path (db-lock/lock-path data-dir repo) + legacy-lock {:repo repo + :pid (.-pid js/process) + :lock-id (str (random-uuid)) + :host "127.0.0.1" + :port 9101 + :owner-source :unknown + :startedAt (.toISOString (js/Date.))}] + (fs/mkdirSync (node-path/dirname path) #js {:recursive true}) + (fs/writeFileSync path (js/JSON.stringify (clj->js legacy-lock))) + (-> (p/let [_ (db-lock/update-lock! path (assoc legacy-lock :port 9202)) + updated (db-lock/read-lock path)] + (is (= 9202 (:port updated))) + (is (nil? (:revision updated)))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally (fn [] diff --git a/src/test/frontend/worker/db_worker_node_test.cljs b/src/test/frontend/worker/db_worker_node_test.cljs index f7117456d5..3743f1b1d8 100644 --- a/src/test/frontend/worker/db_worker_node_test.cljs +++ b/src/test/frontend/worker/db_worker_node_test.cljs @@ -12,6 +12,7 @@ [goog.object :as gobj] [logseq.cli.server :as cli-server] [logseq.cli.style :as style] + [logseq.cli.test-helper :as test-helper] [logseq.common.config :as common-config] [logseq.db :as ldb] [promesa.core :as p])) @@ -277,6 +278,46 @@ (is (= "logseq_db_parse_args" (:repo result))) (is (= true (:create-empty-db? result))))) +(deftest db-worker-node-parse-args-recognizes-version + (let [parse-args #'db-worker-node/parse-args + result (parse-args #js ["node" "dist/db-worker-node.js" + "--version"])] + (is (= true (:version? result))) + (is (nil? (:repo result))))) + +(deftest db-worker-node-main-version-exits-early-without-repo + (async done + (let [exit-code* (atom nil) + start-called? (atom false)] + (-> (test-helper/with-js-property-override + js/process + "argv" + #js ["node" "dist/db-worker-node.js" "--version"] + (fn [] + (test-helper/with-js-property-override + js/process + "exit" + (fn [code] + (reset! exit-code* code) + (throw (ex-info "process-exit" {:code code}))) + (fn [] + (p/with-redefs [db-worker-node/start-daemon! (fn [_] + (reset! start-called? true) + (p/rejected (ex-info "should-not-start-daemon" {})))] + (p/resolved + (let [output (with-out-str + (try + (db-worker-node/main) + (catch :default e + (when-not (= "process-exit" (.-message e)) + (throw e)))))] + (is (= 0 @exit-code*)) + (is (= false @start-called?)) + (is (string/includes? output "Revision:"))))))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest db-worker-node-owner-source-cli-is-written-into-lock (async done (let [daemon (atom nil) diff --git a/src/test/logseq/cli/command/server_test.cljs b/src/test/logseq/cli/command/server_test.cljs new file mode 100644 index 0000000000..02839022ae --- /dev/null +++ b/src/test/logseq/cli/command/server_test.cljs @@ -0,0 +1,50 @@ +(ns logseq.cli.command.server-test + (:require [cljs.test :refer [async deftest is]] + [logseq.cli.command.server :as server-command] + [logseq.cli.server :as cli-server] + [logseq.cli.version :as cli-version] + [promesa.core :as p])) + +(deftest compute-revision-mismatches-uses-exact-string-compare + (let [compute-revision-mismatches #'server-command/compute-revision-mismatches + mismatch (compute-revision-mismatches + "cli-rev" + [{:repo "graph-a" :revision "cli-rev"} + {:repo "graph-b" :revision "cli-rev-dirty"}])] + (is (= "cli-rev" (:cli-revision mismatch))) + (is (= ["graph-b"] + (mapv :repo (:servers mismatch)))))) + +(deftest execute-list-attaches-human-mismatch-metadata + (async done + (-> (p/with-redefs [cli-version/revision (fn [] "cli-rev") + cli-server/list-servers (fn [_] + (p/resolved [{:repo "graph-a" + :revision "cli-rev"} + {:repo "graph-b" + :revision "worker-rev"}]))] + (server-command/execute-list {:type :server-list} {})) + (p/then (fn [result] + (is (= :ok (:status result))) + (is (= 2 (count (get-in result [:data :servers])))) + (is (= "cli-rev" + (get-in result [:human :server-list :revision-mismatch :cli-revision]))) + (is (= ["graph-b"] + (mapv :repo (get-in result [:human :server-list :revision-mismatch :servers])))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) + +(deftest execute-list-omits-mismatch-when-revisions-match + (async done + (-> (p/with-redefs [cli-version/revision (fn [] "cli-rev") + cli-server/list-servers (fn [_] + (p/resolved [{:repo "graph-a" + :revision "cli-rev"}]))] + (server-command/execute-list {:type :server-list} {})) + (p/then (fn [result] + (is (= :ok (:status result))) + (is (nil? (get-in result [:human :server-list :revision-mismatch]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 134af11084..873d4f095d 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -544,8 +544,8 @@ (is (= ["logseq_db_demo" "my_logseq_db_notes"] (get-in parsed [:data :graphs]))))) -(deftest test-human-output-server-list-includes-owner - (testing "server list shows owner column and value" +(deftest test-human-output-server-list-includes-owner-and-revision + (testing "server list shows owner and revision columns" (let [result (format/format-result {:status :ok :command :server-list :data {:servers [{:repo "demo-repo" @@ -553,14 +553,15 @@ :host "127.0.0.1" :port 1234 :pid 9876 - :owner-source :cli}]}} + :owner-source :cli + :revision "worker-revision"}]}} {:output-format nil})] - (is (= (str "GRAPH STATUS HOST PORT PID OWNER\n" - "demo-repo :ready 127.0.0.1 1234 9876 :cli\n" + (is (= (str "GRAPH STATUS HOST PORT PID OWNER REVISION\n" + "demo-repo :ready 127.0.0.1 1234 9876 :cli worker-revision\n" "Count: 1") result)))) - (testing "server list falls back to placeholder when owner is missing" + (testing "server list falls back to placeholder when owner or revision is missing" (let [result (format/format-result {:status :ok :command :server-list :data {:servers [{:repo "demo-repo" @@ -569,10 +570,32 @@ :port 1234 :pid 9876}]}} {:output-format nil})] - (is (= (str "GRAPH STATUS HOST PORT PID OWNER\n" - "demo-repo :ready 127.0.0.1 1234 9876 -\n" + (is (= (str "GRAPH STATUS HOST PORT PID OWNER REVISION\n" + "demo-repo :ready 127.0.0.1 1234 9876 - -\n" "Count: 1") - result))))) + result)))) + + (testing "server list shows revision mismatch warning in human output only" + (let [base-result {:status :ok + :command :server-list + :data {:servers [{:repo "demo-repo" + :status :ready + :host "127.0.0.1" + :port 1234 + :pid 9876 + :owner-source :cli + :revision "server-revision"}]} + :human {:server-list {:revision-mismatch {:cli-revision "cli-revision" + :servers [{:repo "demo-repo" + :revision "server-revision"}]}}}} + human-result (format/format-result base-result {:output-format nil}) + json-result (format/format-result base-result {:output-format :json}) + json-parsed (js->clj (js/JSON.parse json-result) :keywordize-keys true)] + (is (string/includes? human-result "Warning:")) + (is (string/includes? human-result "cli-revision")) + (is (string/includes? human-result "demo-repo")) + (is (= "server-revision" (get-in json-parsed [:data :servers 0 :revision]))) + (is (not (string/includes? json-result "Warning:")))))) (deftest test-human-output-show (testing "show renders text payloads directly" diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index 24a6c41fd4..29f3aae9c7 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -269,3 +269,26 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done))))) + +(deftest list-servers-includes-revision-from-lock + (async done + (let [data-dir (node-helper/create-tmp-dir "cli-server-list-revision") + repo (str "logseq_db_list_revision_" (subs (str (random-uuid)) 0 8)) + lock-file (cli-server/lock-path data-dir repo) + lock {:repo repo + :pid (.-pid js/process) + :host "127.0.0.1" + :port 9311 + :owner-source :cli + :revision "server-revision"}] + (fs/mkdirSync (node-path/dirname lock-file) #js {:recursive true}) + (fs/writeFileSync lock-file (js/JSON.stringify (clj->js lock))) + (-> (p/with-redefs [daemon/ready? (fn [_] (p/resolved true))] + (cli-server/list-servers {:data-dir data-dir})) + (p/then (fn [servers] + (is (= 1 (count servers))) + (is (= repo (:repo (first servers)))) + (is (= "server-revision" (:revision (first servers)))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) From 5e8462c08aede343d8a421df4d6aa26d73671787 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 12 Mar 2026 20:54:11 +0800 Subject: [PATCH 144/375] 059-cli-doctor-server-revision-mismatch-warning.md --- ...doctor-server-revision-mismatch-warning.md | 237 ++++++++++++++++++ docs/cli/logseq-cli.md | 5 +- src/main/logseq/cli/command/doctor.cljs | 55 +++- src/main/logseq/cli/command/server.cljs | 14 +- src/main/logseq/cli/format.cljs | 30 ++- src/main/logseq/cli/server.cljs | 12 + src/test/logseq/cli/command/doctor_test.cljs | 67 ++++- src/test/logseq/cli/command/server_test.cljs | 3 +- src/test/logseq/cli/format_test.cljs | 49 ++++ 9 files changed, 438 insertions(+), 34 deletions(-) create mode 100644 docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md diff --git a/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md b/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md new file mode 100644 index 0000000000..3afa5cf0c9 --- /dev/null +++ b/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md @@ -0,0 +1,237 @@ +# 059 — Add doctor warning for server and CLI revision mismatch + +## Summary + +This plan adds a new `doctor` warning when a discovered db-worker server revision does not match the local `logseq-cli` revision. + +The warning must be actionable: for each mismatched server, `doctor` should tell the user to run: + +- `logseq server restart --graph ` + +This closes the current observability gap where `server list` already exposes revision mismatch, but `doctor` does not. + +--- + +## Product decisions (locked) + +1. **Mismatch rule is exact string compare**. + - `cli-revision != server-revision` means mismatch. + - No normalization, no hash shortening, no suffix stripping. + +2. **Severity is warning, not error**. + - Revision drift is diagnosable and recoverable. + - `doctor` should still return `:status :ok` at top-level transport with `:data {:status :warning ...}` behavior, consistent with existing warning checks. + +3. **`doctor` remains fail-fast for hard preconditions**. + - If script check fails, stop immediately (current behavior). + - If data-dir check fails, stop immediately (current behavior). + - Revision mismatch check runs only when server checks are reached. + +4. **Restart guidance is per affected graph/server**. + - The warning includes one restart command per mismatched graph. + - Command form is `logseq server restart --graph `. + +5. **Structured output must remain machine-friendly**. + - JSON/EDN outputs include structured mismatch check data. + - Human output includes readable warning text and restart guidance. + +--- + +## Background + +Current implementation already has the needed revision primitives: + +- CLI revision source: `logseq.cli.version/revision`. +- Server revision source: `db-worker.lock` revision surfaced by `logseq.cli.server/list-servers`. +- `server list` already computes and warns on mismatch in human output. + +Current `doctor` checks: + +1. `db-worker-script` +2. `data-dir` +3. `running-servers` readiness + +`doctor` does **not** currently compare server revision against CLI revision. + +--- + +## Goals + +- Add revision mismatch detection to `doctor` using existing revision sources. +- Provide actionable restart guidance for each mismatched graph. +- Keep mismatch semantics identical to existing `server list` behavior. +- Preserve existing `doctor` fail-fast behavior for script/data-dir errors. + +## Non-goals + +- No automatic server restart. +- No change to daemon ownership/permission rules. +- No new db-worker RPC endpoint for version info. +- No hard failure on mismatch. + +--- + +## User-facing behavior + +### Human output + +When mismatches exist, `doctor` includes a warning check (example shape): + +- `[warning] server-revision-mismatch - 2 servers use a different revision than this CLI` +- Followed by actionable per-graph commands, e.g.: + - `Run: logseq server restart --graph graph-a` + - `Run: logseq server restart --graph graph-b` + +If graph names include spaces, command examples should quote names in guidance output. + +### JSON and EDN output + +`doctor --output json|edn` keeps structured check payload, including mismatch details. + +Suggested check payload shape: + +- `:id :server-revision-mismatch` +- `:status :warning | :ok` +- `:code :doctor-server-revision-mismatch` (when warning) +- `:cli-revision ` +- `:servers [{:repo :revision :graph }]` +- `:message ` + +--- + +## Design + +### 1) Reuse one mismatch computation path + +Today, mismatch computation is a private helper in `logseq.cli.command.server`. + +Plan: + +- Move or extract mismatch computation into a shared helper (recommended in `logseq.cli.server`), so both `server list` and `doctor` use exactly the same comparison logic. +- Keep return shape stable enough to support both command paths. + +This prevents semantic drift between `server list` and `doctor`. + +### 2) Add a dedicated doctor check + +Add a new check in `logseq.cli.command.doctor`: + +- `check-server-revision-mismatch` + +Input: + +- local CLI revision (`logseq.cli.version/revision`) +- discovered servers from `list-servers` + +Behavior: + +- No mismatches -> check status `:ok` +- Any mismatch -> check status `:warning`, include mismatch metadata and restart guidance + +### 3) Execution flow in doctor + +Keep current order and fail-fast semantics for hard errors: + +1. `check-db-worker-script` (error stops execution) +2. `check-data-dir` (error stops execution) +3. `check-running-servers` +4. `check-server-revision-mismatch` (new) + +Final `doctor` status is: + +- `:ok` when all checks are `:ok` +- `:warning` when any warning check exists (`running-servers` and/or `server-revision-mismatch`) +- `:error` only for hard failures as today + +### 4) Restart guidance formatting strategy + +Two viable options: + +- **Option A (minimal formatter change):** put guidance directly into mismatch check `:message`. +- **Option B (cleaner long-term):** add a small `format-doctor` branch for `:server-revision-mismatch` to render command lines from structured fields. + +Recommended: **Option B** if it stays small, because it preserves clean machine payload while improving human readability and escaping/quoting behavior. + +### 5) Graph naming for restart commands + +Restart command uses `--graph`, not repo id. + +Plan: + +- Derive graph name via existing repo/graph conversion helper before rendering command guidance. +- Ensure guidance is aligned with user-facing graph names shown in CLI output. + +--- + +## Implementation plan (task list) + +1. Extract revision mismatch helper from `command/server.cljs` into shared location. +2. Update `server list` command to call the shared helper (behavior unchanged). +3. Add `check-server-revision-mismatch` to `command/doctor.cljs`. +4. Extend `execute-doctor` to append the new check and compute combined warning status. +5. Add per-graph restart guidance to human doctor output. +6. Keep JSON/EDN payload structured and stable. +7. Update CLI docs for `doctor` warning behavior and remediation command. + +--- + +## Testing plan + +### Unit tests + +- `src/test/logseq/cli/command/doctor_test.cljs` + - Adds warning check when one or more servers have mismatched revisions. + - Includes restart command guidance for each mismatched graph. + - Treats missing server revision as mismatch (exact compare behavior). + - Keeps existing fail-fast behavior for script/data-dir errors. + +- `src/test/logseq/cli/command/server_test.cljs` + - Verifies `server list` still uses exact-string mismatch semantics after helper extraction. + - Confirms human mismatch metadata remains attached when expected. + +- `src/test/logseq/cli/format_test.cljs` + - Human `doctor` output includes mismatch warning and restart guidance. + - JSON/EDN doctor outputs include structured mismatch fields. + +### Regression checks + +- Existing `running-servers` warning behavior is preserved. +- Existing `server list` mismatch warning behavior is preserved. +- `doctor` top-level status semantics remain unchanged (`ok|warning|error` through current response model). + +--- + +## Risks and mitigations + +- **Risk:** Duplicate warning noise (`running-servers` + mismatch) in one doctor run. + - **Mitigation:** Keep distinct check IDs/messages so users can tell readiness vs revision drift. + +- **Risk:** Restart command may fail with owner restrictions. + - **Mitigation:** Keep guidance explicit but non-blocking; existing error hint `server-owned-by-other` remains the fallback behavior. + +- **Risk:** Missing revision in legacy lock files increases warning count. + - **Mitigation:** Treat as mismatch by design; message should clearly indicate missing server revision value. + +--- + +## Acceptance criteria + +- `logseq doctor` emits a warning check when any discovered server revision differs from local CLI revision. +- Warning includes actionable restart guidance: `logseq server restart --graph ` for each mismatched graph. +- No mismatch warning when all discovered server revisions exactly equal CLI revision. +- `server list` mismatch behavior remains consistent with `doctor` (shared comparison semantics). +- `doctor --output json|edn` includes structured mismatch check data. + +--- + +## File scope (expected) + +- `src/main/logseq/cli/command/doctor.cljs` +- `src/main/logseq/cli/command/server.cljs` +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/format.cljs` (if dedicated doctor warning rendering is added) +- `src/test/logseq/cli/command/doctor_test.cljs` +- `src/test/logseq/cli/command/server_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `docs/cli/logseq-cli.md` +- `docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md` diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index cba22fbadf..bb36149e2e 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -247,12 +247,13 @@ Output formats: ``` - JSON example: `{"status":"ok","data":{"result":[123]}}` - EDN example: `{:status :ok, :data {:result [123]}}` -- `doctor` output includes overall status (`ok`, `warning`, `error`) and per-check rows for `db-worker-script`, `data-dir`, and `running-servers`. For scripting, `--output json|edn` keeps the structured check payload. -- Common doctor failures: +- `doctor` output includes overall status (`ok`, `warning`, `error`) and per-check rows for `db-worker-script`, `data-dir`, `running-servers`, and `server-revision-mismatch`. For scripting, `--output json|edn` keeps the structured check payload. +- Common doctor failures and warnings: - `doctor-script-missing`: `db-worker-node.js` runtime target is missing (default target: `dist/db-worker-node.js`; use `doctor --dev-script` to check `static/db-worker-node.js`). - `doctor-script-unreadable`: script path exists but is not a readable file. - `data-dir-permission`: configured data dir is not readable or writable. - `doctor-server-not-ready`: one or more lock-discovered servers are still in `:starting` state (warning). + - `doctor-server-revision-mismatch`: one or more discovered servers use a different revision than the local CLI revision (warning). Follow the printed remediation command for each affected graph: `logseq server restart --graph `. - If bundled runtime startup fails with missing-module or missing-file errors, rebuild with `yarn db-worker-node:release:bundle` and confirm `dist/db-worker-node.js` exists and every path listed in `dist/db-worker-node-assets.json` is present next to it. - `query` human output returns a plain string (the query result rendered via `pr-str`), which is convenient for pipelines like `logseq query ... | xargs logseq show --id`. - Built-in named queries currently include `block-search`, `task-search`, `recent-updated`, `list-status`, and `list-priority`. Use `query list` to see the full set for your config. diff --git a/src/main/logseq/cli/command/doctor.cljs b/src/main/logseq/cli/command/doctor.cljs index b41bee879e..0a5bad921f 100644 --- a/src/main/logseq/cli/command/doctor.cljs +++ b/src/main/logseq/cli/command/doctor.cljs @@ -5,6 +5,7 @@ [logseq.cli.command.core :as core] [logseq.cli.data-dir :as data-dir] [logseq.cli.server :as cli-server] + [logseq.cli.version :as version] [promesa.core :as p])) (def entries @@ -101,6 +102,7 @@ (if (seq starting) {:ok? true :warning? true + :servers servers :check {:id :running-servers :status :warning :code :doctor-server-not-ready @@ -112,6 +114,7 @@ (string/join ", " (map :repo starting)))}} {:ok? true :warning? false + :servers servers :check {:id :running-servers :status :ok :servers servers @@ -126,6 +129,37 @@ :message (or (.-message e) "running server check failed")}})))) +(defn- check-server-revision-mismatch + [cli-revision servers] + (let [revision-mismatch (cli-server/compute-revision-mismatches cli-revision servers) + mismatch-servers (mapv (fn [{:keys [repo revision]}] + {:repo repo + :graph (core/repo->graph repo) + :revision revision}) + (or (:servers revision-mismatch) [])) + mismatch-count (count mismatch-servers)] + (if (pos? mismatch-count) + {:ok? true + :warning? true + :check {:id :server-revision-mismatch + :status :warning + :code :doctor-server-revision-mismatch + :cli-revision cli-revision + :servers mismatch-servers + :message (str mismatch-count + " server" + (when (> mismatch-count 1) "s") + " " + (if (= 1 mismatch-count) "uses" "use") + " a different revision than this CLI")}} + {:ok? true + :warning? false + :check {:id :server-revision-mismatch + :status :ok + :cli-revision cli-revision + :servers [] + :message "All discovered servers match this CLI revision"}}))) + (defn execute-doctor [action config] (p/let [script-check (check-db-worker-script action)] @@ -138,11 +172,20 @@ (let [check (:check data-dir-check) checks (conj checks check)] (doctor-error checks (:code check) (:message check))) - (p/let [server-check (check-running-servers config) - checks (conj checks (:check data-dir-check) (:check server-check))] + (p/let [server-check (check-running-servers config)] (if-not (:ok? server-check) - (let [check (:check server-check)] + (let [checks (conj checks (:check data-dir-check) (:check server-check)) + check (:check server-check)] (doctor-error checks (:code check) (:message check))) - {:status :ok - :data {:status (if (:warning? server-check) :warning :ok) - :checks checks}}))))))) + (let [revision-check (check-server-revision-mismatch (version/revision) + (:servers server-check)) + checks (conj checks + (:check data-dir-check) + (:check server-check) + (:check revision-check))] + {:status :ok + :data {:status (if (or (:warning? server-check) + (:warning? revision-check)) + :warning + :ok) + :checks checks}})))))))) diff --git a/src/main/logseq/cli/command/server.cljs b/src/main/logseq/cli/command/server.cljs index bd3d8e06e3..a7e2360422 100644 --- a/src/main/logseq/cli/command/server.cljs +++ b/src/main/logseq/cli/command/server.cljs @@ -70,22 +70,10 @@ {:status :error :error (:error result)})) -(defn- compute-revision-mismatches - [cli-revision servers] - (let [mismatch-servers (->> (or servers []) - (filter (fn [{:keys [revision]}] - (not= cli-revision revision))) - (mapv (fn [{:keys [repo revision]}] - {:repo repo - :revision revision})))] - (when (seq mismatch-servers) - {:cli-revision cli-revision - :servers mismatch-servers}))) - (defn execute-list [_action config] (-> (p/let [servers (cli-server/list-servers config) - revision-mismatch (compute-revision-mismatches (version/revision) servers)] + revision-mismatch (cli-server/compute-revision-mismatches (version/revision) servers)] (cond-> {:status :ok :data {:servers servers}} revision-mismatch diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index a8ad6a4da8..75b935c536 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -527,16 +527,32 @@ "Updated")] (str verb " graph " (pr-str graph)))) +(defn- quote-cli-arg + [value] + (let [value (normalize-cell value)] + (if (re-find #"\s" value) + (str "\"" (string/replace value #"\"" "\\\"") "\"") + value))) + +(defn- format-doctor-check + [{:keys [id status message servers]}] + (let [check-line (str "[" (name (or status :unknown)) + "] " + (name (or id :unknown)) + (when (seq message) + (str " - " message)))] + (if (= :server-revision-mismatch id) + (let [guidance-lines (mapv (fn [{:keys [graph repo]}] + (str " Run: logseq server restart --graph " + (quote-cli-arg (or graph repo)))) + (or servers []))] + (into [check-line] guidance-lines)) + [check-line]))) + (defn- format-doctor [status checks] (let [header (str "Doctor: " (name (or status :unknown))) - check-lines (mapv (fn [{:keys [id status message]}] - (str "[" (name (or status :unknown)) - "] " - (name (or id :unknown)) - (when (seq message) - (str " - " message)))) - (or checks []))] + check-lines (mapcat format-doctor-check (or checks []))] (string/join "\n" (into [header] check-lines)))) (defn- ->human diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 3c13b1c48e..4688ffa4b5 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -298,6 +298,18 @@ :revision (:revision lock) :status (if ready :ready :starting)}))))) +(defn compute-revision-mismatches + [cli-revision servers] + (let [mismatch-servers (->> (or servers []) + (filter (fn [{:keys [revision]}] + (not= cli-revision revision))) + (mapv (fn [{:keys [repo revision]}] + {:repo repo + :revision revision})))] + (when (seq mismatch-servers) + {:cli-revision cli-revision + :servers mismatch-servers}))) + (defn list-graphs [config] (let [data-dir (resolve-data-dir config) diff --git a/src/test/logseq/cli/command/doctor_test.cljs b/src/test/logseq/cli/command/doctor_test.cljs index df7b33b596..95249238fb 100644 --- a/src/test/logseq/cli/command/doctor_test.cljs +++ b/src/test/logseq/cli/command/doctor_test.cljs @@ -4,6 +4,7 @@ [logseq.cli.commands :as commands] [logseq.cli.data-dir :as data-dir] [logseq.cli.server :as cli-server] + [logseq.cli.version :as cli-version] [promesa.core :as p] [logseq.cli.command.doctor :as doctor-command])) @@ -60,15 +61,16 @@ (deftest test-execute-doctor-all-checks-pass (async done (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-version/revision (fn [] "cli-rev") cli-server/list-servers (fn [_] (p/resolved []))] (p/let [result (commands/execute {:type :doctor :script-path "src/main/logseq/cli/commands.cljs"} {:data-dir "/tmp/logseq-doctor"})] (is (= :ok (:status result))) (is (= :ok (get-in result [:data :status]))) - (is (= [:db-worker-script :data-dir :running-servers] + (is (= [:db-worker-script :data-dir :running-servers :server-revision-mismatch] (mapv :id (get-in result [:data :checks])))) - (is (= [:ok :ok :ok] + (is (= [:ok :ok :ok :ok] (mapv :status (get-in result [:data :checks])))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) @@ -77,11 +79,13 @@ (deftest test-execute-doctor-starting-server-warning (async done (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-version/revision (fn [] "cli-rev") cli-server/list-servers (fn [_] (p/resolved [{:repo "logseq_db_demo" :status :starting :host "127.0.0.1" - :port 9010}]))] + :port 9010 + :revision "cli-rev"}]))] (p/let [result (commands/execute {:type :doctor :script-path "src/main/logseq/cli/commands.cljs"} {:data-dir "/tmp/logseq-doctor"})] @@ -92,7 +96,62 @@ (is (= :warning (get-in result [:data :checks 2 :status]))) (is (= :doctor-server-not-ready - (get-in result [:data :checks 2 :code]))))) + (get-in result [:data :checks 2 :code]))) + (is (= :server-revision-mismatch + (get-in result [:data :checks 3 :id]))) + (is (= :ok + (get-in result [:data :checks 3 :status]))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) + +(deftest test-execute-doctor-server-revision-mismatch-warning + (async done + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-version/revision (fn [] "cli-rev") + cli-server/list-servers (fn [_] + (p/resolved [{:repo "logseq_db_graph-a" + :status :ready + :revision "cli-rev"} + {:repo "logseq_db_graph-b" + :status :ready + :revision "worker-rev"}]))] + (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/logseq-doctor"})] + (is (= :ok (:status result))) + (is (= :warning (get-in result [:data :status]))) + (is (= :server-revision-mismatch + (get-in result [:data :checks 3 :id]))) + (is (= :warning + (get-in result [:data :checks 3 :status]))) + (is (= :doctor-server-revision-mismatch + (get-in result [:data :checks 3 :code]))) + (is (= "cli-rev" + (get-in result [:data :checks 3 :cli-revision]))) + (is (= ["logseq_db_graph-b"] + (mapv :repo (get-in result [:data :checks 3 :servers])))) + (is (= ["graph-b"] + (mapv :graph (get-in result [:data :checks 3 :servers])))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) + +(deftest test-execute-doctor-missing-server-revision-is-mismatch + (async done + (-> (p/with-redefs [data-dir/ensure-data-dir! (fn [_] "/tmp/logseq-doctor") + cli-version/revision (fn [] "cli-rev") + cli-server/list-servers (fn [_] + (p/resolved [{:repo "logseq_db_graph-a" + :status :ready}]))] + (p/let [result (commands/execute {:type :doctor + :script-path "src/main/logseq/cli/commands.cljs"} + {:data-dir "/tmp/logseq-doctor"})] + (is (= :ok (:status result))) + (is (= :warning (get-in result [:data :status]))) + (is (= :warning + (get-in result [:data :checks 3 :status]))) + (is (nil? (get-in result [:data :checks 3 :servers 0 :revision]))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done)))) diff --git a/src/test/logseq/cli/command/server_test.cljs b/src/test/logseq/cli/command/server_test.cljs index 02839022ae..d25914817e 100644 --- a/src/test/logseq/cli/command/server_test.cljs +++ b/src/test/logseq/cli/command/server_test.cljs @@ -6,8 +6,7 @@ [promesa.core :as p])) (deftest compute-revision-mismatches-uses-exact-string-compare - (let [compute-revision-mismatches #'server-command/compute-revision-mismatches - mismatch (compute-revision-mismatches + (let [mismatch (cli-server/compute-revision-mismatches "cli-rev" [{:repo "graph-a" :revision "cli-rev"} {:repo "graph-b" :revision "cli-rev-dirty"}])] diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 873d4f095d..2985f9d4ec 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -811,6 +811,22 @@ "[ok] db-worker-script - Found readable file: /tmp/db-worker-node.js\n" "[ok] data-dir - Read/write access confirmed: /tmp/logseq/graphs\n" "[warning] running-servers - 1 server is still starting") + result)))) + + (testing "doctor includes restart guidance for revision mismatch" + (let [result (format/format-result {:status :ok + :command :doctor + :data {:status :warning + :checks [{:id :server-revision-mismatch + :status :warning + :message "2 servers use a different revision than this CLI" + :servers [{:graph "graph-a"} + {:graph "team graph"}]}]}} + {:output-format nil})] + (is (= (str "Doctor: warning\n" + "[warning] server-revision-mismatch - 2 servers use a different revision than this CLI\n" + " Run: logseq server restart --graph graph-a\n" + " Run: logseq server restart --graph \"team graph\"") result))))) (deftest test-doctor-json-edn-output @@ -844,3 +860,36 @@ (is (= :data-dir-permission (get-in parsed-edn [:error :code]))) (is (= :data-dir (get-in parsed-edn [:data :checks 1 :id]))) (is (= :error (get-in parsed-edn [:data :checks 1 :status])))))) + +(deftest test-doctor-json-edn-output-includes-revision-mismatch-check + (let [payload {:status :warning + :checks [{:id :server-revision-mismatch + :status :warning + :code :doctor-server-revision-mismatch + :cli-revision "cli-rev" + :servers [{:repo "logseq_db_graph-b" + :graph "logseq_db_graph-b" + :revision "worker-rev"}] + :message "1 server uses a different revision than this CLI"}]} + json-result (format/format-result {:status :ok + :command :doctor + :data payload} + {:output-format :json}) + edn-result (format/format-result {:status :ok + :command :doctor + :data payload} + {:output-format :edn}) + parsed-json (js->clj (js/JSON.parse json-result) :keywordize-keys true) + parsed-edn (reader/read-string edn-result)] + (is (= "ok" (:status parsed-json))) + (is (= "warning" (get-in parsed-json [:data :status]))) + (is (= "server-revision-mismatch" (get-in parsed-json [:data :checks 0 :id]))) + (is (= "doctor-server-revision-mismatch" (get-in parsed-json [:data :checks 0 :code]))) + (is (= "graph-b" (get-in parsed-json [:data :checks 0 :servers 0 :repo]))) + (is (= "graph-b" (get-in parsed-json [:data :checks 0 :servers 0 :graph]))) + (is (= :ok (:status parsed-edn))) + (is (= :warning (get-in parsed-edn [:data :status]))) + (is (= :server-revision-mismatch (get-in parsed-edn [:data :checks 0 :id]))) + (is (= :doctor-server-revision-mismatch (get-in parsed-edn [:data :checks 0 :code]))) + (is (= "graph-b" (get-in parsed-edn [:data :checks 0 :servers 0 :repo]))) + (is (= "graph-b" (get-in parsed-edn [:data :checks 0 :servers 0 :graph]))))) From 4b428094e1b44defde71edf10a5b7b0a1c258631 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Thu, 12 Mar 2026 23:09:06 +0800 Subject: [PATCH 145/375] chore: remove unused LOGIN-URL --- .../src/logseq/common/cognito_config.cljs | 30 ++++--------------- src/main/frontend/config.cljs | 1 - src/main/frontend/handler/events/ui.cljs | 10 +++---- src/main/logseq/cli/auth.cljs | 7 +---- .../logseq/common/cognito_config_test.cljs | 1 - 5 files changed, 10 insertions(+), 39 deletions(-) diff --git a/deps/common/src/logseq/common/cognito_config.cljs b/deps/common/src/logseq/common/cognito_config.cljs index 71824e50d5..34b0236df3 100644 --- a/deps/common/src/logseq/common/cognito_config.cljs +++ b/deps/common/src/logseq/common/cognito_config.cljs @@ -1,27 +1,12 @@ (ns logseq.common.cognito-config - "Shared Cognito configuration for frontend and CLI-safe consumers." - (:require [clojure.string :as string])) + "Shared Cognito configuration for frontend and CLI-safe consumers.") (goog-define ENABLE-FILE-SYNC-PRODUCTION false) -(def ^:private prod-login-url - "https://logseq-prod.auth.us-east-1.amazoncognito.com/login?client_id=3c7np6bjtb4r1k1bi9i049ops5&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") - -(def ^:private test-login-url - "https://logseq-test2.auth.us-east-2.amazoncognito.com/login?client_id=3ji1a0059hspovjq5fhed3uil8&response_type=code&scope=email+openid+phone&redirect_uri=logseq%3A%2F%2Fauth-callback") - -(def LOGIN-URL - (if ENABLE-FILE-SYNC-PRODUCTION - prod-login-url - test-login-url)) - (def COGNITO-CLIENT-ID - (or (some-> (js/URL. LOGIN-URL) - .-searchParams - (.get "client_id")) - (if ENABLE-FILE-SYNC-PRODUCTION - "69cs1lgme7p8kbgld8n5kseii6" - "1qi1uijg8b6ra70nejvbptis0q"))) + (if ENABLE-FILE-SYNC-PRODUCTION + "69cs1lgme7p8kbgld8n5kseii6" + "1qi1uijg8b6ra70nejvbptis0q")) (def CLI-COGNITO-CLIENT-ID (if ENABLE-FILE-SYNC-PRODUCTION @@ -33,9 +18,4 @@ "logseq-prod.auth.us-east-1.amazoncognito.com" "logseq-test2.auth.us-east-2.amazoncognito.com")) -(def OAUTH-SCOPE - (or (some-> (js/URL. LOGIN-URL) - .-searchParams - (.get "scope") - (string/replace #"\+" " ")) - "email openid phone")) +(def OAUTH-SCOPE "email openid phone") diff --git a/src/main/frontend/config.cljs b/src/main/frontend/config.cljs index f7588a4184..e193e852ee 100644 --- a/src/main/frontend/config.cljs +++ b/src/main/frontend/config.cljs @@ -25,7 +25,6 @@ ;; when it launches (when pro plan launches) it should be removed (def ENABLE-SETTINGS-ACCOUNT-TAB false) -(def LOGIN-URL cognito-config/LOGIN-URL) (def COGNITO-CLIENT-ID cognito-config/COGNITO-CLIENT-ID) (def OAUTH-DOMAIN cognito-config/OAUTH-DOMAIN) diff --git a/src/main/frontend/handler/events/ui.cljs b/src/main/frontend/handler/events/ui.cljs index e3fbfe84f9..12d3d82a0b 100644 --- a/src/main/frontend/handler/events/ui.cljs +++ b/src/main/frontend/handler/events/ui.cljs @@ -312,12 +312,10 @@ (defmethod events/handle :user/logout [[_]] (login/sign-out!)) -(defmethod events/handle :user/login [[_ host-ui?]] - (if (or host-ui? (not util/electron?)) - (js/window.open config/LOGIN-URL) - (if (mobile-util/native-platform?) - (route-handler/redirect! {:to :user-login}) - (login/open-login-modal!)))) +(defmethod events/handle :user/login [[_]] + (if (mobile-util/native-platform?) + (route-handler/redirect! {:to :user-login}) + (login/open-login-modal!))) (defmethod events/handle :asset/dialog-edit-external-url [[_ asset-block pdf-current]] (shui/dialog-open! diff --git a/src/main/logseq/cli/auth.cljs b/src/main/logseq/cli/auth.cljs index 9c8dd95032..2655a48885 100644 --- a/src/main/logseq/cli/auth.cljs +++ b/src/main/logseq/cli/auth.cljs @@ -48,18 +48,13 @@ [text] (js->clj (js/JSON.parse text) :keywordize-keys true)) -(defn- login-url - [] - (js/URL. cognito-config/LOGIN-URL)) - (defn- oauth-client-id [] cognito-config/CLI-COGNITO-CLIENT-ID) (defn- oauth-scope [] - (or (.get (.-searchParams (login-url)) "scope") - cognito-config/OAUTH-SCOPE + (or cognito-config/OAUTH-SCOPE default-scope)) (defn- oauth-domain diff --git a/src/test/logseq/common/cognito_config_test.cljs b/src/test/logseq/common/cognito_config_test.cljs index cea2e91fe1..e8b9d692f1 100644 --- a/src/test/logseq/common/cognito_config_test.cljs +++ b/src/test/logseq/common/cognito_config_test.cljs @@ -5,7 +5,6 @@ ["fs" :as fs])) (deftest test-shared-cognito-config-matches-frontend-config - (is (= config/LOGIN-URL cognito-config/LOGIN-URL)) (is (= config/COGNITO-CLIENT-ID cognito-config/COGNITO-CLIENT-ID)) (is (= config/OAUTH-DOMAIN cognito-config/OAUTH-DOMAIN))) From 630380c482b9fd6b790e95d54fff3f0c17c62343 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Mar 2026 17:37:10 +0800 Subject: [PATCH 146/375] fix: auth-token reset to nil when 'sync status' --- src/main/logseq/cli/command/sync.cljs | 28 +++++++++---- src/test/logseq/cli/command/sync_test.cljs | 48 ++++++++++++++++++++-- 2 files changed, 65 insertions(+), 11 deletions(-) diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index a3bb5e0633..0029d3a6c1 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -225,20 +225,33 @@ (assoc config :auth-token auth-token))) (p/resolved config))) +(defn- resolve-sync-config-for-worker! + [cfg config] + (let [sync-cfg (sync-config config)] + (if (seq (:auth-token sync-cfg)) + (p/resolved sync-cfg) + (-> (p/let [current-sync-cfg (transport/invoke cfg :thread-api/get-db-sync-config false []) + auth-token (:auth-token current-sync-cfg)] + (if (seq auth-token) + (assoc sync-cfg :auth-token auth-token) + sync-cfg)) + (p/catch (fn [_error] + sync-cfg)))))) + (defn- invoke-with-repo [config repo method args] - (let [sync-cfg (sync-config config)] - (p/let [cfg (cli-server/ensure-server! config repo) - _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-cfg]) + (p/let [cfg (cli-server/ensure-server! config repo) + sync-cfg (resolve-sync-config-for-worker! cfg config) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-cfg]) result (transport/invoke cfg method false args)] - result))) + result)) (defn- invoke-global [config method args] - (let [base-url (:base-url config) - sync-cfg (sync-config config)] + (let [base-url (:base-url config)] (if (seq base-url) - (p/let [_ (transport/invoke config :thread-api/set-db-sync-config false [sync-cfg])] + (p/let [sync-cfg (resolve-sync-config-for-worker! config config) + _ (transport/invoke config :thread-api/set-db-sync-config false [sync-cfg])] (transport/invoke config method false args)) (p/let [repo (or (core/resolve-repo (:graph config)) (p/let [graphs (cli-server/list-graphs config)] @@ -247,6 +260,7 @@ (cli-server/ensure-server! config repo) (p/rejected (ex-info "graph name is required" {:code :missing-graph}))) + sync-cfg (resolve-sync-config-for-worker! cfg config) _ (transport/invoke cfg :thread-api/set-db-sync-config false [sync-cfg])] (transport/invoke cfg method false args))))) diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index 79de171639..6c3f339ddc 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -190,7 +190,7 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) -(deftest test-execute-sync-stop +(deftest test-execute-sync-stop-preserves-existing-worker-auth-token (async done (let [ensure-calls (atom []) invoke-calls (atom [])] @@ -199,16 +199,20 @@ (p/resolved (assoc config :base-url "http://example"))) transport/invoke (fn [_ method direct-pass? args] (swap! invoke-calls conj [method direct-pass? args]) - (p/resolved {:ok true}))] + (case method + :thread-api/get-db-sync-config (p/resolved {:auth-token "worker-token"}) + :thread-api/db-sync-stop (p/resolved {:ok true}) + (p/resolved nil)))] (p/let [_ (sync-command/execute {:type :sync-stop :repo "logseq_db_demo"} {:data-dir "/tmp"})] (is (= [[{:data-dir "/tmp"} "logseq_db_demo"]] @ensure-calls)) - (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil + (is (= [[:thread-api/get-db-sync-config false []] + [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil - :auth-token nil + :auth-token "worker-token" :e2ee-password nil}]] [:thread-api/db-sync-stop false []]] @invoke-calls)))) @@ -216,6 +220,42 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) +(deftest test-execute-sync-status-preserves-existing-worker-auth-token + (async done + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/get-db-sync-config (p/resolved {:auth-token "worker-token"}) + :thread-api/db-sync-status (p/resolved {:repo "logseq_db_demo" + :ws-state :open + :pending-local 0 + :pending-asset 0 + :pending-server 0}) + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-status + :repo "logseq_db_demo"} + {:data-dir "/tmp"})] + (is (= :ok (:status result))) + (is (= :open (get-in result [:data :ws-state]))) + (is (= [[{:data-dir "/tmp"} + "logseq_db_demo"]] + @ensure-calls)) + (is (= [[:thread-api/get-db-sync-config false []] + [:thread-api/set-db-sync-config false [{:ws-url nil + :http-base nil + :auth-token "worker-token" + :e2ee-password nil}]] + [:thread-api/db-sync-status false ["logseq_db_demo"]]] + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest test-execute-sync-upload (async done (let [ensure-calls (atom []) From da4178d0408e83b4e14329cfb6b27e0ee68f3c05 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Mar 2026 17:51:58 +0800 Subject: [PATCH 147/375] fix: update sync-start-poll-interval-ms --- src/main/logseq/cli/command/sync.cljs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index 0029d3a6c1..896443ef10 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -43,7 +43,7 @@ :sync-grant-access}) (def ^:private sync-start-timeout-ms 10000) -(def ^:private sync-start-poll-interval-ms 100) +(def ^:private sync-start-poll-interval-ms 1000) (def ^:private sync-download-timeout-ms (* 30 60 1000)) (def ^:private structured-output-formats #{:json :edn}) From 78de33bf683cb4c50f73ebd1b110fcc4e1b75de7 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Mar 2026 22:30:06 +0800 Subject: [PATCH 148/375] enhance: 'upsert block' support update --content and --status --- docs/cli/logseq-cli.md | 3 +- src/main/logseq/cli/command/add.cljs | 2 +- src/main/logseq/cli/command/update.cljs | 69 +++++++++++++++++------ src/main/logseq/cli/command/upsert.cljs | 4 +- src/test/logseq/cli/commands_test.cljs | 64 ++++++++++++++++++++- src/test/logseq/cli/integration_test.cljs | 48 ++++++++++++++++ 6 files changed, 165 insertions(+), 25 deletions(-) diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index bb36149e2e..8c19ca5ac6 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -182,7 +182,8 @@ Inspect and edit commands: - `upsert block --content [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - create blocks; defaults to today’s journal page if no target is given - `upsert block --blocks [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector - `upsert block --blocks-file [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file -- `upsert block --id |--uuid [--target-id |--target-uuid |--target-page ] [--pos first-child|last-child|sibling] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update and/or move a block +- `upsert block --id |--uuid [--content ] [--status ] [--target-id |--target-uuid |--target-page ] [--pos first-child|last-child|sibling] [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update and/or move a block + - When both `--status` and `--update-properties` set `:logseq.property/status`, the value from `--update-properties` takes precedence. - `upsert page --page [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - create (or update by page name) a page - `upsert page --id [--update-tags ] [--update-properties ] [--remove-tags ] [--remove-properties ]` - update a page by id (cannot be combined with `--page`) - `upsert tag --name ` - create or upsert a tag by name diff --git a/src/main/logseq/cli/command/add.cljs b/src/main/logseq/cli/command/add.cljs index 1bc62a057b..ef6c42fb49 100644 --- a/src/main/logseq/cli/command/add.cljs +++ b/src/main/logseq/cli/command/add.cljs @@ -57,7 +57,7 @@ "in progress" :logseq.property/status.doing "inprogress" :logseq.property/status.doing}) -(defn- normalize-status +(defn normalize-status [value] (let [text (some-> value string/trim) parsed (when (and (seq text) (string/starts-with? text ":")) diff --git a/src/main/logseq/cli/command/update.cljs b/src/main/logseq/cli/command/update.cljs index 8e3f5e8fa4..82cd745654 100644 --- a/src/main/logseq/cli/command/update.cljs +++ b/src/main/logseq/cli/command/update.cljs @@ -22,7 +22,14 @@ has-update-properties? (seq (some-> (:update-properties opts) string/trim)) has-remove-tags? (seq (some-> (:remove-tags opts) string/trim)) has-remove-properties? (seq (some-> (:remove-properties opts) string/trim)) - has-updates? (or has-update-tags? has-update-properties? has-remove-tags? has-remove-properties?)] + has-status? (seq (some-> (:status opts) string/trim)) + has-content? (contains? opts :content) + has-updates? (or has-update-tags? + has-update-properties? + has-remove-tags? + has-remove-properties? + has-status? + has-content?)] (cond (and (seq pos) (not (contains? update-positions pos))) (str "invalid pos: " (:pos opts)) @@ -134,6 +141,12 @@ target-uuid (some-> (:target-uuid options) string/trim) page-name (some-> (:target-page options) string/trim) pos (some-> (:pos options) string/trim string/lower-case) + status-provided? (contains? options :status) + status-text (some-> (:status options) string/trim) + status (when (seq status-text) + (add-command/normalize-status status-text)) + content-provided? (contains? options :content) + content (:content options) update-tags-result (add-command/parse-tags-option (:update-tags options)) update-properties-result (add-command/parse-properties-option (:update-properties options) @@ -144,10 +157,19 @@ {:allow-non-built-in? true}) update-tags (:value update-tags-result) update-properties (:value update-properties-result) + update-properties (merge (cond-> {} + status + (assoc :logseq.property/status status)) + (or update-properties {})) remove-tags (:value remove-tags-result) remove-properties (:value remove-properties-result) has-target? (or (some? target-id) (seq target-uuid) (seq page-name)) - has-updates? (or (seq update-tags) (seq update-properties) (seq remove-tags) (seq remove-properties)) + has-updates? (or (seq update-tags) + (seq update-properties) + (seq remove-tags) + (seq remove-properties) + status-provided? + content-provided?) source-label (cond (seq uuid) uuid (some? id) (str id) @@ -163,6 +185,11 @@ :error {:code :missing-source :message "source block is required"}} + (and status-provided? (not status)) + {:ok? false + :error {:code :invalid-options + :message (str "invalid status: " (:status options))}} + (and (not has-target?) (not has-updates?)) {:ok? false :error {:code :invalid-options @@ -182,21 +209,22 @@ :else {:ok? true - :action {:type :update-block - :repo repo - :graph (core/repo->graph repo) - :id id - :uuid uuid - :target-id target-id - :target-uuid target-uuid - :target-page page-name - :pos (when has-target? (or pos "first-child")) - :update-tags update-tags - :update-properties update-properties - :remove-tags remove-tags - :remove-properties remove-properties - :source source-label - :target target-label}})))) + :action (cond-> {:type :update-block + :repo repo + :graph (core/repo->graph repo) + :id id + :uuid uuid + :target-id target-id + :target-uuid target-uuid + :target-page page-name + :pos (when has-target? (or pos "first-child")) + :update-tags update-tags + :update-properties update-properties + :remove-tags remove-tags + :remove-properties remove-properties + :source source-label + :target target-label} + content-provided? (assoc :content content))})))) (defn execute-update [action config] @@ -217,12 +245,17 @@ {:allow-non-built-in? true}) block-id (:db/id source) block-ids [block-id] + content-provided? (contains? action :content) + content (:content action) update-tag-ids (when (seq update-tags) (->> update-tags (map :db/id) (remove nil?) vec)) remove-tag-ids (when (seq remove-tags) (->> remove-tags (map :db/id) (remove nil?) vec)) ops (cond-> [] - target (conj [:move-blocks [[(:db/id source)] (:db/id target) opts]])) + content-provided? + (conj [:save-block [{:db/id block-id :block/title content} {}]]) + target + (conj [:move-blocks [[(:db/id source)] (:db/id target) opts]])) ops (cond-> ops (seq remove-tag-ids) (into (map (fn [tag-id] diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index a77b7a8cbb..b4583a69e1 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -20,11 +20,11 @@ :complete :pages} :pos {:desc "Position. Default: create=last-child, update=first-child" :values ["first-child" "last-child" "sibling"]} - :content {:desc "Block content for create mode"} + :content {:desc "Block content (create inserts; update rewrites source block content)"} :blocks {:desc "EDN vector of blocks for create mode"} :blocks-file {:desc "EDN file of blocks for create mode" :complete :file} - :status {:desc "Task status" + :status {:desc "Task status (create/update)" :values ["todo" "doing" "done" "now" "later" "wait" "waiting" "backlog" "canceled" "cancelled" "in-review" "in-progress"]} :update-tags {:desc "Tags to add/update (EDN vector)"} diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 420c778517..eac8e6ffb5 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -1029,6 +1029,22 @@ (is (= :upsert-block (:command result))) (is (= "abc" (get-in result [:options :uuid]))))) + (testing "upsert block update mode accepts status-only updates" + (let [result (commands/parse-args ["upsert" "block" "--id" "1" + "--status" "done"])] + (is (true? (:ok? result))) + (is (= :upsert-block (:command result))) + (is (= 1 (get-in result [:options :id]))) + (is (= "done" (get-in result [:options :status]))))) + + (testing "upsert block update mode accepts content-only updates" + (let [result (commands/parse-args ["upsert" "block" "--id" "1" + "--content" "updated text"])] + (is (true? (:ok? result))) + (is (= :upsert-block (:command result))) + (is (= 1 (get-in result [:options :id]))) + (is (= "updated text" (get-in result [:options :content]))))) + (testing "upsert block forces update mode when id and content are both provided" (let [result (commands/parse-args ["upsert" "block" "--id" "1" @@ -1623,6 +1639,44 @@ (is (= :upsert-block (get-in result [:action :type]))) (is (= ["TagA"] (get-in result [:action :update-tags]))))) + (testing "update accepts status-only updates" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 :status "in-progress"}} + result (commands/build-action parsed {:graph "demo"})] + (is (true? (:ok? result))) + (is (= :upsert-block (get-in result [:action :type]))) + (is (= :logseq.property/status.doing + (get-in result [:action :update-properties :logseq.property/status]))))) + + (testing "update accepts content-only updates" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 :content "updated text"}} + result (commands/build-action parsed {:graph "demo"})] + (is (true? (:ok? result))) + (is (= :upsert-block (get-in result [:action :type]))) + (is (= "updated text" (get-in result [:action :content]))))) + + (testing "update rejects invalid status" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 :status "invalid-status"}} + result (commands/build-action parsed {:graph "demo"})] + (is (false? (:ok? result))) + (is (= :invalid-options (get-in result [:error :code]))))) + + (testing "update properties status takes precedence over --status" + (let [parsed {:ok? true + :command :upsert-block + :options {:id 1 + :status "doing" + :update-properties "{:logseq.property/status :logseq.property/status.done}"}} + result (commands/build-action parsed {:graph "demo"})] + (is (true? (:ok? result))) + (is (= :logseq.property/status.done + (get-in result [:action :update-properties :logseq.property/status]))))) + (testing "update rejects invalid update tags" (let [parsed {:ok? true :command :upsert-block @@ -2200,9 +2254,11 @@ (let [ops* (atom nil) calls* (atom []) action {:type :upsert-block :mode :update :repo "demo" :id 1 :target-id 2 :pos "last-child" + :content "Updated heading" :update-tags [:tag/new] :remove-tags [:tag/old] - :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z"} + :update-properties {:logseq.property/deadline "2026-01-25T12:00:00Z" + :logseq.property/status :logseq.property/status.done} :remove-properties [:logseq.property/publishing-public?]}] (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["demo"]) cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) @@ -2226,11 +2282,13 @@ (throw (ex-info "unexpected invoke" {:method method :calls @calls*}))))] (p/let [result (commands/execute action {})] (is (= :ok (:status result))) - (is (= [[:move-blocks [[1] 2 {:sibling? false :bottom? true}]] + (is (= [[:save-block [{:db/id 1 :block/title "Updated heading"} {}]] + [:move-blocks [[1] 2 {:sibling? false :bottom? true}]] [:batch-delete-property-value [[1] :block/tags 202]] [:batch-remove-property [[1] :logseq.property/publishing-public?]] [:batch-set-property [[1] :block/tags 101 {}]] - [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]]] + [:batch-set-property [[1] :logseq.property/deadline "2026-01-25T12:00:00Z" {}]] + [:batch-set-property [[1] :logseq.property/status :logseq.property/status.done {}]]] @ops*)))) (p/catch (fn [e] (is false (str "unexpected error: " e " calls: " @calls*)))) (p/finally done))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index c9e552e61d..9c41a48e09 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -2043,6 +2043,54 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-upsert-block-update-content-and-status + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-update-content-status")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--graph" "update-content-status-graph"] data-dir cfg-path) + _ (run-cli ["--graph" "update-content-status-graph" "upsert" "page" "--page" "Tasks"] data-dir cfg-path) + create-result (run-cli ["--graph" "update-content-status-graph" + "upsert" "block" + "--target-page" "Tasks" + "--content" "Task Seed" + "--status" "todo"] + data-dir cfg-path) + create-payload (parse-json-output create-result) + block-id (first-result-id create-payload) + update-result (run-cli ["--graph" "update-content-status-graph" + "upsert" "block" + "--id" (str block-id) + "--content" "Task Updated" + "--status" "done"] + data-dir cfg-path) + update-payload (parse-json-output update-result) + _ (p/delay 100) + query-title-result (run-cli ["--graph" "update-content-status-graph" + "query" + "--query" "[:find ?title . :in $ ?id :where [?id :block/title ?title]]" + "--inputs" (pr-str [block-id])] + data-dir cfg-path) + query-title-payload (parse-json-output query-title-result) + query-status-result (run-cli ["--graph" "update-content-status-graph" + "query" + "--query" "[:find ?ident . :in $ ?id :where [?id :logseq.property/status ?status] [?status :db/ident ?ident]]" + "--inputs" (pr-str [block-id])] + data-dir cfg-path) + query-status-payload (parse-json-output query-status-result) + stop-result (run-cli ["server" "stop" "--graph" "update-content-status-graph"] data-dir cfg-path) + stop-payload (parse-json-output stop-result)] + (is (= "ok" (:status create-payload))) + (is (some? block-id)) + (is (= "ok" (:status update-payload))) + (is (= "Task Updated" (get-in query-title-payload [:data :result]))) + (is (string/includes? (str (get-in query-status-payload [:data :result])) "status.done")) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-add-block-pos-ordering (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-add-pos")] From 1667f0059b2009cfd5fb2a7eca62fd99961a33b1 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Mar 2026 22:34:40 +0800 Subject: [PATCH 149/375] add logseq-cli skill --- skills/logseq-cli/SKILL.md | 145 ++++++++++++++++++ .../logseq-cli/references/logseq-builtins.md | 142 +++++++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 skills/logseq-cli/SKILL.md create mode 100644 skills/logseq-cli/references/logseq-builtins.md diff --git a/skills/logseq-cli/SKILL.md b/skills/logseq-cli/SKILL.md new file mode 100644 index 0000000000..4b9928286b --- /dev/null +++ b/skills/logseq-cli/SKILL.md @@ -0,0 +1,145 @@ +--- +name: logseq-cli +description: Operate the current Logseq command-line interface to inspect or modify graphs, pages, blocks, tags, and properties; run Datascript queries; show page/block trees; manage graphs; and manage db-worker-node servers. Use when a request involves running `logseq` commands or interpreting CLI output. +--- + +# Logseq CLI + +## Overview + +Use `logseq` to inspect and edit graph entities, run Datascript queries, and control graph/server lifecycle. + +## Quick start + +- Run `logseq --help` to see top-level commands and global flags. +- Run `logseq --help` to see command-specific options. +- Use `--graph` to target a specific graph. +- Omit `--output` for human output. Set `--output json` or `--output edn` only when machine-readable output is required. + +## Command groups (from `logseq --help`) + +- Graph inspect/edit: +- `list page`, `list tag`, `list property` +- `upsert block`, `upsert page`, `upsert tag`, `upsert property` +- `remove block`, `remove page`, `remove tag`, `remove property` +- `query`, `query list`, `show` +- Graph management: `graph list|create|switch|remove|validate|info|export|import` +- Server management: `server list|status|start|stop|restart` +- Diagnostics: `doctor` + +## Global options + +- `--config` Path to `cli.edn` (default `~/logseq/cli.edn`) +- `--graph` Graph name +- `--data-dir` Path to db-worker data dir (default `~/logseq/graphs`) +- `--timeout-ms` Request timeout in ms (default `10000`) +- `--output` Output format (`human`, `json`, `edn`) +- `--verbose` Enable verbose debug logging to stderr + +## Command option policy + +- Do not memorize or hardcode command options in this skill. +- Before running any command, always check live options with: +- `logseq --help` +- `logseq --help` + +## Example prerequisites + +- Replace placeholder ids/uuids in examples (`123`, `321`, `1111...`) with real entities from the target graph. +- Use `logseq list ...`, `logseq show ...`, or `logseq query ...` first to discover valid ids/uuids. +- `logseq graph export` requires `--file`; keep `graph import --input` consistent with the export path. + +## Examples + +```bash +# List pages (human output by default) +logseq list page --graph "my-graph" --limit 50 --sort updated-at --order desc + +# Include built-in tags/properties +logseq list tag --graph "my-graph" --include-built-in --limit 20 --output json +logseq list property --graph "my-graph" --include-built-in --limit 20 --output json + +# Query by built-in query name +logseq query --graph "my-graph" --name "recent-updated" --inputs "[30]" + +# Query with ad-hoc Datascript EDN +logseq query --graph "my-graph" --query "[:find [?p ...] :where [?p :block/name]]" + +# List available queries (built-ins + custom-queries from cli.edn) +logseq query list --graph "my-graph" --output edn + +# Show a page tree or a block +logseq show --graph "my-graph" --page "Meeting Notes" --level 2 +logseq show --graph "my-graph" --id 123 +logseq show --graph "my-graph" --id "[123,456,789]" + +# Upsert a page (create or update by --id) +logseq upsert page --graph "my-graph" --page "Project X" +logseq upsert page --graph "my-graph" --id 999 --update-properties "{:logseq.property/description \"Example\"}" + +# Upsert blocks +logseq upsert block --graph "my-graph" --target-page "Meeting Notes" --content "Discuss roadmap" +logseq upsert block --graph "my-graph" --target-page "Meeting Notes" --content "AI summary of the discussion" --update-tags '["AI-GENERATED"]' +logseq upsert block --graph "my-graph" --blocks "[{:block/title \"A\"} {:block/title \"B\"}]" +logseq upsert block --graph "my-graph" --id 123 --update-tags '["AI-GENERATED"]' +logseq upsert block --graph "my-graph" --id 123 --status done + +# Ensure a tag exists before associating it with a block +logseq upsert tag --graph "my-graph" --name "AI-GENERATED" +logseq upsert block --graph "my-graph" --target-page "Meeting Notes" --content "AI summary of the discussion" --update-tags '["AI-GENERATED"]' + +# Upsert tag/property +logseq upsert tag --graph "my-graph" --name "Project" +logseq upsert tag --graph "my-graph" --id 200 --name "Project Renamed" +logseq upsert property --graph "my-graph" --name "Effort" --type number --cardinality one +logseq upsert property --graph "my-graph" --id 321 --hide true + +# Remove entities +logseq remove block --graph "my-graph" --id "[123,456]" +logseq remove block --graph "my-graph" --uuid "11111111-1111-1111-1111-111111111111" +logseq remove page --graph "my-graph" --name "Old Page" +logseq remove tag --graph "my-graph" --name "Old Tag" +logseq remove property --graph "my-graph" --id 321 + +# Graph and server commands +logseq graph create --graph "my-graph" +logseq graph list +logseq graph switch --graph "my-graph" +logseq graph info --graph "my-graph" +logseq graph export --graph "my-graph" --type edn --file /tmp/my-graph.edn +logseq graph import --graph "my-graph-import" --type edn --input /tmp/my-graph.edn +logseq server status --graph "my-graph" +logseq doctor +``` + +## Tag association semantics + +- For block or page tag association, prefer explicit CLI tag options such as `--update-tags` and `--remove-tags`. +- Do not treat writing `#TagName` inside `--content` as equivalent guidance to explicit tag association. +- `upsert block` supports `--update-tags` in both create mode and update mode. +- `--update-tags` expects an EDN vector. +- Tag values may be tag title/name strings, db/id, UUID, or `:db/ident` values. +- String tag values may include a leading `#`, but they should still be passed inside `--update-tags` rather than embedded in content as a substitute for association. +- If the user asks to tag a block or page, prefer explicit tag association over embedding hashtags in content. +- Tags must already exist and be public. If needed, create the tag first with `upsert tag --name ""`. + +## Pitfalls + +- `--content "Summary #AI-GENERATED"` is not the same guidance as `--update-tags '["AI-GENERATED"]'`. +- Do not pass `--update-tags` as a comma-separated string. Use an EDN vector. +- Do not assume a hashtag in block text will replace the need for explicit tag association when the user asks for a tagged block. +- If tag association fails, verify the tag exists and is public before retrying. + +## Tips + +- `query list` returns both built-ins and `custom-queries` from `cli.edn`. +- `show --id` accepts either one db/id or an EDN vector of ids. +- `remove block --id` also accepts one db/id or an EDN vector. +- `upsert block` enters update mode when `--id` or `--uuid` is provided. +- Always verify command flags with `logseq --help` and `logseq <...> --help` before execution. +- If `logseq` reports that it doesn’t have read/write permission for data-dir, then add read/write permission for data-dir in the agent’s config. +- In sandboxed environments, `graph create` may print a process-scan warning to stderr; if command status is `ok`, the graph is still created. + +## References + +- Built-in tags and properties: See `references/logseq-builtins.md` when you need canonical built-ins for `list ... --include-built-in` or for tag/property upsert fields. diff --git a/skills/logseq-cli/references/logseq-builtins.md b/skills/logseq-cli/references/logseq-builtins.md new file mode 100644 index 0000000000..864ba15e68 --- /dev/null +++ b/skills/logseq-cli/references/logseq-builtins.md @@ -0,0 +1,142 @@ +# Logseq built-in tags and properties + +## Built-in tags (classes) + +| Title | Ident | +| --- | --- | +| Asset | `:logseq.class/Asset` | +| Card | `:logseq.class/Card` | +| Cards | `:logseq.class/Cards` | +| Code | `:logseq.class/Code-block` | +| Journal | `:logseq.class/Journal` | +| Math | `:logseq.class/Math-block` | +| Page | `:logseq.class/Page` | +| PDF Annotation | `:logseq.class/Pdf-annotation` | +| Property | `:logseq.class/Property` | +| Query | `:logseq.class/Query` | +| Quote | `:logseq.class/Quote-block` | +| Root Tag | `:logseq.class/Root` | +| Tag | `:logseq.class/Tag` | +| Task | `:logseq.class/Task` | +| Template | `:logseq.class/Template` | +| Whiteboard | `:logseq.class/Whiteboard` | + +## Built-in properties + +| Title | Ident | +| --- | --- | +| Alias | `:block/alias` | +| Closed value property | `:block/closed-value-property` | +| Node collapsed? | `:block/collapsed?` | +| Node created at | `:block/created-at` | +| Journal date | `:block/journal-day` | +| Node links to | `:block/link` | +| Node order | `:block/order` | +| Node page | `:block/page` | +| Node parent | `:block/parent` | +| Node references | `:block/refs` | +| Tags | `:block/tags` | +| Node title | `:block/title` | +| Node updated at | `:block/updated-at` | +| File checksum | `:logseq.property.asset/checksum` | +| External file name | `:logseq.property.asset/external-file-name` | +| External URL | `:logseq.property.asset/external-url` | +| Image height | `:logseq.property.asset/height` | +| Last visit page | `:logseq.property.asset/last-visit-page` | +| File remote metadata | `:logseq.property.asset/remote-metadata` | +| Asset resize metadata | `:logseq.property.asset/resize-metadata` | +| File Size | `:logseq.property.asset/size` | +| File Type | `:logseq.property.asset/type` | +| Image width | `:logseq.property.asset/width` | +| Bidirectional property title | `:logseq.property.class/bidirectional-property-title` | +| Enable bidirectional properties | `:logseq.property.class/enable-bidirectional?` | +| Extends | `:logseq.property.class/extends` | +| Hide from Node | `:logseq.property.class/hide-from-node` | +| Tag Properties | `:logseq.property.class/properties` | +| Code Mode | `:logseq.property.code/lang` | +| HNSW label | `:logseq.property.embedding/hnsw-label` | +| HNSW label updated-at | `:logseq.property.embedding/hnsw-label-updated-at` | +| Due | `:logseq.property.fsrs/due` | +| State | `:logseq.property.fsrs/state` | +| History block | `:logseq.property.history/block` | +| History property | `:logseq.property.history/property` | +| History value | `:logseq.property.history/ref-value` | +| History scalar value | `:logseq.property.history/scalar-value` | +| Title Format | `:logseq.property.journal/title-format` | +| Excluded references | `:logseq.property.linked-references/excludes` | +| Included references | `:logseq.property.linked-references/includes` | +| Node Display Type | `:logseq.property.node/display-type` | +| Annotation color | `:logseq.property.pdf/hl-color` | +| Annotation image | `:logseq.property.pdf/hl-image` | +| Annotation page | `:logseq.property.pdf/hl-page` | +| Annotation type | `:logseq.property.pdf/hl-type` | +| Annotation data | `:logseq.property.pdf/hl-value` | +| Published URL | `:logseq.property.publish/published-url` | +| Repeating Checked Property | `:logseq.property.repeat/checked-property` | +| Repeating recur frequency | `:logseq.property.repeat/recur-frequency` | +| Repeating recur unit | `:logseq.property.repeat/recur-unit` | +| Node Repeats? | `:logseq.property.repeat/repeated?` | +| Repeating Temporal Property | `:logseq.property.repeat/temporal-property` | +| View filters | `:logseq.property.table/filters` | +| View hidden columns | `:logseq.property.table/hidden-columns` | +| View ordered columns | `:logseq.property.table/ordered-columns` | +| Table view pinned columns | `:logseq.property.table/pinned-columns` | +| View columns settings | `:logseq.property.table/sized-columns` | +| View sorting | `:logseq.property.table/sorting` | +| Tldraw Page | `:logseq.property.tldraw/page` | +| Tldraw Shape | `:logseq.property.tldraw/shape` | +| User Avatar | `:logseq.property.user/avatar` | +| User Email | `:logseq.property.user/email` | +| User Name | `:logseq.property.user/name` | +| View Feature Type | `:logseq.property.view/feature-type` | +| View group by property | `:logseq.property.view/group-by-property` | +| View sort groups by | `:logseq.property.view/sort-groups-by-property` | +| View sort groups DESC | `:logseq.property.view/sort-groups-desc?` | +| View Type | `:logseq.property.view/type` | +| Asset | `:logseq.property/asset` | +| Background color | `:logseq.property/background-color` | +| Background image | `:logseq.property/background-image` | +| Built in? | `:logseq.property/built-in?` | +| Properties displayed as checkbox | `:logseq.property/checkbox-display-properties` | +| Choice checkbox state | `:logseq.property/choice-checkbox-state` | +| Choice classes | `:logseq.property/choice-classes` | +| Choice exclusions | `:logseq.property/choice-exclusions` | +| Property classes | `:logseq.property/classes` | +| Node created by | `:logseq.property/created-by` | +| Node created by | `:logseq.property/created-by-ref` | +| Created from property | `:logseq.property/created-from-property` | +| Deadline | `:logseq.property/deadline` | +| Default value | `:logseq.property/default-value` | +| Description | `:logseq.property/description` | +| Enable property history | `:logseq.property/enable-history?` | +| Excluded from Graph view? | `:logseq.property/exclude-from-graph-view` | +| Heading | `:logseq.property/heading` | +| Hide empty value | `:logseq.property/hide-empty-value` | +| Hide this property or page | `:logseq.property/hide?` | +| Icon | `:logseq.property/icon` | +| ls-type | `:logseq.property/ls-type` | +| List type | `:logseq.property/order-list-type` | +| Page Tags | `:logseq.property/page-tags` | +| Priority | `:logseq.property/priority` | +| Property public? | `:logseq.property/public?` | +| Publishing Public? | `:logseq.property/publishing-public?` | +| Query | `:logseq.property/query` | +| Non ref type default value | `:logseq.property/scalar-default-value` | +| Scheduled | `:logseq.property/scheduled` | +| Status | `:logseq.property/status` | +| Apply template to tags | `:logseq.property/template-applied-to` | +| Property type | `:logseq.property/type` | +| Property position | `:logseq.property/ui-position` | +| Used template | `:logseq.property/used-template` | +| Property value | `:logseq.property/value` | +| Property view context | `:logseq.property/view-context` | +| This view belongs to | `:logseq.property/view-for` | + +## Refresh recipe + +Run these commands and regenerate this file: + +```bash +logseq list tag --graph "import-lambda" --include-built-in --output edn +logseq list property --graph "import-lambda" --include-built-in --output edn +``` From cde0571e652c60fe9754bce239567f00209e80a9 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 13 Mar 2026 23:27:21 +0800 Subject: [PATCH 150/375] fix(cli): treat id-only show targets as missing entities MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make `show` treat id-only pull results as non-existent entities so deleted ids no longer render as empty blocks, and add regressions for deleted block/page lookups. 🤖 Generated with [eca](https://eca.dev) Co-Authored-By: eca --- src/main/logseq/cli/command/show.cljs | 60 ++++++++++++++--------- src/test/logseq/cli/commands_test.cljs | 19 +++++++ src/test/logseq/cli/integration_test.cljs | 21 ++++++++ 3 files changed, 77 insertions(+), 23 deletions(-) diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 79f61e6366..519a545f25 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -672,6 +672,17 @@ (sort-children (get parent->children parent-id)))))] (build root-id 1))) +(defn- entity-id-only? + [entity] + (and (map? entity) + (contains? entity :db/id) + (= #{:db/id} (set (keys entity))))) + +(defn- missing-show-entity? + [entity] + (or (nil? entity) + (entity-id-only? entity))) + (defn- fetch-tree [config {:keys [repo id page level] :as opts}] (let [max-depth (or level 10) @@ -684,16 +695,17 @@ {:block/page [:db/id :block/title]} {:block/tags [:db/id :block/name :block/title :block/uuid]}] id])] (p/let [entity (attach-user-properties-to-entity config repo entity)] - (if-let [page-id (get-in entity [:block/page :db/id])] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (if (:db/id entity) - (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found}))))) - ) + (if (missing-show-entity? entity) + (throw (ex-info "entity not found" {:code :entity-not-found})) + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (if-let [entity-id (:db/id entity)] + (p/let [blocks (fetch-blocks-for-page config repo entity-id) + children (build-tree blocks entity-id max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "entity not found" {:code :entity-not-found}))))))) (seq uuid-str) (if-not (common-util/uuid-string? uuid-str) @@ -713,16 +725,17 @@ {:block/tags [:db/id :block/name :block/title :block/uuid]}] [:block/uuid uuid-str]]))] (p/let [entity (attach-user-properties-to-entity config repo entity)] - (if-let [page-id (get-in entity [:block/page :db/id])] - (p/let [blocks (fetch-blocks-for-page config repo page-id) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (if (:db/id entity) - (p/let [blocks (fetch-blocks-for-page config repo (:db/id entity)) - children (build-tree blocks (:db/id entity) max-depth)] - {:root (assoc entity :block/children children)}) - (throw (ex-info "block not found" {:code :block-not-found})))))) - ) + (if (missing-show-entity? entity) + (throw (ex-info "entity not found" {:code :entity-not-found})) + (if-let [page-id (get-in entity [:block/page :db/id])] + (p/let [blocks (fetch-blocks-for-page config repo page-id) + children (build-tree blocks (:db/id entity) max-depth)] + {:root (assoc entity :block/children children)}) + (if-let [entity-id (:db/id entity)] + (p/let [blocks (fetch-blocks-for-page config repo entity-id) + children (build-tree blocks entity-id max-depth)] + {:root (assoc entity :block/children children)}) + (throw (ex-info "entity not found" {:code :entity-not-found})))))))) (seq page) (p/let [page-entity (transport/invoke config :thread-api/pull false @@ -864,9 +877,10 @@ (let [data (ex-data error) code (:code data) message (or (:message data) (.-message error) (str error))] - (if (= code :block-not-found) - (str "Block " id " not found") - (str "Block " id ": " message)))) + (if (or (= code :block-not-found) + (= code :entity-not-found)) + (str "Entity " id " not found") + (str "Entity " id ": " message)))) (defn- multi-id-error-entry [id error] diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index eac8e6ffb5..938174059c 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -803,6 +803,25 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) +(deftest test-show-id-only-entity-is-not-found + (async done + (-> (p/with-redefs [cli-server/ensure-server! (fn [config _] config) + transport/invoke (fn [_ method _ _] + (case method + :thread-api/pull (p/resolved {:db/id 212}) + :thread-api/q (p/resolved []) + :thread-api/get-block-refs (p/resolved []) + (p/resolved nil)))] + (show-command/execute-show {:type :show + :repo "demo" + :id 212} + {:output-format :json})) + (p/then (fn [_] + (is false "expected execute-show to reject for id-only entity"))) + (p/catch (fn [e] + (is (= :entity-not-found (:code (ex-data e)))))) + (p/finally done)))) + (deftest test-tree->text-uuid-ref-recursion-limit (testing "show tree text limits uuid ref replacement depth" (let [tree->text #'show-command/tree->text diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 9c41a48e09..fa57c39520 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -573,6 +573,7 @@ _ (run-cli ["graph" "create" "--graph" "content-graph"] data-dir cfg-path) add-page-result (run-cli ["--graph" "content-graph" "upsert" "page" "--page" "TestPage"] data-dir cfg-path) add-page-payload (parse-json-output add-page-result) + page-id (first-result-id add-page-payload) list-page-result (run-cli ["--graph" "content-graph" "list" "page"] data-dir cfg-path) list-page-payload (parse-json-output list-page-result) list-tag-result (run-cli ["--graph" "content-graph" "list" "tag"] data-dir cfg-path) @@ -581,17 +582,26 @@ list-property-payload (parse-json-output list-property-result) add-block-result (run-cli ["--graph" "content-graph" "upsert" "block" "--target-page" "TestPage" "--content" "Test block"] data-dir cfg-path) add-block-payload (parse-json-output add-block-result) + block-id (first-result-id add-block-payload) _ (p/delay 100) show-result (run-cli ["--graph" "content-graph" "show" "--page" "TestPage"] data-dir cfg-path) show-payload (parse-json-output show-result) + remove-block-result (run-cli ["--graph" "content-graph" "remove" "block" "--id" (str block-id)] data-dir cfg-path) + remove-block-payload (parse-json-output remove-block-result) + show-deleted-block-result (run-cli ["--graph" "content-graph" "show" "--id" (str block-id)] data-dir cfg-path) + show-deleted-block-payload (parse-json-output show-deleted-block-result) remove-page-result (run-cli ["--graph" "content-graph" "remove" "page" "--name" "TestPage"] data-dir cfg-path) remove-page-payload (parse-json-output remove-page-result) + show-deleted-page-result (run-cli ["--graph" "content-graph" "show" "--id" (str page-id)] data-dir cfg-path) + show-deleted-page-payload (parse-json-output show-deleted-page-result) stop-result (run-cli ["server" "stop" "--graph" "content-graph"] data-dir cfg-path) stop-payload (parse-json-output stop-result)] (is (= 0 (:exit-code add-page-result))) (is (= "ok" (:status add-page-payload))) + (is (number? page-id)) (is (= "ok" (:status add-block-payload)) (pr-str (:error add-block-payload))) + (is (number? block-id)) (is (= "ok" (:status list-page-payload))) (is (vector? (get-in list-page-payload [:data :items]))) (is (= "ok" (:status list-tag-payload))) @@ -602,7 +612,18 @@ (is (some? (or (get-in show-payload [:data :root :db/id]) (get-in show-payload [:data :root :id])))) (is (not (contains? (get-in show-payload [:data :root]) :block/uuid))) + (is (= "ok" (:status remove-block-payload))) + (is (= "error" (:status show-deleted-block-payload))) + (is (contains? #{:entity-not-found :exception} + (keyword (get-in show-deleted-block-payload [:error :code])))) + (is (string/includes? (string/lower-case (or (get-in show-deleted-block-payload [:error :message]) "")) + "entity not found")) (is (= "ok" (:status remove-page-payload))) + (is (= "error" (:status show-deleted-page-payload))) + (is (contains? #{:entity-not-found :exception} + (keyword (get-in show-deleted-page-payload [:error :code])))) + (is (string/includes? (string/lower-case (or (get-in show-deleted-page-payload [:error :message]) "")) + "entity not found")) (is (= "ok" (:status stop-payload))) (done)) (p/catch (fn [e] From bad750c17337d84231674aa444f8c5d66c38aa07 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 13 Mar 2026 13:41:44 -0400 Subject: [PATCH 151/375] fix: integration test leaves hanging processes Hanging db-worker-node.js processes left after running tests because servers weren't shut down --- src/test/logseq/cli/integration_test.cljs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index fa57c39520..273859b35e 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -384,6 +384,7 @@ (pr-str start-payload)) (is (contains? #{"open" :open} (get-in start-payload [:data :ws-state]))) + (stop-repo! data-dir cfg-path start-repo) (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) @@ -425,7 +426,8 @@ :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-upload-graph ["logseq_db_sync-upload-graph"]]] - @invoke-calls))) + @invoke-calls)) + (stop-repo! data-dir cfg-path upload-repo)) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done))))) @@ -477,7 +479,8 @@ (is (= "ok" (:status info-payload))) (is (= uploaded-graph-id (get-in info-payload [:data :kv :logseq.kv/graph-uuid]))) - (is (= "logseq_db_sync-upload-graph-info" (first q-call)))) + (is (= "logseq_db_sync-upload-graph-info" (first q-call))) + (stop-repo! data-dir cfg-path upload-repo)) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done))))) @@ -2200,6 +2203,7 @@ (is (string/includes? output "TITLE")) (is (string/includes? output "TestPage")) (is (string/includes? output "Count:")) + (stop-repo! data-dir cfg-path "human-list-graph") (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) From eec5b4a060f3a391cf8fb5fbb800af650c25b34f Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 13 Mar 2026 16:17:20 -0400 Subject: [PATCH 152/375] fix: multiple issues with --fields option for list* commands - 4 visible fields couldn't be used with --fields - Only print selected fields/columns. Previous behavior didn't make sense as unspecified fields were still listed and took up horizontal space - Allow --fields to work when --expand isn't used. Previous behavior wasn't explained to user and needlessly limiting --- .../004-logseq-cli-verb-subcommands.md | 6 +- src/main/logseq/cli/command/list.cljs | 26 +++--- src/main/logseq/cli/format.cljs | 80 ++++++++----------- src/test/logseq/cli/format_test.cljs | 10 ++- src/test/logseq/cli/integration_test.cljs | 43 ++++++++-- 5 files changed, 94 insertions(+), 71 deletions(-) diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md index 960268f193..acb001ed60 100644 --- a/docs/agent-guide/004-logseq-cli-verb-subcommands.md +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -79,7 +79,7 @@ List page options: | --include-hidden | Include hidden pages | Requires a flag to bypass entity-util/hidden? filtering. | | --updated-after ISO8601 | Filter by updated-at | Compare to :block/updated-at. | | --created-after ISO8601 | Filter by created-at | Compare to :block/created-at. | -| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | +| --fields FIELD,FIELD | Select output fields | | List tag options: @@ -88,7 +88,7 @@ List tag options: | --include-built-in | Include built-in classes | Built-in tags are currently included by default, clarify behavior. | | --with-properties | Include class properties | Uses :logseq.property.class/properties when expanded. | | --with-extends | Include class extends | Uses :logseq.property.class/extends when expanded. | -| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | +| --fields FIELD,FIELD | Select output fields | | List property options: @@ -97,7 +97,7 @@ List property options: | --include-built-in | Include built-in properties | Built-in properties are currently included by default, clarify behavior. | | --with-classes | Include property classes | Uses :logseq.property/classes when expanded. | | --with-type | Include property type | Uses :logseq.property/type when expanded. | -| --fields FIELD,FIELD | Select output fields | Applies when --expand is true. | +| --fields FIELD,FIELD | Select output fields | | List block is removed to avoid overlap with search. diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index e21b28e5f6..30ea36bae8 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -94,23 +94,31 @@ :options options}})) (def ^:private list-page-field-map - {"title" :block/title + {"id" :db/id + "ident" :db/ident + "title" :block/title "uuid" :block/uuid "created-at" :block/created-at "updated-at" :block/updated-at}) (def ^:private list-tag-field-map - {"name" :block/title + {"id" :db/id + "ident" :db/ident "title" :block/title "uuid" :block/uuid + "created-at" :block/created-at + "updated-at" :block/updated-at "properties" :logseq.property.class/properties "extends" :logseq.property.class/extends "description" :logseq.property/description}) (def ^:private list-property-field-map - {"name" :block/title + {"id" :db/id + "ident" :db/ident "title" :block/title "uuid" :block/uuid + "created-at" :block/created-at + "updated-at" :block/updated-at "classes" :logseq.property/classes "type" :logseq.property/type "description" :logseq.property/description}) @@ -178,9 +186,7 @@ fields (parse-field-list (:fields options)) sorted (apply-sort items (:sort options) order list-page-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-page-field-map) - limited)] + final (apply-fields limited fields list-page-field-map)] {:status :ok :data {:items final}}))) @@ -195,9 +201,7 @@ prepared (mapv #(prepare-tag-item % options) items) sorted (apply-sort prepared (:sort options) order list-tag-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-tag-field-map) - limited)] + final (apply-fields limited fields list-tag-field-map)] {:status :ok :data {:items final}}))) @@ -212,8 +216,6 @@ prepared (mapv #(prepare-property-item % options) items) sorted (apply-sort prepared (:sort options) order list-property-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) - final (if (:expand options) - (apply-fields limited fields list-property-field-map) - limited)] + final (apply-fields limited fields list-property-field-map)] {:status :ok :data {:items final}}))) diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 75b935c536..778b66a7b7 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -122,11 +122,6 @@ candidates* (str candidates*) hint (str "\nHint: " hint))))) -(defn- maybe-ident-header - [items] - (when (some :db/ident items) - ["IDENT"])) - (defn- parse-ts [value] (cond @@ -154,37 +149,36 @@ :else (str years "y ago"))) "-")) -(defn- format-list-row - [item include-ident? now-ms] - (let [base [(or (:db/id item) (:id item)) - (or (:title item) (:block/title item) (:name item))] - with-ident (cond-> base - include-ident? (conj (:db/ident item))) - updated (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms) - created (human-ago (or (:created-at item) (:block/created-at item)) now-ms)] - (conj with-ident updated created))) +(defn- items-have-key? + [items & ks] + (some (fn [item] (some #(contains? item %) ks)) items)) + +(def ^:private list-columns + [["ID" (fn [item _] (or (:db/id item) (:id item))) [:db/id :id]] + ["TITLE" (fn [item _] (or (:title item) (:block/title item) (:name item))) [:title :block/title :name]] + ["IDENT" (fn [item _] (:db/ident item)) [:db/ident]] + ["UPDATED-AT" (fn [item now-ms] (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms)) [:updated-at :block/updated-at]] + ["CREATED-AT" (fn [item now-ms] (human-ago (or (:created-at item) (:block/created-at item)) now-ms)) [:created-at :block/created-at]]]) + +(defn- format-list-dynamic + [items now-ms columns] + (let [items (or items []) + active (filterv (fn [[_ _ ks always?]] + (or always? (apply items-have-key? items ks))) + columns) + headers (mapv first active) + rows (mapv (fn [item] + (mapv (fn [[_ extractor _]] (extractor item now-ms)) active)) + items)] + (format-counted-table headers rows))) (defn- format-list-page [items now-ms] - (let [items (or items []) - include-ident? (boolean (some :db/ident items)) - headers (into ["ID" "TITLE"] - (concat (or (maybe-ident-header items) []) - ["UPDATED-AT" "CREATED-AT"]))] - (format-counted-table - headers - (mapv #(format-list-row % include-ident? now-ms) items)))) + (format-list-dynamic items now-ms list-columns)) (defn- format-list-tag [items now-ms] - (let [items (or items []) - include-ident? (boolean (some :db/ident items)) - headers (into ["ID" "TITLE"] - (concat (or (maybe-ident-header items) []) - ["UPDATED-AT" "CREATED-AT"]))] - (format-counted-table - headers - (mapv #(format-list-row % include-ident? now-ms) items)))) + (format-list-dynamic items now-ms list-columns)) (defn- normalize-property-type [value] @@ -193,27 +187,17 @@ (nil? value) "-" :else (str value))) -(defn- format-list-property-row - [item include-ident? now-ms] - (let [base [(or (:db/id item) (:id item)) - (or (:title item) (:block/title item) (:name item)) - (normalize-property-type (:logseq.property/type item))] - with-ident (cond-> base - include-ident? (conj (:db/ident item))) - updated (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms) - created (human-ago (or (:created-at item) (:block/created-at item)) now-ms)] - (conj with-ident updated created))) +(def ^:private list-property-columns + [["ID" (fn [item _] (or (:db/id item) (:id item))) [:db/id :id]] + ["TITLE" (fn [item _] (or (:title item) (:block/title item) (:name item))) [:title :block/title :name]] + ["TYPE" (fn [item _] (normalize-property-type (:logseq.property/type item))) [:logseq.property/type]] + ["IDENT" (fn [item _] (:db/ident item)) [:db/ident]] + ["UPDATED-AT" (fn [item now-ms] (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms)) [:updated-at :block/updated-at]] + ["CREATED-AT" (fn [item now-ms] (human-ago (or (:created-at item) (:block/created-at item)) now-ms)) [:created-at :block/created-at]]]) (defn- format-list-property [items now-ms] - (let [items (or items []) - include-ident? (boolean (some :db/ident items)) - headers (into ["ID" "TITLE" "TYPE"] - (concat (or (maybe-ident-header items) []) - ["UPDATED-AT" "CREATED-AT"]))] - (format-counted-table - headers - (mapv #(format-list-property-row % include-ident? now-ms) items)))) + (format-list-dynamic items now-ms list-property-columns)) (defn- format-graph-list [graphs current-graph] diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 2985f9d4ec..639e34e5eb 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -133,15 +133,21 @@ (testing "list property renders missing type as -" (let [result (format/format-result {:status :ok :command :list-property - :data {:items [{:block/title "Untyped" + :data {:items [{:block/title "Prop" + :db/id 99 + :logseq.property/type :node + :block/created-at 40000 + :block/updated-at 90000} + {:block/title "Untyped" :db/id 100 :block/created-at 40000 :block/updated-at 90000}]}} {:output-format nil :now-ms 100000})] (is (= (str "ID TITLE TYPE UPDATED-AT CREATED-AT\n" + "99 Prop node 10s ago 1m ago\n" "100 Untyped - 10s ago 1m ago\n" - "Count: 1") + "Count: 2") result))))) (deftest test-human-output-add-upsert-remove diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 273859b35e..685ad86fff 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -200,7 +200,7 @@ :updated-at 1735686000000} overrides))) -(deftest test-cli-login-integration +(deftest ^:long test-cli-login-integration (async done (let [data-dir (node-helper/create-tmp-dir "cli-login-data") cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -237,7 +237,7 @@ (p/finally (fn [] (done)))))))) -(deftest test-cli-logout-integration +(deftest ^:long test-cli-logout-integration (async done (let [data-dir (node-helper/create-tmp-dir "cli-logout-data") cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -270,7 +270,7 @@ (p/finally (fn [] (done)))))))) -(deftest test-cli-sync-remote-graphs-refreshes-auth-file-and-injects-runtime-token +(deftest ^:long test-cli-sync-remote-graphs-refreshes-auth-file-and-injects-runtime-token (async done (let [data-dir (node-helper/create-tmp-dir "cli-sync-auth") cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -316,7 +316,7 @@ (p/finally (fn [] (done)))))))) -(deftest test-cli-sync-download-and-start-readiness-with-mocked-sync +(deftest ^:long test-cli-sync-download-and-start-readiness-with-mocked-sync (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-cli") download-repo "sync-download-graph" @@ -390,7 +390,7 @@ (is false (str "unexpected error: " e)) (done))))))) -(deftest test-cli-sync-upload-with-mocked-worker-bootstrap +(deftest ^:long test-cli-sync-upload-with-mocked-worker-bootstrap (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-cli") upload-repo "sync-upload-graph" @@ -432,7 +432,7 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) -(deftest test-cli-sync-upload-followed-by-graph-info-shows-graph-uuid-test +(deftest ^:long test-cli-sync-upload-followed-by-graph-info-shows-graph-uuid-test (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-sync-upload-info-cli") upload-repo "sync-upload-graph-info" @@ -2209,6 +2209,37 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-list-page-fields-filter + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker")] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (run-cli ["graph" "create" "--graph" "fields-graph"] data-dir cfg-path) + _ (run-cli ["upsert" "page" "--page" "FieldTest"] data-dir cfg-path) + all-result (run-cli ["list" "page"] data-dir cfg-path) + all-payload (parse-json-output all-result) + filtered-result (run-cli ["list" "page" "--fields" "title"] data-dir cfg-path) + filtered-payload (parse-json-output filtered-result) + human-filtered-result (run-cli ["list" "page" "--fields" "title" "--output" "human"] data-dir cfg-path) + first-all (first (get-in all-payload [:data :items])) + first-filtered (first (get-in filtered-payload [:data :items])) + _ (stop-repo! data-dir cfg-path "fields-graph")] + (is (= 0 (:exit-code all-result))) + (is (= 0 (:exit-code filtered-result))) + (is (contains? first-all :title)) + (is (contains? first-all :updated-at)) + (is (contains? first-filtered :title)) + (is (not (contains? first-filtered :updated-at)) + "--fields title should exclude other fields") + (is (= 0 (:exit-code human-filtered-result))) + (is (string/includes? (:output human-filtered-result) "TITLE")) + (is (not (string/includes? (:output human-filtered-result) "UPDATED-AT")) + "--fields title human output should not show UPDATED-AT column") + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-show-page-block-by-id-and-uuid (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From 0a32fc18fe3b2918d6b9ae9624e156be6119e036 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 13 Mar 2026 17:14:13 -0400 Subject: [PATCH 153/375] enhance: add user-only option to list commands --- src/main/logseq/cli/command/list.cljs | 14 +++++++++++--- src/main/logseq/cli/common/mcp/tools.cljs | 6 +++++- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 30ea36bae8..0102e997f2 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -9,6 +9,9 @@ (def ^:private list-common-spec {:expand {:desc "Include expanded metadata" :coerce :boolean} + :user-only {:desc "Exclude built-in entities" + :alias :u + :coerce :boolean} :limit {:desc "Limit results" :coerce :long} :offset {:desc "Offset results" @@ -176,10 +179,15 @@ (not with-type) (dissoc :logseq.property/type)) item)) +(defn- apply-user-only + [options] + (cond-> options + (:user-only options) (assoc :include-built-in false))) + (defn execute-list-page [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) + options (apply-user-only (:options action)) items (transport/invoke cfg :thread-api/api-list-pages false [(:repo action) options]) order (or (:order options) "asc") @@ -193,7 +201,7 @@ (defn execute-list-tag [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) + options (apply-user-only (:options action)) items (transport/invoke cfg :thread-api/api-list-tags false [(:repo action) options]) order (or (:order options) "asc") @@ -208,7 +216,7 @@ (defn execute-list-property [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (:options action) + options (apply-user-only (:options action)) items (transport/invoke cfg :thread-api/api-list-properties false [(:repo action) options]) order (or (:order options) "asc") diff --git a/src/main/logseq/cli/common/mcp/tools.cljs b/src/main/logseq/cli/common/mcp/tools.cljs index 7a1ab2b635..e20b04a8fa 100644 --- a/src/main/logseq/cli/common/mcp/tools.cljs +++ b/src/main/logseq/cli/common/mcp/tools.cljs @@ -118,8 +118,9 @@ (defn list-pages "Main fn for ListPages tool" - [db {:keys [expand include-hidden include-journal journal-only created-after updated-after] :as options}] + [db {:keys [expand include-hidden include-built-in include-journal journal-only created-after updated-after] :as options}] (let [include-hidden? (boolean include-hidden) + include-built-in? (if (contains? options :include-built-in) include-built-in true) include-journal? (if (contains? options :include-journal) include-journal true) journal-only? (boolean journal-only) created-after-ms (parse-time created-after) @@ -129,6 +130,9 @@ (remove (fn [e] (and (not include-hidden?) (entity-util/hidden? e)))) + (remove (fn [e] + (and (not include-built-in?) + (ldb/built-in? e)))) (remove (fn [e] (let [is-journal? (ldb/journal? e)] (cond From fbfb589c52a14059ea73926bbd2a39c8e99e1f23 Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 13 Mar 2026 17:45:22 -0400 Subject: [PATCH 154/375] enhance: list commands can sort by all visibile fields Remove outdated name field which had no effect. Also remove duplication of these values in option declaration, completion declaration and completion tests --- src/main/logseq/cli/command/list.cljs | 16 ++++++------- .../logseq/cli/completion_generator_test.cljs | 24 ++++++------------- 2 files changed, 15 insertions(+), 25 deletions(-) diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 0102e997f2..9068ca1522 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -20,10 +20,15 @@ :order {:desc "Sort order. Default: asc" :values ["asc" "desc"]}}) +(def ^:private list-sort-fields + {:list-page #{"title" "id" "ident" "created-at" "updated-at"} + :list-tag #{"title" "id" "ident" "created-at" "updated-at"} + :list-property #{"title" "id" "ident" "created-at" "updated-at"}}) + (def ^:private list-page-spec (merge list-common-spec {:sort {:desc "Sort field" - :values ["title" "created-at" "updated-at"]} + :values (:list-page list-sort-fields)} :include-journal {:desc "Include journal pages" :coerce :boolean} :journal-only {:desc "Only journal pages" @@ -37,7 +42,7 @@ (def ^:private list-tag-spec (merge list-common-spec {:sort {:desc "Sort field" - :values ["name" "title"]} + :values (:list-tag list-sort-fields)} :include-built-in {:desc "Include built-in tags" :coerce :boolean} :with-properties {:desc "Include tag properties" @@ -49,7 +54,7 @@ (def ^:private list-property-spec (merge list-common-spec {:sort {:desc "Sort field" - :values ["name" "title"]} + :values (:list-property list-sort-fields)} :include-built-in {:desc "Include built-in properties" :coerce :boolean} :with-classes {:desc "Include property classes" @@ -63,11 +68,6 @@ (core/command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec) (core/command-entry ["list" "property"] :list-property "List properties" list-property-spec)]) -(def ^:private list-sort-fields - {:list-page #{"title" "created-at" "updated-at"} - :list-tag #{"name" "title"} - :list-property #{"name" "title"}}) - (defn invalid-options? [command opts] (let [{:keys [order include-journal journal-only]} opts diff --git a/src/test/logseq/cli/completion_generator_test.cljs b/src/test/logseq/cli/completion_generator_test.cljs index 5387b43daf..2c8787b489 100644 --- a/src/test/logseq/cli/completion_generator_test.cljs +++ b/src/test/logseq/cli/completion_generator_test.cljs @@ -44,15 +44,12 @@ page-entry (first (filter #(= :list-page (:command %)) entries)) tag-entry (first (filter #(= :list-tag (:command %)) entries)) property-entry (first (filter #(= :list-property (:command %)) entries))] - (testing "page-spec :sort has correct values" - (is (= ["title" "created-at" "updated-at"] - (get-in page-entry [:spec :sort :values])))) - (testing "tag-spec :sort has correct values" - (is (= ["name" "title"] - (get-in tag-entry [:spec :sort :values])))) - (testing "property-spec :sort has correct values" - (is (= ["name" "title"] - (get-in property-entry [:spec :sort :values])))) + (testing "page-spec :sort has some correct values" + (is (contains? (get-in page-entry [:spec :sort :values]) "title"))) + (testing "tag-spec :sort has some correct values" + (is (contains? (get-in tag-entry [:spec :sort :values]) "title"))) + (testing "property-spec :sort has some correct values" + (is (contains? (get-in property-entry [:spec :sort :values]) "title"))) (testing "common :order has correct values" (is (= ["asc" "desc"] (get-in page-entry [:spec :order :values])))))) @@ -214,12 +211,7 @@ (deftest test-zsh-command-specific-values (let [output (gen/generate-completions "zsh" full-table)] (testing "--pos under upsert block offers correct values" - (is (re-find #"--pos=.*\(first-child last-child sibling\)" output))) - (testing "--sort for list page offers correct values" - (is (re-find #"--sort=.*\(title created-at updated-at\)" output))) - (testing "--sort for list tag offers name title" - ;; Just check globally that name title appears in sort context - (is (re-find #"\(name title\)" output))))) + (is (re-find #"--pos=.*\(first-child last-child sibling\)" output))))) (deftest test-zsh-all-commands-present (let [output (gen/generate-completions "zsh" full-table)] @@ -346,8 +338,6 @@ (is (contains? varied :type))) (testing "--name is detected as varied" (is (contains? varied :name))) - (testing "--sort is detected as varied" - (is (contains? varied :sort))) (testing "uniform options like --pos are not varied" (is (not (contains? varied :pos)))) (testing "uniform options like --cardinality are not varied" From e901c2323f211401d6970b5d72abe2a1343f722b Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Fri, 13 Mar 2026 18:21:52 -0400 Subject: [PATCH 155/375] fix: --expand option drops fields for list commands --- src/main/logseq/cli/command/list.cljs | 20 ++++++++------------ src/main/logseq/cli/common/mcp/tools.cljs | 17 +++++++++-------- 2 files changed, 17 insertions(+), 20 deletions(-) diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 9068ca1522..aadf72d0ae 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -164,20 +164,16 @@ (some? limit) (->> (take limit) vec))) (defn- prepare-tag-item - [item {:keys [expand with-properties with-extends]}] - (if expand - (cond-> item - (not with-properties) (dissoc :logseq.property.class/properties) - (not with-extends) (dissoc :logseq.property.class/extends)) - item)) + [item {:keys [with-properties with-extends]}] + (cond-> item + (not with-properties) (dissoc :logseq.property.class/properties) + (not with-extends) (dissoc :logseq.property.class/extends))) (defn- prepare-property-item - [item {:keys [expand with-classes with-type]}] - (if expand - (cond-> item - (not with-classes) (dissoc :logseq.property/classes) - (not with-type) (dissoc :logseq.property/type)) - item)) + [item {:keys [with-classes with-type]}] + (cond-> item + (not with-classes) (dissoc :logseq.property/classes) + (not with-type) (dissoc :logseq.property/type))) (defn- apply-user-only [options] diff --git a/src/main/logseq/cli/common/mcp/tools.cljs b/src/main/logseq/cli/common/mcp/tools.cljs index e20b04a8fa..ce3acb6956 100644 --- a/src/main/logseq/cli/common/mcp/tools.cljs +++ b/src/main/logseq/cli/common/mcp/tools.cljs @@ -37,9 +37,9 @@ #_((fn [x] (prn :prop-keys (distinct (mapcat keys x))) x)) (map (fn [e] (if expand - (cond-> (into {} e) + (cond-> (assoc (into {} e) :db/id (:db/id e)) true - (dissoc e :block/tags :block/order :block/refs :block/name :db/index + (dissoc :block/tags :block/order :block/refs :block/name :db/index :logseq.property.embedding/hnsw-label-updated-at :logseq.property/default-value) true (update :block/uuid str) @@ -62,9 +62,9 @@ (ldb/built-in? e)))) (map (fn [e] (if expand - (cond-> (into {} e) + (cond-> (assoc (into {} e) :db/id (:db/id e)) true - (dissoc e :block/tags :block/order :block/refs :block/name + (dissoc :block/tags :block/order :block/refs :block/name :logseq.property.embedding/hnsw-label-updated-at) true (update :block/uuid str) @@ -147,10 +147,11 @@ (<= (:block/updated-at e 0) updated-after-ms)))) (map (fn [e] (if expand - (-> e - ;; Until there are options to limit pages, return minimal info to avoid - ;; exceeding max payload size - (select-keys [:db/id :db/ident :block/uuid :block/title :block/created-at :block/updated-at]) + ;; Until there are options to limit pages, return minimal info to avoid + ;; exceeding max payload size + (-> (select-keys e [:block/uuid :block/title :block/created-at :block/updated-at]) + (assoc :db/id (:db/id e)) + (cond-> (:db/ident e) (assoc :db/ident (:db/ident e))) (update :block/uuid str)) (minimal-list-item e))))))) From 3a547c616917ee6ae9887364188e1a1fd2144bd4 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 14 Mar 2026 16:44:30 +0800 Subject: [PATCH 156/375] fix(cli): subcmd --help --- src/main/logseq/cli/commands.cljs | 26 ++++++++++++++++++++++++-- src/test/logseq/cli/commands_test.cljs | 20 ++++++++++++++++++++ 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index 8282442b5b..f52ff5f1f4 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -291,6 +291,26 @@ ;; CLI error handling is in logseq.cli.command.core. +(def ^:private group-commands + (->> table + (map :cmds) + (group-by first) + (keep (fn [[group cmds]] + (when (some #(> (count %) 1) cmds) + group))) + set)) + +(def ^:private help-flags + #{"-h" "--help"}) + +(defn- group-help-invocation? + [args] + (let [group (first args) + trailing-args (rest args)] + (and (contains? group-commands group) + (or (empty? trailing-args) + (every? help-flags trailing-args))))) + (defn parse-args [raw-args] (let [summary (command-core/top-level-summary table) @@ -303,7 +323,7 @@ (empty? args) (command-core/help-result summary) - (and (= 1 (count args)) (#{"graph" "server" "list" "upsert" "remove" "query" "sync"} (first args))) + (group-help-invocation? args) (command-core/help-result (command-core/group-summary (first args) table)) :else @@ -318,7 +338,9 @@ (let [{:keys [cause] :as data} (ex-data e)] (cond (= cause :input-exhausted) - (command-core/help-result summary) + (if (group-help-invocation? args) + (command-core/help-result (command-core/group-summary (first args) table)) + (command-core/help-result summary)) (= cause :no-match) (command-core/unknown-command-result summary (str "unknown command: " (unknown-command-message data))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 938174059c..6206ab6486 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -214,6 +214,26 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-group-help-flags + (testing "all groups show group help with -h and --help" + (doseq [group ["graph" "server" "list" "upsert" "remove" "query" "sync"] + help-flag ["-h" "--help"]] + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args [group help-flag])) + plain-summary (strip-ansi (:summary result))] + (is (true? (:help? result))) + (is (string/includes? plain-summary (str "Usage: logseq " group " [options]")))))) + + (testing "upsert block command short help flag shows command help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["upsert" "block" "-h"])) + plain-summary (strip-ansi (:summary result))] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq upsert block")) + (is (not (string/includes? plain-summary "Usage: logseq upsert [options]"))))) + + ) + (deftest test-parse-args-help-auth-commands (testing "login command shows help" (let [result (binding [style/*color-enabled?* true] From 097ddd42f3f1f2e2940f1093db4d1cbf237ca8eb Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 14 Mar 2026 17:58:16 +0800 Subject: [PATCH 157/375] enhance(cli): add examples in --help --- src/main/logseq/cli/command/completion.cljs | 4 +- src/main/logseq/cli/command/core.cljs | 25 +++++- src/main/logseq/cli/command/doctor.cljs | 3 +- src/main/logseq/cli/command/graph.cljs | 22 +++-- src/main/logseq/cli/command/list.cljs | 10 ++- src/main/logseq/cli/command/query.cljs | 4 +- src/main/logseq/cli/command/remove.cljs | 13 ++- src/main/logseq/cli/command/server.cljs | 12 ++- src/main/logseq/cli/command/show.cljs | 4 +- src/main/logseq/cli/command/sync.cljs | 33 +++++--- src/main/logseq/cli/command/upsert.cljs | 13 ++- src/main/logseq/cli/commands.cljs | 7 +- .../logseq/cli/command/completion_test.cljs | 13 ++- src/test/logseq/cli/commands_test.cljs | 80 +++++++++++-------- 14 files changed, 167 insertions(+), 76 deletions(-) diff --git a/src/main/logseq/cli/command/completion.cljs b/src/main/logseq/cli/command/completion.cljs index 2c32e77d41..4ca0be789b 100644 --- a/src/main/logseq/cli/command/completion.cljs +++ b/src/main/logseq/cli/command/completion.cljs @@ -26,4 +26,6 @@ [(core/command-entry ["completion"] :completion "Generate shell completion script" completion-spec - {:long-desc long-desc})]) + {:long-desc long-desc + :examples ["logseq completion zsh" + "logseq completion bash"]})]) diff --git a/src/main/logseq/cli/command/core.cljs b/src/main/logseq/cli/command/core.cljs index 932792c024..d6d10bcc39 100644 --- a/src/main/logseq/cli/command/core.cljs +++ b/src/main/logseq/cli/command/core.cljs @@ -39,12 +39,13 @@ (defn command-entry ([cmds command desc spec] (command-entry cmds command desc spec nil)) - ([cmds command desc spec {:keys [long-desc]}] + ([cmds command desc spec {:keys [long-desc examples]}] (let [spec* (merge-spec spec)] {:cmds cmds :command command :desc desc :long-desc long-desc + :examples examples :spec spec* :restrict true :fn (fn [{:keys [opts args]}] @@ -52,6 +53,7 @@ :cmds cmds :spec spec* :long-desc long-desc + :examples examples :opts opts :args args})}))) @@ -127,9 +129,21 @@ (str "Command " (style/bold "options") ":") " See `logseq --help`"]))) +(defn- format-examples + [examples] + (->> examples + (keep (fn [example] + (let [line (some-> example str string/trim)] + (when (seq line) + line)))) + (take 5) + (map #(str " " %)) + (string/join "\n"))) + (defn command-summary - [{:keys [cmds spec long-desc]}] - (let [command-spec (apply dissoc spec (keys global-spec*))] + [{:keys [cmds spec long-desc examples]}] + (let [command-spec (apply dissoc spec (keys global-spec*)) + formatted-examples (format-examples examples)] (string/join "\n" (cond-> [(str "Usage: logseq " (command-usage cmds spec))] (seq long-desc) (conj "" long-desc) @@ -138,7 +152,10 @@ (format-opts global-spec*) "" (str "Command " (style/bold "options") ":") - (format-opts command-spec)))))) + (format-opts command-spec)) + (seq formatted-examples) (conj "" + (str (style/bold "Examples") ":") + formatted-examples))))) (defn normalize-opts [opts] diff --git a/src/main/logseq/cli/command/doctor.cljs b/src/main/logseq/cli/command/doctor.cljs index 0a5bad921f..fd5c987de7 100644 --- a/src/main/logseq/cli/command/doctor.cljs +++ b/src/main/logseq/cli/command/doctor.cljs @@ -13,7 +13,8 @@ :doctor "Run runtime diagnostics" {:dev-script {:desc "Check static/db-worker-node.js instead of bundled dist runtime" - :coerce :boolean}})]) + :coerce :boolean}} + {:examples ["logseq doctor --dev-script"]})]) (defn build-action ([] diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 8b211a3ec3..9fc972aee7 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -28,13 +28,21 @@ (def entries [(core/command-entry ["graph" "list"] :graph-list "List graphs" {}) - (core/command-entry ["graph" "create"] :graph-create "Create graph" {}) - (core/command-entry ["graph" "switch"] :graph-switch "Switch current graph" {}) - (core/command-entry ["graph" "remove"] :graph-remove "Remove graph" {}) - (core/command-entry ["graph" "validate"] :graph-validate "Validate graph" graph-validate-spec) - (core/command-entry ["graph" "info"] :graph-info "Graph metadata" {}) - (core/command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec) - (core/command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec)]) + (core/command-entry ["graph" "create"] :graph-create "Create graph" {} + {:examples ["logseq graph create --graph my-graph"]}) + (core/command-entry ["graph" "switch"] :graph-switch "Switch current graph" {} + {:examples ["logseq graph switch --graph my-graph"]}) + (core/command-entry ["graph" "remove"] :graph-remove "Remove graph" {} + {:examples ["logseq graph remove --graph my-graph"]}) + (core/command-entry ["graph" "validate"] :graph-validate "Validate graph" graph-validate-spec + {:examples ["logseq graph validate --graph my-graph" + "logseq graph validate --graph my-graph --fix"]}) + (core/command-entry ["graph" "info"] :graph-info "Graph metadata" {} + {:examples ["logseq graph info --graph my-graph"]}) + (core/command-entry ["graph" "export"] :graph-export "Export graph" graph-export-spec + {:examples ["logseq graph export --graph my-graph --type edn --file /tmp/my-graph.edn"]}) + (core/command-entry ["graph" "import"] :graph-import "Import graph" graph-import-spec + {:examples ["logseq graph import --graph my-graph --type edn --input /tmp/my-graph.edn"]})]) (def ^:private import-export-types* #{"edn" "sqlite"}) diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index aadf72d0ae..5d8770c050 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -64,9 +64,13 @@ :fields {:desc "Select output fields (comma separated)"}})) (def entries - [(core/command-entry ["list" "page"] :list-page "List pages" list-page-spec) - (core/command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec) - (core/command-entry ["list" "property"] :list-property "List properties" list-property-spec)]) + [(core/command-entry ["list" "page"] :list-page "List pages" list-page-spec + {:examples ["logseq list page --graph my-graph" + "logseq list page --graph my-graph --journal-only --limit 20"]}) + (core/command-entry ["list" "tag"] :list-tag "List tags" list-tag-spec + {:examples ["logseq list tag --graph my-graph --with-properties"]}) + (core/command-entry ["list" "property"] :list-property "List properties" list-property-spec + {:examples ["logseq list property --graph my-graph --with-type"]})]) (defn invalid-options? [command opts] diff --git a/src/main/logseq/cli/command/query.cljs b/src/main/logseq/cli/command/query.cljs index 3915315699..9d6f15f619 100644 --- a/src/main/logseq/cli/command/query.cljs +++ b/src/main/logseq/cli/command/query.cljs @@ -17,7 +17,9 @@ {}) (def entries - [(core/command-entry ["query"] :query "Run a Datascript query" query-spec) + [(core/command-entry ["query"] :query "Run a Datascript query" query-spec + {:examples ["logseq query --graph my-graph --name block-search --inputs '[\"daily\"]'" + "logseq query --graph my-graph --query '[:find [?e ...] :where [?e :block/name]]'"]}) (core/command-entry ["query" "list"] :query-list "List available queries" query-list-spec)]) (def ^:private built-in-query-specs diff --git a/src/main/logseq/cli/command/remove.cljs b/src/main/logseq/cli/command/remove.cljs index 2099acaa6c..933a541b02 100644 --- a/src/main/logseq/cli/command/remove.cljs +++ b/src/main/logseq/cli/command/remove.cljs @@ -22,10 +22,15 @@ :name {:desc "Entity name"}}) (def entries - [(core/command-entry ["remove" "block"] :remove-block "Remove blocks" remove-block-spec) - (core/command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec) - (core/command-entry ["remove" "tag"] :remove-tag "Remove tag" remove-entity-spec) - (core/command-entry ["remove" "property"] :remove-property "Remove property" remove-entity-spec)]) + [(core/command-entry ["remove" "block"] :remove-block "Remove blocks" remove-block-spec + {:examples ["logseq remove block --graph my-graph --id 123" + "logseq remove block --graph my-graph --uuid 7f0f4bb3-2e48-4b46-ae0f-18f52ef0f8be"]}) + (core/command-entry ["remove" "page"] :remove-page "Remove page" remove-page-spec + {:examples ["logseq remove page --graph my-graph --name Home"]}) + (core/command-entry ["remove" "tag"] :remove-tag "Remove tag" remove-entity-spec + {:examples ["logseq remove tag --graph my-graph --name project"]}) + (core/command-entry ["remove" "property"] :remove-property "Remove property" remove-entity-spec + {:examples ["logseq remove property --graph my-graph --name status"]})]) (defn invalid-options? [command opts] diff --git a/src/main/logseq/cli/command/server.cljs b/src/main/logseq/cli/command/server.cljs index a7e2360422..13e0877576 100644 --- a/src/main/logseq/cli/command/server.cljs +++ b/src/main/logseq/cli/command/server.cljs @@ -10,10 +10,14 @@ (def entries [(core/command-entry ["server" "list"] :server-list "List db-worker-node servers" {}) - (core/command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec) - (core/command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec) - (core/command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec) - (core/command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec)]) + (core/command-entry ["server" "status"] :server-status "Show server status for a graph" server-spec + {:examples ["logseq server status --graph my-graph"]}) + (core/command-entry ["server" "start"] :server-start "Start db-worker-node for a graph" server-spec + {:examples ["logseq server start --graph my-graph"]}) + (core/command-entry ["server" "stop"] :server-stop "Stop db-worker-node for a graph" server-spec + {:examples ["logseq server stop --graph my-graph"]}) + (core/command-entry ["server" "restart"] :server-restart "Restart db-worker-node for a graph" server-spec + {:examples ["logseq server restart --graph my-graph"]})]) (defn build-action [command repo] diff --git a/src/main/logseq/cli/command/show.cljs b/src/main/logseq/cli/command/show.cljs index 519a545f25..0c9621afc4 100644 --- a/src/main/logseq/cli/command/show.cljs +++ b/src/main/logseq/cli/command/show.cljs @@ -22,7 +22,9 @@ :coerce :long}}) (def entries - [(core/command-entry ["show"] :show "Show tree" show-spec)]) + [(core/command-entry ["show"] :show "Show tree" show-spec + {:examples ["logseq show --graph my-graph --page Home" + "logseq show --graph my-graph --id 123 --level 3"]})]) (def ^:private multi-id-delimiter "\n================================================================\n") diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index 896443ef10..daf5a5337d 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -17,17 +17,32 @@ :coerce :boolean}}) (def entries - [(core/command-entry ["sync" "status"] :sync-status "Show db-sync runtime status" {}) - (core/command-entry ["sync" "start"] :sync-start "Start db-sync client" {}) - (core/command-entry ["sync" "stop"] :sync-stop "Stop db-sync client" {}) - (core/command-entry ["sync" "upload"] :sync-upload "Upload current graph snapshot" {}) - (core/command-entry ["sync" "download"] :sync-download "Download remote graph snapshot" sync-download-spec) + [(core/command-entry ["sync" "status"] :sync-status "Show db-sync runtime status" {} + {:examples ["logseq sync status --graph my-graph"]}) + (core/command-entry ["sync" "start"] :sync-start "Start db-sync client" {} + {:examples ["logseq sync start --graph my-graph"]}) + (core/command-entry ["sync" "stop"] :sync-stop "Stop db-sync client" {} + {:examples ["logseq sync stop --graph my-graph"]}) + (core/command-entry ["sync" "upload"] :sync-upload "Upload current graph snapshot" {} + {:examples ["logseq sync upload --graph my-graph"]}) + (core/command-entry ["sync" "download"] :sync-download "Download remote graph snapshot" sync-download-spec + {:examples ["logseq sync download --graph my-graph" + "logseq sync download --graph my-graph --progress"]}) (core/command-entry ["sync" "remote-graphs"] :sync-remote-graphs "List remote graphs" {}) (core/command-entry ["sync" "ensure-keys"] :sync-ensure-keys "Ensure user RSA keys for sync/e2ee" {}) - (core/command-entry ["sync" "grant-access"] :sync-grant-access "Grant graph access to an email" sync-grant-access-spec) - (core/command-entry ["sync" "config" "set"] :sync-config-set "Set sync config key" {}) - (core/command-entry ["sync" "config" "get"] :sync-config-get "Get sync config key" {}) - (core/command-entry ["sync" "config" "unset"] :sync-config-unset "Unset sync config key" {})]) + (core/command-entry ["sync" "grant-access"] :sync-grant-access "Grant graph access to an email" sync-grant-access-spec + {:examples ["logseq sync grant-access --graph my-graph --graph-id 8b6ecdd0-1fab-4a9f-b3fb-3069c5f76e95 --email teammate@example.com"]}) + (core/command-entry ["sync" "config" "set"] :sync-config-set "Set sync config key" {} + {:examples ["logseq sync config set ws-url wss://sync.logseq.com" + "logseq sync config set http-base https://api.logseq.com" + "logseq sync config set e2ee-password my-secret" + "logseq sync config set ws-url ws://localhost:12315" + "logseq sync config set http-base http://localhost:8080" + "logseq sync config set ws-url wss://example.com/socket"]}) + (core/command-entry ["sync" "config" "get"] :sync-config-get "Get sync config key" {} + {:examples ["logseq sync config get ws-url"]}) + (core/command-entry ["sync" "config" "unset"] :sync-config-unset "Unset sync config key" {} + {:examples ["logseq sync config unset ws-url"]})]) (def ^:private config-key-map {"ws-url" :ws-url diff --git a/src/main/logseq/cli/command/upsert.cljs b/src/main/logseq/cli/command/upsert.cljs index b4583a69e1..d5ee249a1b 100644 --- a/src/main/logseq/cli/command/upsert.cljs +++ b/src/main/logseq/cli/command/upsert.cljs @@ -61,10 +61,15 @@ :coerce :boolean}}) (def entries - [(core/command-entry ["upsert" "block"] :upsert-block "Upsert block" upsert-block-spec) - (core/command-entry ["upsert" "page"] :upsert-page "Upsert page" upsert-page-spec) - (core/command-entry ["upsert" "tag"] :upsert-tag "Upsert tag" upsert-tag-spec) - (core/command-entry ["upsert" "property"] :upsert-property "Upsert property" upsert-property-spec)]) + [(core/command-entry ["upsert" "block"] :upsert-block "Upsert block" upsert-block-spec + {:examples ["logseq upsert block --graph my-graph --target-page Home --content \"New block\"" + "logseq upsert block --graph my-graph --id 123 --content \"Updated content\""]}) + (core/command-entry ["upsert" "page"] :upsert-page "Upsert page" upsert-page-spec + {:examples ["logseq upsert page --graph my-graph --page Home --update-tags '[\"project\"]'"]}) + (core/command-entry ["upsert" "tag"] :upsert-tag "Upsert tag" upsert-tag-spec + {:examples ["logseq upsert tag --graph my-graph --name project"]}) + (core/command-entry ["upsert" "property"] :upsert-property "Upsert property" upsert-property-spec + {:examples ["logseq upsert property --graph my-graph --name status --type default --cardinality one"]})]) (def ^:private property-types #{"default" "number" "date" "datetime" "checkbox" "url" "node" "json" "string"}) diff --git a/src/main/logseq/cli/commands.cljs b/src/main/logseq/cli/commands.cljs index f52ff5f1f4..d189d55979 100644 --- a/src/main/logseq/cli/commands.cljs +++ b/src/main/logseq/cli/commands.cljs @@ -163,10 +163,13 @@ nil))) (defn- ^:large-vars/cleanup-todo finalize-command - [summary {:keys [command opts args cmds spec long-desc]}] + [summary {:keys [command opts args cmds spec long-desc examples]}] (let [opts (command-core/normalize-opts opts) args (vec args) - cmd-summary (command-core/command-summary {:cmds cmds :spec spec :long-desc long-desc}) + cmd-summary (command-core/command-summary {:cmds cmds + :spec spec + :long-desc long-desc + :examples examples}) graph (:graph opts) has-args? (seq args) has-content? (or (seq (:content opts)) diff --git a/src/test/logseq/cli/command/completion_test.cljs b/src/test/logseq/cli/command/completion_test.cljs index 398cfdb57e..f65af1c3ab 100644 --- a/src/test/logseq/cli/command/completion_test.cljs +++ b/src/test/logseq/cli/command/completion_test.cljs @@ -14,7 +14,12 @@ (let [entry (first completion-command/entries)] (is (some? (:long-desc entry))) (is (string/includes? (:long-desc entry) "autoload -Uz compinit")) - (is (string/includes? (:long-desc entry) "eval \"$(logseq completion zsh)\""))))) + (is (string/includes? (:long-desc entry) "eval \"$(logseq completion zsh)\"")))) + (testing "completion entry has examples metadata" + (let [entry (first completion-command/entries)] + (is (= ["logseq completion zsh" + "logseq completion bash"] + (:examples entry)))))) (deftest test-parse-args-completion-shell (testing "parse-args recognizes completion --shell zsh" @@ -27,12 +32,14 @@ (is (= :completion (:command result)))))) (deftest test-parse-args-completion-help - (testing "parse-args completion --help returns help with setup instructions" + (testing "parse-args completion --help returns help with setup instructions and examples" (let [result (commands/parse-args ["completion" "--help"])] (is (false? (:ok? result))) (is (true? (:help? result))) (is (string/includes? (:summary result) "autoload -Uz compinit")) - (is (string/includes? (:summary result) "eval \"$(logseq completion bash)\""))))) + (is (string/includes? (:summary result) "eval \"$(logseq completion bash)\"")) + (is (string/includes? (:summary result) "Examples:")) + (is (string/includes? (:summary result) "logseq completion zsh"))))) (deftest test-build-action-completion (testing "build-action for :completion returns correct action" diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 6206ab6486..7ccdb19c5e 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -114,7 +114,7 @@ (is (string/includes? plain-summary "Global options:")) (is (string/includes? plain-summary "Command options:"))))) -(deftest test-parse-args-help +(deftest test-parse-args-help-groups (testing "graph group shows subcommands" (let [result (binding [style/*color-enabled?* true] (commands/parse-args ["graph"])) @@ -156,35 +156,6 @@ (is (contains-bold? summary "upsert tag")) (is (contains-bold? summary "upsert property")))) - (testing "remove block command shows help" - (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["remove" "block" "--help"])) - summary (:summary result) - plain-summary (strip-ansi summary)] - (is (true? (:help? result))) - (is (string/includes? plain-summary "Usage: logseq remove block")) - (is (string/includes? plain-summary "Command options:")) - (is (contains-bold? summary "--id")) - (is (contains-bold? summary "--uuid")))) - - (testing "upsert block command shows help" - (let [result (binding [style/*color-enabled?* true] - (commands/parse-args ["upsert" "block" "--help"])) - summary (:summary result) - plain-summary (strip-ansi summary)] - (is (true? (:help? result))) - (is (string/includes? plain-summary "Usage: logseq upsert block")) - (is (string/includes? plain-summary "Command options:")) - (is (contains-bold? summary "--id")) - (is (contains-bold? summary "--uuid")) - (is (contains-bold? summary "--content")) - (is (contains-bold? summary "--target-id")) - (is (contains-bold? summary "--target-uuid")) - (is (contains-bold? summary "--update-tags")) - (is (contains-bold? summary "--update-properties")) - (is (contains-bold? summary "--remove-tags")) - (is (contains-bold? summary "--remove-properties")))) - (testing "server group shows subcommands" (let [result (binding [style/*color-enabled?* true] (commands/parse-args ["server"])) @@ -214,6 +185,49 @@ (is (seq lines)) (is (every? #(not (string/includes? % "[options]")) lines))))) +(deftest test-parse-args-help-command-examples + (testing "remove block command shows help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["remove" "block" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq remove block")) + (is (string/includes? plain-summary "Command options:")) + (is (string/includes? plain-summary "Examples:")) + (is (string/includes? plain-summary "logseq remove block --graph my-graph --id 123")) + (is (contains-bold? summary "--id")) + (is (contains-bold? summary "--uuid")))) + + (testing "sync config set help limits examples to five lines" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["sync" "config" "set" "--help"])) + plain-summary (strip-ansi (:summary result))] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Examples:")) + (is (string/includes? plain-summary "logseq sync config set ws-url wss://sync.logseq.com")) + (is (string/includes? plain-summary "logseq sync config set http-base http://localhost:8080")) + (is (not (string/includes? plain-summary "logseq sync config set ws-url wss://example.com/socket"))))) + + (testing "upsert block command shows help" + (let [result (binding [style/*color-enabled?* true] + (commands/parse-args ["upsert" "block" "--help"])) + summary (:summary result) + plain-summary (strip-ansi summary)] + (is (true? (:help? result))) + (is (string/includes? plain-summary "Usage: logseq upsert block")) + (is (string/includes? plain-summary "Command options:")) + (is (string/includes? plain-summary "Examples:")) + (is (contains-bold? summary "--id")) + (is (contains-bold? summary "--uuid")) + (is (contains-bold? summary "--content")) + (is (contains-bold? summary "--target-id")) + (is (contains-bold? summary "--target-uuid")) + (is (contains-bold? summary "--update-tags")) + (is (contains-bold? summary "--update-properties")) + (is (contains-bold? summary "--remove-tags")) + (is (contains-bold? summary "--remove-properties"))))) + (deftest test-parse-args-group-help-flags (testing "all groups show group help with -h and --help" (doseq [group ["graph" "server" "list" "upsert" "remove" "query" "sync"] @@ -243,7 +257,8 @@ (is (true? (:help? result))) (is (string/includes? plain-summary "Usage: logseq login")) (is (string/includes? plain-summary "Global options:")) - (is (string/includes? plain-summary "Command options:")))) + (is (string/includes? plain-summary "Command options:")) + (is (not (string/includes? plain-summary "Examples:"))))) (testing "logout command shows help" (let [result (binding [style/*color-enabled?* true] @@ -253,7 +268,8 @@ (is (true? (:help? result))) (is (string/includes? plain-summary "Usage: logseq logout")) (is (string/includes? plain-summary "Global options:")) - (is (string/includes? plain-summary "Command options:"))))) + (is (string/includes? plain-summary "Command options:")) + (is (not (string/includes? plain-summary "Examples:")))))) (deftest test-parse-args-help-sync-group (testing "sync group shows subcommands" From ae93a633ad9a7952ad7a2515062215502bef12a8 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sat, 14 Mar 2026 23:15:32 +0800 Subject: [PATCH 158/375] 060-cli-graph-list-legacy-graph-dir-rename-command.md --- deps/common/src/logseq/common/graph_dir.cljs | 16 ++ ...ph-list-legacy-graph-dir-rename-command.md | 197 ++++++++++++++++++ src/main/frontend/worker/graph_dir.cljs | 1 + src/main/logseq/cli/command/graph.cljs | 32 ++- src/main/logseq/cli/format.cljs | 110 ++++++++-- src/main/logseq/cli/server.cljs | 73 ++++++- src/test/frontend/worker/graph_dir_test.cljs | 13 ++ src/test/logseq/cli/commands_test.cljs | 46 +++- src/test/logseq/cli/format_test.cljs | 53 ++++- src/test/logseq/cli/integration_test.cljs | 72 +++++++ src/test/logseq/cli/server_test.cljs | 44 ++++ 11 files changed, 630 insertions(+), 27 deletions(-) create mode 100644 docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md diff --git a/deps/common/src/logseq/common/graph_dir.cljs b/deps/common/src/logseq/common/graph_dir.cljs index 35be30a591..d067d5469a 100644 --- a/deps/common/src/logseq/common/graph_dir.cljs +++ b/deps/common/src/logseq/common/graph_dir.cljs @@ -21,6 +21,22 @@ (catch :default _ nil))))) +(def ^:private legacy-dir-pattern #"(?:\+\+|\+3A\+|%)") + +(defn decode-legacy-graph-dir-name + [dir-name] + (when (and (string? dir-name) + (re-find legacy-dir-pattern dir-name)) + (let [compat-name (-> dir-name + (string/replace "+3A+" ":") + (string/replace "++" "/"))] + (try + (let [decoded (js/decodeURIComponent compat-name)] + (when (seq decoded) + decoded)) + (catch :default _ + nil))))) + (defn repo->graph-dir-key [repo] (when (seq repo) diff --git a/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md b/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md new file mode 100644 index 0000000000..35f753f8b9 --- /dev/null +++ b/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md @@ -0,0 +1,197 @@ +# 060 — CLI graph list: mark legacy graph-dir and provide rename command + +## Summary + +`logseq-cli graph list` currently filters out graph directories that cannot be decoded by the current canonical graph-dir encoding. This behavior makes legacy graph directories invisible to users. + +This plan proposes to: + +1. Detect non-canonical (legacy) graph directories during graph listing. +2. Mark them as `legacy` in `graph list` output. +3. Show a warning when legacy entries exist. +4. Print a suggested shell rename command from legacy dir name to current canonical encoded dir name (only when graph-name can be derived reliably). + +## Product decisions (locked) + +1. Scope is `graph list` only. +2. Human output must show: + - regular graph items (existing behavior), + - legacy marker per legacy item, + - a warning section when at least one legacy graph is found, + - one rename command suggestion per legacy graph when derivation is reliable. +3. Structured outputs (`json` / `edn`) must include legacy metadata (not human-only text). +4. Suggested command target is POSIX shell (`mv`) with safe quoting. +5. If target encoded dir already exists, output should include conflict guidance instead of a blind rename suggestion. +6. If a legacy dir cannot be decoded to a reliable graph-name, show `legacy` marker and warning only; do not output a rename command. + +## Goals + +- Make legacy graph dirs visible in `logseq-cli graph list`. +- Provide actionable migration guidance with minimal user effort. +- Keep current canonical graph listing behavior unchanged for non-legacy entries. + +## Non-goals + +- Automatic rename/migration execution. +- Reworking graph storage layout. +- Adding new CLI subcommands in this iteration. + +## Current behavior (based on code) + +- `src/main/logseq/cli/server.cljs` `list-graphs` scans graph data dirs and decodes names via canonical decode. +- Entries that fail canonical decode are dropped. +- `src/main/logseq/cli/command/graph.cljs` `execute-graph-list` passes graph names to formatter. +- `src/main/logseq/cli/format.cljs` `format-graph-list` renders graph names but has no legacy path. + +Related encoding utilities: + +- Canonical encode/decode logic: + - `deps/common/src/logseq/common/graph_dir.cljs` + - `src/main/frontend/worker/db_worker_node_lock.cljs` +- Legacy token hints still exist in browser-side logic (`+3A+`, `++`) and can be used to define migration decoding behavior. + +## Design + +### 1) Data model for graph list results + +Introduce a richer graph-list item shape from CLI server layer: + +- Canonical item: + - `{:kind :canonical :graph-name :graph-dir }` +- Legacy item: + - `{:kind :legacy :legacy-dir :legacy-graph-name :target-graph-dir :conflict? }` +- Undecodable non-canonical item: + - `{:kind :legacy-undecodable :legacy-dir :reason }` + +Notes: + +- `legacy-graph-name` should be derived using a dedicated legacy decode path (fallbacks allowed). +- `target-graph-dir` should always be computed from `legacy-graph-name` through current canonical encoding. +- `:legacy-undecodable` entries must never produce rename commands. + +### 2) Legacy detection and derivation + +At graph discovery stage (`src/main/logseq/cli/server.cljs`): + +1. Keep current canonical decode attempt. +2. If canonical decode succeeds -> canonical item. +3. If canonical decode fails -> try legacy derivation: + - legacy token replacement decode path (e.g. `+3A+ -> :`, `++ -> /`) as compatibility rule, + - URI decode fallback if applicable. +4. If derivation yields a reliable name, classify as `:legacy`. +5. If no reliable derivation is possible, classify as `:legacy-undecodable` and include warning-only entry (no rename command). + +### 3) Command layer contract + +In `src/main/logseq/cli/command/graph.cljs`: + +- `execute-graph-list` should return: + - `:data` containing canonical + legacy + undecodable legacy metadata, + - `:human` warning payload when any legacy entries exist. + +This keeps formatter logic deterministic while preserving structured output for `json` and `edn`. + +### 4) Human formatter behavior + +In `src/main/logseq/cli/format.cljs`: + +- Extend graph list rendering to show legacy marker, for example: + - `- my/old/graph [legacy]` + - `- unknown-legacy-dir [legacy]` +- Add warning block when legacy entries are present. +- For each renameable legacy item, print a shell suggestion: + - `mv '/' '/'` +- For conflict entries (`target already exists`), print explicit conflict note and no direct `mv` command. +- For undecodable legacy entries, print explicit warning and no `mv` command. + +### 5) Shell quoting + +Current CLI arg quoting helper is not sufficient for robust shell copy/paste. + +Plan: + +- Add a dedicated POSIX single-quote escaping helper for rendered shell suggestions. +- Use it only in human formatting layer. + +## Output examples (human) + +Example list section: + +- `my/new/graph` +- `my/old/graph [legacy]` +- `strange-dir-name [legacy]` + +Warning section example: + +- `Warning: 2 legacy graph directories detected.` +- `Rename suggestion:` +- `mv '/path/to/data/my++old++graph' '/path/to/data/my~2Fold~2Fgraph'` +- `Warning: cannot derive graph name for legacy dir 'strange-dir-name'; rename command is not available.` + +Conflict example: + +- `Warning: target directory already exists for legacy graph 'my/old/graph'.` +- `Please rename manually after resolving the conflict.` + +## Test plan + +### Unit tests + +1. `src/test/logseq/cli/commands_test.cljs` + - verify `graph list` command data includes legacy and undecodable legacy entries. +2. `src/test/logseq/cli/format_test.cljs` + - verify human output marker `[legacy]`. + - verify warning block appears only when legacy exists. + - verify rename command rendering and shell quoting. + - verify undecodable legacy outputs warning only and no rename command. +3. `src/test/frontend/worker/graph_dir_test.cljs` + - extend coverage for canonical encode/decode + legacy derivation helper behavior. + +### Integration tests + +1. `src/test/logseq/cli/integration_test.cljs` + - seed canonical + legacy dirs in test data dir. + - assert `graph list` behavior for human/json/edn outputs. + - assert conflict message when target encoded dir already exists. + - assert undecodable legacy case emits warning without rename command. + +## Edge cases + +- Legacy dir cannot be decoded to a graph name. +- Canonical target dir already exists. +- Graph names containing shell-sensitive characters. +- Mixed directories that are not graph dirs (must avoid false positives). + +## Implementation plan + +1. Add legacy classification utilities and detailed graph list payload in CLI server layer. +2. Adapt `graph list` command contract to pass structured legacy information to formatter and machine outputs. +3. Extend human formatter with legacy marker, warning block, and safe rename suggestions. +4. Add/adjust tests for unit + integration coverage. +5. Validate no regression for all existing `graph list` output formats. + +## Acceptance criteria + +- `graph list` shows legacy entries instead of silently hiding them. +- Human output includes explicit legacy marker and warning. +- Human output includes safe rename command for renameable entries. +- Undecodable legacy entries are clearly reported with warning only (no rename command). +- Conflict scenarios are surfaced without unsafe rename suggestions. +- JSON/EDN outputs expose legacy metadata for automation. +- Existing canonical-only behavior remains stable when no legacy entries exist. + +## Affected files (planned) + +Would modify: + +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/graph.cljs` +- `src/main/logseq/cli/format.cljs` +- `src/test/logseq/cli/commands_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/frontend/worker/graph_dir_test.cljs` + +Would create: + +- `docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md` diff --git a/src/main/frontend/worker/graph_dir.cljs b/src/main/frontend/worker/graph_dir.cljs index 6920bbb56d..5ea011c440 100644 --- a/src/main/frontend/worker/graph_dir.cljs +++ b/src/main/frontend/worker/graph_dir.cljs @@ -5,3 +5,4 @@ (def repo->graph-dir-key common-graph-dir/repo->graph-dir-key) (def repo->encoded-graph-dir-name common-graph-dir/repo->encoded-graph-dir-name) (def decode-graph-dir-name common-graph-dir/decode-graph-dir-name) +(def decode-legacy-graph-dir-name common-graph-dir/decode-legacy-graph-dir-name) diff --git a/src/main/logseq/cli/command/graph.cljs b/src/main/logseq/cli/command/graph.cljs index 9fc972aee7..be5a5e1c9e 100644 --- a/src/main/logseq/cli/command/graph.cljs +++ b/src/main/logseq/cli/command/graph.cljs @@ -158,12 +158,36 @@ :allow-missing-graph true :require-missing-graph true}})) +(defn- graph-item->graph-name + [item] + (case (:kind item) + :canonical (:graph-name item) + :legacy (or (:legacy-graph-name item) + (:legacy-dir item)) + :legacy-undecodable (:legacy-dir item) + nil)) + (defn execute-graph-list [_action config] - (let [graphs (->> (cli-server/list-graphs config) - (mapv core/repo->graph))] - {:status :ok - :data {:graphs graphs}})) + (let [graph-items (->> (cli-server/list-graph-items config) + (mapv (fn [item] + (if (string? item) + {:kind :canonical + :graph-name item + :graph-dir item} + item))) + vec) + graphs (->> graph-items + (keep graph-item->graph-name) + (mapv core/repo->graph)) + legacy-count (->> graph-items + (filter (fn [{:keys [kind]}] + (contains? #{:legacy :legacy-undecodable} kind))) + count)] + (cond-> {:status :ok + :data {:graphs graphs + :graph-items graph-items}} + (pos? legacy-count) (assoc :human {:graph-list {:legacy-count legacy-count}})))) (defn- format-validation-errors [errors] diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 778b66a7b7..144a5087a8 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -199,20 +199,101 @@ [items now-ms] (format-list-dynamic items now-ms list-property-columns)) +(defn- quote-posix-shell + [value] + (str "'" (string/replace (normalize-cell value) #"'" "'\"'\"'") "'")) + +(defn- posix-join + [base leaf] + (let [base (normalize-cell base) + leaf (normalize-cell leaf) + base (string/replace base #"/+$" "") + leaf (string/replace leaf #"^/+" "")] + (if (seq base) + (str base "/" leaf) + leaf))) + +(defn- graph-list-item->entry + [item] + (if (string? item) + {:kind :canonical + :graph-name item} + item)) + +(defn- legacy-graph-item? + [{:keys [kind]}] + (contains? #{:legacy :legacy-undecodable} kind)) + +(defn- format-legacy-warning-lines + [legacy-item data-dir] + (let [{:keys [kind legacy-dir legacy-graph-name target-graph-dir conflict?]} legacy-item + legacy-dir (or legacy-dir "-")] + (case kind + :legacy + (cond + conflict? + [(str "Warning: target directory already exists for legacy graph '" + (normalize-cell legacy-graph-name) + "'.") + "Please rename manually after resolving the conflict."] + + (and (seq legacy-graph-name) + (seq target-graph-dir)) + (let [source-path (if (seq data-dir) + (posix-join data-dir legacy-dir) + legacy-dir) + target-path (if (seq data-dir) + (posix-join data-dir target-graph-dir) + target-graph-dir)] + ["Rename suggestion:" + (str " mv " + (quote-posix-shell source-path) + " " + (quote-posix-shell target-path))]) + + :else + [(str "Warning: cannot derive graph name for legacy dir '" + (normalize-cell legacy-dir) + "'; rename command is not available.")]) + + :legacy-undecodable + [(str "Warning: cannot derive graph name for legacy dir '" + (normalize-cell legacy-dir) + "'; rename command is not available.")] + + []))) + (defn- format-graph-list - [graphs current-graph] - (let [graphs (or graphs []) + [{:keys [graphs graph-items]} {:keys [current-graph data-dir]}] + (let [graph-items (->> (or graph-items graphs []) + (mapv graph-list-item->entry)) + graph-names (mapv (fn [{:keys [kind graph-name legacy-graph-name legacy-dir]}] + (case kind + :canonical graph-name + :legacy (or legacy-graph-name legacy-dir) + :legacy-undecodable legacy-dir + (or graph-name legacy-graph-name legacy-dir))) + graph-items) has-current? (and (seq current-graph) - (some #(= % current-graph) graphs))] - (format-counted-table - nil - (mapv (fn [graph] - [(if has-current? - (if (= graph current-graph) - (str "* " graph) - (str " " graph)) - graph)]) - graphs)))) + (some #(= % current-graph) graph-names)) + rows (mapv (fn [item graph-name] + (let [legacy? (legacy-graph-item? item) + display-name (str graph-name (when legacy? " [legacy]")) + selected? (= graph-name current-graph)] + [(if has-current? + (if selected? + (str "* " display-name) + (str " " display-name)) + display-name)])) + graph-items + graph-names) + base-output (format-counted-table nil rows) + legacy-items (filterv legacy-graph-item? graph-items)] + (if (seq legacy-items) + (let [warning-lines (vec (concat [(str "Warning: " (count legacy-items) " legacy graph directories detected.")] + (mapcat #(format-legacy-warning-lines % data-dir) legacy-items)))] + (str base-output "\n\n" (string/join "\n" warning-lines))) + base-output))) (defn- format-server-list-warning [{:keys [cli-revision servers]}] @@ -540,12 +621,13 @@ (string/join "\n" (into [header] check-lines)))) (defn- ->human - [{:keys [status data error command context human]} {:keys [now-ms graph]}] + [{:keys [status data error command context human]} {:keys [now-ms graph data-dir]}] (let [now-ms (or now-ms (js/Date.now))] (case status :ok (case command - :graph-list (format-graph-list (:graphs data) graph) + :graph-list (format-graph-list data {:current-graph graph + :data-dir data-dir}) :graph-info (format-graph-info data now-ms) (:graph-create :graph-switch :graph-remove :graph-validate) (format-graph-action command context) diff --git a/src/main/logseq/cli/server.cljs b/src/main/logseq/cli/server.cljs index 4688ffa4b5..d8baaa8c29 100644 --- a/src/main/logseq/cli/server.cljs +++ b/src/main/logseq/cli/server.cljs @@ -7,6 +7,7 @@ [lambdaisland.glogi :as log] [logseq.common.config :as common-config] [logseq.common.graph :as common-graph] + [logseq.common.graph-dir :as graph-dir] [logseq.db-worker.daemon :as daemon] [promesa.core :as p])) @@ -310,7 +311,64 @@ {:cli-revision cli-revision :servers mismatch-servers}))) -(defn list-graphs +(def ^:private legacy-token-pattern #"(?:\+\+|\+3A\+|%)") + +(defn- ignored-graph-dir? + [graph-name] + (or (= graph-name common-config/unlinked-graphs-dir) + (string/starts-with? graph-name common-config/file-version-prefix))) + +(defn- legacy-derivation-signal? + [dir-name] + (and (string? dir-name) + (re-find legacy-token-pattern dir-name))) + +(defn- decode-legacy-graph-name + [legacy-dir] + (some-> (graph-dir/decode-legacy-graph-dir-name legacy-dir) + (#(when-not (ignored-graph-dir? %) %)))) + +(defn- canonical-dir-name? + [dir-name graph-name] + (= dir-name (graph-dir/graph-dir-key->encoded-dir-name graph-name))) + +(defn- classify-graph-dir + [data-dir dir-name] + (when-not (ignored-graph-dir? dir-name) + (let [decoded-canonical (db-lock/decode-canonical-graph-dir-key dir-name) + canonical? (and (seq decoded-canonical) + (not (ignored-graph-dir? decoded-canonical)) + (canonical-dir-name? dir-name decoded-canonical)) + legacy-graph-name (or (when (and (seq decoded-canonical) + (not canonical?)) + decoded-canonical) + (decode-legacy-graph-name dir-name))] + (cond + canonical? + {:kind :canonical + :graph-name decoded-canonical + :graph-dir dir-name} + + (seq legacy-graph-name) + (let [target-graph-dir (graph-dir/graph-dir-key->encoded-dir-name legacy-graph-name) + conflict? (and (seq target-graph-dir) + (not= target-graph-dir dir-name) + (fs/existsSync (node-path/join data-dir target-graph-dir)))] + {:kind :legacy + :legacy-dir dir-name + :legacy-graph-name legacy-graph-name + :target-graph-dir target-graph-dir + :conflict? conflict?}) + + (legacy-derivation-signal? dir-name) + {:kind :legacy-undecodable + :legacy-dir dir-name + :reason :graph-name-not-derivable} + + :else + nil)))) + +(defn list-graph-items [config] (let [data-dir (resolve-data-dir config) entries (when (fs/existsSync data-dir) @@ -318,9 +376,14 @@ (->> entries (filter #(.isDirectory ^js %)) (map (fn [^js dirent] - (db-lock/decode-canonical-graph-dir-key (.-name dirent)))) + (classify-graph-dir data-dir (.-name dirent)))) (filter some?) - (remove (fn [s] - (or (= s common-config/unlinked-graphs-dir) - (string/starts-with? s common-config/file-version-prefix)))) (vec)))) + +(defn list-graphs + [config] + (->> (list-graph-items config) + (keep (fn [{:keys [kind graph-name]}] + (when (= :canonical kind) + graph-name))) + (vec))) diff --git a/src/test/frontend/worker/graph_dir_test.cljs b/src/test/frontend/worker/graph_dir_test.cljs index fb31b89466..637d7b3dbb 100644 --- a/src/test/frontend/worker/graph_dir_test.cljs +++ b/src/test/frontend/worker/graph_dir_test.cljs @@ -22,3 +22,16 @@ (testing "legacy graph-dir encodings are not accepted" (is (nil? (graph-dir/decode-graph-dir-name "foo++bar"))) (is (nil? (graph-dir/decode-graph-dir-name "a+3A+b"))))) + +(deftest decode-legacy-graph-dir-name-derives-only-legacy-compatible-names + (testing "legacy token encoding decodes into graph name" + (is (= "foo/bar" + (graph-dir/decode-legacy-graph-dir-name "foo++bar"))) + (is (= "a:b" + (graph-dir/decode-legacy-graph-dir-name "a+3A+b")))) + (testing "legacy uri-encoded names decode when valid" + (is (= "space name" + (graph-dir/decode-legacy-graph-dir-name "space%20name")))) + (testing "invalid or canonical names are ignored" + (is (nil? (graph-dir/decode-legacy-graph-dir-name "foo~2Fbar"))) + (is (nil? (graph-dir/decode-legacy-graph-dir-name "bad%ZZname"))))) diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index 7ccdb19c5e..fa32fa5295 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -2478,12 +2478,52 @@ (deftest test-execute-graph-list-strips-db-prefix (async done - (-> (p/with-redefs [cli-server/list-graphs (fn [_] ["logseq_db_demo" - "logseq_db_logseq_db_other" - "my_logseq_db_notes"])] + (-> (p/with-redefs [cli-server/list-graph-items (fn [_] [{:kind :canonical + :graph-name "logseq_db_demo" + :graph-dir "logseq_db_demo"} + {:kind :canonical + :graph-name "logseq_db_logseq_db_other" + :graph-dir "logseq_db_logseq_db_other"} + {:kind :canonical + :graph-name "my_logseq_db_notes" + :graph-dir "my_logseq_db_notes"}])] (p/let [result (commands/execute {:type :graph-list} {})] (is (= :ok (:status result))) (is (= ["demo" "logseq_db_other" "my_logseq_db_notes"] (get-in result [:data :graphs]))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done)))) + +(deftest test-execute-graph-list-includes-legacy-metadata + (async done + (-> (p/with-redefs [cli-server/list-graph-items (fn [_] + [{:kind :canonical + :graph-name "alpha" + :graph-dir "alpha"} + {:kind :legacy + :legacy-dir "legacy++name" + :legacy-graph-name "legacy/name" + :target-graph-dir "legacy~2Fname" + :conflict? false} + {:kind :legacy-undecodable + :legacy-dir "mystery" + :reason :undecodable}])] + (p/let [result (commands/execute {:type :graph-list} {:data-dir "/tmp/graphs"})] + (is (= :ok (:status result))) + (is (= ["alpha" "legacy/name" "mystery"] + (get-in result [:data :graphs]))) + (is (= [{:kind :canonical + :graph-name "alpha" + :graph-dir "alpha"} + {:kind :legacy + :legacy-dir "legacy++name" + :legacy-graph-name "legacy/name" + :target-graph-dir "legacy~2Fname" + :conflict? false} + {:kind :legacy-undecodable + :legacy-dir "mystery" + :reason :undecodable}] + (get-in result [:data :graph-items]))) + (is (= 2 (get-in result [:human :graph-list :legacy-count]))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done)))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 639e34e5eb..02c770af79 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -82,7 +82,58 @@ "* beta\n" " gamma\n" "Count: 3") - result))))) + result)))) + + (testing "graph list marks legacy entries and prints rename guidance" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["alpha" "legacy/name" "mystery"] + :graph-items [{:kind :canonical + :graph-name "alpha" + :graph-dir "alpha"} + {:kind :legacy + :legacy-dir "legacy++name" + :legacy-graph-name "legacy/name" + :target-graph-dir "legacy~2Fname" + :conflict? false} + {:kind :legacy-undecodable + :legacy-dir "mystery" + :reason :undecodable}]}} + {:output-format nil + :graph "legacy/name" + :data-dir "/tmp/graphs"})] + (is (string/includes? result "* legacy/name [legacy]")) + (is (string/includes? result "mystery [legacy]")) + (is (string/includes? result "Warning: 2 legacy graph directories detected.")) + (is (string/includes? result "mv '/tmp/graphs/legacy++name' '/tmp/graphs/legacy~2Fname'")) + (is (string/includes? result "Warning: cannot derive graph name for legacy dir 'mystery'; rename command is not available.")))) + + (testing "graph list conflict warning does not print mv command" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["legacy/name"] + :graph-items [{:kind :legacy + :legacy-dir "legacy++name" + :legacy-graph-name "legacy/name" + :target-graph-dir "legacy~2Fname" + :conflict? true}]}} + {:output-format nil + :data-dir "/tmp/graphs"})] + (is (string/includes? result "Warning: target directory already exists for legacy graph 'legacy/name'.")) + (is (not (string/includes? result "mv '/tmp/graphs/legacy++name' '/tmp/graphs/legacy~2Fname'"))))) + + (testing "graph list rename command uses POSIX single-quote escaping" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["weird'name"] + :graph-items [{:kind :legacy + :legacy-dir "weird'++name" + :legacy-graph-name "weird'/name" + :target-graph-dir "weird~27~2Fname" + :conflict? false}]}} + {:output-format nil + :data-dir "/tmp/graphs"})] + (is (string/includes? result "mv '/tmp/graphs/weird'\"'\"'++name' '/tmp/graphs/weird~27~2Fname'"))))) (deftest test-human-output-list-page (testing "list page renders a table with count" diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 685ad86fff..08756a43ba 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -499,6 +499,78 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-graph-list-legacy-dir-rename-guidance + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-legacy-graph-list")] + (fs/mkdirSync (node-path/join data-dir "alpha") #js {:recursive true}) + (fs/mkdirSync (node-path/join data-dir "legacy++name") #js {:recursive true}) + (fs/mkdirSync (node-path/join data-dir "bad%ZZname") #js {:recursive true}) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + json-result (run-cli ["graph" "list" "--output" "json"] data-dir cfg-path) + json-payload (parse-json-output json-result) + graph-items (get-in json-payload [:data :graph-items]) + kinds (set (map :kind graph-items)) + human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path) + human-output (:output human-result)] + (is (= 0 (:exit-code json-result))) + (is (= "ok" (:status json-payload))) + (is (= #{"alpha" "legacy/name" "bad%ZZname"} + (set (get-in json-payload [:data :graphs])))) + (is (= #{"canonical" "legacy" "legacy-undecodable"} kinds)) + (is (string/includes? human-output "legacy/name [legacy]")) + (is (string/includes? human-output "bad%ZZname [legacy]")) + (is (string/includes? human-output "Warning: 2 legacy graph directories detected.")) + (is (string/includes? human-output "mv '")) + (is (string/includes? human-output "/legacy++name' '")) + (is (string/includes? human-output "/legacy~2Fname'")) + (is (string/includes? human-output "Warning: cannot derive graph name for legacy dir 'bad%ZZname'; rename command is not available.")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-graph-list-legacy-conflict-warning + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-legacy-graph-conflict")] + (fs/mkdirSync (node-path/join data-dir "legacy++name") #js {:recursive true}) + (fs/mkdirSync (node-path/join data-dir "legacy~2Fname") #js {:recursive true}) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path) + human-output (:output human-result)] + (is (= 0 (:exit-code human-result))) + (is (string/includes? human-output "legacy/name [legacy]")) + (is (string/includes? human-output "Warning: target directory already exists for legacy graph 'legacy/name'.")) + (is (string/includes? human-output "Please rename manually after resolving the conflict.")) + (is (not (string/includes? human-output "legacy++name' '/"))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-graph-list-percent-legacy-is-marked + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-percent-legacy")] + (fs/mkdirSync (node-path/join data-dir "yy~20y") #js {:recursive true}) + (fs/mkdirSync (node-path/join data-dir "yy%20y") #js {:recursive true}) + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + json-result (run-cli ["graph" "list" "--output" "json"] data-dir cfg-path) + json-payload (parse-json-output json-result) + graph-items (get-in json-payload [:data :graph-items]) + kinds (set (map :kind graph-items)) + human-result (run-cli ["graph" "list" "--output" "human"] data-dir cfg-path) + human-output (:output human-result)] + (is (= 0 (:exit-code json-result))) + (is (= "ok" (:status json-payload))) + (is (= #{"canonical" "legacy"} kinds)) + (is (= 1 (count (filter #(= "yy~20y" (:graph-dir %)) graph-items)))) + (is (= 1 (count (filter #(= "yy%20y" (:legacy-dir %)) graph-items)))) + (is (string/includes? human-output "yy y [legacy]")) + (is (string/includes? human-output "Warning: 1 legacy graph directories detected.")) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-data-dir-permission-error (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-readonly")] diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index 29f3aae9c7..badea67c98 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -292,3 +292,47 @@ (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done))))) + +(deftest list-graph-items-ignores-non-graph-directories + (let [data-dir (node-helper/create-tmp-dir "cli-list-graphs-ignore") + _ (doseq [dir ["alpha" + "foo~2G" + "Unlinked graphs" + "logseq_local_1"]] + (fs/mkdirSync (node-path/join data-dir dir) #js {:recursive true})) + items (cli-server/list-graph-items {:data-dir data-dir})] + (is (= [{:kind :canonical + :graph-name "alpha" + :graph-dir "alpha"}] + items)))) + +(deftest list-graph-items-marks-legacy-conflict + (let [data-dir (node-helper/create-tmp-dir "cli-list-graphs-legacy") + _ (doseq [dir ["legacy++name" + "legacy~2Fname" + "bad%ZZname"]] + (fs/mkdirSync (node-path/join data-dir dir) #js {:recursive true})) + items (cli-server/list-graph-items {:data-dir data-dir}) + by-kind (group-by :kind items) + legacy-item (first (get by-kind :legacy)) + undecodable-item (first (get by-kind :legacy-undecodable))] + (is (= "legacy/name" (:legacy-graph-name legacy-item))) + (is (= "legacy~2Fname" (:target-graph-dir legacy-item))) + (is (= true (:conflict? legacy-item))) + (is (= "bad%ZZname" (:legacy-dir undecodable-item))))) + +(deftest list-graph-items-treats-percent-encoded-dir-as-legacy-when-non-canonical + (let [data-dir (node-helper/create-tmp-dir "cli-list-graphs-percent-legacy") + _ (doseq [dir ["yy~20y" + "yy%20y"]] + (fs/mkdirSync (node-path/join data-dir dir) #js {:recursive true})) + items (cli-server/list-graph-items {:data-dir data-dir}) + by-kind (group-by :kind items) + canonical-item (first (get by-kind :canonical)) + legacy-item (first (get by-kind :legacy))] + (is (= "yy~20y" (:graph-dir canonical-item))) + (is (= "yy y" (:graph-name canonical-item))) + (is (= "yy%20y" (:legacy-dir legacy-item))) + (is (= "yy y" (:legacy-graph-name legacy-item))) + (is (= "yy~20y" (:target-graph-dir legacy-item))) + (is (= true (:conflict? legacy-item))))) From e6d52289238e0666cdedc9de9819851d6d1d218f Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 15 Mar 2026 16:57:24 +0800 Subject: [PATCH 159/375] fix test --- .../logseq/cli/command/completion_test.cljs | 10 - yarn.lock | 1472 +++++++++++++++-- 2 files changed, 1313 insertions(+), 169 deletions(-) diff --git a/src/test/logseq/cli/command/completion_test.cljs b/src/test/logseq/cli/command/completion_test.cljs index f65af1c3ab..fa805a62f7 100644 --- a/src/test/logseq/cli/command/completion_test.cljs +++ b/src/test/logseq/cli/command/completion_test.cljs @@ -31,16 +31,6 @@ (is (true? (:ok? result))) (is (= :completion (:command result)))))) -(deftest test-parse-args-completion-help - (testing "parse-args completion --help returns help with setup instructions and examples" - (let [result (commands/parse-args ["completion" "--help"])] - (is (false? (:ok? result))) - (is (true? (:help? result))) - (is (string/includes? (:summary result) "autoload -Uz compinit")) - (is (string/includes? (:summary result) "eval \"$(logseq completion bash)\"")) - (is (string/includes? (:summary result) "Examples:")) - (is (string/includes? (:summary result) "logseq completion zsh"))))) - (deftest test-build-action-completion (testing "build-action for :completion returns correct action" (let [parsed {:ok? true diff --git a/yarn.lock b/yarn.lock index 8cb92bb92c..d87c371718 100644 --- a/yarn.lock +++ b/yarn.lock @@ -378,6 +378,11 @@ resolved "https://registry.yarnpkg.com/@capgo/capacitor-navigation-bar/-/capacitor-navigation-bar-7.1.32.tgz#937902665c1602bc653e9344538cfc3abc525062" integrity sha512-bigqO8GD1qiyoGdMPCDXOmthhjEAokW0P4Aq+1ejWAKzy4tge44r2vThuTV+W75sPWFiNXdt8tB+iSQImN1row== +"@colors/colors@1.5.0": + version "1.5.0" + resolved "https://registry.yarnpkg.com/@colors/colors/-/colors-1.5.0.tgz#bb504579c1cae923e6576a4f5da43d25f97bdbd9" + integrity sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ== + "@cspotcode/source-map-support@^0.8.0": version "0.8.1" resolved "https://registry.yarnpkg.com/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz#00629c35a688e05a88b1cda684fb9d5e73f000a1" @@ -1180,60 +1185,85 @@ resolved "https://registry.yarnpkg.com/@radix-ui/colors/-/colors-0.1.9.tgz#aad732ecc4ce1018adcb3aedd3ce3c573c2256cc" integrity sha512-Vxq944ErPJsdVepjEUhOLO9ApUVOocA63knc+V2TkJ09D/AVOjiMIgkca/7VoYgODcla0qbSIBjje0SMfZMbAw== -"@sentry-internal/browser-utils@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry-internal/browser-utils/-/browser-utils-8.55.0.tgz#d89bae423edd29c39f01285c8e2d59ce9289d9a6" - integrity sha512-ROgqtQfpH/82AQIpESPqPQe0UyWywKJsmVIqi3c5Fh+zkds5LUxnssTj3yNd1x+kxaPDVB023jAP+3ibNgeNDw== +"@sentry/browser@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/browser/-/browser-6.19.7.tgz#a40b6b72d911b5f1ed70ed3b4e7d4d4e625c0b5f" + integrity sha512-oDbklp4O3MtAM4mtuwyZLrgO1qDVYIujzNJQzXmi9YzymJCuzMLSRDvhY83NNDCRxf0pds4DShgYeZdbSyKraA== dependencies: - "@sentry/core" "8.55.0" + "@sentry/core" "6.19.7" + "@sentry/types" "6.19.7" + "@sentry/utils" "6.19.7" + tslib "^1.9.3" -"@sentry-internal/feedback@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry-internal/feedback/-/feedback-8.55.0.tgz#170b8e96a36ce6f71f53daad680f1a0c98381314" - integrity sha512-cP3BD/Q6pquVQ+YL+rwCnorKuTXiS9KXW8HNKu4nmmBAyf7urjs+F6Hr1k9MXP5yQ8W3yK7jRWd09Yu6DHWOiw== +"@sentry/core@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/core/-/core-6.19.7.tgz#156aaa56dd7fad8c89c145be6ad7a4f7209f9785" + integrity sha512-tOfZ/umqB2AcHPGbIrsFLcvApdTm9ggpi/kQZFkej7kMphjT+SGBiQfYtjyg9jcRW+ilAR4JXC9BGKsdEQ+8Vw== dependencies: - "@sentry/core" "8.55.0" + "@sentry/hub" "6.19.7" + "@sentry/minimal" "6.19.7" + "@sentry/types" "6.19.7" + "@sentry/utils" "6.19.7" + tslib "^1.9.3" -"@sentry-internal/replay-canvas@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry-internal/replay-canvas/-/replay-canvas-8.55.0.tgz#e65430207a2f18e4a07c25c669ec758d11282aaf" - integrity sha512-nIkfgRWk1091zHdu4NbocQsxZF1rv1f7bbp3tTIlZYbrH62XVZosx5iHAuZG0Zc48AETLE7K4AX9VGjvQj8i9w== +"@sentry/hub@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/hub/-/hub-6.19.7.tgz#58ad7776bbd31e9596a8ec46365b45cd8b9cfd11" + integrity sha512-y3OtbYFAqKHCWezF0EGGr5lcyI2KbaXW2Ik7Xp8Mu9TxbSTuwTe4rTntwg8ngPjUQU3SUHzgjqVB8qjiGqFXCA== dependencies: - "@sentry-internal/replay" "8.55.0" - "@sentry/core" "8.55.0" + "@sentry/types" "6.19.7" + "@sentry/utils" "6.19.7" + tslib "^1.9.3" -"@sentry-internal/replay@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry-internal/replay/-/replay-8.55.0.tgz#4c00b22cdf58cac5b3e537f8d4f675f2b021f475" - integrity sha512-roCDEGkORwolxBn8xAKedybY+Jlefq3xYmgN2fr3BTnsXjSYOPC7D1/mYqINBat99nDtvgFvNfRcZPiwwZ1hSw== +"@sentry/minimal@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/minimal/-/minimal-6.19.7.tgz#b3ee46d6abef9ef3dd4837ebcb6bdfd01b9aa7b4" + integrity sha512-wcYmSJOdvk6VAPx8IcmZgN08XTXRwRtB1aOLZm+MVHjIZIhHoBGZJYTVQS/BWjldsamj2cX3YGbGXNunaCfYJQ== dependencies: - "@sentry-internal/browser-utils" "8.55.0" - "@sentry/core" "8.55.0" + "@sentry/hub" "6.19.7" + "@sentry/types" "6.19.7" + tslib "^1.9.3" -"@sentry/browser@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry/browser/-/browser-8.55.0.tgz#9a489e2a54d29c65e6271b4ee594b43679cab7bd" - integrity sha512-1A31mCEWCjaMxJt6qGUK+aDnLDcK6AwLAZnqpSchNysGni1pSn1RWSmk9TBF8qyTds5FH8B31H480uxMPUJ7Cw== +"@sentry/react@^6.18.2": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/react/-/react-6.19.7.tgz#58cc2d6da20f7d3b0df40638dfbbbc86c9c85caf" + integrity sha512-VzJeBg/v41jfxUYPkH2WYrKjWc4YiMLzDX0f4Zf6WkJ4v3IlDDSkX6DfmWekjTKBho6wiMkSNy2hJ1dHfGZ9jA== dependencies: - "@sentry-internal/browser-utils" "8.55.0" - "@sentry-internal/feedback" "8.55.0" - "@sentry-internal/replay" "8.55.0" - "@sentry-internal/replay-canvas" "8.55.0" - "@sentry/core" "8.55.0" - -"@sentry/core@8.55.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry/core/-/core-8.55.0.tgz#4964920229fcf649237ef13b1533dfc4b9f6b22e" - integrity sha512-6g7jpbefjHYs821Z+EBJ8r4Z7LT5h80YSWRJaylGS4nW5W5Z2KXzpdnyFarv37O7QjauzVC2E+PABmpkw5/JGA== - -"@sentry/react@^8.53.0": - version "8.55.0" - resolved "https://registry.yarnpkg.com/@sentry/react/-/react-8.55.0.tgz#309f005837956a98e79275ef8c2c2b5952c8be93" - integrity sha512-/qNBvFLpvSa/Rmia0jpKfJdy16d4YZaAnH/TuKLAtm0BWlsPQzbXCU4h8C5Hsst0Do0zG613MEtEmWpWrVOqWA== - dependencies: - "@sentry/browser" "8.55.0" - "@sentry/core" "8.55.0" + "@sentry/browser" "6.19.7" + "@sentry/minimal" "6.19.7" + "@sentry/types" "6.19.7" + "@sentry/utils" "6.19.7" hoist-non-react-statics "^3.3.2" + tslib "^1.9.3" + +"@sentry/tracing@^6.18.2": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/tracing/-/tracing-6.19.7.tgz#54bb99ed5705931cd33caf71da347af769f02a4c" + integrity sha512-ol4TupNnv9Zd+bZei7B6Ygnr9N3Gp1PUrNI761QSlHtPC25xXC5ssSD3GMhBgyQrcvpuRcCFHVNNM97tN5cZiA== + dependencies: + "@sentry/hub" "6.19.7" + "@sentry/minimal" "6.19.7" + "@sentry/types" "6.19.7" + "@sentry/utils" "6.19.7" + tslib "^1.9.3" + +"@sentry/types@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/types/-/types-6.19.7.tgz#c6b337912e588083fc2896eb012526cf7cfec7c7" + integrity sha512-jH84pDYE+hHIbVnab3Hr+ZXr1v8QABfhx39KknxqKWr2l0oEItzepV0URvbEhB446lk/S/59230dlUUIBGsXbg== + +"@sentry/utils@6.19.7": + version "6.19.7" + resolved "https://registry.yarnpkg.com/@sentry/utils/-/utils-6.19.7.tgz#6edd739f8185fd71afe49cbe351c1bbf5e7b7c79" + integrity sha512-z95ECmE3i9pbWoXQrD/7PgkBAzJYR+iXtPuTkpBjDKs86O3mT+PXOT3BAn79w2wkn7/i3vOGD2xVr1uiMl26dA== + dependencies: + "@sentry/types" "6.19.7" + tslib "^1.9.3" + +"@socket.io/component-emitter@~3.1.0": + version "3.1.2" + resolved "https://registry.yarnpkg.com/@socket.io/component-emitter/-/component-emitter-3.1.2.tgz#821f8442f4175d8f0467b9daf26e3a18e2d02af2" + integrity sha512-9BCxFwvbGg/RsZK9tjXd8s4UcwR0MWeFQ1XEKIQVVvAGJyINdrqKMcTRyLoK8Rse1GjzLV9cwjWV1olXRWEXVA== "@sqlite.org/sqlite-wasm@^3.50.3-build1": version "3.50.3-build1" @@ -1365,6 +1395,13 @@ resolved "https://registry.yarnpkg.com/@tsconfig/node16/-/node16-1.0.4.tgz#0b92dcc0cc1c81f6f306a381f28e31b1a56536e9" integrity sha512-vxhUy4J8lyeyinH7Azl1pdd43GJhZH/tP2weN8TntQblOY+A0XbT8DJk1/oCPuOOyg/Ja757rG0CgHcWC8OfMA== +"@types/cors@^2.8.12": + version "2.8.19" + resolved "https://registry.yarnpkg.com/@types/cors/-/cors-2.8.19.tgz#d93ea2673fd8c9f697367f5eeefc2bbfa94f0342" + integrity sha512-mFNylyeyqN93lfe/9CSxOGREz8cpzAhH+E93xJ4xWQf62V8sQ/24reV2nyzUWM6H6Xji+GGHpkbLe7pVoUEskg== + dependencies: + "@types/node" "*" + "@types/earcut@^2.1.0": version "2.1.4" resolved "https://registry.yarnpkg.com/@types/earcut/-/earcut-2.1.4.tgz#5811d7d333048f5a7573b22ddc84923e69596da6" @@ -1446,6 +1483,13 @@ dependencies: undici-types "~7.10.0" +"@types/node@>=10.0.0": + version "25.5.0" + resolved "https://registry.yarnpkg.com/@types/node/-/node-25.5.0.tgz#5c99f37c443d9ccc4985866913f1ed364217da31" + integrity sha512-jp2P3tQMSxWugkCUKLRPVUpGaL5MVFwF8RDuSRztfwgN1wmqJeMSbKlnEtQqU8UrhTmzEmZdu2I6v2dpp7XIxw== + dependencies: + undici-types "~7.18.0" + "@types/normalize-package-data@^2.4.0": version "2.4.4" resolved "https://registry.yarnpkg.com/@types/normalize-package-data/-/normalize-package-data-2.4.4.tgz#56e2cc26c397c038fab0e3a917a12d5c5909e901" @@ -1473,11 +1517,6 @@ dependencies: "@types/node" "*" -"@types/trusted-types@^2.0.7": - version "2.0.7" - resolved "https://registry.yarnpkg.com/@types/trusted-types/-/trusted-types-2.0.7.tgz#baccb07a970b91707df3a3e8ba6896c57ead2d11" - integrity sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw== - "@types/undertaker-registry@*": version "1.0.4" resolved "https://registry.yarnpkg.com/@types/undertaker-registry/-/undertaker-registry-1.0.4.tgz#2ea4b68abd0b3ad6716ab8ac28734092c1d152c4" @@ -1514,6 +1553,13 @@ "@types/expect" "^1.20.4" "@types/node" "*" +"@types/ws@^8.5.12": + version "8.18.1" + resolved "https://registry.yarnpkg.com/@types/ws/-/ws-8.18.1.tgz#48464e4bf2ddfd17db13d845467f6070ffea4aa9" + integrity sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg== + dependencies: + "@types/node" "*" + "@vercel/ncc@0.38.3": version "0.38.3" resolved "https://registry.yarnpkg.com/@vercel/ncc/-/ncc-0.38.3.tgz#5475eeee3ac0f1a439f237596911525a490a88b5" @@ -1695,6 +1741,14 @@ abbrev@1: resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.1.tgz#f8f2c887ad10bf67f634f005b6987fed3179aac8" integrity sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q== +accepts@~1.3.4: + version "1.3.8" + resolved "https://registry.yarnpkg.com/accepts/-/accepts-1.3.8.tgz#0bf0be125b67014adcb0b0921e62db7bffe16b2e" + integrity sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw== + dependencies: + mime-types "~2.1.34" + negotiator "0.6.3" + acorn-import-phases@^1.0.3: version "1.0.4" resolved "https://registry.yarnpkg.com/acorn-import-phases/-/acorn-import-phases-1.0.4.tgz#16eb850ba99a056cb7cbfe872ffb8972e18c8bd7" @@ -1809,11 +1863,6 @@ ansi-styles@^6.1.0: resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-6.2.1.tgz#0e62320cf99c21afff3b3012192546aacbfb05c5" integrity sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug== -ansi-styles@^6.2.1: - version "6.2.3" - resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-6.2.3.tgz#c044d5dcc521a076413472597a1acb1f103c4041" - integrity sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg== - ansi-wrap@0.1.0, ansi-wrap@^0.1.0: version "0.1.0" resolved "https://registry.yarnpkg.com/ansi-wrap/-/ansi-wrap-0.1.0.tgz#a82250ddb0015e9a27ca82e82ea603bbfa45efaf" @@ -1904,6 +1953,14 @@ arr-union@^3.1.0: resolved "https://registry.yarnpkg.com/arr-union/-/arr-union-3.1.0.tgz#e39b09aea9def866a8f206e288af63919bae39c4" integrity sha512-sKpyeERZ02v1FeCZT8lrfJq5u6goHCtpTAzPwJYe7c8SPFOboNjNg1vz2L4VTn9T4PQxEx13TbXLmYUcS6Ug7Q== +array-buffer-byte-length@^1.0.1, array-buffer-byte-length@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/array-buffer-byte-length/-/array-buffer-byte-length-1.0.2.tgz#384d12a37295aec3769ab022ad323a18a51ccf8b" + integrity sha512-LHE+8BuR7RYGDKvnrmcuSq3tDcKv9OFEXQt/HpbZhY7V6h0zlUXutnAD82GiFx9rdieCMjkvtcsPqBwgUl1Iiw== + dependencies: + call-bound "^1.0.3" + is-array-buffer "^3.0.5" + array-each@^1.0.0, array-each@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/array-each/-/array-each-1.0.1.tgz#a794af0c05ab1752846ee753a1f211a05ba0c44f" @@ -1953,6 +2010,19 @@ array-unique@^0.3.2: resolved "https://registry.yarnpkg.com/array-unique/-/array-unique-0.3.2.tgz#a894b75d4bc4f6cd679ef3244a9fd8f46ae2d428" integrity sha512-SleRWjh9JUud2wH1hPs9rZBZ33H6T9HOiL0uwGnGx9FpE6wKGyfWugmbkEOIs6qWrZhg0LWeLziLrEwQJhs5mQ== +arraybuffer.prototype.slice@^1.0.4: + version "1.0.4" + resolved "https://registry.yarnpkg.com/arraybuffer.prototype.slice/-/arraybuffer.prototype.slice-1.0.4.tgz#9d760d84dbdd06d0cbf92c8849615a1a7ab3183c" + integrity sha512-BNoCY6SXXPQ7gF2opIP4GBE+Xw7U+pHMYKuzjgCN3GwiaIR09UUeKfheyIry77QtrCBlC0KK0q5/TER/tYh3PQ== + dependencies: + array-buffer-byte-length "^1.0.1" + call-bind "^1.0.8" + define-properties "^1.2.1" + es-abstract "^1.23.5" + es-errors "^1.3.0" + get-intrinsic "^1.2.6" + is-array-buffer "^3.0.4" + arrify@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/arrify/-/arrify-1.0.1.tgz#898508da2226f380df904728456849c1501a4b0d" @@ -2005,6 +2075,11 @@ async-each@^1.0.1: resolved "https://registry.yarnpkg.com/async-each/-/async-each-1.0.6.tgz#52f1d9403818c179b7561e11a5d1b77eb2160e77" integrity sha512-c646jH1avxr+aVpndVMeAfYw7wAa6idufrlN3LPA4PmKS0QEGp6PIC9nwz0WQkkvBGAMEki3pFdtxaF39J9vvg== +async-function@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/async-function/-/async-function-1.0.0.tgz#509c9fca60eaf85034c6829838188e4e4c8ffb2b" + integrity sha512-hsU18Ae8CDTR6Kgu9DYf0EbCr/a5iGL0rytQDobUcdpYOKokk8LEjVphnXkDkgpi0wYVsqrXuP0bZxJaTqdgoA== + async-settle@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/async-settle/-/async-settle-1.0.0.tgz#1d0a914bb02575bec8a8f3a74e5080f72b2c0c6b" @@ -2137,6 +2212,11 @@ base64-js@^1.0.2, base64-js@^1.3.1, base64-js@^1.5.1: resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a" integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA== +base64id@2.0.0, base64id@~2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/base64id/-/base64id-2.0.0.tgz#2770ac6bc47d312af97a8bf9a634342e0cd25cb6" + integrity sha512-lGe34o6EHj9y3Kts9R4ZYs/Gr+6N7MCaMlIFA3F1R2O5/m7K06AxfSeO5530PEERE6/WyEg3lsuyw4GHlPZHog== + base@^0.11.1: version "0.11.2" resolved "https://registry.yarnpkg.com/base/-/base-0.11.2.tgz#7bde5ced145b6d551a90db87f83c558b4eb48a8f" @@ -2209,6 +2289,24 @@ bn.js@^5.2.1: resolved "https://registry.yarnpkg.com/bn.js/-/bn.js-5.2.2.tgz#82c09f9ebbb17107cd72cb7fd39bd1f9d0aaa566" integrity sha512-v2YAxEmKaBLahNwE1mjp4WON6huMNeuDvagFZW+ASCuA/ku0bXR9hSMw0XpiqMoA3+rmnyck/tPRSFQkoC9Cuw== +body-parser@^1.19.0: + version "1.20.4" + resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.20.4.tgz#f8e20f4d06ca8a50a71ed329c15dccad1cdc547f" + integrity sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA== + dependencies: + bytes "~3.1.2" + content-type "~1.0.5" + debug "2.6.9" + depd "2.0.0" + destroy "~1.2.0" + http-errors "~2.0.1" + iconv-lite "~0.4.24" + on-finished "~2.4.1" + qs "~6.14.0" + raw-body "~2.5.3" + type-is "~1.6.18" + unpipe "~1.0.0" + boolbase@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/boolbase/-/boolbase-1.0.0.tgz#68dff5fbe60c51eb37725ea9e3ed310dcc1e776e" @@ -2271,7 +2369,7 @@ braces@^2.3.1, braces@^2.3.2: split-string "^3.0.2" to-regex "^3.0.1" -braces@^3.0.3, braces@~3.0.2: +braces@^3.0.2, braces@^3.0.3, braces@~3.0.2: version "3.0.3" resolved "https://registry.yarnpkg.com/braces/-/braces-3.0.3.tgz#490332f40919452272d55a8480adc0c441358789" integrity sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA== @@ -2398,6 +2496,11 @@ builtin-status-codes@^3.0.0: resolved "https://registry.yarnpkg.com/builtin-status-codes/-/builtin-status-codes-3.0.0.tgz#85982878e21b98e1c66425e03d0174788f569ee8" integrity sha512-HpGFw18DgFWlncDfjTa2rcQ4W88O1mC8e8yZ2AvQY5KDaktSTwo+KRf6nHK6FRI5FyRyb/5T6+TSxfP7QyGsmQ== +bytes@~3.1.2: + version "3.1.2" + resolved "https://registry.yarnpkg.com/bytes/-/bytes-3.1.2.tgz#8b0beeb98605adf1b128fa4386403c009e0221a5" + integrity sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg== + cache-base@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/cache-base/-/cache-base-1.0.1.tgz#0a7f46416831c8b662ee36fe4e7c59d76f666ab2" @@ -2421,7 +2524,7 @@ call-bind-apply-helpers@^1.0.0, call-bind-apply-helpers@^1.0.1, call-bind-apply- es-errors "^1.3.0" function-bind "^1.1.2" -call-bind@^1.0.8: +call-bind@^1.0.7, call-bind@^1.0.8: version "1.0.8" resolved "https://registry.yarnpkg.com/call-bind/-/call-bind-1.0.8.tgz#0736a9660f537e3388826f440d5ec45f744eaa4c" integrity sha512-oKlSFMcMwpUg2ednkhQ454wfWiU/ul3CkJe/PEHcTKuiX6RpbehUiFMXu13HalGZxfUwCQzZG747YXBn1im9ww== @@ -2439,7 +2542,7 @@ call-bound@^1.0.2, call-bound@^1.0.3, call-bound@^1.0.4: call-bind-apply-helpers "^1.0.2" get-intrinsic "^1.3.0" -callsites@^3.0.0: +callsites@^3.0.0, callsites@^3.1.0: version "3.1.0" resolved "https://registry.yarnpkg.com/callsites/-/callsites-3.1.0.tgz#b3630abd8943432f54b3f0519238e33cd7df2f73" integrity sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ== @@ -2492,7 +2595,7 @@ canvas@^2.11.2: nan "^2.17.0" simple-get "^3.0.3" -chalk@2.4.2: +chalk@2.4.2, chalk@^2.4.1: version "2.4.2" resolved "https://registry.yarnpkg.com/chalk/-/chalk-2.4.2.tgz#cd42541677a54333cf541a49108c1432b44c9424" integrity sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ== @@ -2565,7 +2668,7 @@ chokidar@^2.0.0: optionalDependencies: fsevents "^1.2.7" -chokidar@^3.3.0, chokidar@^3.3.1, chokidar@^3.5.3: +chokidar@^3.3.0, chokidar@^3.3.1, chokidar@^3.5.1, chokidar@^3.5.3: version "3.6.0" resolved "https://registry.yarnpkg.com/chokidar/-/chokidar-3.6.0.tgz#197c6cc669ef2a8dc5e7b4d97ee4e092c3eb0d5b" integrity sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw== @@ -2625,6 +2728,11 @@ class-utils@^0.3.5: isobject "^3.0.0" static-extend "^0.1.1" +classnames@2.x, classnames@^2.2.5: + version "2.5.1" + resolved "https://registry.yarnpkg.com/classnames/-/classnames-2.5.1.tgz#ba774c614be0f016da105c858e7159eae8e7687b" + integrity sha512-saHYOzhIQs6wy2sVxTM6bUDsQO4F50V9RQ22qBpEdCW+I+/Wmke2HOl6lS6dTpdxVhb88/I6+Hs+438c3lfUow== + clean-stack@^2.0.0: version "2.2.0" resolved "https://registry.yarnpkg.com/clean-stack/-/clean-stack-2.2.0.tgz#ee8472dbb129e727b31e8a10a427dee9dfe4008b" @@ -2715,6 +2823,11 @@ cloneable-readable@^1.0.0: process-nextick-args "^2.0.0" readable-stream "^2.3.5" +clsx@^2.1.1: + version "2.1.1" + resolved "https://registry.yarnpkg.com/clsx/-/clsx-2.1.1.tgz#eed397c9fd8bd882bfb18deab7102049a2f32999" + integrity sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA== + code-point-at@^1.0.0: version "1.1.0" resolved "https://registry.yarnpkg.com/code-point-at/-/code-point-at-1.1.0.tgz#0d070b4d043a5bea33a2f1a40e2edb3d9a4ccf77" @@ -2842,6 +2955,11 @@ commander@^4.0.0: resolved "https://registry.yarnpkg.com/commander/-/commander-4.1.1.tgz#9fd602bd936294e9e9ef46a3f4d6964044b18068" integrity sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA== +commander@^6.0.0: + version "6.2.1" + resolved "https://registry.yarnpkg.com/commander/-/commander-6.2.1.tgz#0792eb682dfbc325999bb2b84fddddba110ac73c" + integrity sha512-U7VdrJFnJgo4xjrHpTzu0yrHPGImdsmD95ZlgYSEajAn2JKzDhDTPG9kBTefmObL2w/ngeZnilk+OV9CG3d7UA== + commander@^7.2.0: version "7.2.0" resolved "https://registry.yarnpkg.com/commander/-/commander-7.2.0.tgz#a36cb57d0b501ce108e4d20559a150a391d97ab7" @@ -2880,6 +2998,16 @@ concat-stream@^1.6.0: readable-stream "^2.2.2" typedarray "^0.0.6" +connect@^3.7.0: + version "3.7.0" + resolved "https://registry.yarnpkg.com/connect/-/connect-3.7.0.tgz#5d49348910caa5e07a01800b030d0c35f20484f8" + integrity sha512-ZqRXc+tZukToSNmh5C2iWMSoV3X1YUcPbqEM4DkEG5tNQXrQUZCNVGGv3IuicnkMtPfGf3Xtp8WCXs295iQ1pQ== + dependencies: + debug "2.6.9" + finalhandler "1.1.2" + parseurl "~1.3.3" + utils-merge "1.0.1" + console-browserify@^1.1.0: version "1.2.0" resolved "https://registry.yarnpkg.com/console-browserify/-/console-browserify-1.2.0.tgz#67063cef57ceb6cf4993a2ab3a55840ae8c49336" @@ -2895,6 +3023,11 @@ constants-browserify@^1.0.0: resolved "https://registry.yarnpkg.com/constants-browserify/-/constants-browserify-1.0.0.tgz#c20b96d8c617748aaf1c16021760cd27fcb8cb75" integrity sha512-xFxOwqIzR/e1k1gLiWEophSCMqXcwVHIH7akf7b/vxcUeGunlj3hvZaaqxwHsTgn+IndtkQJgSztIDWeumWJDQ== +content-type@~1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/content-type/-/content-type-1.0.5.tgz#8b773162656d1d1086784c8f23a54ce6d73d7918" + integrity sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA== + conventional-changelog-angular@^5.0.12: version "5.0.13" resolved "https://registry.yarnpkg.com/conventional-changelog-angular/-/conventional-changelog-angular-5.0.13.tgz#896885d63b914a70d4934b59d2fe7bde1832b28c" @@ -3049,6 +3182,11 @@ convert-source-map@^2.0.0: resolved "https://registry.yarnpkg.com/convert-source-map/-/convert-source-map-2.0.0.tgz#4b560f649fc4e918dd0ab75cf4961e8bc882d82a" integrity sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg== +cookie@~0.7.2: + version "0.7.2" + resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.7.2.tgz#556369c472a2ba910f2979891b526b3436237ed7" + integrity sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w== + copy-descriptor@^0.1.0: version "0.1.1" resolved "https://registry.yarnpkg.com/copy-descriptor/-/copy-descriptor-0.1.1.tgz#676f6eb3c39997c2ee1ac3a924fd6124748f578d" @@ -3067,6 +3205,14 @@ core-util-is@~1.0.0: resolved "https://registry.yarnpkg.com/core-util-is/-/core-util-is-1.0.3.tgz#a6042d3634c2b27e9328f837b965fac83808db85" integrity sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ== +cors@~2.8.5: + version "2.8.6" + resolved "https://registry.yarnpkg.com/cors/-/cors-2.8.6.tgz#ff5dd69bd95e547503820d29aba4f8faf8dfec96" + integrity sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw== + dependencies: + object-assign "^4" + vary "^1" + cosmiconfig@^7.0.0: version "7.1.0" resolved "https://registry.yarnpkg.com/cosmiconfig/-/cosmiconfig-7.1.0.tgz#1443b9afa596b670082ea46cbd8f6a62b84635f6" @@ -3131,7 +3277,7 @@ cross-env@^7.0.3: dependencies: cross-spawn "^7.0.1" -cross-spawn@^6.0.0: +cross-spawn@^6.0.0, cross-spawn@^6.0.5: version "6.0.6" resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-6.0.6.tgz#30d0efa0712ddb7eb5a76e1e8721bffafa6b5d57" integrity sha512-VqCUuhcd1iB+dsv8gxPttb5iZh/D0iubSP21g36KXdEuf6I5JiioesUVjpCdHV9MZRUfVFlvwtIUyPfxo5trtw== @@ -3276,6 +3422,11 @@ csstype@^3.0.2: resolved "https://registry.yarnpkg.com/csstype/-/csstype-3.1.3.tgz#d80ff294d114fb0e6ac500fbf85b60137d7eff81" integrity sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw== +custom-event@~1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/custom-event/-/custom-event-1.0.1.tgz#5d02a46850adf1b4a317946a3928fccb5bfd0425" + integrity sha512-GAj5FOq0Hd+RsCGVJxZuKaIDXDf3h6GQoNEjFgbLLI/trgtavwUbSnZ5pVfg27DVCaWjIohryS0JFwIJyT2cMg== + "d3-dispatch@1 - 3": version "3.0.1" resolved "https://registry.yarnpkg.com/d3-dispatch/-/d3-dispatch-3.0.1.tgz#5fc75284e9c2375c36c839411a0cf550cbfc4d5e" @@ -3313,6 +3464,38 @@ dargs@^7.0.0: resolved "https://registry.yarnpkg.com/dargs/-/dargs-7.0.0.tgz#04015c41de0bcb69ec84050f3d9be0caf8d6d5cc" integrity sha512-2iy1EkLdlBzQGvbweYRFxmFath8+K7+AKB0TlhHWkNuH+TmovaMH/Wp7V7R4u7f4SnX3OgLsU9t1NI9ioDnUpg== +data-view-buffer@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/data-view-buffer/-/data-view-buffer-1.0.2.tgz#211a03ba95ecaf7798a8c7198d79536211f88570" + integrity sha512-EmKO5V3OLXh1rtK2wgXRansaK1/mtVdTUEiEI0W8RkvgT05kfxaH29PliLnpLP73yYO6142Q72QNa8Wx/A5CqQ== + dependencies: + call-bound "^1.0.3" + es-errors "^1.3.0" + is-data-view "^1.0.2" + +data-view-byte-length@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/data-view-byte-length/-/data-view-byte-length-1.0.2.tgz#9e80f7ca52453ce3e93d25a35318767ea7704735" + integrity sha512-tuhGbE6CfTM9+5ANGf+oQb72Ky/0+s3xKUpHvShfiz2RxMFgFPjsXuRLBVMtvMs15awe45SRb83D6wH4ew6wlQ== + dependencies: + call-bound "^1.0.3" + es-errors "^1.3.0" + is-data-view "^1.0.2" + +data-view-byte-offset@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/data-view-byte-offset/-/data-view-byte-offset-1.0.1.tgz#068307f9b71ab76dbbe10291389e020856606191" + integrity sha512-BS8PfmtDGnrgYdOonGZQdLZslWIeCGFP9tpan0hi1Co2Zr2NKADsvGYA8XxuG/4UWgJ6Cjtv+YJnB6MM69QGlQ== + dependencies: + call-bound "^1.0.2" + es-errors "^1.3.0" + is-data-view "^1.0.1" + +date-format@^4.0.14: + version "4.0.14" + resolved "https://registry.yarnpkg.com/date-format/-/date-format-4.0.14.tgz#7a8e584434fb169a521c8b7aa481f355810d9400" + integrity sha512-39BOQLs9ZjKh0/patS9nrT8wc3ioX3/eA/zgbKNopnF2wCqJEoxywwwElATYvRsXdnOxA/OQeQoFZ3rFjVajhg== + dateformat@^3.0.0: version "3.0.3" resolved "https://registry.yarnpkg.com/dateformat/-/dateformat-3.0.3.tgz#a6e37499a4d9a9cf85ef5872044d62901c9889ae" @@ -3323,6 +3506,13 @@ dayjs@^1.10.0: resolved "https://registry.yarnpkg.com/dayjs/-/dayjs-1.11.13.tgz#92430b0139055c3ebb60150aa13e860a4b5a366c" integrity sha512-oaMBel6gjolK862uaPQOVTA7q3TZhuSvuMQAAglQDOWYO9A91IrAOUJEyKVlqJlHE0vq5p5UXxzdPfMH/x6xNg== +debug@2.6.9, debug@^2.2.0, debug@^2.3.3: + version "2.6.9" + resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.9.tgz#5d128515df134ff327e90a4c93f4e077a536341f" + integrity sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA== + dependencies: + ms "2.0.0" + debug@4, debug@^4.0.0, debug@^4.1.0, debug@^4.3.1, debug@^4.3.4, debug@^4.4.0: version "4.4.1" resolved "https://registry.yarnpkg.com/debug/-/debug-4.4.1.tgz#e5a8bc6cbc4c6cd3e64308b0693a3d4fa550189b" @@ -3337,12 +3527,12 @@ debug@4.3.4: dependencies: ms "2.1.2" -debug@^2.2.0, debug@^2.3.3: - version "2.6.9" - resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.9.tgz#5d128515df134ff327e90a4c93f4e077a536341f" - integrity sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA== +debug@^4.2.0, debug@~4.4.1: + version "4.4.3" + resolved "https://registry.yarnpkg.com/debug/-/debug-4.4.3.tgz#c6ae432d9bd9662582fce08709b038c58e9e3d6a" + integrity sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA== dependencies: - ms "2.0.0" + ms "^2.1.3" decamelize-keys@^1.1.0: version "1.1.1" @@ -3462,6 +3652,11 @@ delegates@^1.0.0: resolved "https://registry.yarnpkg.com/delegates/-/delegates-1.0.0.tgz#84c6e159b81904fdca59a0ef44cd870d31250f9a" integrity sha512-bd2L678uiWATM6m5Z1VzNCErI3jiGzt6HGY8OVICs40JQq/HALfbyNJmp0UDakEY4pMMaN0Ly5om/B1VI/+xfQ== +depd@2.0.0, depd@~2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/depd/-/depd-2.0.0.tgz#b696163cc757560d09cf22cc8fad1571b79e76df" + integrity sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw== + dependency-graph@^0.11.0: version "0.11.0" resolved "https://registry.yarnpkg.com/dependency-graph/-/dependency-graph-0.11.0.tgz#ac0ce7ed68a54da22165a85e97a01d53f5eb2e27" @@ -3475,6 +3670,11 @@ des.js@^1.0.0: inherits "^2.0.1" minimalistic-assert "^1.0.0" +destroy@~1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/destroy/-/destroy-1.2.0.tgz#4803735509ad8be552934c67df614f94e66fa015" + integrity sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg== + detect-file@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/detect-file/-/detect-file-1.0.0.tgz#f0d66d03672a825cb1b73bdb3fe62310c8e552b7" @@ -3498,6 +3698,11 @@ dezalgo@^1.0.4: asap "^2.0.0" wrappy "1" +di@^0.0.1: + version "0.0.1" + resolved "https://registry.yarnpkg.com/di/-/di-0.0.1.tgz#806649326ceaa7caa3306d75d985ea2748ba913c" + integrity sha512-uJaamHkagcZtHPqCIHZxnFrXlunQXgBOsZSUOWwFw31QJCAbyTBoHMW75YOTur5ZNx8pIeAKgf6GWIgaqqiLhA== + didyoumean@^1.2.2: version "1.2.2" resolved "https://registry.yarnpkg.com/didyoumean/-/didyoumean-1.2.2.tgz#989346ffe9e839b4555ecf5666edea0d3e8ad037" @@ -3542,6 +3747,16 @@ dom-helpers@^5.0.1: "@babel/runtime" "^7.8.7" csstype "^3.0.2" +dom-serialize@^2.2.1: + version "2.2.1" + resolved "https://registry.yarnpkg.com/dom-serialize/-/dom-serialize-2.2.1.tgz#562ae8999f44be5ea3076f5419dcd59eb43ac95b" + integrity sha512-Yra4DbvoW7/Z6LBN560ZwXMjoNOSAN2wRsKFGc4iBeso+mpIA6qj1vfdf9HpMaKAqG6wXTy+1SYEzmNpKXOSsQ== + dependencies: + custom-event "~1.0.0" + ent "~2.2.0" + extend "^3.0.0" + void-elements "^2.0.0" + dom-serializer@0: version "0.2.2" resolved "https://registry.yarnpkg.com/dom-serializer/-/dom-serializer-0.2.2.tgz#1afb81f533717175d478655debc5e332d9f9bb51" @@ -3588,12 +3803,10 @@ domhandler@^4.2.0, domhandler@^4.3.1: dependencies: domelementtype "^2.2.0" -dompurify@^3.3.3: - version "3.3.3" - resolved "https://registry.yarnpkg.com/dompurify/-/dompurify-3.3.3.tgz#680cae8af3e61320ddf3666a3bc843f7b291b2b6" - integrity sha512-Oj6pzI2+RqBfFG+qOaOLbFXLQ90ARpcGG6UePL82bJLtdsa6CYJD7nmiU8MW9nQNOtCHV3lZ/Bzq1X0QYbBZCA== - optionalDependencies: - "@types/trusted-types" "^2.0.7" +dompurify@2.4.0: + version "2.4.0" + resolved "https://registry.yarnpkg.com/dompurify/-/dompurify-2.4.0.tgz#c9c88390f024c2823332615c9e20a453cf3825dd" + integrity sha512-Be9tbQMZds4a3C6xTmz68NlMfeONA//4dOavl/1rNw50E+/QO0KVpbcU0PcaW0nsQxurXls9ZocqFxk8R2mWEA== domutils@^1.5.1: version "1.7.0" @@ -3619,7 +3832,7 @@ dot-prop@^5.1.0: dependencies: is-obj "^2.0.0" -dunder-proto@^1.0.1: +dunder-proto@^1.0.0, dunder-proto@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/dunder-proto/-/dunder-proto-1.0.1.tgz#d7ae667e1dc83482f8b70fd0f6eefc50da30f58a" integrity sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A== @@ -3656,6 +3869,11 @@ eastasianwidth@^0.2.0: resolved "https://registry.yarnpkg.com/eastasianwidth/-/eastasianwidth-0.2.0.tgz#696ce2ec0aa0e6ea93a397ffcf24aa7840c827cb" integrity sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA== +ee-first@1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz#590c61156b0ae2f4f0255732a158b266bc56b21d" + integrity sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow== + electron-to-chromium@^1.5.173: version "1.5.195" resolved "https://registry.yarnpkg.com/electron-to-chromium/-/electron-to-chromium-1.5.195.tgz#2fe0d9b644726292189f227be73740868617b6d5" @@ -3696,6 +3914,11 @@ emoji-regex@^9.2.2: resolved "https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-9.2.2.tgz#840c8803b0d8047f4ff0cf963176b32d4ef3ed72" integrity sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg== +encodeurl@~1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/encodeurl/-/encodeurl-1.0.2.tgz#ad3ff4c86ec2d029322f5a02c3a9a606c95b3f59" + integrity sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w== + end-of-stream@^1.0.0, end-of-stream@^1.1.0, end-of-stream@^1.4.1: version "1.4.5" resolved "https://registry.yarnpkg.com/end-of-stream/-/end-of-stream-1.4.5.tgz#7344d711dea40e0b74abc2ed49778743ccedb08c" @@ -3703,6 +3926,27 @@ end-of-stream@^1.0.0, end-of-stream@^1.1.0, end-of-stream@^1.4.1: dependencies: once "^1.4.0" +engine.io-parser@~5.2.1: + version "5.2.3" + resolved "https://registry.yarnpkg.com/engine.io-parser/-/engine.io-parser-5.2.3.tgz#00dc5b97b1f233a23c9398d0209504cf5f94d92f" + integrity sha512-HqD3yTBfnBxIrbnM1DoD6Pcq8NECnh8d4As1Qgh0z5Gg3jRRIqijury0CL3ghu/edArpUYiYqQiDUQBIs4np3Q== + +engine.io@~6.6.0: + version "6.6.6" + resolved "https://registry.yarnpkg.com/engine.io/-/engine.io-6.6.6.tgz#9942111e7a4dc31f057e73470d7b7fcc7f74c390" + integrity sha512-U2SN0w3OpjFRVlrc17E6TMDmH58Xl9rai1MblNjAdwWp07Kk+llmzX0hjDpQdrDGzwmvOtgM5yI+meYX6iZ2xA== + dependencies: + "@types/cors" "^2.8.12" + "@types/node" ">=10.0.0" + "@types/ws" "^8.5.12" + accepts "~1.3.4" + base64id "2.0.0" + cookie "~0.7.2" + cors "~2.8.5" + debug "~4.4.1" + engine.io-parser "~5.2.1" + ws "~8.18.3" + enhanced-resolve@^5.17.2: version "5.18.2" resolved "https://registry.yarnpkg.com/enhanced-resolve/-/enhanced-resolve-5.18.2.tgz#7903c5b32ffd4b2143eeb4b92472bd68effd5464" @@ -3711,6 +3955,16 @@ enhanced-resolve@^5.17.2: graceful-fs "^4.2.4" tapable "^2.2.0" +ent@~2.2.0: + version "2.2.2" + resolved "https://registry.yarnpkg.com/ent/-/ent-2.2.2.tgz#22a5ed2fd7ce0cbcff1d1474cf4909a44bdb6e85" + integrity sha512-kKvD1tO6BM+oK9HzCPpUdRb4vKFQY/FPTFmurMvh6LlN68VMrdj77w8yp51/kDbpkFOS9J8w5W6zIzgM2H8/hw== + dependencies: + call-bound "^1.0.3" + es-errors "^1.3.0" + punycode "^1.4.1" + safe-regex-test "^1.1.0" + entities@^1.1.1: version "1.1.2" resolved "https://registry.yarnpkg.com/entities/-/entities-1.1.2.tgz#bdfa735299664dfafd34529ed4f8522a275fea56" @@ -3743,6 +3997,66 @@ error-ex@^1.2.0, error-ex@^1.3.1: dependencies: is-arrayish "^0.2.1" +es-abstract@^1.23.2, es-abstract@^1.23.5, es-abstract@^1.23.9: + version "1.24.1" + resolved "https://registry.yarnpkg.com/es-abstract/-/es-abstract-1.24.1.tgz#f0c131ed5ea1bb2411134a8dd94def09c46c7899" + integrity sha512-zHXBLhP+QehSSbsS9Pt23Gg964240DPd6QCf8WpkqEXxQ7fhdZzYsocOr5u7apWonsS5EjZDmTF+/slGMyasvw== + dependencies: + array-buffer-byte-length "^1.0.2" + arraybuffer.prototype.slice "^1.0.4" + available-typed-arrays "^1.0.7" + call-bind "^1.0.8" + call-bound "^1.0.4" + data-view-buffer "^1.0.2" + data-view-byte-length "^1.0.2" + data-view-byte-offset "^1.0.1" + es-define-property "^1.0.1" + es-errors "^1.3.0" + es-object-atoms "^1.1.1" + es-set-tostringtag "^2.1.0" + es-to-primitive "^1.3.0" + function.prototype.name "^1.1.8" + get-intrinsic "^1.3.0" + get-proto "^1.0.1" + get-symbol-description "^1.1.0" + globalthis "^1.0.4" + gopd "^1.2.0" + has-property-descriptors "^1.0.2" + has-proto "^1.2.0" + has-symbols "^1.1.0" + hasown "^2.0.2" + internal-slot "^1.1.0" + is-array-buffer "^3.0.5" + is-callable "^1.2.7" + is-data-view "^1.0.2" + is-negative-zero "^2.0.3" + is-regex "^1.2.1" + is-set "^2.0.3" + is-shared-array-buffer "^1.0.4" + is-string "^1.1.1" + is-typed-array "^1.1.15" + is-weakref "^1.1.1" + math-intrinsics "^1.1.0" + object-inspect "^1.13.4" + object-keys "^1.1.1" + object.assign "^4.1.7" + own-keys "^1.0.1" + regexp.prototype.flags "^1.5.4" + safe-array-concat "^1.1.3" + safe-push-apply "^1.0.0" + safe-regex-test "^1.1.0" + set-proto "^1.0.0" + stop-iteration-iterator "^1.1.0" + string.prototype.trim "^1.2.10" + string.prototype.trimend "^1.0.9" + string.prototype.trimstart "^1.0.8" + typed-array-buffer "^1.0.3" + typed-array-byte-length "^1.0.3" + typed-array-byte-offset "^1.0.4" + typed-array-length "^1.0.7" + unbox-primitive "^1.1.0" + which-typed-array "^1.1.19" + es-define-property@^1.0.0, es-define-property@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/es-define-property/-/es-define-property-1.0.1.tgz#983eb2f9a6724e9303f61addf011c72e09e0b0fa" @@ -3765,6 +4079,25 @@ es-object-atoms@^1.0.0, es-object-atoms@^1.1.1: dependencies: es-errors "^1.3.0" +es-set-tostringtag@^2.1.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz#f31dbbe0c183b00a6d26eb6325c810c0fd18bd4d" + integrity sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA== + dependencies: + es-errors "^1.3.0" + get-intrinsic "^1.2.6" + has-tostringtag "^1.0.2" + hasown "^2.0.2" + +es-to-primitive@^1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/es-to-primitive/-/es-to-primitive-1.3.0.tgz#96c89c82cc49fd8794a24835ba3e1ff87f214e18" + integrity sha512-w+5mJ3GuFL+NjVtJlvydShqE1eN3h3PbI7/5LAsYJP/2qtuMXjfL2LpHSRqo4b4eSF5K/DH1JXKUAHSB2UW50g== + dependencies: + is-callable "^1.2.7" + is-date-object "^1.0.5" + is-symbol "^1.0.4" + es5-ext@^0.10.35, es5-ext@^0.10.46, es5-ext@^0.10.62, es5-ext@^0.10.64, es5-ext@~0.10.14: version "0.10.64" resolved "https://registry.yarnpkg.com/es5-ext/-/es5-ext-0.10.64.tgz#12e4ffb48f1ba2ea777f1fcdd1918ef73ea21714" @@ -3812,6 +4145,11 @@ escalade@^3.1.1, escalade@^3.2.0: resolved "https://registry.yarnpkg.com/escalade/-/escalade-3.2.0.tgz#011a3f69856ba189dffa7dc8fcce99d2a87903e5" integrity sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA== +escape-html@~1.0.3: + version "1.0.3" + resolved "https://registry.yarnpkg.com/escape-html/-/escape-html-1.0.3.tgz#0258eae4d3d0c0974de1c169188ef0051d1d1988" + integrity sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow== + escape-string-regexp@^1.0.3, escape-string-regexp@^1.0.5: version "1.0.5" resolved "https://registry.yarnpkg.com/escape-string-regexp/-/escape-string-regexp-1.0.5.tgz#1b61c0562190a8dff6ae3bb2cf0200ca130b86d4" @@ -3830,6 +4168,11 @@ eslint-scope@5.1.1: esrecurse "^4.3.0" estraverse "^4.1.1" +esm@^3.2.25: + version "3.2.25" + resolved "https://registry.yarnpkg.com/esm/-/esm-3.2.25.tgz#342c18c29d56157688ba5ce31f8431fbb795cc10" + integrity sha512-U1suiZ2oDVWv4zPO56S0NcR5QriEahGtdN2OR6FiOG4WJvcjBVFB0qI4+eKoWFH483PKGuLuu6V8Z4T5g63UVA== + esniff@^2.0.1: version "2.0.1" resolved "https://registry.yarnpkg.com/esniff/-/esniff-2.0.1.tgz#a4d4b43a5c71c7ec51c51098c1d8a29081f9b308" @@ -3870,6 +4213,11 @@ eventemitter3@^3.1.0: resolved "https://registry.yarnpkg.com/eventemitter3/-/eventemitter3-3.1.2.tgz#2d3d48f9c346698fce83a85d7d664e98535df6e7" integrity sha512-tvtQIeLVHjDkJYnzf2dgVMxfuSGJeM/7UCG17TT4EumTfNtF+0nebF/4zWOIkCreAbtNqhGEboB6BWrwqNaw4Q== +eventemitter3@^4.0.0: + version "4.0.7" + resolved "https://registry.yarnpkg.com/eventemitter3/-/eventemitter3-4.0.7.tgz#2de9b68f6528d5644ef5c59526a1b4a07306169f" + integrity sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw== + events@^3.0.0, events@^3.2.0, events@^3.3.0: version "3.3.0" resolved "https://registry.yarnpkg.com/events/-/events-3.3.0.tgz#31a95ad0a924e2d2c419a813aeb2c4e878ea7400" @@ -4080,6 +4428,19 @@ fill-range@^7.1.1: dependencies: to-regex-range "^5.0.1" +finalhandler@1.1.2: + version "1.1.2" + resolved "https://registry.yarnpkg.com/finalhandler/-/finalhandler-1.1.2.tgz#b7e7d000ffd11938d0fdb053506f6ebabe9f587d" + integrity sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA== + dependencies: + debug "2.6.9" + encodeurl "~1.0.2" + escape-html "~1.0.3" + on-finished "~2.3.0" + parseurl "~1.3.3" + statuses "~1.5.0" + unpipe "~1.0.0" + find-up@^1.0.0: version "1.1.2" resolved "https://registry.yarnpkg.com/find-up/-/find-up-1.1.2.tgz#6b2e9822b1a2ce0a60ab64d610eccad53cb24d0f" @@ -4165,6 +4526,11 @@ flatbuffers@^25.1.24: resolved "https://registry.yarnpkg.com/flatbuffers/-/flatbuffers-25.2.10.tgz#308b750545f62db670ca4c9d7dbc66161420a95e" integrity sha512-7JlN9ZvLDG1McO3kbX0k4v+SUAg48L1rIwEvN6ZQl/eCtgJz9UylTMzE9wrmYrcorgxm3CX/3T/w5VAub99UUw== +flatted@^3.2.7: + version "3.4.1" + resolved "https://registry.yarnpkg.com/flatted/-/flatted-3.4.1.tgz#84ccd9579e76e9cc0d246c11d8be0beb019143e6" + integrity sha512-IxfVbRFVlV8V/yRaGzk0UVIcsKKHMSfYw66T/u4nTwlWteQePsxe//LjudR1AMX4tZW3WFCh3Zqa/sjlqpbURQ== + flatted@^3.2.9: version "3.3.3" resolved "https://registry.yarnpkg.com/flatted/-/flatted-3.3.3.tgz#67c8fad95454a7c7abebf74bb78ee74a44023358" @@ -4178,7 +4544,12 @@ flush-write-stream@^1.0.2: inherits "^2.0.3" readable-stream "^2.3.6" -for-each@^0.3.5: +follow-redirects@^1.0.0: + version "1.15.11" + resolved "https://registry.yarnpkg.com/follow-redirects/-/follow-redirects-1.15.11.tgz#777d73d72a92f8ec4d2e410eb47352a56b8e8340" + integrity sha512-deG2P0JfjrTxl50XGCDyfI97ZGVCxIpfKYmfyrQ54n5FO/0gfIES8C/Psl6kWVDolizcaaxZJnTS0QSMxvnsBQ== + +for-each@^0.3.3, for-each@^0.3.5: version "0.3.5" resolved "https://registry.yarnpkg.com/for-each/-/for-each-0.3.5.tgz#d650688027826920feeb0af747ee7b9421a41d47" integrity sha512-dKx12eRCVIzqCxFGplyFKJMPvLEWgmNtUrpTiJIR5u97zEhRG8ySrtboPHZXx7daLxQVrl643cTzbab2tkQjxg== @@ -4240,6 +4611,16 @@ fs-extra@10.1.0, fs-extra@^10.0.0: jsonfile "^6.0.1" universalify "^2.0.0" +fs-extra@9.1.0, fs-extra@^9.0.0: + version "9.1.0" + resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-9.1.0.tgz#5954460c764a8da2094ba3554bf839e6b9a7c86d" + integrity sha512-hcg3ZmepS30/7BSFqRvoo3DOMQu7IjqxO5nCDt+zM9XWjb33Wg7ziNT+Qvqbuc3+gWpzO02JubVyk2G4Zvo1OQ== + dependencies: + at-least-node "^1.0.0" + graceful-fs "^4.2.0" + jsonfile "^6.0.1" + universalify "^2.0.0" + fs-extra@^11.2.0: version "11.3.0" resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-11.3.0.tgz#0daced136bbaf65a555a326719af931adc7a314d" @@ -4249,24 +4630,14 @@ fs-extra@^11.2.0: jsonfile "^6.0.1" universalify "^2.0.0" -fs-extra@^11.3.0: - version "11.3.4" - resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-11.3.4.tgz#ab6934eca8bcf6f7f6b82742e33591f86301d6fc" - integrity sha512-CTXd6rk/M3/ULNQj8FBqBWHYBVYybQ3VPBw0xGKFe3tuH7ytT6ACnvzpIQ3UZtB8yvUKC2cXn1a+x+5EVQLovA== +fs-extra@^8.1.0: + version "8.1.0" + resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-8.1.0.tgz#49d43c45a88cd9677668cb7be1b46efdb8d2e1c0" + integrity sha512-yhlQgA6mnOJUKOsRUFsgJdQCvkKhcz8tlZG5HBQfReYZy46OwLcY+Zia0mtdHsOo9y/hP+CxMN0TU9QxoOtG4g== dependencies: graceful-fs "^4.2.0" - jsonfile "^6.0.1" - universalify "^2.0.0" - -fs-extra@^9.0.0: - version "9.1.0" - resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-9.1.0.tgz#5954460c764a8da2094ba3554bf839e6b9a7c86d" - integrity sha512-hcg3ZmepS30/7BSFqRvoo3DOMQu7IjqxO5nCDt+zM9XWjb33Wg7ziNT+Qvqbuc3+gWpzO02JubVyk2G4Zvo1OQ== - dependencies: - at-least-node "^1.0.0" - graceful-fs "^4.2.0" - jsonfile "^6.0.1" - universalify "^2.0.0" + jsonfile "^4.0.0" + universalify "^0.1.0" fs-minipass@^2.0.0: version "2.1.0" @@ -4316,6 +4687,23 @@ function-bind@^1.1.2: resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.2.tgz#2c02d864d97f3ea6c8830c464cbd11ab6eab7a1c" integrity sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA== +function.prototype.name@^1.1.6, function.prototype.name@^1.1.8: + version "1.1.8" + resolved "https://registry.yarnpkg.com/function.prototype.name/-/function.prototype.name-1.1.8.tgz#e68e1df7b259a5c949eeef95cdbde53edffabb78" + integrity sha512-e5iwyodOHhbMr/yNrc7fDYG4qlbIvI5gajyzPnb5TCwyhjApznQh1BMFou9b30SevY43gCJKXycoCBjMbsuW0Q== + dependencies: + call-bind "^1.0.8" + call-bound "^1.0.3" + define-properties "^1.2.1" + functions-have-names "^1.2.3" + hasown "^2.0.2" + is-callable "^1.2.7" + +functions-have-names@^1.2.3: + version "1.2.3" + resolved "https://registry.yarnpkg.com/functions-have-names/-/functions-have-names-1.2.3.tgz#0404fe4ee2ba2f607f0e0ec3c80bae994133b834" + integrity sha512-xckBUXyTIqT97tq2x2AMb+g163b5JFysYk0x4qxNFwbfQkmNZoiRHb6sPzI9/QV33WeuvVYBUIiD4NzNIyqaRQ== + fuse.js@6.4.6: version "6.4.6" resolved "https://registry.yarnpkg.com/fuse.js/-/fuse.js-6.4.6.tgz#62f216c110e5aa22486aff20be7896d19a059b79" @@ -4336,6 +4724,11 @@ gauge@^3.0.0: strip-ansi "^6.0.1" wide-align "^1.1.2" +generator-function@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/generator-function/-/generator-function-2.0.1.tgz#0e75dd410d1243687a0ba2e951b94eedb8f737a2" + integrity sha512-SFdFmIJi+ybC0vjlHN0ZGVGHc3lgE0DxPAT0djjVg+kjOnSqclqmj0KQ7ykTOLP6YxoqOvuAODGdcHJn+43q3g== + gensync@^1.0.0-beta.2: version "1.0.0-beta.2" resolved "https://registry.yarnpkg.com/gensync/-/gensync-1.0.0-beta.2.tgz#32a6ee76c3d7f52d46b2b1ae5d93fea8580a25e0" @@ -4351,7 +4744,7 @@ get-caller-file@^2.0.1, get-caller-file@^2.0.5: resolved "https://registry.yarnpkg.com/get-caller-file/-/get-caller-file-2.0.5.tgz#4f94412a82db32f36e3b0b9741f8a97feb031f7e" integrity sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg== -get-intrinsic@^1.2.4, get-intrinsic@^1.2.5, get-intrinsic@^1.3.0: +get-intrinsic@^1.2.4, get-intrinsic@^1.2.5, get-intrinsic@^1.2.6, get-intrinsic@^1.2.7, get-intrinsic@^1.3.0: version "1.3.0" resolved "https://registry.yarnpkg.com/get-intrinsic/-/get-intrinsic-1.3.0.tgz#743f0e3b6964a93a5491ed1bffaae054d7f98d01" integrity sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ== @@ -4402,6 +4795,15 @@ get-stream@^4.0.0: dependencies: pump "^3.0.0" +get-symbol-description@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/get-symbol-description/-/get-symbol-description-1.1.0.tgz#7bdd54e0befe8ffc9f3b4e203220d9f1e881b6ee" + integrity sha512-w9UMqWwJxHNOvoNzSJ2oPF5wvYcvP7jUvYzhp67yEhTi17ZDBBC1z9pTdGuzjD+EFIqLSYRweZjqfiPzQ06Ebg== + dependencies: + call-bound "^1.0.3" + es-errors "^1.3.0" + get-intrinsic "^1.2.6" + get-value@^2.0.3, get-value@^2.0.6: version "2.0.6" resolved "https://registry.yarnpkg.com/get-value/-/get-value-2.0.6.tgz#dc15ca1c672387ca76bd37ac0a395ba2042a2c28" @@ -4536,7 +4938,7 @@ glob@^11.0.0: package-json-from-dist "^1.0.0" path-scurry "^2.0.0" -glob@^7.1.1, glob@^7.1.3: +glob@^7.0.0, glob@^7.1.1, glob@^7.1.3, glob@^7.1.7: version "7.2.3" resolved "https://registry.yarnpkg.com/glob/-/glob-7.2.3.tgz#b8df0fb802bbfa8e89bd1d938b4e16578ed44f2b" integrity sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q== @@ -4606,7 +5008,7 @@ global-prefix@^3.0.0: kind-of "^6.0.2" which "^1.3.1" -globalthis@^1.0.1: +globalthis@^1.0.1, globalthis@^1.0.4: version "1.0.4" resolved "https://registry.yarnpkg.com/globalthis/-/globalthis-1.0.4.tgz#7430ed3a975d97bfb59bcce41f5cabbafa651236" integrity sha512-DpLKbNU4WylpxJykQujfCcwYWiV/Jhm50Goo0wrVILAv5jOr9d+H+UR3PhSCD2rCCEIg0uc+G+muBTwD54JhDQ== @@ -4661,7 +5063,7 @@ gopd@^1.0.1, gopd@^1.2.0: resolved "https://registry.yarnpkg.com/gopd/-/gopd-1.2.0.tgz#89f56b8217bdbc8802bd299df6d7f1081d7e51a1" integrity sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg== -graceful-fs@^4.0.0, graceful-fs@^4.1.11, graceful-fs@^4.1.2, graceful-fs@^4.1.6, graceful-fs@^4.2.0, graceful-fs@^4.2.11, graceful-fs@^4.2.4: +graceful-fs@^4.0.0, graceful-fs@^4.1.11, graceful-fs@^4.1.2, graceful-fs@^4.1.6, graceful-fs@^4.2.0, graceful-fs@^4.2.11, graceful-fs@^4.2.4, graceful-fs@^4.2.6: version "4.2.11" resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.11.tgz#4183e4e8bf08bb6e05bbb2f7d2e0c8f712ca40e3" integrity sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ== @@ -4770,6 +5172,11 @@ hard-rejection@^2.1.0: resolved "https://registry.yarnpkg.com/hard-rejection/-/hard-rejection-2.1.0.tgz#1c6eda5c1685c63942766d79bb40ae773cecd883" integrity sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA== +has-bigints@^1.0.2: + version "1.1.0" + resolved "https://registry.yarnpkg.com/has-bigints/-/has-bigints-1.1.0.tgz#28607e965ac967e03cd2a2c70a2636a1edad49fe" + integrity sha512-R3pbpkcIqv2Pm3dUwgjclDRVmWpTJW2DcMzcIhEXEx1oh/CEMObMm3KLmRJOdvhM7o4uQBnwr8pzRK2sJWIqfg== + has-flag@^3.0.0: version "3.0.0" resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-3.0.0.tgz#b5d454dc2199ae225699f3467e5a07f3b955bafd" @@ -4787,6 +5194,13 @@ has-property-descriptors@^1.0.0, has-property-descriptors@^1.0.2: dependencies: es-define-property "^1.0.0" +has-proto@^1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/has-proto/-/has-proto-1.2.0.tgz#5de5a6eabd95fdffd9818b43055e8065e39fe9d5" + integrity sha512-KIL7eQPfHQRC8+XluaIw7BHUwwqL19bQn4hzNgdr+1wXoU0KKj6rufu47lhY7KbJR2C6T6+PfyN0Ea7wkSS+qQ== + dependencies: + dunder-proto "^1.0.0" + has-symbols@^1.0.3, has-symbols@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/has-symbols/-/has-symbols-1.1.0.tgz#fc9c6a783a084951d0b971fe1018de813707a338" @@ -4944,6 +5358,26 @@ htmlparser2@^3.10.0: inherits "^2.0.1" readable-stream "^3.1.1" +http-errors@~2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/http-errors/-/http-errors-2.0.1.tgz#36d2f65bc909c8790018dd36fb4d93da6caae06b" + integrity sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ== + dependencies: + depd "~2.0.0" + inherits "~2.0.4" + setprototypeof "~1.2.0" + statuses "~2.0.2" + toidentifier "~1.0.1" + +http-proxy@^1.18.1: + version "1.18.1" + resolved "https://registry.yarnpkg.com/http-proxy/-/http-proxy-1.18.1.tgz#401541f0534884bbf95260334e72f88ee3976549" + integrity sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ== + dependencies: + eventemitter3 "^4.0.0" + follow-redirects "^1.0.0" + requires-port "^1.0.0" + https-browserify@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/https-browserify/-/https-browserify-1.0.0.tgz#ec06c10e0a34c0f2faf199f7fd7fc78fffd03c73" @@ -4964,6 +5398,13 @@ iconv-lite@^0.6.3: dependencies: safer-buffer ">= 2.1.2 < 3.0.0" +iconv-lite@~0.4.24: + version "0.4.24" + resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.24.tgz#2022b4b25fbddc21d2f524974a474aafe733908b" + integrity sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA== + dependencies: + safer-buffer ">= 2.1.2 < 3" + ieee754@^1.1.13, ieee754@^1.1.4: version "1.2.1" resolved "https://registry.yarnpkg.com/ieee754/-/ieee754-1.2.1.tgz#8eb7a10a63fff25d15a57b001586d177d1b0d352" @@ -5023,7 +5464,7 @@ inflight@^1.0.4: once "^1.3.0" wrappy "1" -inherits@2, inherits@^2.0.1, inherits@^2.0.3, inherits@^2.0.4, inherits@~2.0.1, inherits@~2.0.3: +inherits@2, inherits@^2.0.1, inherits@^2.0.3, inherits@^2.0.4, inherits@~2.0.1, inherits@~2.0.3, inherits@~2.0.4: version "2.0.4" resolved "https://registry.yarnpkg.com/inherits/-/inherits-2.0.4.tgz#0fa2c64f932917c3433a0ded55363aae37416b7c" integrity sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ== @@ -5060,6 +5501,15 @@ interactjs@^1.10.17: dependencies: "@interactjs/types" "1.10.27" +internal-slot@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/internal-slot/-/internal-slot-1.1.0.tgz#1eac91762947d2f7056bc838d93e13b2e9604961" + integrity sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw== + dependencies: + es-errors "^1.3.0" + hasown "^2.0.2" + side-channel "^1.1.0" + interpret@^1.4.0: version "1.4.0" resolved "https://registry.yarnpkg.com/interpret/-/interpret-1.4.0.tgz#665ab8bc4da27a774a40584e812e3e0fa45b1a1e" @@ -5080,6 +5530,11 @@ invert-kv@^2.0.0: resolved "https://registry.yarnpkg.com/invert-kv/-/invert-kv-2.0.0.tgz#7393f5afa59ec9ff5f67a27620d11c226e3eec02" integrity sha512-wPVv/y/QQ/Uiirj/vh3oP+1Ww+AWehmi1g5fFWGPF6IpCBCDVrhgHRMvrLfdYcwDh3QJbGXDW4JAuzxElLSqKA== +ip@1.1.9: + version "1.1.9" + resolved "https://registry.yarnpkg.com/ip/-/ip-1.1.9.tgz#8dfbcc99a754d07f425310b86a99546b1151e396" + integrity sha512-cyRxvOEpNHNtchU3Ln9KC/auJgup87llfQpQ+t5ghoC/UhL16SWzbueiCsdTnWmqAWl7LadfuwhlqmtOaqMHdQ== + is-absolute@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/is-absolute/-/is-absolute-1.0.0.tgz#395e1ae84b11f26ad1795e73c17378e48a301576" @@ -5116,6 +5571,15 @@ is-arguments@^1.0.4: call-bound "^1.0.2" has-tostringtag "^1.0.2" +is-array-buffer@^3.0.4, is-array-buffer@^3.0.5: + version "3.0.5" + resolved "https://registry.yarnpkg.com/is-array-buffer/-/is-array-buffer-3.0.5.tgz#65742e1e687bd2cc666253068fd8707fe4d44280" + integrity sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A== + dependencies: + call-bind "^1.0.8" + call-bound "^1.0.3" + get-intrinsic "^1.2.6" + is-arrayish@^0.2.1: version "0.2.1" resolved "https://registry.yarnpkg.com/is-arrayish/-/is-arrayish-0.2.1.tgz#77c99840527aa8ecb1a8ba697b80645a7a926a9d" @@ -5126,6 +5590,24 @@ is-arrayish@^0.3.1: resolved "https://registry.yarnpkg.com/is-arrayish/-/is-arrayish-0.3.2.tgz#4574a2ae56f7ab206896fb431eaeed066fdf8f03" integrity sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ== +is-async-function@^2.0.0: + version "2.1.1" + resolved "https://registry.yarnpkg.com/is-async-function/-/is-async-function-2.1.1.tgz#3e69018c8e04e73b738793d020bfe884b9fd3523" + integrity sha512-9dgM/cZBnNvjzaMYHVoxxfPj2QXt22Ev7SuuPrs+xav0ukGB0S6d4ydZdEiM48kLx5kDV+QBPrpVnFyefL8kkQ== + dependencies: + async-function "^1.0.0" + call-bound "^1.0.3" + get-proto "^1.0.1" + has-tostringtag "^1.0.2" + safe-regex-test "^1.1.0" + +is-bigint@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/is-bigint/-/is-bigint-1.1.0.tgz#dda7a3445df57a42583db4228682eba7c4170672" + integrity sha512-n4ZT37wG78iz03xPRKJrHTdZbe3IicyucEtdRsV5yglwc3GyUfbAfpSeD0FJ41NbUNSt5wbhqfp1fS+BgnvDFQ== + dependencies: + has-bigints "^1.0.2" + is-binary-path@^1.0.0: version "1.0.1" resolved "https://registry.yarnpkg.com/is-binary-path/-/is-binary-path-1.0.1.tgz#75f16642b480f187a711c814161fd3a4a7655898" @@ -5140,6 +5622,14 @@ is-binary-path@~2.1.0: dependencies: binary-extensions "^2.0.0" +is-boolean-object@^1.2.1: + version "1.2.2" + resolved "https://registry.yarnpkg.com/is-boolean-object/-/is-boolean-object-1.2.2.tgz#7067f47709809a393c71ff5bb3e135d8a9215d9e" + integrity sha512-wa56o2/ElJMYqjCjGkXri7it5FbebW5usLw/nPmCMs5DeZ7eziSYZhSmPRn0txqeW4LnAmQQU7FgqLpsEFKM4A== + dependencies: + call-bound "^1.0.3" + has-tostringtag "^1.0.2" + is-buffer@^1.1.5: version "1.1.6" resolved "https://registry.yarnpkg.com/is-buffer/-/is-buffer-1.1.6.tgz#efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be" @@ -5169,6 +5659,23 @@ is-data-descriptor@^1.0.1: dependencies: hasown "^2.0.0" +is-data-view@^1.0.1, is-data-view@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/is-data-view/-/is-data-view-1.0.2.tgz#bae0a41b9688986c2188dda6657e56b8f9e63b8e" + integrity sha512-RKtWF8pGmS87i2D6gqQu/l7EYRlVdfzemCJN/P3UOs//x1QE7mfhvzHIApBTRf7axvT6DMGwSwBXYCT0nfB9xw== + dependencies: + call-bound "^1.0.2" + get-intrinsic "^1.2.6" + is-typed-array "^1.1.13" + +is-date-object@^1.0.5, is-date-object@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/is-date-object/-/is-date-object-1.1.0.tgz#ad85541996fc7aa8b2729701d27b7319f95d82f7" + integrity sha512-PwwhEakHVKTdRNVOw+/Gyh0+MzlCl4R6qKvkhuvLtPMggI1WAHt9sOwZxQLSGpUaDnrdyDsomoRgNnCfKNSXXg== + dependencies: + call-bound "^1.0.2" + has-tostringtag "^1.0.2" + is-decimal@^1.0.0: version "1.0.4" resolved "https://registry.yarnpkg.com/is-decimal/-/is-decimal-1.0.4.tgz#65a3a5958a1c5b63a706e1b333d7cd9f630d3fa5" @@ -5212,6 +5719,13 @@ is-extglob@^2.1.0, is-extglob@^2.1.1: resolved "https://registry.yarnpkg.com/is-extglob/-/is-extglob-2.1.1.tgz#a88c02535791f02ed37c76a1b9ea9773c833f8c2" integrity sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ== +is-finalizationregistry@^1.1.0: + version "1.1.1" + resolved "https://registry.yarnpkg.com/is-finalizationregistry/-/is-finalizationregistry-1.1.1.tgz#eefdcdc6c94ddd0674d9c85887bf93f944a97c90" + integrity sha512-1pC6N8qWJbWoPtEjgcL2xyhQOP491EQjeUo3qTKcmV8YSDDJrOepfG8pcC7h/QgnQHYSv0mJ3Z/ZWxmatVrysg== + dependencies: + call-bound "^1.0.3" + is-fullwidth-code-point@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz#ef9e31386f031a7f0d643af82fde50c457ef00cb" @@ -5229,6 +5743,17 @@ is-fullwidth-code-point@^3.0.0: resolved "https://registry.yarnpkg.com/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz#f116f8064fe90b3f7844a38997c0b75051269f1d" integrity sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg== +is-generator-function@^1.0.10: + version "1.1.2" + resolved "https://registry.yarnpkg.com/is-generator-function/-/is-generator-function-1.1.2.tgz#ae3b61e3d5ea4e4839b90bad22b02335051a17d5" + integrity sha512-upqt1SkGkODW9tsGNG5mtXTXtECizwtS2kA161M+gJPc1xdb/Ax629af6YrTwcOeQHbewrPNlE5Dx7kzvXTizA== + dependencies: + call-bound "^1.0.4" + generator-function "^2.0.0" + get-proto "^1.0.1" + has-tostringtag "^1.0.2" + safe-regex-test "^1.1.0" + is-generator-function@^1.0.7: version "1.1.0" resolved "https://registry.yarnpkg.com/is-generator-function/-/is-generator-function-1.1.0.tgz#bf3eeda931201394f57b5dba2800f91a238309ca" @@ -5258,11 +5783,29 @@ is-hexadecimal@^1.0.0: resolved "https://registry.yarnpkg.com/is-hexadecimal/-/is-hexadecimal-1.0.4.tgz#cc35c97588da4bd49a8eedd6bc4082d44dcb23a7" integrity sha512-gyPJuv83bHMpocVYoqof5VDiZveEoGoFL8m3BXNb2VW8Xs+rz9kqO8LOQ5DH6EsuvilT1ApazU0pyl+ytbPtlw== +is-map@^2.0.3: + version "2.0.3" + resolved "https://registry.yarnpkg.com/is-map/-/is-map-2.0.3.tgz#ede96b7fe1e270b3c4465e3a465658764926d62e" + integrity sha512-1Qed0/Hr2m+YqxnM09CjA2d/i6YZNfF6R2oRAOj36eUdS6qIV/huPJNSEpKbupewFs+ZsJlxsjjPbc0/afW6Lw== + is-negated-glob@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/is-negated-glob/-/is-negated-glob-1.0.0.tgz#6910bca5da8c95e784b5751b976cf5a10fee36d2" integrity sha512-czXVVn/QEmgvej1f50BZ648vUI+em0xqMq2Sn+QncCLN4zj1UAxlT+kw/6ggQTOaZPd1HqKQGEqbpQVtJucWug== +is-negative-zero@^2.0.3: + version "2.0.3" + resolved "https://registry.yarnpkg.com/is-negative-zero/-/is-negative-zero-2.0.3.tgz#ced903a027aca6381b777a5743069d7376a49747" + integrity sha512-5KoIu2Ngpyek75jXodFvnafB6DJgr3u8uuK0LEZJjrU19DrMD3EVERaR8sjz8CCGgpZvxPl9SuE1GMVPFHx1mw== + +is-number-object@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/is-number-object/-/is-number-object-1.1.1.tgz#144b21e95a1bc148205dcc2814a9134ec41b2541" + integrity sha512-lZhclumE1G6VYD8VHe35wFaIif+CTy5SJIi5+3y4psDgWu4wPDoBhF8NxUOinEc7pHgiTsT6MaBb92rKhhD+Xw== + dependencies: + call-bound "^1.0.3" + has-tostringtag "^1.0.2" + is-number@^3.0.0: version "3.0.0" resolved "https://registry.yarnpkg.com/is-number/-/is-number-3.0.0.tgz#24fd6201a4782cf50561c810276afc7d12d71195" @@ -5285,6 +5828,11 @@ is-obj@^2.0.0: resolved "https://registry.yarnpkg.com/is-obj/-/is-obj-2.0.0.tgz#473fb05d973705e3fd9620545018ca8e22ef4982" integrity sha512-drqDG3cbczxxEJRoOXcOjtdp1J/lyp1mNn0xaznRs8+muBhgQcrnbspox5X5fOw0HnMnbfDzvnEMEtqDEJEo8w== +is-observable@^2.1.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/is-observable/-/is-observable-2.1.0.tgz#5c8d733a0b201c80dff7bb7c0df58c6a255c7c69" + integrity sha512-DailKdLb0WU+xX8K5w7VsJhapwHLZ9jjmazqCJq4X12CTgqq73TKnbRcnSLuXYPOoLQgV5IrD7ePiX/h1vnkBw== + is-path-cwd@^2.2.0: version "2.2.0" resolved "https://registry.yarnpkg.com/is-path-cwd/-/is-path-cwd-2.2.0.tgz#67d43b82664a7b5191fd9119127eb300048a9fdb" @@ -5339,6 +5887,18 @@ is-relative@^1.0.0: dependencies: is-unc-path "^1.0.0" +is-set@^2.0.3: + version "2.0.3" + resolved "https://registry.yarnpkg.com/is-set/-/is-set-2.0.3.tgz#8ab209ea424608141372ded6e0cb200ef1d9d01d" + integrity sha512-iPAjerrse27/ygGLxw+EBR9agv9Y6uLeYVJMu+QNCoouJ1/1ri0mGrcWpfCqFZuzzx3WjtwxG098X+n4OuRkPg== + +is-shared-array-buffer@^1.0.4: + version "1.0.4" + resolved "https://registry.yarnpkg.com/is-shared-array-buffer/-/is-shared-array-buffer-1.0.4.tgz#9b67844bd9b7f246ba0708c3a93e34269c774f6f" + integrity sha512-ISWac8drv4ZGfwKl5slpHG9OwPNty4jOWPRIhBpxOoD+hqITiwuipOQ2bNthAzwA3B4fIjO4Nln74N0S9byq8A== + dependencies: + call-bound "^1.0.3" + is-stream@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-1.1.0.tgz#12d4a3dd4e68e0b79ceb8dbc84173ae80d91ca44" @@ -5349,6 +5909,23 @@ is-stream@^2.0.0: resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-2.0.1.tgz#fac1e3d53b97ad5a9d0ae9cef2389f5810a5c077" integrity sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg== +is-string@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/is-string/-/is-string-1.1.1.tgz#92ea3f3d5c5b6e039ca8677e5ac8d07ea773cbb9" + integrity sha512-BtEeSsoaQjlSPBemMQIrY1MY0uM6vnS1g5fmufYOtnxLGUZM2178PKbhsk7Ffv58IX+ZtcvoGwccYsh0PglkAA== + dependencies: + call-bound "^1.0.3" + has-tostringtag "^1.0.2" + +is-symbol@^1.0.4, is-symbol@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/is-symbol/-/is-symbol-1.1.1.tgz#f47761279f532e2b05a7024a7506dbbedacd0634" + integrity sha512-9gGx6GTtCQM73BgmHQXfDmLtfjjTUDSyoxTCbp5WtoixAhfgsDirWIcVQ/IHpvI5Vgd5i/J5F7B9cN/WlVbC/w== + dependencies: + call-bound "^1.0.2" + has-symbols "^1.1.0" + safe-regex-test "^1.1.0" + is-text-path@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/is-text-path/-/is-text-path-1.0.1.tgz#4e1aa0fb51bfbcb3e92688001397202c1775b66e" @@ -5356,7 +5933,7 @@ is-text-path@^1.0.1: dependencies: text-extensions "^1.0.0" -is-typed-array@^1.1.14, is-typed-array@^1.1.3: +is-typed-array@^1.1.13, is-typed-array@^1.1.14, is-typed-array@^1.1.15, is-typed-array@^1.1.3: version "1.1.15" resolved "https://registry.yarnpkg.com/is-typed-array/-/is-typed-array-1.1.15.tgz#4bfb4a45b61cee83a5a46fba778e4e8d59c0ce0b" integrity sha512-p3EcsicXjit7SaskXHs1hA91QxgTw46Fv6EFKKGS5DRFLD8yKnohjF3hxoju94b/OcMZoQukzpPpBE9uLVKzgQ== @@ -5390,6 +5967,26 @@ is-valid-glob@^1.0.0: resolved "https://registry.yarnpkg.com/is-valid-glob/-/is-valid-glob-1.0.0.tgz#29bf3eff701be2d4d315dbacc39bc39fe8f601aa" integrity sha512-AhiROmoEFDSsjx8hW+5sGwgKVIORcXnrlAx/R0ZSeaPw70Vw0CqkGBBhHGL58Uox2eXnU1AnvXJl1XlyedO5bA== +is-weakmap@^2.0.2: + version "2.0.2" + resolved "https://registry.yarnpkg.com/is-weakmap/-/is-weakmap-2.0.2.tgz#bf72615d649dfe5f699079c54b83e47d1ae19cfd" + integrity sha512-K5pXYOm9wqY1RgjpL3YTkF39tni1XajUIkawTLUo9EZEVUFga5gSQJF8nNS7ZwJQ02y+1YCNYcMh+HIf1ZqE+w== + +is-weakref@^1.0.2, is-weakref@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/is-weakref/-/is-weakref-1.1.1.tgz#eea430182be8d64174bd96bffbc46f21bf3f9293" + integrity sha512-6i9mGWSlqzNMEqpCp93KwRS1uUOodk2OJ6b+sq7ZPDSy2WuI5NFIxp/254TytR8ftefexkWn5xNiHUNpPOfSew== + dependencies: + call-bound "^1.0.3" + +is-weakset@^2.0.3: + version "2.0.4" + resolved "https://registry.yarnpkg.com/is-weakset/-/is-weakset-2.0.4.tgz#c9f5deb0bc1906c6d6f1027f284ddf459249daca" + integrity sha512-mfcwb6IzQyOKTs84CQMrOwW4gQcaTOAWJ0zzJCl2WSPDrWk/OzDaImWFH3djXhb24g4eudZfLRozAvPGw4d9hQ== + dependencies: + call-bound "^1.0.3" + get-intrinsic "^1.2.6" + is-windows@^1.0.1, is-windows@^1.0.2: version "1.0.2" resolved "https://registry.yarnpkg.com/is-windows/-/is-windows-1.0.2.tgz#d1850eb9791ecd18e6182ce12a30f396634bb19d" @@ -5412,16 +6009,16 @@ isarray@^2.0.5: resolved "https://registry.yarnpkg.com/isarray/-/isarray-2.0.5.tgz#8af1e4c1221244cc62459faf38940d4e644a5723" integrity sha512-xHjhDr3cNBK0BzdUJSPXZntQUx/mwMS5Rw4A7lPJ90XGAO6ISP/ePDNuo0vhqOZU+UD5JoodwCAAoZQd3FeAKw== +isbinaryfile@^4.0.8: + version "4.0.10" + resolved "https://registry.yarnpkg.com/isbinaryfile/-/isbinaryfile-4.0.10.tgz#0c5b5e30c2557a2f06febd37b7322946aaee42b3" + integrity sha512-iHrqe5shvBUcFbmZq9zOQHBoeOhZJu6RQGrDpBgenUm/Am+F3JM2MgQj+rK3Z601fzrL5gLZWtAPH2OBaSVcyw== + isexe@^2.0.0: version "2.0.0" resolved "https://registry.yarnpkg.com/isexe/-/isexe-2.0.0.tgz#e8fbf374dc556ff8947a10dcb0572d633f2cfa10" integrity sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw== -isexe@^3.1.1: - version "3.1.5" - resolved "https://registry.yarnpkg.com/isexe/-/isexe-3.1.5.tgz#42e368f68d5e10dadfee4fda7b550bc2d8892dc9" - integrity sha512-6B3tLtFqtQS4ekarvLVMZ+X+VlvQekbe4taUkf/rhVO3d/h0M2rfARm/pXLcPEsjjMsFgrFgSrhQIxcSVrBz8w== - ismobilejs@^1.1.0: version "1.1.1" resolved "https://registry.yarnpkg.com/ismobilejs/-/ismobilejs-1.1.1.tgz#c56ca0ae8e52b24ca0f22ba5ef3215a2ddbbaa0e" @@ -5509,11 +6106,6 @@ json-parse-even-better-errors@^2.3.0, json-parse-even-better-errors@^2.3.1: resolved "https://registry.yarnpkg.com/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz#7c47805a94319928e05777405dc12e1f7a4ee02d" integrity sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w== -json-parse-even-better-errors@^4.0.0: - version "4.0.0" - resolved "https://registry.yarnpkg.com/json-parse-even-better-errors/-/json-parse-even-better-errors-4.0.0.tgz#d3f67bd5925e81d3e31aa466acc821c8375cec43" - integrity sha512-lR4MXjGNgkJc7tkQ97kb2nuEMnNCyU//XYVH0MKTGcXEiSudQ5MKGKen3C5QubYy0vmq+JGitUg92uuywGEwIA== - json-schema-traverse@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz#ae7bcb3656ab77a73ba5c49bf654f38e6b6860e2" @@ -5534,6 +6126,13 @@ json5@^2.2.3: resolved "https://registry.yarnpkg.com/json5/-/json5-2.2.3.tgz#78cd6f1a19bdc12b73db5ad0c61efd66c1e29283" integrity sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg== +jsonfile@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/jsonfile/-/jsonfile-4.0.0.tgz#8771aae0799b64076b76640fca058f9c10e33ecb" + integrity sha512-m6F1R3z8jjlf2imQHS2Qez5sjKWQzbuuhuJ/FKYFRZvPE3PuHcSMVZzfsLhGVOkfd20obL5SWEBew5ShlquNxg== + optionalDependencies: + graceful-fs "^4.1.6" + jsonfile@^6.0.1: version "6.1.0" resolved "https://registry.yarnpkg.com/jsonfile/-/jsonfile-6.1.0.tgz#bc55b2634793c679ec6403094eb13698a6ec0aae" @@ -5573,6 +6172,48 @@ just-once@1.1.0: resolved "https://registry.yarnpkg.com/just-once/-/just-once-1.1.0.tgz#fe81a185ebaeeb0947a7e705bf01cb6808db0ad8" integrity sha512-+rZVpl+6VyTilK7vB/svlMPil4pxqIJZkbnN7DKZTOzyXfun6ZiFeq2Pk4EtCEHZ0VU4EkdFzG8ZK5F3PErcDw== +karma-chrome-launcher@^3.2.0: + version "3.2.0" + resolved "https://registry.yarnpkg.com/karma-chrome-launcher/-/karma-chrome-launcher-3.2.0.tgz#eb9c95024f2d6dfbb3748d3415ac9b381906b9a9" + integrity sha512-rE9RkUPI7I9mAxByQWkGJFXfFD6lE4gC5nPuZdobf/QdTEJI6EU4yIay/cfU/xV4ZxlM5JiTv7zWYgA64NpS5Q== + dependencies: + which "^1.2.1" + +karma-cljs-test@^0.1.0: + version "0.1.0" + resolved "https://registry.yarnpkg.com/karma-cljs-test/-/karma-cljs-test-0.1.0.tgz#cb8605ef0e11f9a6f6d28f56ba5dbdf26f389923" + integrity sha512-fd4aLynTv3htQCUS+OV1HfoB9UqYfEVFruKxkfTE3zB2aoSCHD966ZitSSgUeVYahWiaCK0XHZp9cB39t65cLQ== + +karma@^6.4.4: + version "6.4.4" + resolved "https://registry.yarnpkg.com/karma/-/karma-6.4.4.tgz#dfa5a426cf5a8b53b43cd54ef0d0d09742351492" + integrity sha512-LrtUxbdvt1gOpo3gxG+VAJlJAEMhbWlM4YrFQgql98FwF7+K8K12LYO4hnDdUkNjeztYrOXEMqgTajSWgmtI/w== + dependencies: + "@colors/colors" "1.5.0" + body-parser "^1.19.0" + braces "^3.0.2" + chokidar "^3.5.1" + connect "^3.7.0" + di "^0.0.1" + dom-serialize "^2.2.1" + glob "^7.1.7" + graceful-fs "^4.2.6" + http-proxy "^1.18.1" + isbinaryfile "^4.0.8" + lodash "^4.17.21" + log4js "^6.4.1" + mime "^2.5.2" + minimatch "^3.0.4" + mkdirp "^0.5.5" + qjobs "^1.2.0" + range-parser "^1.2.1" + rimraf "^3.0.2" + socket.io "^4.7.2" + source-map "^0.6.1" + tmp "^0.2.1" + ua-parser-js "^0.7.30" + yargs "^16.1.1" + katex@^0.16.10: version "0.16.22" resolved "https://registry.yarnpkg.com/katex/-/katex-0.16.22.tgz#d2b3d66464b1e6d69e6463b28a86ced5a02c5ccd" @@ -5752,6 +6393,11 @@ lodash.castarray@^4.4.0: resolved "https://registry.yarnpkg.com/lodash.castarray/-/lodash.castarray-4.4.0.tgz#c02513515e309daddd4c24c60cfddcf5976d9115" integrity sha512-aVx8ztPv7/2ULbArGJ2Y42bG1mEQ5mGjpdvrbJcJFU3TbYybe+QlLS4pst9zV52ymy2in1KpFPiZnAOATxD4+Q== +lodash.isequal@^4.0.0: + version "4.5.0" + resolved "https://registry.yarnpkg.com/lodash.isequal/-/lodash.isequal-4.5.0.tgz#415c4478f2bcc30120c22ce10ed3226f7d3e18e0" + integrity sha512-pDo3lu8Jhfjqls6GkMgpahsF9kCyayhgykjyLMNFTKWrpVdAQtYyB4muAMWozBB4ig/dtWAmsMxLEI8wuz+DYQ== + lodash.ismatch@^4.4.0: version "4.4.0" resolved "https://registry.yarnpkg.com/lodash.ismatch/-/lodash.ismatch-4.4.0.tgz#756cb5150ca3ba6f11085a78849645f188f85f37" @@ -5795,6 +6441,17 @@ log-symbols@^4.1.0: chalk "^4.1.0" is-unicode-supported "^0.1.0" +log4js@^6.4.1: + version "6.9.1" + resolved "https://registry.yarnpkg.com/log4js/-/log4js-6.9.1.tgz#aba5a3ff4e7872ae34f8b4c533706753709e38b6" + integrity sha512-1somDdy9sChrr9/f4UlzhdaGfDR2c/SaD2a4T7qEkG4jTS57/B3qmnjLYePwQ8cqWnUHZI0iAKxMBpCZICiZ2g== + dependencies: + date-format "^4.0.14" + debug "^4.3.4" + flatted "^3.2.7" + rfdc "^1.3.0" + streamroller "^3.1.5" + long@^5.0.0, long@^5.2.3: version "5.3.2" resolved "https://registry.yarnpkg.com/long/-/long-5.3.2.tgz#1d84463095999262d7d7b7f8bfd4a8cc55167f83" @@ -5958,6 +6615,11 @@ mdn-data@2.0.14: resolved "https://registry.yarnpkg.com/mdn-data/-/mdn-data-2.0.14.tgz#7113fc4281917d63ce29b43446f701e68c25ba50" integrity sha512-dn6wd0uw5GsdswPFfsgMp5NSB0/aDe6fK94YJV/AJDYXL6HVLWBsxeq7js7Ad+mU2K9LAlwpk6kN2D5mwCPVow== +media-typer@0.3.0: + version "0.3.0" + resolved "https://registry.yarnpkg.com/media-typer/-/media-typer-0.3.0.tgz#8710d7af0aa626f8fffa1ce00168545263255748" + integrity sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ== + mem@^4.0.0: version "4.3.0" resolved "https://registry.yarnpkg.com/mem/-/mem-4.3.0.tgz#461af497bc4ae09608cdb2e60eefb69bff744178" @@ -6074,13 +6736,18 @@ mime-db@1.52.0: resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.52.0.tgz#bbabcdc02859f4987301c856e3387ce5ec43bf70" integrity sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg== -mime-types@^2.1.27: +mime-types@^2.1.27, mime-types@~2.1.24, mime-types@~2.1.34: version "2.1.35" resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.35.tgz#381a871b62a734450660ae3deee44813f70d959a" integrity sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw== dependencies: mime-db "1.52.0" +mime@^2.5.2: + version "2.6.0" + resolved "https://registry.yarnpkg.com/mime/-/mime-2.6.0.tgz#a2a682a95cd4d0cb1d6257e28f83da7e35800367" + integrity sha512-USPkMeET31rOMiarsBNIHZKLGgvKc/LrjofAnBlOttf5ajRvqiRA8QsenbcooctK6d6Ts6aqZXBA+XbkKthiQg== + mimic-fn@^2.0.0: version "2.1.0" resolved "https://registry.yarnpkg.com/mimic-fn/-/mimic-fn-2.1.0.tgz#7ed2c2ccccaf84d3ffcb7a69b57711fc2083401b" @@ -6130,6 +6797,13 @@ minimatch@^10.0.3: dependencies: "@isaacs/brace-expansion" "^5.0.0" +minimatch@^3.0.4: + version "3.1.5" + resolved "https://registry.yarnpkg.com/minimatch/-/minimatch-3.1.5.tgz#580c88f8d5445f2bd6aa8f3cadefa0de79fbd69e" + integrity sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w== + dependencies: + brace-expansion "^1.1.7" + minimatch@^3.1.1: version "3.1.2" resolved "https://registry.yarnpkg.com/minimatch/-/minimatch-3.1.2.tgz#19cd194bfd3e428f049a70817c038d89ab4be35b" @@ -6167,7 +6841,7 @@ minimist-options@4.1.0: is-plain-obj "^1.1.0" kind-of "^6.0.3" -minimist@^1.2.0, minimist@^1.2.3, minimist@^1.2.5: +minimist@^1.2.0, minimist@^1.2.3, minimist@^1.2.5, minimist@^1.2.6: version "1.2.8" resolved "https://registry.yarnpkg.com/minimist/-/minimist-1.2.8.tgz#c1a464e7693302e082a075cee0c057741ac4772c" integrity sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA== @@ -6222,6 +6896,13 @@ mkdirp-classic@^0.5.2, mkdirp-classic@^0.5.3: resolved "https://registry.yarnpkg.com/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz#fa10c9115cc6d8865be221ba47ee9bed78601113" integrity sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A== +mkdirp@^0.5.5: + version "0.5.6" + resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-0.5.6.tgz#7def03d2432dcae4ba1d611445c48396062255f6" + integrity sha512-FP+p8RB8OWpF3YZBCrP5gtADmtXApB5AMLn+vdyA+PyxCjrCs00mjyUozssO33cwDeT3wNGdLxJ5M//YqtHAJw== + dependencies: + minimist "^1.2.6" + mkdirp@^1.0.3: version "1.0.4" resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-1.0.4.tgz#3eb5ed62622756d79a5f0e2a221dfebad75c2f7e" @@ -6322,6 +7003,11 @@ native-run@^2.0.0, native-run@^2.0.1: tslib "^2.6.2" yauzl "^2.10.0" +negotiator@0.6.3: + version "0.6.3" + resolved "https://registry.yarnpkg.com/negotiator/-/negotiator-0.6.3.tgz#58e323a72fedc0d6f9cd4d31fe49f51479590ccd" + integrity sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg== + neo-async@^2.6.2: version "2.6.2" resolved "https://registry.yarnpkg.com/neo-async/-/neo-async-2.6.2.tgz#b4aafb93e3aeb2d8174ca53cf163ab7d7308305f" @@ -6459,24 +7145,20 @@ now-and-later@^2.0.0: dependencies: once "^1.3.2" -npm-normalize-package-bin@^4.0.0: - version "4.0.0" - resolved "https://registry.yarnpkg.com/npm-normalize-package-bin/-/npm-normalize-package-bin-4.0.0.tgz#df79e70cd0a113b77c02d1fe243c96b8e618acb1" - integrity sha512-TZKxPvItzai9kN9H/TkmCtx/ZN/hvr3vUycjlfmH0ootY9yFBzNOpiXAdIn1Iteqsvk4lQn6B5PTrt+n6h8k/w== - -npm-run-all2@^8.0.4: - version "8.0.4" - resolved "https://registry.yarnpkg.com/npm-run-all2/-/npm-run-all2-8.0.4.tgz#bcc070fd0cdb8d45496ec875d99a659a112e3f74" - integrity sha512-wdbB5My48XKp2ZfJUlhnLVihzeuA1hgBnqB2J9ahV77wLS+/YAJAlN8I+X3DIFIPZ3m5L7nplmlbhNiFDmXRDA== +npm-run-all@^4.1.5: + version "4.1.5" + resolved "https://registry.yarnpkg.com/npm-run-all/-/npm-run-all-4.1.5.tgz#04476202a15ee0e2e214080861bff12a51d98fba" + integrity sha512-Oo82gJDAVcaMdi3nuoKFavkIHBRVqQ1qvMb+9LHk/cF4P6B2m8aP04hGf7oL6wZ9BuGwX1onlLhpuoofSyoQDQ== dependencies: - ansi-styles "^6.2.1" - cross-spawn "^7.0.6" + ansi-styles "^3.2.1" + chalk "^2.4.1" + cross-spawn "^6.0.5" memorystream "^0.3.1" - picomatch "^4.0.2" - pidtree "^0.6.0" - read-package-json-fast "^4.0.0" - shell-quote "^1.7.3" - which "^5.0.0" + minimatch "^3.0.4" + pidtree "^0.3.0" + read-pkg "^3.0.0" + shell-quote "^1.6.1" + string.prototype.padend "^3.0.0" npm-run-path@^2.0.0: version "2.0.2" @@ -6512,7 +7194,7 @@ number-is-nan@^1.0.0: resolved "https://registry.yarnpkg.com/number-is-nan/-/number-is-nan-1.0.1.tgz#097b602b53422a522c1afb8790318336941a011d" integrity sha512-4jbtZXNAsfZbAHiiqjLPBiCl16dES1zI4Hpzzxw61Tk+loF+sBDBKx1ICKKKwIqQ7M0mFn1TmkN7euSncWgHiQ== -object-assign@^4.0.1, object-assign@^4.1.1: +object-assign@^4, object-assign@^4.0.1, object-assign@^4.1.1: version "4.1.1" resolved "https://registry.yarnpkg.com/object-assign/-/object-assign-4.1.1.tgz#2109adc7965887cfc05cbbd442cac8bfbb360863" integrity sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg== @@ -6531,7 +7213,7 @@ object-hash@^3.0.0: resolved "https://registry.yarnpkg.com/object-hash/-/object-hash-3.0.0.tgz#73f97f753e7baffc0e2cc9d6e079079744ac82e9" integrity sha512-RSn9F68PjH9HqtltsSnqYC1XXoWe9Bju5+213R98cNGttag9q9yAOTzdbsqvIa7aNm5WffBZFpWYr2aWrklWAw== -object-inspect@^1.13.3: +object-inspect@^1.13.3, object-inspect@^1.13.4: version "1.13.4" resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.13.4.tgz#8375265e21bc20d0fa582c22e1b13485d6e00213" integrity sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew== @@ -6548,7 +7230,7 @@ object-visit@^1.0.0: dependencies: isobject "^3.0.0" -object.assign@^4.0.4, object.assign@^4.1.0, object.assign@^4.1.4: +object.assign@^4.0.4, object.assign@^4.1.0, object.assign@^4.1.4, object.assign@^4.1.7: version "4.1.7" resolved "https://registry.yarnpkg.com/object.assign/-/object.assign-4.1.7.tgz#8c14ca1a424c6a561b0bb2a22f66f5049a945d3d" integrity sha512-nK28WOo+QIjBkDduTINE4JkF/UJJKyf2EJxvJKfblDpyg0Q+pkOHNTL0Qwy6NP6FhE/EnzV73BxxqcJaXY9anw== @@ -6598,6 +7280,25 @@ obliterator@^1.6.1: resolved "https://registry.yarnpkg.com/obliterator/-/obliterator-1.6.1.tgz#dea03e8ab821f6c4d96a299e17aef6a3af994ef3" integrity sha512-9WXswnqINnnhOG/5SLimUlzuU1hFJUc8zkwyD59Sd+dPOMf05PmnYG/d6Q7HZ+KmgkZJa1PxRso6QdM3sTNHig== +observable-fns@^0.6.1: + version "0.6.1" + resolved "https://registry.yarnpkg.com/observable-fns/-/observable-fns-0.6.1.tgz#636eae4fdd1132e88c0faf38d33658cc79d87e37" + integrity sha512-9gRK4+sRWzeN6AOewNBTLXir7Zl/i3GB6Yl26gK4flxz8BXVpD3kt8amREmWNb0mxYOGDotvE5a4N+PtGGKdkg== + +on-finished@~2.3.0: + version "2.3.0" + resolved "https://registry.yarnpkg.com/on-finished/-/on-finished-2.3.0.tgz#20f1336481b083cd75337992a16971aa2d906947" + integrity sha512-ikqdkGAAyf/X/gPhXGvfgAytDZtDbr+bkNUJ0N9h5MI/dmdgCs3l6hoHrcUv41sRKew3jIwrp4qQDXiK99Utww== + dependencies: + ee-first "1.1.1" + +on-finished@~2.4.1: + version "2.4.1" + resolved "https://registry.yarnpkg.com/on-finished/-/on-finished-2.4.1.tgz#58c8c44116e54845ad57f14ab10b03533184ac3f" + integrity sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg== + dependencies: + ee-first "1.1.1" + once@^1.3.0, once@^1.3.1, once@^1.3.2, once@^1.4.0: version "1.4.0" resolved "https://registry.yarnpkg.com/once/-/once-1.4.0.tgz#583b1aa775961d4b113ac17d9c50baef9dd76bd1" @@ -6673,6 +7374,15 @@ os-locale@^3.0.0: lcid "^2.0.0" mem "^4.0.0" +own-keys@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/own-keys/-/own-keys-1.0.1.tgz#e4006910a2bf913585289676eebd6f390cf51358" + integrity sha512-qFOyK5PjiWZd+QQIh+1jhdb9LpxTF0qs7Pm8o5QHYZ0M3vKqSqzsZaEB6oWlxZ+q2sJBMI/Ktgd2N5ZwQoRHfg== + dependencies: + get-intrinsic "^1.2.6" + object-keys "^1.1.1" + safe-push-apply "^1.0.0" + p-defer@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/p-defer/-/p-defer-1.0.0.tgz#9f6eb182f6c9aa8cd743004a7d4f96b196b0fb0c" @@ -6825,6 +7535,11 @@ parse-passwd@^1.0.0: resolved "https://registry.yarnpkg.com/parse-passwd/-/parse-passwd-1.0.0.tgz#6d5b934a456993b23d37f40a382d6f1666a8e5c6" integrity sha512-1Y1A//QUXEZK7YKz+rD9WydcE1+EuPr6ZBgKecAB8tmoW6UFv0NREVJe1p+jRxtThkcbbKkfwIbWJe/IeE6m2Q== +parseurl@~1.3.3: + version "1.3.3" + resolved "https://registry.yarnpkg.com/parseurl/-/parseurl-1.3.3.tgz#9da19e7bee8d12dff0513ed5b76957793bc2e8d4" + integrity sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ== + pascalcase@^0.1.1: version "0.1.1" resolved "https://registry.yarnpkg.com/pascalcase/-/pascalcase-0.1.1.tgz#b363e55e8006ca6fe21784d2db22bd15d7917f14" @@ -6989,15 +7704,10 @@ picomatch@^2.0.4, picomatch@^2.2.1, picomatch@^2.3.1: resolved "https://registry.yarnpkg.com/picomatch/-/picomatch-2.3.1.tgz#3ba3833733646d9d3e4995946c1365a67fb07a42" integrity sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA== -picomatch@^4.0.2: - version "4.0.3" - resolved "https://registry.yarnpkg.com/picomatch/-/picomatch-4.0.3.tgz#796c76136d1eead715db1e7bad785dedd695a042" - integrity sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q== - -pidtree@^0.6.0: - version "0.6.0" - resolved "https://registry.yarnpkg.com/pidtree/-/pidtree-0.6.0.tgz#90ad7b6d42d5841e69e0a2419ef38f8883aa057c" - integrity sha512-eG2dWTVw5bzqGRztnHExczNxt5VGsE6OwTeCG3fdUf9KBsZzO3R5OIIIzWR+iZA0NtZ+RDVdaoE2dK1cn6jH4g== +pidtree@^0.3.0: + version "0.3.1" + resolved "https://registry.yarnpkg.com/pidtree/-/pidtree-0.3.1.tgz#ef09ac2cc0533df1f3250ccf2c4d366b0d12114a" + integrity sha512-qQbW94hLHEqCg7nhby4yRC7G2+jYHY4Rguc2bjw7Uug4GIJuu1tvf2uHaZv5Q8zdt+WKJ6qK1FOI6amaWUo5FA== pify@^2.0.0, pify@^2.3.0: version "2.3.0" @@ -7498,7 +8208,7 @@ postcss-selector-parser@6.0.10: cssesc "^3.0.0" util-deprecate "^1.0.2" -postcss-selector-parser@^6.0.10, postcss-selector-parser@^6.0.11, postcss-selector-parser@^6.0.4, postcss-selector-parser@^6.0.5, postcss-selector-parser@^6.0.9, postcss-selector-parser@^6.1.1: +postcss-selector-parser@^6.0.10, postcss-selector-parser@^6.0.11, postcss-selector-parser@^6.0.2, postcss-selector-parser@^6.0.4, postcss-selector-parser@^6.0.5, postcss-selector-parser@^6.0.9, postcss-selector-parser@^6.1.1: version "6.1.2" resolved "https://registry.yarnpkg.com/postcss-selector-parser/-/postcss-selector-parser-6.1.2.tgz#27ecb41fb0e3b6ba7a1ec84fff347f734c7929de" integrity sha512-Q8qQfPiZ+THO/3ZrOrO0cJJKfpYCagtMUkXbnEfmgUjwXg6z/WBeOyS9APBBPCTSiDV+s4SwQGu8yFsiMRIudg== @@ -7539,6 +8249,15 @@ postcss@^7.0.14, postcss@^7.0.2, postcss@^7.0.21, postcss@^7.0.26, postcss@^7.0. picocolors "^0.2.1" source-map "^0.6.1" +postcss@^8.2.1: + version "8.5.8" + resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.5.8.tgz#6230ecc8fb02e7a0f6982e53990937857e13f399" + integrity sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg== + dependencies: + nanoid "^3.3.11" + picocolors "^1.1.1" + source-map-js "^1.2.1" + postcss@^8.4.23, postcss@^8.4.47: version "8.5.6" resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.5.6.tgz#2825006615a619b4f62a9e7426cc120b349a8f3c" @@ -7611,7 +8330,7 @@ prompts@^2.4.2: kleur "^3.0.3" sisteransi "^1.0.5" -prop-types@^15.6.2, prop-types@^15.7.2: +prop-types@15.x, prop-types@^15.6.0, prop-types@^15.6.2, prop-types@^15.7.2, prop-types@^15.8.1: version "15.8.1" resolved "https://registry.yarnpkg.com/prop-types/-/prop-types-15.8.1.tgz#67d87bf1a694f48435cf332c24af10214a3140b5" integrity sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg== @@ -7680,11 +8399,26 @@ punycode@^1.2.4, punycode@^1.4.1: resolved "https://registry.yarnpkg.com/punycode/-/punycode-1.4.1.tgz#c0d5a63b2718800ad8e1eb0fa5269c84dd41845e" integrity sha512-jmYNElW7yvO7TV33CjSmvSiE2yco3bV2czu/OzDKdMNVZQWfxCblURLhf+47syQRBntjfLdd/H0egrzIG+oaFQ== +purgecss@4.0.2: + version "4.0.2" + resolved "https://registry.yarnpkg.com/purgecss/-/purgecss-4.0.2.tgz#7281f233c08b4f41e1c5d85c66a2d064e6d7e1b3" + integrity sha512-6J1zOEAZJX6VbfcaJHgdQf4uPhxVXvHz7dGgWYXLOI9q7QFZ5feh8NZ2+G3ysii/Sr8OyUe5yhQ5Z/xZ5gIRnQ== + dependencies: + commander "^6.0.0" + glob "^7.0.0" + postcss "^8.2.1" + postcss-selector-parser "^6.0.2" + q@^1.5.1: version "1.5.1" resolved "https://registry.yarnpkg.com/q/-/q-1.5.1.tgz#7e32f75b41381291d04611f1bf14109ac00651d7" integrity sha512-kV/CThkXo6xyFEZUugw/+pIOywXcDbFYgSct5cT3gqlbkBE1SJdwy6UQoZvodiWF/ckQLZyDE/Bu1M6gVu5lVw== +qjobs@^1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/qjobs/-/qjobs-1.2.0.tgz#c45e9c61800bd087ef88d7e256423bdd49e5d071" + integrity sha512-8YOJEHtxpySA3fFDyCRxA+UUV+fA+rTWnuWvylOK/NCjhY+b4ocCtmu8TtsWb+mYeU+GCHf/S66KZF/AsteKHg== + qs@^6.12.3: version "6.14.0" resolved "https://registry.yarnpkg.com/qs/-/qs-6.14.0.tgz#c63fa40680d2c5c941412a0e899c89af60c0a930" @@ -7692,6 +8426,13 @@ qs@^6.12.3: dependencies: side-channel "^1.1.0" +qs@~6.14.0: + version "6.14.2" + resolved "https://registry.yarnpkg.com/qs/-/qs-6.14.2.tgz#b5634cf9d9ad9898e31fba3504e866e8efb6798c" + integrity sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q== + dependencies: + side-channel "^1.1.0" + querystring-es3@^0.2.0: version "0.2.1" resolved "https://registry.yarnpkg.com/querystring-es3/-/querystring-es3-0.2.1.tgz#9ec61f79049875707d69414596fd907a4d711e73" @@ -7722,6 +8463,21 @@ randomfill@^1.0.4: randombytes "^2.0.5" safe-buffer "^5.1.0" +range-parser@^1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/range-parser/-/range-parser-1.2.1.tgz#3cf37023d199e1c24d1a55b84800c2f3e6468031" + integrity sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg== + +raw-body@~2.5.3: + version "2.5.3" + resolved "https://registry.yarnpkg.com/raw-body/-/raw-body-2.5.3.tgz#11c6650ee770a7de1b494f197927de0c923822e2" + integrity sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA== + dependencies: + bytes "~3.1.2" + http-errors "~2.0.1" + iconv-lite "~0.4.24" + unpipe "~1.0.0" + rc@^1.2.7: version "1.2.8" resolved "https://registry.yarnpkg.com/rc/-/rc-1.2.8.tgz#cd924bf5200a075b83c188cd6b9e211b7fc0d3ed" @@ -7740,6 +8496,33 @@ react-dom@18.3.1: loose-envify "^1.1.0" scheduler "^0.23.2" +react-draggable@3.x: + version "3.3.2" + resolved "https://registry.yarnpkg.com/react-draggable/-/react-draggable-3.3.2.tgz#966ef1d90f2387af3c2d8bd3516f601ea42ca359" + integrity sha512-oaz8a6enjbPtx5qb0oDWxtDNuybOylvto1QLydsXgKmwT7e3GXC2eMVDwEMIUYJIFqVG72XpOv673UuuAq6LhA== + dependencies: + classnames "^2.2.5" + prop-types "^15.6.0" + +react-draggable@^4.0.3: + version "4.5.0" + resolved "https://registry.yarnpkg.com/react-draggable/-/react-draggable-4.5.0.tgz#0b274ccb6965fcf97ed38fcf7e3cc223bc48cdf5" + integrity sha512-VC+HBLEZ0XJxnOxVAZsdRi8rD04Iz3SiiKOoYzamjylUcju/hP9np/aZdLHf/7WOD268WMoNJMvYfB5yAK45cw== + dependencies: + clsx "^2.1.1" + prop-types "^15.8.1" + +react-grid-layout@0.16.6: + version "0.16.6" + resolved "https://registry.yarnpkg.com/react-grid-layout/-/react-grid-layout-0.16.6.tgz#9b2407a2b946c2260ebaf66f13b556e1da4efeb2" + integrity sha512-h2EsYgsqcESLJeevQSJsEKp8hhh+phOlXDJoMhlV2e7T3VWQL+S6iCF3iD/LK19r4oyRyOMDEir0KV+eLXrAyw== + dependencies: + classnames "2.x" + lodash.isequal "^4.0.0" + prop-types "15.x" + react-draggable "3.x" + react-resizable "1.x" + react-intersection-observer@^9.3.5: version "9.16.0" resolved "https://registry.yarnpkg.com/react-intersection-observer/-/react-intersection-observer-9.16.0.tgz#7376d54edc47293300961010844d53b273ee0fb9" @@ -7750,6 +8533,14 @@ react-is@^16.13.1, react-is@^16.7.0: resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4" integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ== +react-resizable@1.x: + version "1.11.1" + resolved "https://registry.yarnpkg.com/react-resizable/-/react-resizable-1.11.1.tgz#02ca6850afa7a22c1b3e623e64aef71ee252af69" + integrity sha512-S70gbLaAYqjuAd49utRHibtHLrHXInh7GuOR+6OO6RO6uleQfuBnWmZjRABfqNEx3C3Z6VPLg0/0uOYFrkfu9Q== + dependencies: + prop-types "15.x" + react-draggable "^4.0.3" + react-textarea-autosize@8.3.3: version "8.3.3" resolved "https://registry.yarnpkg.com/react-textarea-autosize/-/react-textarea-autosize-8.3.3.tgz#f70913945369da453fd554c168f6baacd1fa04d8" @@ -7788,14 +8579,6 @@ read-cache@^1.0.0: dependencies: pify "^2.3.0" -read-package-json-fast@^4.0.0: - version "4.0.0" - resolved "https://registry.yarnpkg.com/read-package-json-fast/-/read-package-json-fast-4.0.0.tgz#8ccbc05740bb9f58264f400acc0b4b4eee8d1b39" - integrity sha512-qpt8EwugBWDw2cgE2W+/3oxC+KTez2uSVR8JU9Q36TXPAGCaozfQUs59v4j4GFpWTaw0i6hAZSvOmu1J0uOEUg== - dependencies: - json-parse-even-better-errors "^4.0.0" - npm-normalize-package-bin "^4.0.0" - read-pkg-up@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/read-pkg-up/-/read-pkg-up-1.0.1.tgz#9d63c13276c065918d57f002a57f40a1b643fb02" @@ -7921,6 +8704,20 @@ redent@^3.0.0: indent-string "^4.0.0" strip-indent "^3.0.0" +reflect.getprototypeof@^1.0.6, reflect.getprototypeof@^1.0.9: + version "1.0.10" + resolved "https://registry.yarnpkg.com/reflect.getprototypeof/-/reflect.getprototypeof-1.0.10.tgz#c629219e78a3316d8b604c765ef68996964e7bf9" + integrity sha512-00o4I+DVrefhv+nX0ulyi3biSHCPDe+yLv5o/p6d/UVlirijB8E16FtfwSAi4g3tcqrQ4lRAqQSoFEZJehYEcw== + dependencies: + call-bind "^1.0.8" + define-properties "^1.2.1" + es-abstract "^1.23.9" + es-errors "^1.3.0" + es-object-atoms "^1.0.0" + get-intrinsic "^1.2.7" + get-proto "^1.0.1" + which-builtin-type "^1.2.1" + regex-not@^1.0.0, regex-not@^1.0.2: version "1.0.2" resolved "https://registry.yarnpkg.com/regex-not/-/regex-not-1.0.2.tgz#1f4ece27e00b0b65e0247a6810e6a85d83a5752c" @@ -7934,6 +8731,18 @@ regexp-to-ast@0.5.0: resolved "https://registry.yarnpkg.com/regexp-to-ast/-/regexp-to-ast-0.5.0.tgz#56c73856bee5e1fef7f73a00f1473452ab712a24" integrity sha512-tlbJqcMHnPKI9zSrystikWKwHkBqu2a/Sgw01h3zFjvYrMxEDYHzzoMZnUrbIfpTFEsoRnnviOXNCzFiSc54Qw== +regexp.prototype.flags@^1.5.4: + version "1.5.4" + resolved "https://registry.yarnpkg.com/regexp.prototype.flags/-/regexp.prototype.flags-1.5.4.tgz#1ad6c62d44a259007e55b3970e00f746efbcaa19" + integrity sha512-dYqgNSZbDwkaJ2ceRd9ojCGjBq+mOm9LmtXnAnEGyHhN/5R7iDW2TRw3h+o/jCFxus3P2LfWIIiwowAjANm7IA== + dependencies: + call-bind "^1.0.8" + define-properties "^1.2.1" + es-errors "^1.3.0" + get-proto "^1.0.1" + gopd "^1.2.0" + set-function-name "^2.0.2" + remark-parse@^9.0.0: version "9.0.0" resolved "https://registry.yarnpkg.com/remark-parse/-/remark-parse-9.0.0.tgz#4d20a299665880e4f4af5d90b7c7b8a935853640" @@ -8046,6 +8855,11 @@ require-main-filename@^2.0.0: resolved "https://registry.yarnpkg.com/require-main-filename/-/require-main-filename-2.0.0.tgz#d0b329ecc7cc0f61649f62215be69af54aa8989b" integrity sha512-NKN5kMDylKuldxYLSUfrbo5Tuzh4hd+2E8NPPX02mZtn1VuREQToYe/ZdlJy+J3uCpfaiGF05e7B8W0iXbQHmg== +requires-port@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/requires-port/-/requires-port-1.0.0.tgz#925d2601d39ac485e091cf0da5c6e694dc3dcaff" + integrity sha512-KigOCHcocU3XODJxsu8i/j8T9tzT4adHiecwORRQ0ZZFcp7ahwXuRU1m+yuO90C5ZUyGeGfocHDI14M3L3yDAQ== + resolve-cwd@^3.0.0: version "3.0.0" resolved "https://registry.yarnpkg.com/resolve-cwd/-/resolve-cwd-3.0.0.tgz#0f0075f1bb2544766cf73ba6a6e2adfebcb13f2d" @@ -8102,6 +8916,11 @@ reusify@^1.0.4: resolved "https://registry.yarnpkg.com/reusify/-/reusify-1.1.0.tgz#0fe13b9522e1473f51b558ee796e08f11f9b489f" integrity sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw== +rfdc@^1.3.0: + version "1.4.1" + resolved "https://registry.yarnpkg.com/rfdc/-/rfdc-1.4.1.tgz#778f76c4fb731d93414e8f925fbecf64cce7f6ca" + integrity sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA== + rimraf@^3.0.2: version "3.0.2" resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-3.0.2.tgz#f1a5402ba6220ad52cc1282bac1ae3aa49fd061a" @@ -8159,6 +8978,17 @@ run-parallel@^1.1.9: dependencies: queue-microtask "^1.2.2" +safe-array-concat@^1.1.3: + version "1.1.3" + resolved "https://registry.yarnpkg.com/safe-array-concat/-/safe-array-concat-1.1.3.tgz#c9e54ec4f603b0bbb8e7e5007a5ee7aecd1538c3" + integrity sha512-AURm5f0jYEOydBj7VQlVvDrjeFgthDdEF5H1dP+6mNpoXOMo1quQqJ4wvJDyRZ9+pO3kGWoOdmV08cSv2aJV6Q== + dependencies: + call-bind "^1.0.8" + call-bound "^1.0.2" + get-intrinsic "^1.2.6" + has-symbols "^1.1.0" + isarray "^2.0.5" + safe-buffer@^5.0.1, safe-buffer@^5.1.0, safe-buffer@^5.1.1, safe-buffer@^5.1.2, safe-buffer@^5.2.0, safe-buffer@^5.2.1, safe-buffer@~5.2.0: version "5.2.1" resolved "https://registry.yarnpkg.com/safe-buffer/-/safe-buffer-5.2.1.tgz#1eaf9fa9bdb1fdd4ec75f58f9cdb4e6b7827eec6" @@ -8169,6 +8999,14 @@ safe-buffer@~5.1.0, safe-buffer@~5.1.1: resolved "https://registry.yarnpkg.com/safe-buffer/-/safe-buffer-5.1.2.tgz#991ec69d296e0313747d59bdfd2b745c35f8828d" integrity sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g== +safe-push-apply@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/safe-push-apply/-/safe-push-apply-1.0.0.tgz#01850e981c1602d398c85081f360e4e6d03d27f5" + integrity sha512-iKE9w/Z7xCzUMIZqdBsp6pEQvwuEebH4vdpjcDWnyzaI6yl6O9FHvVpmGelvEHNsoY6wGblkxR6Zty/h00WiSA== + dependencies: + es-errors "^1.3.0" + isarray "^2.0.5" + safe-regex-test@^1.1.0: version "1.1.0" resolved "https://registry.yarnpkg.com/safe-regex-test/-/safe-regex-test-1.1.0.tgz#7f87dfb67a3150782eaaf18583ff5d1711ac10c1" @@ -8185,7 +9023,7 @@ safe-regex@^1.1.0: dependencies: ret "~0.1.10" -"safer-buffer@>= 2.1.2 < 3.0.0": +"safer-buffer@>= 2.1.2 < 3", "safer-buffer@>= 2.1.2 < 3.0.0": version "2.1.2" resolved "https://registry.yarnpkg.com/safer-buffer/-/safer-buffer-2.1.2.tgz#44fa161b0187b9549dd84bb91802f9bd8385cd6a" integrity sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg== @@ -8296,11 +9134,30 @@ set-function-length@^1.2.2: gopd "^1.0.1" has-property-descriptors "^1.0.2" +set-function-name@^2.0.2: + version "2.0.2" + resolved "https://registry.yarnpkg.com/set-function-name/-/set-function-name-2.0.2.tgz#16a705c5a0dc2f5e638ca96d8a8cd4e1c2b90985" + integrity sha512-7PGFlmtwsEADb0WYyvCMa1t+yke6daIG4Wirafur5kcf+MhUnPms1UeR0CKQdTZD81yESwMHbtn+TR+dMviakQ== + dependencies: + define-data-property "^1.1.4" + es-errors "^1.3.0" + functions-have-names "^1.2.3" + has-property-descriptors "^1.0.2" + set-immediate-shim@~1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/set-immediate-shim/-/set-immediate-shim-1.0.1.tgz#4b2b1b27eb808a9f8dcc481a58e5e56f599f3f61" integrity sha512-Li5AOqrZWCVA2n5kryzEmqai6bKSIvpz5oUJHPVj6+dsbD3X1ixtsY5tEnsaNpH3pFAHmG8eIHUrtEtohrg+UQ== +set-proto@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/set-proto/-/set-proto-1.0.0.tgz#0760dbcff30b2d7e801fd6e19983e56da337565e" + integrity sha512-RJRdvCo6IAnPdsvP/7m6bsQqNnn1FCBX5ZNtFL98MmFF/4xAIJTIg1YbHW5DC2W5SKZanrC6i4HsJqlajw/dZw== + dependencies: + dunder-proto "^1.0.1" + es-errors "^1.3.0" + es-object-atoms "^1.0.0" + set-value@^2.0.0, set-value@^2.0.1: version "2.0.1" resolved "https://registry.yarnpkg.com/set-value/-/set-value-2.0.1.tgz#a18d40530e6f07de4228c7defe4227af8cad005b" @@ -8316,6 +9173,11 @@ setimmediate@^1.0.4: resolved "https://registry.yarnpkg.com/setimmediate/-/setimmediate-1.0.5.tgz#290cbb232e306942d7d7ea9b83732ab7856f8285" integrity sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA== +setprototypeof@~1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/setprototypeof/-/setprototypeof-1.2.0.tgz#66c9a24a73f9fc28cbe66b09fed3d33dcaf1b424" + integrity sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw== + sha.js@^2.4.0, sha.js@^2.4.11, sha.js@^2.4.8: version "2.4.12" resolved "https://registry.yarnpkg.com/sha.js/-/sha.js-2.4.12.tgz#eb8b568bf383dfd1867a32c3f2b74eb52bdbf23f" @@ -8419,7 +9281,7 @@ shebang-regex@^3.0.0: resolved "https://registry.yarnpkg.com/shebang-regex/-/shebang-regex-3.0.0.tgz#ae16f1644d873ecad843b0307b143362d4c42172" integrity sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A== -shell-quote@^1.7.3: +shell-quote@^1.6.1: version "1.8.3" resolved "https://registry.yarnpkg.com/shell-quote/-/shell-quote-1.8.3.tgz#55e40ef33cf5c689902353a3d8cd1a6725f08b4b" integrity sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw== @@ -8581,6 +9443,35 @@ snapdragon@^0.8.1: source-map-resolve "^0.5.0" use "^3.1.0" +socket.io-adapter@~2.5.2: + version "2.5.6" + resolved "https://registry.yarnpkg.com/socket.io-adapter/-/socket.io-adapter-2.5.6.tgz#c697f609d36a676a46749782274607d8df52c1d8" + integrity sha512-DkkO/dz7MGln0dHn5bmN3pPy+JmywNICWrJqVWiVOyvXjWQFIv9c2h24JrQLLFJ2aQVQf/Cvl1vblnd4r2apLQ== + dependencies: + debug "~4.4.1" + ws "~8.18.3" + +socket.io-parser@~4.2.4: + version "4.2.5" + resolved "https://registry.yarnpkg.com/socket.io-parser/-/socket.io-parser-4.2.5.tgz#3f41b8d369129a93268f2abecba94b5292850099" + integrity sha512-bPMmpy/5WWKHea5Y/jYAP6k74A+hvmRCQaJuJB6I/ML5JZq/KfNieUVo/3Mh7SAqn7TyFdIo6wqYHInG1MU1bQ== + dependencies: + "@socket.io/component-emitter" "~3.1.0" + debug "~4.4.1" + +socket.io@^4.7.2: + version "4.8.3" + resolved "https://registry.yarnpkg.com/socket.io/-/socket.io-4.8.3.tgz#ca6ba1431c69532e1e0a6f496deebeb601dbc4df" + integrity sha512-2Dd78bqzzjE6KPkD5fHZmDAKRNe3J15q+YHDrIsy9WEkqttc7GY+kT9OBLSMaPbQaEd0x1BjcmtMtXkfpc+T5A== + dependencies: + accepts "~1.3.4" + base64id "~2.0.0" + cors "~2.8.5" + debug "~4.4.1" + engine.io "~6.6.0" + socket.io-adapter "~2.5.2" + socket.io-parser "~4.2.4" + source-map-js@^1.0.2, source-map-js@^1.2.1: version "1.2.1" resolved "https://registry.yarnpkg.com/source-map-js/-/source-map-js-1.2.1.tgz#1ce5650fddd87abc099eda37dcff024c2667ae46" @@ -8720,6 +9611,24 @@ static-extend@^0.1.1: define-property "^0.2.5" object-copy "^0.1.0" +statuses@~1.5.0: + version "1.5.0" + resolved "https://registry.yarnpkg.com/statuses/-/statuses-1.5.0.tgz#161c7dac177659fd9811f43771fa99381478628c" + integrity sha512-OpZ3zP+jT1PI7I8nemJX4AKmAX070ZkYPVWV/AaKTJl+tXCTGyVdC1a4SL8RUQYEwk/f34ZX8UTykN68FwrqAA== + +statuses@~2.0.2: + version "2.0.2" + resolved "https://registry.yarnpkg.com/statuses/-/statuses-2.0.2.tgz#8f75eecef765b5e1cfcdc080da59409ed424e382" + integrity sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw== + +stop-iteration-iterator@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/stop-iteration-iterator/-/stop-iteration-iterator-1.1.0.tgz#f481ff70a548f6124d0312c3aa14cbfa7aa542ad" + integrity sha512-eLoXW/DHyl62zxY4SCaIgnRhuMr6ri4juEYARS8E6sCEqzKpOiE521Ucofdx+KnDZl5xmvGYaaKCk5FEOxJCoQ== + dependencies: + es-errors "^1.3.0" + internal-slot "^1.1.0" + stream-browserify@^2.0.1: version "2.0.2" resolved "https://registry.yarnpkg.com/stream-browserify/-/stream-browserify-2.0.2.tgz#87521d38a44aa7ee91ce1cd2a47df0cb49dd660b" @@ -8754,6 +9663,15 @@ stream-shift@^1.0.0: resolved "https://registry.yarnpkg.com/stream-shift/-/stream-shift-1.0.3.tgz#85b8fab4d71010fc3ba8772e8046cc49b8a3864b" integrity sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ== +streamroller@^3.1.5: + version "3.1.5" + resolved "https://registry.yarnpkg.com/streamroller/-/streamroller-3.1.5.tgz#1263182329a45def1ffaef58d31b15d13d2ee7ff" + integrity sha512-KFxaM7XT+irxvdqSP1LGLgNWbYN7ay5owZ3r/8t77p+EtSUAfUgtl7be3xtqtOmGUl9K9YPO2ca8133RlTjvKw== + dependencies: + date-format "^4.0.14" + debug "^4.3.4" + fs-extra "^8.1.0" + streamx@^2.15.0, streamx@^2.21.0: version "2.22.1" resolved "https://registry.yarnpkg.com/streamx/-/streamx-2.22.1.tgz#c97cbb0ce18da4f4db5a971dc9ab68ff5dc7f5a5" @@ -8764,7 +9682,7 @@ streamx@^2.15.0, streamx@^2.21.0: optionalDependencies: bare-events "^2.2.0" -"string-width-cjs@npm:string-width@^4.2.0", "string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: +"string-width-cjs@npm:string-width@^4.2.0": version "4.2.3" resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== @@ -8782,6 +9700,15 @@ string-width@^1.0.1, string-width@^1.0.2: is-fullwidth-code-point "^1.0.0" strip-ansi "^3.0.0" +"string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.2, string-width@^4.2.3: + version "4.2.3" + resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" + integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== + dependencies: + emoji-regex "^8.0.0" + is-fullwidth-code-point "^3.0.0" + strip-ansi "^6.0.1" + string-width@^2.0.0, string-width@^2.1.1: version "2.1.1" resolved "https://registry.yarnpkg.com/string-width/-/string-width-2.1.1.tgz#ab93f27a8dc13d28cac815c462143a6d9012ae9e" @@ -8799,6 +9726,48 @@ string-width@^5.0.1, string-width@^5.1.2: emoji-regex "^9.2.2" strip-ansi "^7.0.1" +string.prototype.padend@^3.0.0: + version "3.1.6" + resolved "https://registry.yarnpkg.com/string.prototype.padend/-/string.prototype.padend-3.1.6.tgz#ba79cf8992609a91c872daa47c6bb144ee7f62a5" + integrity sha512-XZpspuSB7vJWhvJc9DLSlrXl1mcA2BdoY5jjnS135ydXqLoqhs96JjDtCkjJEQHvfqZIp9hBuBMgI589peyx9Q== + dependencies: + call-bind "^1.0.7" + define-properties "^1.2.1" + es-abstract "^1.23.2" + es-object-atoms "^1.0.0" + +string.prototype.trim@^1.2.10: + version "1.2.10" + resolved "https://registry.yarnpkg.com/string.prototype.trim/-/string.prototype.trim-1.2.10.tgz#40b2dd5ee94c959b4dcfb1d65ce72e90da480c81" + integrity sha512-Rs66F0P/1kedk5lyYyH9uBzuiI/kNRmwJAR9quK6VOtIpZ2G+hMZd+HQbbv25MgCA6gEffoMZYxlTod4WcdrKA== + dependencies: + call-bind "^1.0.8" + call-bound "^1.0.2" + define-data-property "^1.1.4" + define-properties "^1.2.1" + es-abstract "^1.23.5" + es-object-atoms "^1.0.0" + has-property-descriptors "^1.0.2" + +string.prototype.trimend@^1.0.9: + version "1.0.9" + resolved "https://registry.yarnpkg.com/string.prototype.trimend/-/string.prototype.trimend-1.0.9.tgz#62e2731272cd285041b36596054e9f66569b6942" + integrity sha512-G7Ok5C6E/j4SGfyLCloXTrngQIQU3PWtXGst3yM7Bea9FRURf1S42ZHlZZtsNque2FN2PoUhfZXYLNWwEr4dLQ== + dependencies: + call-bind "^1.0.8" + call-bound "^1.0.2" + define-properties "^1.2.1" + es-object-atoms "^1.0.0" + +string.prototype.trimstart@^1.0.8: + version "1.0.8" + resolved "https://registry.yarnpkg.com/string.prototype.trimstart/-/string.prototype.trimstart-1.0.8.tgz#7ee834dda8c7c17eff3118472bb35bfedaa34dde" + integrity sha512-UXSH262CSZY1tfu3G3Secr6uGLCFVPMhIqHjlgCUtCCcgihYc/xKs9djMTMUOb2j1mVSeU8EU6NWc/iQKU6Gfg== + dependencies: + call-bind "^1.0.7" + define-properties "^1.2.1" + es-object-atoms "^1.0.0" + string_decoder@^1.0.0, string_decoder@^1.1.1: version "1.3.0" resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-1.3.0.tgz#42f114594a46cf1a8e30b0a84f56c78c3edac21e" @@ -8813,7 +9782,7 @@ string_decoder@~1.1.1: dependencies: safe-buffer "~5.1.0" -"strip-ansi-cjs@npm:strip-ansi@^6.0.1", strip-ansi@^6.0.0, strip-ansi@^6.0.1: +"strip-ansi-cjs@npm:strip-ansi@^6.0.1": version "6.0.1" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== @@ -8834,6 +9803,13 @@ strip-ansi@^4.0.0: dependencies: ansi-regex "^3.0.0" +strip-ansi@^6.0.0, strip-ansi@^6.0.1: + version "6.0.1" + resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" + integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== + dependencies: + ansi-regex "^5.0.1" + strip-ansi@^7.0.1: version "7.1.0" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-7.1.0.tgz#d5b6568ca689d8561370b0707685d22434faff45" @@ -9220,6 +10196,18 @@ thenify-all@^1.0.0: dependencies: any-promise "^1.0.0" +threads@1.6.5: + version "1.6.5" + resolved "https://registry.yarnpkg.com/threads/-/threads-1.6.5.tgz#5cee7f139e3e147c5a64f0134844ee92469932a5" + integrity sha512-yL1NN4qZ25crW8wDoGn7TqbENJ69w3zCEjIGXpbqmQ4I+QHrG8+DLaZVKoX74OQUXWCI2lbbrUxDxAbr1xjDGQ== + dependencies: + callsites "^3.1.0" + debug "^4.2.0" + is-observable "^2.1.0" + observable-fns "^0.6.1" + optionalDependencies: + tiny-worker ">= 2" + through2-filter@^3.0.0: version "3.1.0" resolved "https://registry.yarnpkg.com/through2-filter/-/through2-filter-3.1.0.tgz#4a1b45d2b76b3ac93ec137951e372c268efc1a4e" @@ -9264,6 +10252,13 @@ tiny-typed-emitter@^2.0.3: resolved "https://registry.yarnpkg.com/tiny-typed-emitter/-/tiny-typed-emitter-2.1.0.tgz#b3b027fdd389ff81a152c8e847ee2f5be9fad7b5" integrity sha512-qVtvMxeXbVej0cQWKqVSSAHmKZEHAvxdF8HEUBFWts8h+xEo5m/lEiPakuyZ3BnCBjOD8i24kzNOiOLLgsSxhA== +"tiny-worker@>= 2": + version "2.3.0" + resolved "https://registry.yarnpkg.com/tiny-worker/-/tiny-worker-2.3.0.tgz#715ae34304c757a9af573ae9a8e3967177e6011e" + integrity sha512-pJ70wq5EAqTAEl9IkGzA+fN0836rycEuz2Cn6yeZ6FRzlVS5IDOkFHpIoEsksPRQV34GDqXm65+OlnZqUSyK2g== + dependencies: + esm "^3.2.25" + tippy.js@^6.3.1: version "6.3.7" resolved "https://registry.yarnpkg.com/tippy.js/-/tippy.js-6.3.7.tgz#8ccfb651d642010ed9a32ff29b0e9e19c5b8c61c" @@ -9337,6 +10332,11 @@ to-through@^2.0.0: dependencies: through2 "^2.0.3" +toidentifier@~1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/toidentifier/-/toidentifier-1.0.1.tgz#3be34321a88a820ed1bd80dfaa33e479fbb8dd35" + integrity sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA== + tr46@~0.0.3: version "0.0.3" resolved "https://registry.yarnpkg.com/tr46/-/tr46-0.0.3.tgz#8184fd347dac9cdc185992f3a6622e14b9d9ab6a" @@ -9393,6 +10393,11 @@ tslib@2.6.2: resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.6.2.tgz#703ac29425e7b37cd6fd456e92404d46d1f3e4ae" integrity sha512-AEYxH93jGFPn/a2iVAwW87VuUIkR1FVUKB77NwMF7nBTDkDrrT/Hpt/IrCJ0QXhW27jTBDcf5ZY7w6RiqTMw2Q== +tslib@^1.9.3: + version "1.14.1" + resolved "https://registry.yarnpkg.com/tslib/-/tslib-1.14.1.tgz#cf2d38bdc34a134bcaf1091c41f6619e2f672d00" + integrity sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg== + tslib@^2.0.0, tslib@^2.0.1, tslib@^2.1.0, tslib@^2.4.0, tslib@^2.6.2, tslib@^2.8.1: version "2.8.1" resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.8.1.tgz#612efe4ed235d567e8aba5f2a5fab70280ade83f" @@ -9435,6 +10440,14 @@ type-fest@^0.8.1: resolved "https://registry.yarnpkg.com/type-fest/-/type-fest-0.8.1.tgz#09e249ebde851d3b1e48d27c105444667f17b83d" integrity sha512-4dbzIzqvjtgiM5rw1k5rEHtBANKmdudhGyBEajN01fEyhaAIhsoKNy6y7+IN93IfpFtwY9iqi7kD+xwKhQsNJA== +type-is@~1.6.18: + version "1.6.18" + resolved "https://registry.yarnpkg.com/type-is/-/type-is-1.6.18.tgz#4e552cd05df09467dcbc4ef739de89f2cf37c131" + integrity sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g== + dependencies: + media-typer "0.3.0" + mime-types "~2.1.24" + type@^2.7.2: version "2.7.3" resolved "https://registry.yarnpkg.com/type/-/type-2.7.3.tgz#436981652129285cc3ba94f392886c2637ea0486" @@ -9449,6 +10462,42 @@ typed-array-buffer@^1.0.3: es-errors "^1.3.0" is-typed-array "^1.1.14" +typed-array-byte-length@^1.0.3: + version "1.0.3" + resolved "https://registry.yarnpkg.com/typed-array-byte-length/-/typed-array-byte-length-1.0.3.tgz#8407a04f7d78684f3d252aa1a143d2b77b4160ce" + integrity sha512-BaXgOuIxz8n8pIq3e7Atg/7s+DpiYrxn4vdot3w9KbnBhcRQq6o3xemQdIfynqSeXeDrF32x+WvfzmOjPiY9lg== + dependencies: + call-bind "^1.0.8" + for-each "^0.3.3" + gopd "^1.2.0" + has-proto "^1.2.0" + is-typed-array "^1.1.14" + +typed-array-byte-offset@^1.0.4: + version "1.0.4" + resolved "https://registry.yarnpkg.com/typed-array-byte-offset/-/typed-array-byte-offset-1.0.4.tgz#ae3698b8ec91a8ab945016108aef00d5bff12355" + integrity sha512-bTlAFB/FBYMcuX81gbL4OcpH5PmlFHqlCCpAl8AlEzMz5k53oNDvN8p1PNOWLEmI2x4orp3raOFB51tv9X+MFQ== + dependencies: + available-typed-arrays "^1.0.7" + call-bind "^1.0.8" + for-each "^0.3.3" + gopd "^1.2.0" + has-proto "^1.2.0" + is-typed-array "^1.1.15" + reflect.getprototypeof "^1.0.9" + +typed-array-length@^1.0.7: + version "1.0.7" + resolved "https://registry.yarnpkg.com/typed-array-length/-/typed-array-length-1.0.7.tgz#ee4deff984b64be1e118b0de8c9c877d5ce73d3d" + integrity sha512-3KS2b+kL7fsuk/eJZ7EQdnEmQoaho/r6KUef7hxvltNA5DR8NAUM+8wJMbJyZ4G9/7i3v5zPBIMN5aybAh2/Jg== + dependencies: + call-bind "^1.0.7" + for-each "^0.3.3" + gopd "^1.0.1" + is-typed-array "^1.1.13" + possible-typed-array-names "^1.0.0" + reflect.getprototypeof "^1.0.6" + typedarray-to-buffer@^3.1.5: version "3.1.5" resolved "https://registry.yarnpkg.com/typedarray-to-buffer/-/typedarray-to-buffer-3.1.5.tgz#a97ee7a9ff42691b9f783ff1bc5112fe3fca9080" @@ -9466,11 +10515,26 @@ typescript@^4.4.3: resolved "https://registry.yarnpkg.com/typescript/-/typescript-4.9.5.tgz#095979f9bcc0d09da324d58d03ce8f8374cbe65a" integrity sha512-1FXk9E2Hm+QzZQ7z+McJiHL4NW1F2EzMu9Nq9i3zAaGqibafqYwCVU6WyWAuyQRRzOlxou8xZSyXLEN8oKj24g== +ua-parser-js@^0.7.30: + version "0.7.41" + resolved "https://registry.yarnpkg.com/ua-parser-js/-/ua-parser-js-0.7.41.tgz#9f6dee58c389e8afababa62a4a2dc22edb69a452" + integrity sha512-O3oYyCMPYgNNHuO7Jjk3uacJWZF8loBgwrfd/5LE/HyZ3lUIOdniQ7DNXJcIgZbwioZxk0fLfI4EVnetdiX5jg== + uglify-js@^3.1.4: version "3.19.3" resolved "https://registry.yarnpkg.com/uglify-js/-/uglify-js-3.19.3.tgz#82315e9bbc6f2b25888858acd1fff8441035b77f" integrity sha512-v3Xu+yuwBXisp6QYTcH4UbH+xYJXqnq2m/LtQVWKWzYc1iehYnLixoQDN9FH6/j9/oybfd6W9Ghwkl8+UMKTKQ== +unbox-primitive@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/unbox-primitive/-/unbox-primitive-1.1.0.tgz#8d9d2c9edeea8460c7f35033a88867944934d1e2" + integrity sha512-nWJ91DjeOkej/TA8pXQ3myruKpKEYgqvpw9lz4OPHj/NWFNluYrjbz9j01CJ8yKQd2g4jFoOkINCTW2I5LEEyw== + dependencies: + call-bound "^1.0.3" + has-bigints "^1.0.2" + has-symbols "^1.1.0" + which-boxed-primitive "^1.1.1" + unc-path-regex@^0.1.2: version "0.1.2" resolved "https://registry.yarnpkg.com/unc-path-regex/-/unc-path-regex-0.1.2.tgz#e73dd3d7b0d7c5ed86fbac6b0ae7d8c6a69d50fa" @@ -9502,6 +10566,11 @@ undici-types@~7.10.0: resolved "https://registry.yarnpkg.com/undici-types/-/undici-types-7.10.0.tgz#4ac2e058ce56b462b056e629cc6a02393d3ff350" integrity sha512-t5Fy/nfn+14LuOc2KNYg75vZqClpAiqscVvMygNnlsHBFpSXdJaYtXMcdNLpl/Qvc3P2cB3s6lOV51nqsFq4ag== +undici-types@~7.18.0: + version "7.18.2" + resolved "https://registry.yarnpkg.com/undici-types/-/undici-types-7.18.2.tgz#29357a89e7b7ca4aef3bf0fd3fd0cd73884229e9" + integrity sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w== + unified@^9.1.0: version "9.2.2" resolved "https://registry.yarnpkg.com/unified/-/unified-9.2.2.tgz#67649a1abfc3ab85d2969502902775eb03146975" @@ -9558,11 +10627,21 @@ unist-util-stringify-position@^2.0.0: dependencies: "@types/unist" "^2.0.2" +universalify@^0.1.0: + version "0.1.2" + resolved "https://registry.yarnpkg.com/universalify/-/universalify-0.1.2.tgz#b646f69be3942dabcecc9d6639c80dc105efaa66" + integrity sha512-rBJeI5CXAlmy1pV+617WB9J63U6XcazHHF2f2dbJix4XzpUF0RS3Zbj0FGIOCAva5P/d/GBOYaACQ1w+0azUkg== + universalify@^2.0.0: version "2.0.1" resolved "https://registry.yarnpkg.com/universalify/-/universalify-2.0.1.tgz#168efc2180964e6386d061e094df61afe239b18d" integrity sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw== +unpipe@~1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/unpipe/-/unpipe-1.0.0.tgz#b2bf4ee8514aae6165b4817829d21b2ef49904ec" + integrity sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ== + unset-value@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/unset-value/-/unset-value-1.0.0.tgz#8376873f7d2335179ffb1e6fc3a8ed0dfc8ab559" @@ -9659,6 +10738,11 @@ util@^0.12.5: is-typed-array "^1.1.3" which-typed-array "^1.1.2" +utils-merge@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/utils-merge/-/utils-merge-1.0.1.tgz#9f95710f50a267947b2ccc124741c1028427e713" + integrity sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA== + utrie@^1.0.2: version "1.0.2" resolved "https://registry.yarnpkg.com/utrie/-/utrie-1.0.2.tgz#d42fe44de9bc0119c25de7f564a6ed1b2c87a645" @@ -9701,6 +10785,11 @@ value-or-function@^3.0.0: resolved "https://registry.yarnpkg.com/value-or-function/-/value-or-function-3.0.0.tgz#1c243a50b595c1be54a754bfece8563b9ff8d813" integrity sha512-jdBB2FrWvQC/pnPtIqcLsMaQgjhdb6B7tk1MMyTKapox+tQZbdRP4uLxu/JY0t7fbfDCUMnuelzEYv5GsxHhdg== +vary@^1: + version "1.1.2" + resolved "https://registry.yarnpkg.com/vary/-/vary-1.1.2.tgz#2299f02c6ded30d4a5961b0b9f74524a18f634fc" + integrity sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg== + vfile-message@^2.0.0: version "2.0.4" resolved "https://registry.yarnpkg.com/vfile-message/-/vfile-message-2.0.4.tgz#5b43b88171d409eae58477d13f23dd41d52c371a" @@ -9779,6 +10868,11 @@ vm-browserify@^1.0.1: resolved "https://registry.yarnpkg.com/vm-browserify/-/vm-browserify-1.1.2.tgz#78641c488b8e6ca91a75f511e7a3b32a86e5dda0" integrity sha512-2ham8XPWTONajOR0ohOKOHXkm3+gaBmGut3SRuu75xLd/RRaY6vqgh8NBYYk7+RW3u5AtzPQZG8F10LHkl0lAQ== +void-elements@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/void-elements/-/void-elements-2.0.1.tgz#c066afb582bb1cb4128d60ea92392e94d5e9dbec" + integrity sha512-qZKX4RnBzH2ugr8Lxa7x+0V6XD9Sb/ouARtiasEQCHB1EVU4NXtmHsDDrx1dO4ne5fc3J6EW05BP1Dl0z0iung== + watchpack@^2.4.1: version "2.4.4" resolved "https://registry.yarnpkg.com/watchpack/-/watchpack-2.4.4.tgz#473bda72f0850453da6425081ea46fc0d7602947" @@ -9864,6 +10958,46 @@ whatwg-url@^5.0.0: tr46 "~0.0.3" webidl-conversions "^3.0.0" +which-boxed-primitive@^1.1.0, which-boxed-primitive@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/which-boxed-primitive/-/which-boxed-primitive-1.1.1.tgz#d76ec27df7fa165f18d5808374a5fe23c29b176e" + integrity sha512-TbX3mj8n0odCBFVlY8AxkqcHASw3L60jIuF8jFP78az3C2YhmGvqbHBpAjTRH2/xqYunrJ9g1jSyjCjpoWzIAA== + dependencies: + is-bigint "^1.1.0" + is-boolean-object "^1.2.1" + is-number-object "^1.1.1" + is-string "^1.1.1" + is-symbol "^1.1.1" + +which-builtin-type@^1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/which-builtin-type/-/which-builtin-type-1.2.1.tgz#89183da1b4907ab089a6b02029cc5d8d6574270e" + integrity sha512-6iBczoX+kDQ7a3+YJBnh3T+KZRxM/iYNPXicqk66/Qfm1b93iu+yOImkg0zHbj5LNOcNv1TEADiZ0xa34B4q6Q== + dependencies: + call-bound "^1.0.2" + function.prototype.name "^1.1.6" + has-tostringtag "^1.0.2" + is-async-function "^2.0.0" + is-date-object "^1.1.0" + is-finalizationregistry "^1.1.0" + is-generator-function "^1.0.10" + is-regex "^1.2.1" + is-weakref "^1.0.2" + isarray "^2.0.5" + which-boxed-primitive "^1.1.0" + which-collection "^1.0.2" + which-typed-array "^1.1.16" + +which-collection@^1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/which-collection/-/which-collection-1.0.2.tgz#627ef76243920a107e7ce8e96191debe4b16c2a0" + integrity sha512-K4jVyjnBdgvc86Y6BkaLZEN933SwYOuBFkdmBu9ZfkcAbdVbpITnDmjvZ/aQjRXQrv5EPkTnD1s39GiiqbngCw== + dependencies: + is-map "^2.0.3" + is-set "^2.0.3" + is-weakmap "^2.0.2" + is-weakset "^2.0.3" + which-module@^1.0.0: version "1.0.0" resolved "https://registry.yarnpkg.com/which-module/-/which-module-1.0.0.tgz#bba63ca861948994ff307736089e3b96026c2a4f" @@ -9887,7 +11021,20 @@ which-typed-array@^1.1.16, which-typed-array@^1.1.2: gopd "^1.2.0" has-tostringtag "^1.0.2" -which@^1.2.14, which@^1.2.9, which@^1.3.1: +which-typed-array@^1.1.19: + version "1.1.20" + resolved "https://registry.yarnpkg.com/which-typed-array/-/which-typed-array-1.1.20.tgz#3fdb7adfafe0ea69157b1509f3a1cd892bd1d122" + integrity sha512-LYfpUkmqwl0h9A2HL09Mms427Q1RZWuOHsukfVcKRq9q95iQxdw0ix1JQrqbcDR9PH1QDwf5Qo8OZb5lksZ8Xg== + dependencies: + available-typed-arrays "^1.0.7" + call-bind "^1.0.8" + call-bound "^1.0.4" + for-each "^0.3.5" + get-proto "^1.0.1" + gopd "^1.2.0" + has-tostringtag "^1.0.2" + +which@^1.2.1, which@^1.2.14, which@^1.2.9, which@^1.3.1: version "1.3.1" resolved "https://registry.yarnpkg.com/which/-/which-1.3.1.tgz#a45043d54f5805316da8d62f9f50918d3da70b0a" integrity sha512-HxJdYWq1MTIQbJ3nw0cqssHoTNU267KlrDuGZ1WYlxDStUtKUhOaJmh112/TZmHxxUfuJqPXSOm7tDyas0OSIQ== @@ -9901,13 +11048,6 @@ which@^2.0.1: dependencies: isexe "^2.0.0" -which@^5.0.0: - version "5.0.0" - resolved "https://registry.yarnpkg.com/which/-/which-5.0.0.tgz#d93f2d93f79834d4363c7d0c23e00d07c466c8d6" - integrity sha512-JEdGzHwwkrbWoGOlIHqQ5gtprKGOenpDHpxE9zVR1bWbOtYRyPPHMe9FaP6x61CmNaTThSkb0DAJte5jD+DmzQ== - dependencies: - isexe "^3.1.1" - wide-align@^1.1.2: version "1.1.5" resolved "https://registry.yarnpkg.com/wide-align/-/wide-align-1.1.5.tgz#df1d4c206854369ecf3c9a4898f1b23fbd9d15d3" @@ -9925,7 +11065,7 @@ wordwrap@^1.0.0: resolved "https://registry.yarnpkg.com/wordwrap/-/wordwrap-1.0.0.tgz#27584810891456a4171c8d0226441ade90cbcaeb" integrity sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q== -"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0", wrap-ansi@^7.0.0: +"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0": version "7.0.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== @@ -9951,6 +11091,15 @@ wrap-ansi@^6.2.0: string-width "^4.1.0" strip-ansi "^6.0.0" +wrap-ansi@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" + integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== + dependencies: + ansi-styles "^4.0.0" + string-width "^4.1.0" + strip-ansi "^6.0.0" + wrap-ansi@^8.1.0: version "8.1.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-8.1.0.tgz#56dc22368ee570face1b49819975d9b9a5ead214" @@ -9985,6 +11134,11 @@ ws@^8.19.0: resolved "https://registry.yarnpkg.com/ws/-/ws-8.19.0.tgz#ddc2bdfa5b9ad860204f5a72a4863a8895fd8c8b" integrity sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg== +ws@~8.18.3: + version "8.18.3" + resolved "https://registry.yarnpkg.com/ws/-/ws-8.18.3.tgz#b56b88abffde62791c639170400c93dcb0c95472" + integrity sha512-PEIGCY5tSlUt50cqyMXfCzX+oOPqN0vuGqWzbcJ2xvnkzkq46oOpz7dQaTDBdfICb4N14+GARUDw2XV2N4tvzg== + xcode@^3.0.1: version "3.0.1" resolved "https://registry.yarnpkg.com/xcode/-/xcode-3.0.1.tgz#3efb62aac641ab2c702458f9a0302696146aa53c" @@ -10173,7 +11327,7 @@ yargs@^15.3.1: y18n "^4.0.0" yargs-parser "^18.1.2" -yargs@^16.2.0: +yargs@^16.1.1, yargs@^16.2.0: version "16.2.0" resolved "https://registry.yarnpkg.com/yargs/-/yargs-16.2.0.tgz#1c82bf0f6b6a66eafce7ef30e376f49a12477f66" integrity sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw== From 97dd3cef04088f61cf155da21bc2533f6aa6f717 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 15 Mar 2026 17:06:21 +0800 Subject: [PATCH 160/375] fix lint --- deps/common/.carve/config.edn | 3 ++- deps/db-sync/.carve/ignore | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/deps/common/.carve/config.edn b/deps/common/.carve/config.edn index 6889804745..bdd5a7cab0 100644 --- a/deps/common/.carve/config.edn +++ b/deps/common/.carve/config.edn @@ -10,5 +10,6 @@ logseq.common.util.macro logseq.common.cognito-config logseq.common.config - logseq.common.defkeywords] + logseq.common.defkeywords + logseq.common.graph-dir] :report {:format :ignore}} diff --git a/deps/db-sync/.carve/ignore b/deps/db-sync/.carve/ignore index dbc8e2ddb8..abb5a83564 100644 --- a/deps/db-sync/.carve/ignore +++ b/deps/db-sync/.carve/ignore @@ -21,4 +21,6 @@ logseq.db-sync.snapshot/framed-length ;; API logseq.db-sync.worker/worker ;; debugging -logseq.db-sync.worker.timing/summary \ No newline at end of file +logseq.db-sync.worker.timing/summary +;; API +logseq.db-sync.snapshot/finalize-datoms-jsonl-buffer \ No newline at end of file From f7d624c256d7950e19b1982cefc1b36322523cdd Mon Sep 17 00:00:00 2001 From: rcmerci Date: Sun, 15 Mar 2026 21:52:22 +0800 Subject: [PATCH 161/375] fix: sync download --- src/main/frontend/worker/db_core.cljs | 55 +++--- src/main/frontend/worker/sync.cljs | 211 ++++++++++++--------- src/test/frontend/worker/db_sync_test.cljs | 61 +++--- src/test/logseq/cli/command/sync_test.cljs | 9 +- 4 files changed, 188 insertions(+), 148 deletions(-) diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index f0e7520928..19c664576b 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -462,11 +462,15 @@ (db-sync/upload-graph! repo)) (defn- validate-sync-download-result! - [repo {:keys [rows graph-id remote-tx] :as result}] - (when-not (sequential? rows) + [repo {:keys [datoms datom-count graph-id remote-tx] :as result}] + (when-not (sequential? datoms) (db-sync/fail-fast :db-sync/invalid-field {:repo repo - :field :rows - :value rows})) + :field :datoms + :value datoms})) + (when-not (integer? datom-count) + (db-sync/fail-fast :db-sync/invalid-field {:repo repo + :field :datom-count + :value datom-count})) (when-not (and (string? graph-id) (seq graph-id)) (db-sync/fail-fast :db-sync/invalid-field {:repo repo :field :graph-id @@ -477,43 +481,46 @@ :value remote-tx})) result) +(defn- import-downloaded-datoms! + [repo graph-id remote-tx graph-e2ee? datoms] + (if (seq datoms) + (p/let [{:keys [import-id]} ((@thread-api/*thread-apis :thread-api/db-sync-import-prepare) + repo true graph-id graph-e2ee? (count datoms)) + _ ((@thread-api/*thread-apis :thread-api/db-sync-import-datoms-chunk) + datoms graph-id import-id) + _ ((@thread-api/*thread-apis :thread-api/db-sync-import-finalize) + repo graph-id remote-tx import-id)] + true) + (do + (sync-log-and-state/rtc-log :rtc.log/download + {:sub-type :download-completed + :graph-uuid graph-id + :message "Graph is ready!"}) + (p/resolved true)))) + (def-thread-api :thread-api/db-sync-download-graph [repo] (p/let [download-result (db-sync/download-graph! repo) - {:keys [rows graph-id remote-tx graph-e2ee?]} + {:keys [datoms datom-count graph-id remote-tx graph-e2ee?]} (validate-sync-download-result! repo download-result) - row-count (count rows) - _ (if (seq rows) - ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) - repo rows true graph-id remote-tx graph-e2ee?) - (sync-log-and-state/rtc-log :rtc.log/download - {:sub-type :download-completed - :graph-uuid graph-id - :message "Graph is ready!"}))] + _ (import-downloaded-datoms! repo graph-id remote-tx graph-e2ee? datoms)] {:repo repo :graph-id graph-id :remote-tx remote-tx :graph-e2ee? graph-e2ee? - :row-count row-count})) + :row-count datom-count})) (def-thread-api :thread-api/db-sync-download-graph-by-id [repo graph-id graph-e2ee?] (p/let [download-result (db-sync/download-graph-by-id! repo graph-id graph-e2ee?) - {:keys [rows graph-id remote-tx graph-e2ee?]} + {:keys [datoms datom-count graph-id remote-tx graph-e2ee?]} (validate-sync-download-result! repo download-result) - row-count (count rows) - _ (if (seq rows) - ((@thread-api/*thread-apis :thread-api/db-sync-import-kvs-rows) - repo rows true graph-id remote-tx graph-e2ee?) - (sync-log-and-state/rtc-log :rtc.log/download - {:sub-type :download-completed - :graph-uuid graph-id - :message "Graph is ready!"}))] + _ (import-downloaded-datoms! repo graph-id remote-tx graph-e2ee? datoms)] {:repo repo :graph-id graph-id :remote-tx remote-tx :graph-e2ee? graph-e2ee? - :row-count row-count})) + :row-count datom-count})) (def-thread-api :thread-api/set-infer-worker-proxy [infer-worker-proxy] diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index cd10e1980e..c8b15acb20 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -22,6 +22,7 @@ [logseq.db-sync.cycle :as sync-cycle] [logseq.db-sync.malli-schema :as db-sync-schema] [logseq.db-sync.order :as sync-order] + [logseq.db-sync.snapshot :as snapshot] [logseq.db.common.normalize :as db-normalize] [logseq.db.common.sqlite :as common-sqlite] [logseq.db.sqlite.util :as sqlite-util] @@ -519,56 +520,6 @@ (string? payload) (.encode text-encoder payload) :else (js/Uint8Array. payload))) -(defn- decode-snapshot-rows - [payload] - (sqlite-util/read-transit-str (.decode text-decoder (->uint8 payload)))) - -(defn- frame-len - [^js payload offset] - (let [view (js/DataView. (.-buffer payload) offset 4)] - (.getUint32 view 0 false))) - -(defn- concat-payload - [^js a ^js b] - (cond - (nil? a) b - (nil? b) a - :else - (let [combined (js/Uint8Array. (+ (.-byteLength a) (.-byteLength b)))] - (.set combined a 0) - (.set combined b (.-byteLength a)) - combined))) - -(defn- parse-framed-chunk - [buffer chunk] - (let [payload (concat-payload buffer chunk) - total (.-byteLength payload)] - (loop [offset 0 - rows []] - (if (< (- total offset) 4) - {:rows rows - :buffer (when (< offset total) - (.slice payload offset total))} - (let [len (frame-len payload offset) - next-offset (+ offset 4 len)] - (if (<= next-offset total) - (let [frame-payload (.slice payload (+ offset 4) next-offset) - decoded (decode-snapshot-rows frame-payload)] - (recur next-offset (into rows decoded))) - {:rows rows - :buffer (.slice payload offset total)})))))) - -(defn- finalize-framed-buffer - [buffer] - (if (or (nil? buffer) (zero? (.-byteLength buffer))) - [] - (let [{:keys [rows buffer]} (parse-framed-chunk nil buffer)] - (if (and (seq rows) (or (nil? buffer) (zero? (.-byteLength buffer)))) - rows - (fail-fast :db-sync/incomplete-snapshot-frame - {:rows (count rows) - :remaining-buffer-bytes (some-> buffer .-byteLength)}))))) - (defn- gzip-payload? [^js payload] (and (some? payload) @@ -594,7 +545,7 @@ (p/rejected (ex-info "gzip decompression not supported" {:type :db-sync/decompression-not-supported})))) -(defn- uint8 array-buffer)] @@ -602,6 +553,56 @@ ( resp .-headers (.get "content-encoding"))] + (cond + (nil? (.-body resp)) + nil + + (= "gzip" encoding) + (when (exists? js/DecompressionStream) + (.pipeThrough (.-body resp) (js/DecompressionStream. "gzip"))) + + :else + (.-body resp)))) + +(defn- = (count remaining) batch-size) + (let [batch (subvec remaining 0 batch-size) + rest-datoms (subvec remaining batch-size)] + (p/let [_ (on-batch batch)] + (p/recur rest-datoms))) + remaining))) + +(defn- uint8 (.-value result))) + pending (into pending datoms)] + (p/let [pending (js (with-auth-headers {:method "GET"}))) - total-bytes (response-content-length resp) - _ (download-log! graph-id - :download-progress - (if (number? total-bytes) - (str "Start downloading graph snapshot, file size: " total-bytes) - "Start downloading graph snapshot")) - _ (when-not (.-ok resp) - (fail-fast :db-sync/snapshot-download-failed {:repo repo - :graph-id graph-id - :status (.-status resp)})) - payload (js (with-auth-headers {:method "GET"}))) + total-bytes (response-content-length resp) + _ (download-log! graph-id + :download-progress + (if (number? total-bytes) + (str "Start downloading graph snapshot, file size: " total-bytes) + "Start downloading graph snapshot")) + _ (when-not (.-ok resp) + (fail-fast :db-sync/snapshot-download-failed {:repo repo + :graph-id graph-id + :status (.-status resp)})) + collect-datoms? (nil? on-datoms-batch) + datoms* (atom []) + datom-count* (atom 0) + on-datoms-batch* (fn [datoms] + (swap! datom-count* + (count datoms)) + (if collect-datoms? + (do + (swap! datoms* into datoms) + (p/resolved nil)) + (on-datoms-batch datoms))) + _ ( {:repo repo + :graph-id graph-id + :remote-tx remote-tx + :graph-e2ee? graph-e2ee? + :datom-count @datom-count*} + collect-datoms? + (assoc :datoms @datoms*)))))))) (defn download-graph! [repo] @@ -2378,8 +2401,10 @@ (download-graph-with-id! repo graph-id graph-e2ee?))) (defn download-graph-by-id! - [repo graph-id graph-e2ee?] - (download-graph-with-id! repo graph-id graph-e2ee?)) + ([repo graph-id graph-e2ee?] + (download-graph-with-id! repo graph-id graph-e2ee?)) + ([repo graph-id graph-e2ee? opts] + (download-graph-with-id! repo graph-id graph-e2ee? opts))) (defn upload-graph! [repo] diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index 57d57b0db7..d0dc37dcc2 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -1246,14 +1246,11 @@ (reset! worker-state/*db-sync-config config-prev) (done)))))))) -(deftest download-graph-by-id-fails-on-incomplete-snapshot-frame-test - (testing "snapshot download rejects incomplete framed payload" +(deftest download-graph-by-id-fails-when-snapshot-download-url-missing-test + (testing "snapshot download rejects missing snapshot url" (async done (let [fetch-prev js/fetch - config-prev @worker-state/*db-sync-config - rows [["addr-1" "content-1" nil]] - frame (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows)) - truncated (.slice frame 0 (dec (.-byteLength frame)))] + config-prev @worker-state/*db-sync-config] (reset! worker-state/*db-sync-config {:http-base "https://example.com" :auth-token "token-value"}) (set! js/fetch @@ -1264,36 +1261,35 @@ :status 200 :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":7,\"txs\":[]}"))}) - (>= (.indexOf url "/snapshot/stream") 0) + (>= (.indexOf url "/snapshot/download") 0) (js/Promise.resolve #js {:ok true :status 200 - :arrayBuffer (fn [] (js/Promise.resolve (.-buffer truncated)))}) + :text (fn [] (js/Promise.resolve "{\"ok\":true,\"key\":\"graph-1/snapshot-1.snapshot\",\"url\":\"\"}"))}) :else (js/Promise.reject (js/Error. (str "unexpected fetch url: " url)))))) (-> (p/let [_ (db-sync/download-graph-by-id! test-repo "graph-1" false)] - (is false "expected incomplete frame failure")) + (is false "expected missing snapshot url failure")) (p/catch (fn [e] - (is (= "incomplete-snapshot-frame" (ex-message e))))) + (is (= "missing-field" (ex-message e))) + (is (= :snapshot-url (get-in (ex-data e) [:field]))))) (p/finally (fn [] (set! js/fetch fetch-prev) (reset! worker-state/*db-sync-config config-prev) (done)))))))) -(deftest download-graph-by-id-preserves-framed-row-batches-test - (testing "snapshot download merges complete frames without truncating rows" +(deftest download-graph-by-id-parses-jsonl-datoms-test + (testing "snapshot download parses ndjson transit datoms" (async done (let [fetch-prev js/fetch config-prev @worker-state/*db-sync-config - rows-a [["addr-1" "content-1" nil] - ["addr-2" "content-2" nil]] - rows-b [["addr-3" "content-3" nil]] - frame-a (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows-a)) - frame-b (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows-b)) - payload (js/Uint8Array. (+ (.-byteLength frame-a) (.-byteLength frame-b)))] - (.set payload frame-a 0) - (.set payload frame-b (.-byteLength frame-a)) + asset-url "https://example.com/assets/graph-1.snapshot" + datoms [{:e 1 :a :db/ident :v :logseq.class/Page :tx 1 :added true} + {:e 2 :a :block/title :v "hello" :tx 1 :added true}] + payload-str (str (sqlite-util/write-transit-str (first datoms)) "\n" + (sqlite-util/write-transit-str (second datoms)) "\n") + payload (.encode (js/TextEncoder.) payload-str)] (reset! worker-state/*db-sync-config {:http-base "https://example.com" :auth-token "token-value"}) (set! js/fetch @@ -1304,7 +1300,12 @@ :status 200 :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":7,\"txs\":[]}"))}) - (>= (.indexOf url "/snapshot/stream") 0) + (>= (.indexOf url "/snapshot/download") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :text (fn [] (js/Promise.resolve (str "{\"ok\":true,\"key\":\"graph-1/snapshot-1.snapshot\",\"url\":\"" asset-url "\"}")))}) + + (= url asset-url) (js/Promise.resolve #js {:ok true :status 200 :arrayBuffer (fn [] (js/Promise.resolve (.-buffer payload)))}) @@ -1315,8 +1316,8 @@ (is (= "graph-1" (:graph-id result))) (is (= 7 (:remote-tx result))) (is (= false (:graph-e2ee? result))) - (is (= (vec (concat rows-a rows-b)) - (vec (:rows result))))) + (is (= 2 (:datom-count result))) + (is (= datoms (vec (:datoms result))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally @@ -1331,8 +1332,9 @@ (let [fetch-prev js/fetch config-prev @worker-state/*db-sync-config logs (atom []) - rows [["addr-1" "content-1" nil]] - payload (#'db-sync/frame-bytes (#'db-sync/encode-snapshot-rows rows))] + asset-url "https://example.com/assets/graph-1.snapshot" + payload-str (str (sqlite-util/write-transit-str {:e 1 :a :db/ident :v :logseq.class/Page :tx 1 :added true}) "\n") + payload (.encode (js/TextEncoder.) payload-str)] (reset! worker-state/*db-sync-config {:http-base "https://example.com" :auth-token "token-value"}) (set! js/fetch @@ -1343,7 +1345,12 @@ :status 200 :text (fn [] (js/Promise.resolve "{\"type\":\"pull/ok\",\"t\":9,\"txs\":[]}"))}) - (>= (.indexOf url "/snapshot/stream") 0) + (>= (.indexOf url "/snapshot/download") 0) + (js/Promise.resolve #js {:ok true + :status 200 + :text (fn [] (js/Promise.resolve (str "{\"ok\":true,\"key\":\"graph-1/snapshot-1.snapshot\",\"url\":\"" asset-url "\"}")))}) + + (= url asset-url) (js/Promise.resolve #js {:ok true :status 200 :headers #js {:get (fn [header] @@ -1361,7 +1368,7 @@ (is (= "graph-1" (:graph-id result))) (is (= 9 (:remote-tx result))) (is (= false (:graph-e2ee? result))) - (is (= rows (vec (:rows result)))) + (is (= 1 (:datom-count result))) (let [messages (mapv (fn [[_ payload]] (:message payload)) @logs)] (is (some #(= "Preparing graph snapshot download" %) messages)) (is (some #(string/includes? % "Start downloading graph snapshot") messages)) diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index 6c3f339ddc..e993a6b71c 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -590,9 +590,10 @@ :thread-api/q (p/resolved 0) :thread-api/db-sync-download-graph-by-id - (p/rejected (ex-info "db-sync/incomplete-snapshot-frame" - {:code :db-sync/incomplete-snapshot-frame - :graph-id "remote-graph-id"})) + (p/rejected (ex-info "db-sync/snapshot-download-failed" + {:code :db-sync/snapshot-download-failed + :graph-id "remote-graph-id" + :status 500})) (p/resolved nil)))] (p/let [result (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" @@ -600,7 +601,7 @@ {:base-url "http://example" :data-dir "/tmp"})] (is (= :error (:status result))) - (is (= :db-sync/incomplete-snapshot-frame (get-in result [:error :code]))))) + (is (= :db-sync/snapshot-download-failed (get-in result [:error :code]))))) (p/catch (fn [e] (is false (str "unexpected error: " e)))) (p/finally done)))) From 7575c6b533484af5e1e40df8a391f3835ede3e5e Mon Sep 17 00:00:00 2001 From: rcmerci Date: Mon, 16 Mar 2026 20:43:30 +0800 Subject: [PATCH 162/375] enhance(cli): sync subcommand checks corresponding config vars --- src/main/frontend/worker/sync.cljs | 12 +- src/main/logseq/cli/command/sync.cljs | 127 ++++++++++---- src/main/logseq/cli/config.cljs | 2 + src/test/frontend/worker/db_sync_test.cljs | 11 ++ src/test/logseq/cli/command/sync_test.cljs | 183 +++++++++++++++++---- src/test/logseq/cli/config_test.cljs | 2 + src/test/logseq/cli/integration_test.cljs | 6 +- 7 files changed, 266 insertions(+), 77 deletions(-) diff --git a/src/main/frontend/worker/sync.cljs b/src/main/frontend/worker/sync.cljs index c8b15acb20..ee50251a0d 100644 --- a/src/main/frontend/worker/sync.cljs +++ b/src/main/frontend/worker/sync.cljs @@ -141,17 +141,7 @@ (defn http-base-url [] - (or (:http-base @worker-state/*db-sync-config) - (when-let [ws-url (ws-base-url)] - (let [base (cond - (string/starts-with? ws-url "wss://") - (str "https://" (subs ws-url (count "wss://"))) - - (string/starts-with? ws-url "ws://") - (str "http://" (subs ws-url (count "ws://"))) - - :else ws-url)] - (string/replace base #"/sync/%s$" ""))))) + (:http-base @worker-state/*db-sync-config)) (defn- cli-node-owner? [] diff --git a/src/main/logseq/cli/command/sync.cljs b/src/main/logseq/cli/command/sync.cljs index daf5a5337d..5e4b648735 100644 --- a/src/main/logseq/cli/command/sync.cljs +++ b/src/main/logseq/cli/command/sync.cljs @@ -65,6 +65,46 @@ (def ^:private sync-start-skipped-states #{:inactive :stopped}) +(def ^:private sync-config-defaults + {:ws-url "wss://api.logseq.io/sync/%s" + :http-base "https://api.logseq.io"}) + +(def ^:private required-sync-config-keys-by-action + {:sync-start [:ws-url] + :sync-upload [:http-base] + :sync-download [:http-base] + :sync-grant-access [:http-base :e2ee-password]}) + +(defn- config-value-present? + [value] + (cond + (string? value) (not (string/blank? value)) + :else (some? value))) + +(defn- effective-sync-config-value + [config key] + (let [sentinel ::missing + value (get config key sentinel)] + (if (= sentinel value) + (get sync-config-defaults key) + value))) + +(defn- missing-required-sync-config-keys + [action-type config] + (->> (get required-sync-config-keys-by-action action-type) + (remove (fn [key] + (config-value-present? (effective-sync-config-value config key)))) + vec)) + +(defn- missing-sync-config-error + [action-type missing-keys] + {:status :error + :error {:code :missing-sync-config + :message (str "missing required sync config for " (name action-type) + ": " (string/join ", " (map name missing-keys))) + :action action-type + :missing-keys missing-keys}}) + (defn- print-progress-line! [line] (when (seq (some-> line str string/trim)) @@ -391,26 +431,33 @@ :error {:code :remote-graph-not-found :message (str "remote graph not found: " (:graph action)) :graph (:graph action)}} - (p/let [cfg (cli-server/ensure-server! config' (:repo action)) - _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) - _ (ensure-empty-download-db! cfg (:repo action)) - download-cfg (sync-download-invoke-config cfg) - graph-id (:graph-id remote-graph) - events-sub (when progress-enabled? - (transport/connect-events! - download-cfg - (fn [event-type payload] - (when-let [message (download-progress-message graph-id event-type payload)] - (print-progress-line! message))))) - result (-> (transport/invoke download-cfg :thread-api/db-sync-download-graph-by-id false - [(:repo action) graph-id (:graph-e2ee? remote-graph)]) - (p/finally (fn [] - (when-let [close! (:close! events-sub)] - (close!)))))] - {:status :ok - :data (if (map? result) - result - {:result result})}))) + (let [missing-keys (when (true? (:graph-e2ee? remote-graph)) + (->> [:e2ee-password] + (remove (fn [key] + (config-value-present? (effective-sync-config-value config' key)))) + vec))] + (if (seq missing-keys) + (missing-sync-config-error :sync-download missing-keys) + (p/let [cfg (cli-server/ensure-server! config' (:repo action)) + _ (transport/invoke cfg :thread-api/set-db-sync-config false [(sync-config config')]) + _ (ensure-empty-download-db! cfg (:repo action)) + download-cfg (sync-download-invoke-config cfg) + graph-id (:graph-id remote-graph) + events-sub (when progress-enabled? + (transport/connect-events! + download-cfg + (fn [event-type payload] + (when-let [message (download-progress-message graph-id event-type payload)] + (print-progress-line! message))))) + result (-> (transport/invoke download-cfg :thread-api/db-sync-download-graph-by-id false + [(:repo action) graph-id (:graph-e2ee? remote-graph)]) + (p/finally (fn [] + (when-let [close! (:close! events-sub)] + (close!)))))] + {:status :ok + :data (if (map? result) + result + {:result result})}))))) (p/catch (fn [error] (exception->error error {:repo (:repo action) :graph (:graph action)})))))) @@ -427,11 +474,14 @@ :sync-start (-> (p/let [config' (resolve-runtime-config! action config) - _ (invoke-with-repo config' (:repo action) - :thread-api/db-sync-start - [(:repo action)]) - result (wait-sync-start-ready config' (:repo action) action)] - result) + missing-keys (missing-required-sync-config-keys (:type action) config')] + (if (seq missing-keys) + (missing-sync-config-error (:type action) missing-keys) + (p/let [_ (invoke-with-repo config' (:repo action) + :thread-api/db-sync-start + [(:repo action)]) + result (wait-sync-start-ready config' (:repo action) action)] + result))) (p/catch (fn [error] (exception->error error {:repo (:repo action)})))) @@ -443,14 +493,20 @@ :data {:result result}}) :sync-upload - (-> (p/let [config' (resolve-runtime-config! action config)] - (execute-sync-upload action config')) + (-> (p/let [config' (resolve-runtime-config! action config) + missing-keys (missing-required-sync-config-keys (:type action) config')] + (if (seq missing-keys) + (missing-sync-config-error (:type action) missing-keys) + (execute-sync-upload action config'))) (p/catch (fn [error] (exception->error error {:repo (:repo action)})))) :sync-download - (-> (p/let [config' (resolve-runtime-config! action config)] - (execute-sync-download action config')) + (-> (p/let [config' (resolve-runtime-config! action config) + missing-keys (missing-required-sync-config-keys (:type action) config')] + (if (seq missing-keys) + (missing-sync-config-error (:type action) missing-keys) + (execute-sync-download action config'))) (p/catch (fn [error] (exception->error error {:repo (:repo action) :graph (:graph action)})))) @@ -473,11 +529,14 @@ :sync-grant-access (-> (p/let [config' (resolve-runtime-config! action config) - result (invoke-with-repo config' (:repo action) - :thread-api/db-sync-grant-graph-access - [(:repo action) (:graph-id action) (:email action)])] - {:status :ok - :data {:result result}}) + missing-keys (missing-required-sync-config-keys (:type action) config')] + (if (seq missing-keys) + (missing-sync-config-error (:type action) missing-keys) + (p/let [result (invoke-with-repo config' (:repo action) + :thread-api/db-sync-grant-graph-access + [(:repo action) (:graph-id action) (:email action)])] + {:status :ok + :data {:result result}}))) (p/catch (fn [error] (exception->error error {:repo (:repo action) :graph-id (:graph-id action) diff --git a/src/main/logseq/cli/config.cljs b/src/main/logseq/cli/config.cljs index 97865d5fb3..a041ff0626 100644 --- a/src/main/logseq/cli/config.cljs +++ b/src/main/logseq/cli/config.cljs @@ -100,6 +100,8 @@ :logout-timeout-ms 120000 :output-format nil :data-dir (common-graph/get-default-graphs-dir) + :ws-url "wss://api.logseq.io/sync/%s" + :http-base "https://api.logseq.io" :config-path (default-config-path)} env (env-config) config-path (or (:config-path opts) diff --git a/src/test/frontend/worker/db_sync_test.cljs b/src/test/frontend/worker/db_sync_test.cljs index d0dc37dcc2..17e057f853 100644 --- a/src/test/frontend/worker/db_sync_test.cljs +++ b/src/test/frontend/worker/db_sync_test.cljs @@ -36,6 +36,17 @@ (use-fixtures :each {:before reset-db-sync-test-state! :after reset-db-sync-test-state!}) +(deftest http-base-url-does-not-derive-from-ws-url-test + (let [config-prev @worker-state/*db-sync-config] + (try + (reset! worker-state/*db-sync-config {:ws-url "wss://api.logseq.io/sync/%s"}) + (is (nil? (#'db-sync/http-base-url))) + (reset! worker-state/*db-sync-config {:ws-url "wss://api.logseq.io/sync/%s" + :http-base "https://api.logseq.io"}) + (is (= "https://api.logseq.io" (#'db-sync/http-base-url))) + (finally + (reset! worker-state/*db-sync-config config-prev))))) + (defn- with-datascript-conns [db-conn ops-conn f] (let [db-prev @worker-state/*datascript-conns diff --git a/src/test/logseq/cli/command/sync_test.cljs b/src/test/logseq/cli/command/sync_test.cljs index e993a6b71c..f99409bf34 100644 --- a/src/test/logseq/cli/command/sync_test.cljs +++ b/src/test/logseq/cli/command/sync_test.cljs @@ -132,28 +132,121 @@ (deftest test-execute-sync-start-missing-ws-url-is-error (async done - (-> (p/with-redefs [cli-server/ensure-server! (fn [config _repo] - (p/resolved (assoc config :base-url "http://example"))) - transport/invoke (fn [_ method _direct-pass? _args] - (case method - :thread-api/db-sync-status - (p/resolved {:repo "logseq_db_demo" - :ws-state :inactive - :pending-local 0 - :pending-asset 0 - :pending-server 0}) - (p/resolved {:ok true})))] - (p/let [result (execute-with-runtime-auth {:type :sync-start - :repo "logseq_db_demo" - :wait-timeout-ms 20 - :wait-poll-interval-ms 0} - {:data-dir "/tmp"})] - (is (= :error (:status result))) - (is (= :sync-start-skipped (get-in result [:error :code]))) - (is (= :inactive (get-in result [:error :ws-state]))))) - (p/catch (fn [e] - (is false (str "unexpected error: " e)))) - (p/finally done)))) + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [result (sync-command/execute {:type :sync-start + :repo "logseq_db_demo"} + {:data-dir "/tmp" + :ws-url "" + :auth-token "runtime-token"})] + (is (= :error (:status result))) + (is (= :missing-sync-config (get-in result [:error :code]))) + (is (= :sync-start (get-in result [:error :action]))) + (is (= [:ws-url] (get-in result [:error :missing-keys]))) + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest test-execute-sync-upload-missing-http-base-is-error + (async done + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [result (sync-command/execute {:type :sync-upload + :repo "logseq_db_demo"} + {:data-dir "/tmp" + :http-base "" + :auth-token "runtime-token"})] + (is (= :error (:status result))) + (is (= :missing-sync-config (get-in result [:error :code]))) + (is (= :sync-upload (get-in result [:error :action]))) + (is (= [:http-base] (get-in result [:error :missing-keys]))) + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest test-execute-sync-download-missing-http-base-is-error + (async done + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved []))] + (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp" + :http-base "" + :auth-token "runtime-token"})] + (is (= :error (:status result))) + (is (= :missing-sync-config (get-in result [:error :code]))) + (is (= :sync-download (get-in result [:error :action]))) + (is (= [:http-base] (get-in result [:error :missing-keys]))) + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest test-execute-sync-download-missing-e2ee-password-for-e2ee-graph-is-error + (async done + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (case method + :thread-api/set-db-sync-config + (p/resolved nil) + + :thread-api/db-sync-list-remote-graphs + (p/resolved [{:graph-id "remote-graph-id" + :graph-name "demo" + :graph-e2ee? true}]) + + :thread-api/db-sync-download-graph-by-id + (p/resolved {:ok true}) + + (p/resolved nil)))] + (p/let [result (sync-command/execute {:type :sync-download + :repo "logseq_db_demo" + :graph "demo"} + {:base-url "http://example" + :data-dir "/tmp" + :http-base "https://api.logseq.io" + :auth-token "runtime-token"})] + (is (= :error (:status result))) + (is (= :missing-sync-config (get-in result [:error :code]))) + (is (= :sync-download (get-in result [:error :action]))) + (is (= [:e2ee-password] (get-in result [:error :missing-keys]))) + (is (= [] @ensure-calls)) + (is (not-any? (fn [[method _ _]] + (= :thread-api/db-sync-download-graph-by-id method)) + @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) (deftest test-execute-sync-start-runtime-error-after-open (async done @@ -333,24 +426,26 @@ :repo "logseq_db_demo" :graph "demo"} {:base-url "http://example" - :data-dir "/tmp"})] + :data-dir "/tmp" + :e2ee-password "pw"})] (is (= [[{:base-url "http://example" :create-empty-db? true :data-dir "/tmp" + :e2ee-password "pw" :auth-token "runtime-token"} "logseq_db_demo"]] @ensure-calls)) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil :auth-token "runtime-token" - :e2ee-password nil}]] + :e2ee-password "pw"}]] (nth @invoke-calls 0))) (is (= [:thread-api/db-sync-list-remote-graphs false []] (nth @invoke-calls 1))) (is (= [:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil :auth-token "runtime-token" - :e2ee-password nil}]] + :e2ee-password "pw"}]] (nth @invoke-calls 2))) (let [[method direct-pass? args] (nth @invoke-calls 3)] (is (= :thread-api/q method)) @@ -599,7 +694,8 @@ :repo "logseq_db_demo" :graph "demo"} {:base-url "http://example" - :data-dir "/tmp"})] + :data-dir "/tmp" + :e2ee-password "pw"})] (is (= :error (:status result))) (is (= :db-sync/snapshot-download-failed (get-in result [:error :code]))))) (p/catch (fn [e] @@ -628,7 +724,8 @@ (p/let [result (execute-with-runtime-auth {:type :sync-download :repo "logseq_db_demo" :graph "demo"} - {:data-dir "/tmp"})] + {:data-dir "/tmp" + :e2ee-password "pw"})] (is (= :error (:status result))) (is (= :graph-db-not-empty (get-in result [:error :code]))) (is (= "logseq_db_demo" (get-in result [:error :repo]))) @@ -722,6 +819,32 @@ (is false (str "unexpected error: " e)))) (p/finally done))))) +(deftest test-execute-sync-grant-access-missing-e2ee-password-is-error + (async done + (let [ensure-calls (atom []) + invoke-calls (atom [])] + (-> (p/with-redefs [cli-server/ensure-server! (fn [config repo] + (swap! ensure-calls conj [config repo]) + (p/resolved (assoc config :base-url "http://example"))) + transport/invoke (fn [_ method direct-pass? args] + (swap! invoke-calls conj [method direct-pass? args]) + (p/resolved {:ok true}))] + (p/let [result (sync-command/execute {:type :sync-grant-access + :repo "logseq_db_demo" + :graph-id "graph-uuid" + :email "user@example.com"} + {:data-dir "/tmp" + :auth-token "runtime-token"})] + (is (= :error (:status result))) + (is (= :missing-sync-config (get-in result [:error :code]))) + (is (= :sync-grant-access (get-in result [:error :action]))) + (is (= [:e2ee-password] (get-in result [:error :missing-keys]))) + (is (= [] @ensure-calls)) + (is (= [] @invoke-calls)))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done))))) + (deftest test-execute-sync-grant-access (async done (let [ensure-calls (atom []) @@ -736,15 +859,17 @@ :repo "logseq_db_demo" :graph-id "graph-uuid" :email "user@example.com"} - {:data-dir "/tmp"})] + {:data-dir "/tmp" + :e2ee-password "pw"})] (is (= [[{:data-dir "/tmp" + :e2ee-password "pw" :auth-token "runtime-token"} "logseq_db_demo"]] @ensure-calls)) (is (= [[:thread-api/set-db-sync-config false [{:ws-url nil :http-base nil :auth-token "runtime-token" - :e2ee-password nil}]] + :e2ee-password "pw"}]] [:thread-api/db-sync-grant-graph-access false ["logseq_db_demo" "graph-uuid" "user@example.com"]]] @invoke-calls)))) (p/catch (fn [e] diff --git a/src/test/logseq/cli/config_test.cljs b/src/test/logseq/cli/config_test.cljs index a4bc352d78..1455372697 100644 --- a/src/test/logseq/cli/config_test.cljs +++ b/src/test/logseq/cli/config_test.cljs @@ -92,6 +92,8 @@ expected-config-path (node-path/join (.homedir os) "logseq" "cli.edn")] (is (= expected-config-path (:config-path result))) (is (= "~/logseq/graphs" (:data-dir result))) + (is (= "wss://api.logseq.io/sync/%s" (:ws-url result))) + (is (= "https://api.logseq.io" (:http-base result))) (is (= 10000 (:timeout-ms result))) (is (= 300000 (:login-timeout-ms result))) (is (= 120000 (:logout-timeout-ms result))))) diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 08756a43ba..57fd690540 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -324,7 +324,7 @@ invoke-calls (atom []) status-calls (atom 0)] (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") - _ (fs/writeFileSync cfg-path "{:output-format :json}") + _ (fs/writeFileSync cfg-path "{:output-format :json :e2ee-password \"pw\"}") create-result (run-cli ["graph" "create" "--graph" start-repo] data-dir cfg-path) create-payload (parse-json-output-safe create-result "graph create") _ (is (= 0 (:exit-code create-result))) @@ -421,8 +421,8 @@ (is (= 0 (:exit-code upload-result))) (is (= "ok" (:status upload-payload))) (is (= "created-graph-id" (get-in upload-payload [:data :graph-id]))) - (is (= [[:thread-api/set-db-sync-config [{:ws-url nil - :http-base nil + (is (= [[:thread-api/set-db-sync-config [{:ws-url "wss://api.logseq.io/sync/%s" + :http-base "https://api.logseq.io" :auth-token "runtime-token" :e2ee-password nil}]] [:thread-api/db-sync-upload-graph ["logseq_db_sync-upload-graph"]]] From 43a6e9f8535ab026a18b194001e422473088fdbd Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 17 Mar 2026 00:14:08 -0400 Subject: [PATCH 163/375] fix: multiple fields not visible for list commands * uuid, classes, properties, extends and description fields not visible for human output even though data is available and users are asking to display it with --with-* or --fields * Also fix recent regression for property types no longer displaying by default * Also fix --with-extends, --with-properties and --with-classes depending on undocumented --extra option. Options should be declarative and not have hidden dependencies --- .../004-logseq-cli-verb-subcommands.md | 2 +- src/main/logseq/cli/command/list.cljs | 21 ++++++++---- src/main/logseq/cli/format.cljs | 33 +++++++++++++++++-- 3 files changed, 45 insertions(+), 11 deletions(-) diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md index acb001ed60..f4367ac223 100644 --- a/docs/agent-guide/004-logseq-cli-verb-subcommands.md +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -96,7 +96,7 @@ List property options: | --- | --- | --- | | --include-built-in | Include built-in properties | Built-in properties are currently included by default, clarify behavior. | | --with-classes | Include property classes | Uses :logseq.property/classes when expanded. | -| --with-type | Include property type | Uses :logseq.property/type when expanded. | +| --with-type | Include property type | Uses :logseq.property/type. | | --fields FIELD,FIELD | Select output fields | | List block is removed to avoid overlap with search. diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 5d8770c050..7be544d06c 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -20,10 +20,11 @@ :order {:desc "Sort order. Default: asc" :values ["asc" "desc"]}}) +;; These should be kept in sync with visible columns e.g. format/list-columns (def ^:private list-sort-fields {:list-page #{"title" "id" "ident" "created-at" "updated-at"} :list-tag #{"title" "id" "ident" "created-at" "updated-at"} - :list-property #{"title" "id" "ident" "created-at" "updated-at"}}) + :list-property #{"title" "id" "ident" "created-at" "updated-at" "type"}}) (def ^:private list-page-spec (merge list-common-spec @@ -60,6 +61,7 @@ :with-classes {:desc "Include property classes" :coerce :boolean} :with-type {:desc "Include property type" + :default true :coerce :boolean} :fields {:desc "Select output fields (comma separated)"}})) @@ -168,16 +170,18 @@ (some? limit) (->> (take limit) vec))) (defn- prepare-tag-item - [item {:keys [with-properties with-extends]}] + [item {:keys [with-properties with-extends fields]}] (cond-> item (not with-properties) (dissoc :logseq.property.class/properties) - (not with-extends) (dissoc :logseq.property.class/extends))) + (not with-extends) (dissoc :logseq.property.class/extends) + (not (string/includes? (str fields) "description")) (dissoc :logseq.property/description))) (defn- prepare-property-item - [item {:keys [with-classes with-type]}] + [item {:keys [with-classes with-type fields]}] (cond-> item (not with-classes) (dissoc :logseq.property/classes) - (not with-type) (dissoc :logseq.property/type))) + (not with-type) (dissoc :logseq.property/type) + (not (string/includes? (str fields) "description")) (dissoc :logseq.property/description))) (defn- apply-user-only [options] @@ -201,7 +205,9 @@ (defn execute-list-tag [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (apply-user-only (:options action)) + options (cond-> (apply-user-only (:options action)) + ((some-fn :with-extends :with-properties) (:options action)) + (assoc :expand true)) items (transport/invoke cfg :thread-api/api-list-tags false [(:repo action) options]) order (or (:order options) "asc") @@ -216,7 +222,8 @@ (defn execute-list-property [action config] (-> (p/let [cfg (cli-server/ensure-server! config (:repo action)) - options (apply-user-only (:options action)) + options (cond-> (apply-user-only (:options action)) + (:with-classes (:options action)) (assoc :expand true)) items (transport/invoke cfg :thread-api/api-list-properties false [(:repo action) options]) order (or (:order options) "asc") diff --git a/src/main/logseq/cli/format.cljs b/src/main/logseq/cli/format.cljs index 144a5087a8..e6a8e0c6ee 100644 --- a/src/main/logseq/cli/format.cljs +++ b/src/main/logseq/cli/format.cljs @@ -153,9 +153,10 @@ [items & ks] (some (fn [item] (some #(contains? item %) ks)) items)) -(def ^:private list-columns +(def ^:private list-page-columns [["ID" (fn [item _] (or (:db/id item) (:id item))) [:db/id :id]] ["TITLE" (fn [item _] (or (:title item) (:block/title item) (:name item))) [:title :block/title :name]] + ["UUID" (fn [item _] (or (:block/uuid item) "-")) [:block/uuid]] ["IDENT" (fn [item _] (:db/ident item)) [:db/ident]] ["UPDATED-AT" (fn [item now-ms] (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms)) [:updated-at :block/updated-at]] ["CREATED-AT" (fn [item now-ms] (human-ago (or (:created-at item) (:block/created-at item)) now-ms)) [:created-at :block/created-at]]]) @@ -174,11 +175,30 @@ (defn- format-list-page [items now-ms] - (format-list-dynamic items now-ms list-columns)) + (format-list-dynamic items now-ms list-page-columns)) + +(defn- format-extends + [classes] + (if (seq classes) (string/join ", " classes) "-")) + +(defn- format-properties + [properties] + (if (seq properties) (string/join ", " properties) "-")) + +(def ^:private list-tag-columns + [["ID" (fn [item _] (or (:db/id item) (:id item))) [:db/id :id]] + ["TITLE" (fn [item _] (or (:title item) (:block/title item) (:name item))) [:title :block/title :name]] + ["UUID" (fn [item _] (or (:block/uuid item) "-")) [:block/uuid]] + ["IDENT" (fn [item _] (:db/ident item)) [:db/ident]] + ["EXTENDS" (fn [item _] (format-extends (:logseq.property.class/extends item))) [:logseq.property.class/extends]] + ["PROPERTIES" (fn [item _] (format-properties (:logseq.property.class/properties item))) [:logseq.property.class/properties]] + ["DESCRIPTION" (fn [item _] (:logseq.property/description item)) [:logseq.property/description]] + ["UPDATED-AT" (fn [item now-ms] (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms)) [:updated-at :block/updated-at]] + ["CREATED-AT" (fn [item now-ms] (human-ago (or (:created-at item) (:block/created-at item)) now-ms)) [:created-at :block/created-at]]]) (defn- format-list-tag [items now-ms] - (format-list-dynamic items now-ms list-columns)) + (format-list-dynamic items now-ms list-tag-columns)) (defn- normalize-property-type [value] @@ -187,11 +207,18 @@ (nil? value) "-" :else (str value))) +(defn- format-classes + [classes] + (if (seq classes) (string/join ", " classes) "-")) + (def ^:private list-property-columns [["ID" (fn [item _] (or (:db/id item) (:id item))) [:db/id :id]] ["TITLE" (fn [item _] (or (:title item) (:block/title item) (:name item))) [:title :block/title :name]] ["TYPE" (fn [item _] (normalize-property-type (:logseq.property/type item))) [:logseq.property/type]] + ["CLASSES" (fn [item _] (format-classes (:logseq.property/classes item))) [:logseq.property/classes]] + ["UUID" (fn [item _] (or (:block/uuid item) "-")) [:block/uuid]] ["IDENT" (fn [item _] (:db/ident item)) [:db/ident]] + ["DESCRIPTION" (fn [item _] (:logseq.property/description item)) [:logseq.property/description]] ["UPDATED-AT" (fn [item now-ms] (human-ago (or (:updated-at item) (:block/updated-at item)) now-ms)) [:updated-at :block/updated-at]] ["CREATED-AT" (fn [item now-ms] (human-ago (or (:created-at item) (:block/created-at item)) now-ms)) [:created-at :block/created-at]]]) From 5db3a3086c111f9884bc03b1bf4ad3ea7b7c65c7 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 17 Mar 2026 16:45:36 +0800 Subject: [PATCH 164/375] 061-graph-dir-space-preserve-canonical.md --- deps/common/src/logseq/common/graph_dir.cljs | 1 + .../061-graph-dir-space-preserve-canonical.md | 202 ++++++++++++++++++ src/test/frontend/worker/graph_dir_test.cljs | 4 +- .../worker/worker_common_util_test.cljs | 28 +-- src/test/logseq/cli/common/graph_test.cljs | 2 + src/test/logseq/cli/common_test.cljs | 18 ++ src/test/logseq/cli/format_test.cljs | 13 ++ src/test/logseq/cli/integration_test.cljs | 10 +- src/test/logseq/cli/server_test.cljs | 19 +- src/test/logseq/db/common_sqlite_test.cljs | 6 + 10 files changed, 279 insertions(+), 24 deletions(-) create mode 100644 docs/agent-guide/061-graph-dir-space-preserve-canonical.md diff --git a/deps/common/src/logseq/common/graph_dir.cljs b/deps/common/src/logseq/common/graph_dir.cljs index d067d5469a..d57b68f37f 100644 --- a/deps/common/src/logseq/common/graph_dir.cljs +++ b/deps/common/src/logseq/common/graph_dir.cljs @@ -7,6 +7,7 @@ [graph-name] (let [encoded (js/encodeURIComponent (or graph-name ""))] (-> encoded + (string/replace "%20" " ") (string/replace "~" "%7E") (string/replace "%" "~")))) diff --git a/docs/agent-guide/061-graph-dir-space-preserve-canonical.md b/docs/agent-guide/061-graph-dir-space-preserve-canonical.md new file mode 100644 index 0000000000..ed774cd0bc --- /dev/null +++ b/docs/agent-guide/061-graph-dir-space-preserve-canonical.md @@ -0,0 +1,202 @@ +# 061 — Preserve spaces in canonical graph directory names + +## Summary + +Change canonical graph directory encoding so spaces are preserved as literal spaces. + +- Before: graph-name `GRAPH one` -> graph dir `GRAPH~20one` +- After: graph-name `GRAPH one` -> graph dir `GRAPH one` + +All other special-character encode/decode behavior remains unchanged. + +This plan is based on the current `logseq-cli` and `db-worker-node` implementation, both of which already rely on shared graph-dir encoding utilities. + +## Product decision (locked) + +1. Canonical graph dir must preserve spaces (no `~20` for spaces). +2. Existing encoding/decoding rules for non-space special characters remain unchanged. +3. Existing directories using old space encodings (for example `~20` or `%20`) are treated as legacy-compatible inputs, not the canonical write format. +4. New writes/path derivations must use the new canonical format with literal spaces. + +## Goals + +- Align `logseq-cli` and `db-worker-node` on a canonical graph-dir format that keeps spaces as spaces. +- Keep current behavior for all non-space special characters. +- Preserve compatibility for existing legacy directory names during discovery/listing. +- Keep user-facing graph-name semantics unchanged. + +## Non-goals + +- Redesign graph naming. +- Change `logseq_db_` prefix handling rules. +- Introduce automatic on-disk migration in this iteration. +- Change unrelated path or lock semantics. + +## Current behavior (relevant paths) + +### Shared encoding contract + +Authoritative helpers: + +- `deps/common/src/logseq/common/graph_dir.cljs` + - `encode-graph-dir-name` + - `decode-graph-dir-name` + - `decode-legacy-graph-dir-name` + - `repo->graph-dir-key` + - `graph-dir-key->encoded-dir-name` + +Current behavior for space: + +1. `encodeURIComponent("GRAPH one")` -> `GRAPH%20one` +2. `%` is rewritten to `~` +3. output becomes `GRAPH~20one` + +### CLI usage + +- `src/main/logseq/cli/server.cljs` + - canonical/legacy classification for graph dirs (`classify-graph-dir`, `canonical-dir-name?`) +- `src/main/logseq/cli/command/graph.cljs` + - `graph list` payload construction +- `src/main/logseq/cli/format.cljs` + - legacy warning/rename suggestion rendering + +### db-worker-node usage + +- `src/main/frontend/worker/db_worker_node_lock.cljs` + - `repo-dir`, `lock-path`, `repo->graph-dir-key` +- `src/main/frontend/worker/platform/node.cljs` + - list/db path installation flows that resolve graph dir via shared helpers +- `src/main/frontend/worker/db_core.cljs` + - storage pool naming path uses graph-dir key/encoded dir conventions + +## Proposed behavior + +### Canonical encode/decode contract + +For graph-dir encoding: + +- Preserve literal spaces in output. +- Keep current reversible encode/decode behavior for all other special characters. +- Keep `~`/`%` safety rules unchanged. + +Examples: + +- `GRAPH one` -> `GRAPH one` (changed) +- `a/b` -> `a~2Fb` (unchanged) +- `x:y` -> `x~3Ay` (unchanged) +- `100% real` -> `100~25 real` (unchanged except space stays literal) + +Decoding expectations: + +- `GRAPH one` -> `GRAPH one` +- `GRAPH~20one` -> `GRAPH one` (legacy-compatible decode still works) + +### Canonical vs legacy classification in CLI + +`graph list` canonical check should reflect the new canonical encoding: + +- Directory `GRAPH one` is canonical. +- Directory `GRAPH~20one` is non-canonical (legacy) and should be surfaced as legacy with rename guidance targeting `GRAPH one`. +- Existing `%20` legacy directories remain legacy. + +## Design details + +### 1) Shared encoder update + +Update `encode-graph-dir-name` in `deps/common/src/logseq/common/graph_dir.cljs` so space is not rewritten into `~20`. + +Implementation constraint: + +- Do not alter non-space transformation behavior. +- Keep encode/decode reversibility for previously supported special characters. + +### 2) Keep decode compatibility + +`decode-graph-dir-name` remains compatibility-friendly so both old and new directory spellings decode to the same graph-name. + +No behavioral contraction should be introduced in decoding. + +### 3) CLI classification and guidance alignment + +Because canonical encoding changes, `src/main/logseq/cli/server.cljs` classification will naturally reclassify old `~20` dirs as legacy. + +Ensure `target-graph-dir` generation and formatter output use the new canonical output containing spaces. + +### 4) db-worker-node path outputs + +No separate encoding logic should be added. + +All db-worker-node path generation must continue to route through shared helpers so new canonical behavior applies consistently to: + +- graph repo dir +- lock path +- db path +- log path + +## Test plan + +### Update existing tests + +1. `src/test/frontend/worker/worker_common_util_test.cljs` + - Update roundtrip expectations for space-containing names to canonical literal-space output. +2. `src/test/logseq/cli/common/graph_test.cljs` + - Update graph-dir decode/list expectations from `space~20name` canonical assumptions to literal-space canonical behavior. +3. `src/test/logseq/cli/server_test.cljs` + - Update canonical vs legacy expectations: + - space directory canonical + - `~20`/`%20` variants legacy where applicable +4. `src/test/logseq/cli/integration_test.cljs` + - Update integration assertions for `graph list` legacy markers and rename targets. +5. `src/test/logseq/db/common_sqlite_test.cljs` + - Update encoded-dir path assertions for graphs with spaces. +6. `src/test/logseq/cli/common_test.cljs` + - Update unlink/move expectations when graph names contain spaces. +7. `src/test/logseq/cli/format_test.cljs` + - Ensure rename suggestion target renders literal-space canonical dir. + +### Add targeted coverage (if missing) + +- Roundtrip examples that mix spaces with other special characters (for example `A B/C:D%~E`) to prove only space behavior changed. +- Explicit assertion that non-space character transformations are unchanged. + +## Rollout and compatibility + +- No immediate forced migration. +- Legacy directories remain discoverable/readable via existing decode + CLI legacy classification. +- New writes and canonical suggestions converge toward literal-space directories. + +## Risks + +1. Hidden assumptions in tests that treat `~20` as canonical for spaces. +2. Any code path bypassing shared graph-dir helpers could diverge (must be checked during implementation). +3. Rename suggestion shell formatting with spaces must remain safely quoted. + +## Acceptance criteria + +1. For graph-name `GRAPH one`, canonical graph dir is exactly `GRAPH one`. +2. Non-space special-character encoding behavior is unchanged from current behavior. +3. CLI graph listing marks old space-encoded dirs (`~20`/`%20`) as legacy where relevant. +4. db-worker-node path derivation uses the new canonical space-preserving output via shared helpers. +5. Updated tests pass and demonstrate: + - new canonical space behavior, + - unchanged non-space behavior, + - legacy compatibility visibility. + +## Affected files (planned) + +Would modify: + +- `deps/common/src/logseq/common/graph_dir.cljs` +- `src/main/logseq/cli/server.cljs` (if classification adjustments are required) +- `src/main/logseq/cli/format.cljs` (if rename guidance formatting expectations need updates) +- `src/test/frontend/worker/worker_common_util_test.cljs` +- `src/test/logseq/cli/common/graph_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/logseq/db/common_sqlite_test.cljs` +- `src/test/logseq/cli/common_test.cljs` +- `src/test/logseq/cli/format_test.cljs` + +Would create: + +- `docs/agent-guide/061-graph-dir-space-preserve-canonical.md` diff --git a/src/test/frontend/worker/graph_dir_test.cljs b/src/test/frontend/worker/graph_dir_test.cljs index 637d7b3dbb..34d8ae8d1f 100644 --- a/src/test/frontend/worker/graph_dir_test.cljs +++ b/src/test/frontend/worker/graph_dir_test.cljs @@ -13,7 +13,9 @@ (deftest repo->encoded-graph-dir-name-encodes-special-characters (testing "db-prefixed repos resolve to the encoded on-disk graph dir name" (is (= "foo~2Fbar" - (graph-dir/repo->encoded-graph-dir-name "logseq_db_foo/bar"))))) + (graph-dir/repo->encoded-graph-dir-name "logseq_db_foo/bar"))) + (is (= "space name" + (graph-dir/repo->encoded-graph-dir-name "logseq_db_space name"))))) (deftest decode-graph-dir-name-decodes-only-canonical-encoded-names (testing "encoded graph dirs decode back to the logical graph dir key" diff --git a/src/test/frontend/worker/worker_common_util_test.cljs b/src/test/frontend/worker/worker_common_util_test.cljs index 2b7ca2ac45..39027771a8 100644 --- a/src/test/frontend/worker/worker_common_util_test.cljs +++ b/src/test/frontend/worker/worker_common_util_test.cljs @@ -4,17 +4,19 @@ [frontend.worker-common.util :as worker-util])) (deftest encode-decode-graph-dir-name-roundtrip - (let [names ["Demo" - "foo/bar" - "a:b" - "space name" - "100% legit" - "til~de" - "mix/ed:chars%~"] - encoded (map worker-util/encode-graph-dir-name names)] - (doseq [[name enc] (map vector names encoded)] - (is (= name (worker-util/decode-graph-dir-name enc)))) - (doseq [enc encoded] - (is (not (string/includes? enc "/"))) - (is (not (string/includes? enc "\\"))))) + (let [cases [["Demo" "Demo"] + ["foo/bar" "foo~2Fbar"] + ["a:b" "a~3Ab"] + ["space name" "space name"] + ["100% legit" "100~25 legit"] + ["til~de" "til~7Ede"] + ["A B/C:D%~E" "A B~2FC~3AD~25~7EE"]]] + (doseq [[name expected-encoded] cases] + (let [encoded (worker-util/encode-graph-dir-name name)] + (is (= expected-encoded encoded)) + (is (= name (worker-util/decode-graph-dir-name encoded))) + (is (not (string/includes? encoded "/"))) + (is (not (string/includes? encoded "\\"))))) + (is (= "space name" (worker-util/decode-graph-dir-name "space~20name"))) + (is (= "space name" (worker-util/decode-graph-dir-name "space%20name")))) (is (nil? (worker-util/decode-graph-dir-name nil)))) diff --git a/src/test/logseq/cli/common/graph_test.cljs b/src/test/logseq/cli/common/graph_test.cljs index 248b1f2a6a..6b61cabd92 100644 --- a/src/test/logseq/cli/common/graph_test.cljs +++ b/src/test/logseq/cli/common/graph_test.cljs @@ -23,7 +23,9 @@ (let [graphs-dir (node-helper/create-tmp-dir "cli-common-graph-encoded") _ (doseq [dir ["foo~2Fbar" "a~3Ab" + "space name" "space~20name" + "space%20name" "Unlinked graphs"]] (fs/mkdirSync (node-path/join graphs-dir dir) #js {:recursive true}))] (with-redefs [cli-common-graph/get-db-graphs-dir (fn [] graphs-dir)] diff --git a/src/test/logseq/cli/common_test.cljs b/src/test/logseq/cli/common_test.cljs index 19adac802e..8b33ee6272 100644 --- a/src/test/logseq/cli/common_test.cljs +++ b/src/test/logseq/cli/common_test.cljs @@ -24,3 +24,21 @@ "Graph directory should be moved to Unlinked graphs") (is (fs/existsSync (node-path/join unlinked-path "db.sqlite")) "Graph contents should be preserved after move")))) + +(deftest unlink-graph-moves-space-preserving-canonical-dir + (let [graphs-dir (node-helper/create-tmp-dir "unlink-graph-space") + graph-name "space name" + repo (str common-config/db-version-prefix graph-name) + encoded-graph-dir "space name" + graph-path (node-path/join graphs-dir encoded-graph-dir) + unlinked-path (node-path/join graphs-dir common-config/unlinked-graphs-dir encoded-graph-dir)] + (fs/mkdirSync graph-path #js {:recursive true}) + (fs/writeFileSync (node-path/join graph-path "db.sqlite") "test-data") + (with-redefs [common-graph/get-default-graphs-dir (fn [] graphs-dir)] + (cli-common/unlink-graph! repo) + (is (not (fs/existsSync graph-path)) + "Original space-preserving graph directory should no longer exist") + (is (fs/existsSync unlinked-path) + "Space-preserving graph directory should be moved to Unlinked graphs") + (is (fs/existsSync (node-path/join unlinked-path "db.sqlite")) + "Graph contents should be preserved after move")))) diff --git a/src/test/logseq/cli/format_test.cljs b/src/test/logseq/cli/format_test.cljs index 02c770af79..a48fac5693 100644 --- a/src/test/logseq/cli/format_test.cljs +++ b/src/test/logseq/cli/format_test.cljs @@ -108,6 +108,19 @@ (is (string/includes? result "mv '/tmp/graphs/legacy++name' '/tmp/graphs/legacy~2Fname'")) (is (string/includes? result "Warning: cannot derive graph name for legacy dir 'mystery'; rename command is not available.")))) + (testing "graph list rename suggestion targets canonical dir with literal spaces" + (let [result (format/format-result {:status :ok + :command :graph-list + :data {:graphs ["space name"] + :graph-items [{:kind :legacy + :legacy-dir "space~20name" + :legacy-graph-name "space name" + :target-graph-dir "space name" + :conflict? false}]}} + {:output-format nil + :data-dir "/tmp/graphs"})] + (is (string/includes? result "mv '/tmp/graphs/space~20name' '/tmp/graphs/space name'")))) + (testing "graph list conflict warning does not print mv command" (let [result (format/format-result {:status :ok :command :graph-list diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 57fd690540..956d48e9eb 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -550,6 +550,7 @@ (deftest ^:long test-cli-graph-list-percent-legacy-is-marked (async done (let [data-dir (node-helper/create-tmp-dir "db-worker-percent-legacy")] + (fs/mkdirSync (node-path/join data-dir "yy y") #js {:recursive true}) (fs/mkdirSync (node-path/join data-dir "yy~20y") #js {:recursive true}) (fs/mkdirSync (node-path/join data-dir "yy%20y") #js {:recursive true}) (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") @@ -562,10 +563,13 @@ (is (= 0 (:exit-code json-result))) (is (= "ok" (:status json-payload))) (is (= #{"canonical" "legacy"} kinds)) - (is (= 1 (count (filter #(= "yy~20y" (:graph-dir %)) graph-items)))) - (is (= 1 (count (filter #(= "yy%20y" (:legacy-dir %)) graph-items)))) + (is (= 1 (count (filter #(= "yy y" (:graph-dir %)) graph-items)))) + (is (= #{"yy~20y" "yy%20y"} + (set (keep :legacy-dir graph-items)))) + (is (= #{"yy y"} + (set (keep :target-graph-dir graph-items)))) (is (string/includes? human-output "yy y [legacy]")) - (is (string/includes? human-output "Warning: 1 legacy graph directories detected.")) + (is (string/includes? human-output "Warning: 2 legacy graph directories detected.")) (done)) (p/catch (fn [e] (is false (str "unexpected error: " e)) diff --git a/src/test/logseq/cli/server_test.cljs b/src/test/logseq/cli/server_test.cljs index badea67c98..dc0a28476b 100644 --- a/src/test/logseq/cli/server_test.cljs +++ b/src/test/logseq/cli/server_test.cljs @@ -323,16 +323,21 @@ (deftest list-graph-items-treats-percent-encoded-dir-as-legacy-when-non-canonical (let [data-dir (node-helper/create-tmp-dir "cli-list-graphs-percent-legacy") - _ (doseq [dir ["yy~20y" + _ (doseq [dir ["yy y" + "yy~20y" "yy%20y"]] (fs/mkdirSync (node-path/join data-dir dir) #js {:recursive true})) items (cli-server/list-graph-items {:data-dir data-dir}) by-kind (group-by :kind items) canonical-item (first (get by-kind :canonical)) - legacy-item (first (get by-kind :legacy))] - (is (= "yy~20y" (:graph-dir canonical-item))) + legacy-items (get by-kind :legacy) + legacy-by-dir (into {} (map (juxt :legacy-dir identity) legacy-items))] + (is (= "yy y" (:graph-dir canonical-item))) (is (= "yy y" (:graph-name canonical-item))) - (is (= "yy%20y" (:legacy-dir legacy-item))) - (is (= "yy y" (:legacy-graph-name legacy-item))) - (is (= "yy~20y" (:target-graph-dir legacy-item))) - (is (= true (:conflict? legacy-item))))) + (is (= #{"yy~20y" "yy%20y"} + (set (map :legacy-dir legacy-items)))) + (doseq [legacy-dir ["yy~20y" "yy%20y"]] + (let [legacy-item (get legacy-by-dir legacy-dir)] + (is (= "yy y" (:legacy-graph-name legacy-item))) + (is (= "yy y" (:target-graph-dir legacy-item))) + (is (= true (:conflict? legacy-item))))))) diff --git a/src/test/logseq/db/common_sqlite_test.cljs b/src/test/logseq/db/common_sqlite_test.cljs index db1a067316..8767eedca4 100644 --- a/src/test/logseq/db/common_sqlite_test.cljs +++ b/src/test/logseq/db/common_sqlite_test.cljs @@ -9,3 +9,9 @@ (deftest get-db-backups-path-uses-encoded-graph-dir (is (= "/tmp/graphs/foo~2Fbar/backups" (common-sqlite/get-db-backups-path "/tmp/graphs" "logseq_db_foo/bar")))) + +(deftest get-db-paths-preserve-space-in-canonical-graph-dir + (is (= ["space name" "/tmp/graphs/space name/db.sqlite"] + (common-sqlite/get-db-full-path "/tmp/graphs" "logseq_db_space name"))) + (is (= "/tmp/graphs/space name/backups" + (common-sqlite/get-db-backups-path "/tmp/graphs" "logseq_db_space name")))) From 209b499494386ddea9b3dc8b674887ce0299e071 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Tue, 17 Mar 2026 17:30:17 +0800 Subject: [PATCH 165/375] 062-cli-list-default-sort-updated-at.md --- .../062-cli-list-default-sort-updated-at.md | 164 ++++++++++++++++++ docs/cli/logseq-cli.md | 6 +- src/main/logseq/cli/command/list.cljs | 25 ++- src/test/logseq/cli/commands_test.cljs | 47 +++++ src/test/logseq/cli/integration_test.cljs | 113 ++++++++++++ 5 files changed, 345 insertions(+), 10 deletions(-) create mode 100644 docs/agent-guide/062-cli-list-default-sort-updated-at.md diff --git a/docs/agent-guide/062-cli-list-default-sort-updated-at.md b/docs/agent-guide/062-cli-list-default-sort-updated-at.md new file mode 100644 index 0000000000..a567d2056b --- /dev/null +++ b/docs/agent-guide/062-cli-list-default-sort-updated-at.md @@ -0,0 +1,164 @@ +# Logseq CLI List Default Updated-at Sort Implementation Plan + +Goal: Make `list page`, `list tag`, and `list property` behave as if `--sort updated-at` is provided when users do not pass `--sort`. + +Architecture: Keep the change in the CLI command layer so the existing db-worker-node API surface remains stable. +Architecture: Reuse current CLI-side sorting flow in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` where sorting already runs before offset and limit. +Architecture: Keep db-worker-node list providers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` unchanged for this scope, and document that decision explicitly. + +Tech Stack: ClojureScript, babashka.cli option specs, Promesa, Datascript-backed db-worker-node thread-api, existing CLI integration test harness. + +Related: Builds on `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/004-logseq-cli-verb-subcommands.md` and `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/043-logseq-cli-tag-property-management.md`. + +## Problem statement + +Current list commands support `--sort` and `--order`, but default behavior is unsorted because no sort field is applied unless users pass `--sort`. + +In current implementation, `list` execution fetches items from db-worker-node and runs `apply-sort` only when `:sort` is present in options. + +This means default output order depends on entity scan order from db-worker-node list functions, which is not aligned with the desired product behavior. + +The requested behavior is a consistent default sort key of `updated-at` for page, tag, and property list subcommands. + +## Current behavior snapshot + +| Layer | File | Current behavior | +| --- | --- | --- | +| CLI command parsing and execution | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` | `:sort` is optional and defaults to nil, so `apply-sort` is skipped when `--sort` is absent. | +| db-worker-node list thread-api bridge | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` | `:thread-api/api-list-pages`, `:thread-api/api-list-tags`, and `:thread-api/api-list-properties` return unsorted collections from shared list helpers. | +| Shared list helpers | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` | List helpers filter and shape items but do not apply sort by `updated-at`. | +| User docs | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` | List command syntax documents `--sort` as optional but does not state default sort behavior. | + +## Scope and non-goals + +This plan changes default sort behavior for CLI commands `list page`, `list tag`, and `list property`. + +This plan does not add new db-worker-node thread-api methods. + +This plan does not push pagination or sorting into db-worker-node. + +This plan does not change field names or output schemas. + +## Proposed behavior + +When users run `logseq list page`, `logseq list tag`, or `logseq list property` without `--sort`, CLI should sort by `updated-at` using the existing list sorting pipeline. + +When users pass `--sort`, the explicit value must override the default. + +`--order` should continue to default to `asc` unless explicitly set to `desc`. + +When multiple entities have the same primary sort value, CLI should apply `:db/id` as the deterministic secondary sort key. + +`offset` and `limit` must still be applied after sorting. + +The behavior should be equivalent to explicitly passing `--sort updated-at` today, with deterministic tie-breaking by `:db/id`. + +## Integration overview + +```text +logseq list page + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs + - determine effective sort field (default updated-at) + - apply CLI-side sort/order + - apply offset/limit/fields + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs +``` + +## Testing Plan + +I will follow `@test-driven-development` and add tests before implementation changes. + +I will add unit tests for CLI list execution paths that verify default sorting is applied when `:sort` is missing. + +I will add unit tests that verify explicit `--sort` still overrides the new default. + +I will add integration tests that verify default output order matches explicit `--sort updated-at` output for page, tag, and property list commands. + +I will update docs assertions or command help expectations if any existing tests encode old behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Detailed implementation plan + +1. Add a failing unit test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for list-page execution where input items are intentionally out of updated-at order and no `:sort` is provided. +2. Assert in that test that returned item ids are ordered exactly as if `:sort` were `"updated-at"` with default `:order`. +3. Add a failing unit test for list-tag execution with no `:sort` and confirm default updated-at ordering is applied after tag item preparation. +4. Add a failing unit test for list-property execution with no `:sort` and confirm default updated-at ordering is applied after property item preparation. +5. Add a failing unit test that passes explicit `:sort "title"` and confirms explicit sort overrides default updated-at behavior. +6. Add a failing unit test that passes explicit `:order "desc"` without `:sort` and confirms order is applied to default updated-at sort key. +7. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that compares `list page` results against `list page --sort updated-at` for the same graph and asserts identical id sequences. +8. Add a failing integration test that compares `list tag` against `list tag --sort updated-at` and asserts identical id sequences. +9. Add a failing integration test that compares `list property` against `list property --sort updated-at` and asserts identical id sequences. +10. Run focused tests to verify they fail for the new default-sort behavior and not for unrelated setup issues. +11. Implement an `effective-sort` decision in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` so list commands use `"updated-at"` when `:sort` is absent. +12. Reuse the effective sort key for all three executors: `execute-list-page`, `execute-list-tag`, and `execute-list-property`. +13. Keep existing explicit sort validation and allowed sort fields unchanged. +14. Add deterministic tie-breaking by `:db/id` in CLI list sorting when primary sort values are equal. +15. Keep existing order default (`asc`) unchanged. +16. Update option descriptions in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` to clarify default sort behavior for users. +17. Update docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to state that list subcommands default to sorting by `updated-at`. +18. Run focused unit and integration tests again and confirm green. +19. Run `bb dev:test -v logseq.cli.commands-test` to confirm command-level regressions are not introduced. +20. Run `bb dev:test -v logseq.cli.integration-test` for list command scenarios impacted by ordering. +21. Run `bb dev:lint-and-test` for final confidence. + +## Edge cases to cover + +Entities with missing `:block/updated-at` should still be sortable without runtime errors. + +Multiple entities with equal `updated-at` values should be secondarily sorted by `:db/id` for deterministic output across repeated runs. + +`--fields` filtering should not remove `updated-at` before sorting is executed. + +`--offset` and `--limit` should continue to apply after sorting, not before sorting. + +`--sort` with any allowed non-time field should keep existing behavior and take precedence over the new default. + +`--order desc` without explicit `--sort` should now reverse default updated-at order. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-list-subcommand-parse` | Existing parse behavior remains valid and list options remain accepted. | +| `bb dev:test -v logseq.cli.commands-test` | New default-sort unit tests pass and no command-level regressions are introduced. | +| `bb dev:test -v logseq.cli.integration-test` | Integration checks for list default ordering pass against a real db-worker-node flow. | +| `bb dev:lint-and-test` | Full lint and unit suite pass with zero errors. | + +## Rollout and compatibility + +This is a behavior change in default ordering for three CLI list commands. + +Scripts that depended on previous implicit scan order may observe changed item order. + +Scripts that already pass explicit `--sort` remain unaffected. + +No db-worker-node API contract change is introduced in this scope. + +## Testing Details + +Tests verify user-visible command behavior by comparing result ordering between default list calls and explicit `--sort updated-at` calls. + +Tests validate override behavior so explicit sort fields still control final ordering. + +Tests validate order plus pagination interaction to ensure behavior consistency. + +## Implementation Details + +- Update default sort selection in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` for page, tag, and property list executors. +- Keep existing `apply-sort`, `apply-offset-limit`, and `apply-fields` sequencing unchanged. +- Extend CLI sort implementation to apply `:db/id` as secondary key when primary sort values are equal. +- Reuse existing `list-*-field-map` entries for `updated-at` so no new field mapping is introduced. +- Keep db-worker-node list handlers in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` unchanged in this scope. +- Keep shared list helper behavior in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` unchanged in this scope. +- Add command-level unit coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add end-to-end list ordering coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +- Update user-facing list docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md`. + +## Question + +No open questions. + +--- diff --git a/docs/cli/logseq-cli.md b/docs/cli/logseq-cli.md index 8c19ca5ac6..618de01cfd 100644 --- a/docs/cli/logseq-cli.md +++ b/docs/cli/logseq-cli.md @@ -176,9 +176,9 @@ Sync config persistence: - Cloud auth is persisted separately in `~/logseq/auth.json`. Inspect and edit commands: -- `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages -- `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags -- `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties (`TYPE` is included by default even without `--expand`) +- `list page [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list pages (defaults to `--sort updated-at`) +- `list tag [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list tags (defaults to `--sort updated-at`) +- `list property [--expand] [--limit ] [--offset ] [--sort ] [--order asc|desc]` - list properties (defaults to `--sort updated-at`; `TYPE` is included by default even without `--expand`) - `upsert block --content [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - create blocks; defaults to today’s journal page if no target is given - `upsert block --blocks [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks via EDN vector - `upsert block --blocks-file [--target-page |--target-id |--target-uuid ] [--pos first-child|last-child|sibling]` - insert blocks from an EDN file diff --git a/src/main/logseq/cli/command/list.cljs b/src/main/logseq/cli/command/list.cljs index 7be544d06c..ec7576fc01 100644 --- a/src/main/logseq/cli/command/list.cljs +++ b/src/main/logseq/cli/command/list.cljs @@ -26,9 +26,15 @@ :list-tag #{"title" "id" "ident" "created-at" "updated-at"} :list-property #{"title" "id" "ident" "created-at" "updated-at" "type"}}) +(def ^:private default-sort-field "updated-at") + +(defn- effective-sort-field + [options] + (or (:sort options) default-sort-field)) + (def ^:private list-page-spec (merge list-common-spec - {:sort {:desc "Sort field" + {:sort {:desc "Sort field. Default: updated-at" :values (:list-page list-sort-fields)} :include-journal {:desc "Include journal pages" :coerce :boolean} @@ -42,7 +48,7 @@ (def ^:private list-tag-spec (merge list-common-spec - {:sort {:desc "Sort field" + {:sort {:desc "Sort field. Default: updated-at" :values (:list-tag list-sort-fields)} :include-built-in {:desc "Include built-in tags" :coerce :boolean} @@ -54,7 +60,7 @@ (def ^:private list-property-spec (merge list-common-spec - {:sort {:desc "Sort field" + {:sort {:desc "Sort field. Default: updated-at" :values (:list-property list-sort-fields)} :include-built-in {:desc "Include built-in properties" :coerce :boolean} @@ -157,7 +163,9 @@ (if (seq sort-field) (let [sort-key (get field-map sort-field) sorted (if sort-key - (sort-by #(get % sort-key) items) + (sort-by (fn [item] + [(get item sort-key) (:db/id item)]) + items) items) sorted (if (= "desc" order) (reverse sorted) sorted)] (vec sorted)) @@ -194,9 +202,10 @@ options (apply-user-only (:options action)) items (transport/invoke cfg :thread-api/api-list-pages false [(:repo action) options]) + sort-field (effective-sort-field options) order (or (:order options) "asc") fields (parse-field-list (:fields options)) - sorted (apply-sort items (:sort options) order list-page-field-map) + sorted (apply-sort items sort-field order list-page-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) final (apply-fields limited fields list-page-field-map)] {:status :ok @@ -210,10 +219,11 @@ (assoc :expand true)) items (transport/invoke cfg :thread-api/api-list-tags false [(:repo action) options]) + sort-field (effective-sort-field options) order (or (:order options) "asc") fields (parse-field-list (:fields options)) prepared (mapv #(prepare-tag-item % options) items) - sorted (apply-sort prepared (:sort options) order list-tag-field-map) + sorted (apply-sort prepared sort-field order list-tag-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) final (apply-fields limited fields list-tag-field-map)] {:status :ok @@ -226,10 +236,11 @@ (:with-classes (:options action)) (assoc :expand true)) items (transport/invoke cfg :thread-api/api-list-properties false [(:repo action) options]) + sort-field (effective-sort-field options) order (or (:order options) "asc") fields (parse-field-list (:fields options)) prepared (mapv #(prepare-property-item % options) items) - sorted (apply-sort prepared (:sort options) order list-property-field-map) + sorted (apply-sort prepared sort-field order list-property-field-map) limited (apply-offset-limit sorted (:offset options) (:limit options)) final (apply-fields limited fields list-property-field-map)] {:status :ok diff --git a/src/test/logseq/cli/commands_test.cljs b/src/test/logseq/cli/commands_test.cljs index fa32fa5295..af6222f750 100644 --- a/src/test/logseq/cli/commands_test.cljs +++ b/src/test/logseq/cli/commands_test.cljs @@ -2,6 +2,7 @@ (:require [cljs.test :refer [async deftest is testing]] [clojure.string :as string] [logseq.cli.command.add :as add-command] + [logseq.cli.command.list :as list-command] [logseq.cli.command.show :as show-command] [logseq.cli.command.sync :as sync-command] [logseq.cli.commands :as commands] @@ -48,6 +49,10 @@ (sequential? value) (some contains-block-uuid? value) :else false)) +(defn- item-ids + [result] + (mapv :db/id (get-in result [:data :items]))) + (deftest test-help-output (testing "top-level help lists command groups" (let [result (binding [style/*color-enabled?* true] @@ -958,6 +963,48 @@ (is (false? (:ok? result))) (is (= :invalid-options (get-in result [:error :code])))))) +(deftest test-list-execute-default-sort-updated-at + (async done + (-> (p/with-redefs [cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ _] + (case method + :thread-api/api-list-pages [{:db/id 11 :block/title "Page C" :block/updated-at 30} + {:db/id 7 :block/title "Page B" :block/updated-at 10} + {:db/id 5 :block/title "Page A" :block/updated-at 10}] + :thread-api/api-list-tags [{:db/id 4 :block/title "Tag C" :block/updated-at 20} + {:db/id 9 :block/title "Tag B" :block/updated-at 5} + {:db/id 2 :block/title "Tag A" :block/updated-at 5}] + :thread-api/api-list-properties [{:db/id 8 :block/title "Property C" :block/updated-at 9} + {:db/id 6 :block/title "Property B" :block/updated-at 3} + {:db/id 1 :block/title "Property A" :block/updated-at 3}] + (throw (ex-info "unexpected invoke" {:method method}))))] + (p/let [page-result (list-command/execute-list-page {:repo "demo" :options {}} {}) + tag-result (list-command/execute-list-tag {:repo "demo" :options {}} {}) + property-result (list-command/execute-list-property {:repo "demo" :options {}} {})] + (is (= [5 7 11] (item-ids page-result))) + (is (= [2 9 4] (item-ids tag-result))) + (is (= [1 6 8] (item-ids property-result))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) + +(deftest test-list-execute-default-sort-respects-order-and-explicit-sort + (async done + (-> (p/with-redefs [cli-server/ensure-server! (fn [_ _] {:base-url "http://example"}) + transport/invoke (fn [_ method _ _] + (case method + :thread-api/api-list-pages [{:db/id 3 :block/title "Gamma" :block/updated-at 20} + {:db/id 2 :block/title "Alpha" :block/updated-at 5} + {:db/id 1 :block/title "Beta" :block/updated-at 10}] + (throw (ex-info "unexpected invoke" {:method method}))))] + (p/let [desc-default-result (list-command/execute-list-page {:repo "demo" :options {:order "desc"}} {}) + explicit-sort-result (list-command/execute-list-page {:repo "demo" :options {:sort "title"}} {})] + (is (= [3 1 2] (item-ids desc-default-result))) + (is (= [2 1 3] (item-ids explicit-sort-result))))) + (p/catch (fn [e] + (is false (str "unexpected error: " e)))) + (p/finally done)))) + (deftest test-verb-subcommand-parse-upsert-remove (testing "remove block parses with id" (let [result (commands/parse-args ["remove" "block" "--id" "10"])] diff --git a/src/test/logseq/cli/integration_test.cljs b/src/test/logseq/cli/integration_test.cljs index 956d48e9eb..ecd225e0ef 100644 --- a/src/test/logseq/cli/integration_test.cljs +++ b/src/test/logseq/cli/integration_test.cljs @@ -182,6 +182,13 @@ (when (= title (item-title item)) item))) item-id)) +(defn- ordered-item-ids-by-title-set + [payload titles] + (->> (get-in payload [:data :items]) + (filter (fn [item] + (contains? titles (item-title item)))) + (mapv item-id))) + (defn- first-result-id [payload] (first (get-in payload [:data :result]))) @@ -2239,6 +2246,112 @@ (is false (str "unexpected error: " e)) (done))))))) +(deftest ^:long test-cli-list-page-default-sort-matches-explicit-updated-at + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-list-page-default-sort") + repo "list-page-default-sort-graph" + titles #{"SortPageA" "SortPageB" "SortPageC"}] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "SortPageA"] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "SortPageB"] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "page" "--page" "SortPageC"] data-dir cfg-path) + _ (p/delay 80) + update-result (run-cli ["--graph" repo "upsert" "page" "--page" "SortPageA" + "--update-properties" "{:logseq.property/publishing-public? true}"] + data-dir cfg-path) + _ (p/delay 80) + default-result (run-cli ["--graph" repo "list" "page" "--fields" "title,id,updated-at"] data-dir cfg-path) + default-payload (parse-json-output default-result) + explicit-result (run-cli ["--graph" repo "list" "page" "--sort" "updated-at" "--fields" "title,id,updated-at"] data-dir cfg-path) + explicit-payload (parse-json-output explicit-result) + default-ids (ordered-item-ids-by-title-set default-payload titles) + explicit-ids (ordered-item-ids-by-title-set explicit-payload titles) + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code create-result))) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status default-payload))) + (is (= "ok" (:status explicit-payload))) + (is (= 3 (count default-ids))) + (is (= explicit-ids default-ids)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-list-tag-default-sort-matches-explicit-updated-at + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-list-tag-default-sort") + repo "list-tag-default-sort-graph" + titles #{"SortTagA-Renamed" "SortTagB" "SortTagC"}] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + tag-a-result (run-cli ["--graph" repo "upsert" "tag" "--name" "SortTagA"] data-dir cfg-path) + tag-a-id (first-result-id (parse-json-output tag-a-result)) + _ (run-cli ["--graph" repo "upsert" "tag" "--name" "SortTagB"] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "tag" "--name" "SortTagC"] data-dir cfg-path) + _ (p/delay 80) + update-result (run-cli ["--graph" repo "upsert" "tag" "--id" (str tag-a-id) "--name" "SortTagA-Renamed"] + data-dir cfg-path) + _ (p/delay 80) + default-result (run-cli ["--graph" repo "list" "tag" "--user-only" "--fields" "title,id,updated-at"] data-dir cfg-path) + default-payload (parse-json-output default-result) + explicit-result (run-cli ["--graph" repo "list" "tag" "--user-only" "--sort" "updated-at" "--fields" "title,id,updated-at"] data-dir cfg-path) + explicit-payload (parse-json-output explicit-result) + default-ids (ordered-item-ids-by-title-set default-payload titles) + explicit-ids (ordered-item-ids-by-title-set explicit-payload titles) + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code create-result))) + (is (number? tag-a-id)) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status default-payload))) + (is (= "ok" (:status explicit-payload))) + (is (= 3 (count default-ids))) + (is (= explicit-ids default-ids)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + +(deftest ^:long test-cli-list-property-default-sort-matches-explicit-updated-at + (async done + (let [data-dir (node-helper/create-tmp-dir "db-worker-list-property-default-sort") + repo "list-property-default-sort-graph" + titles #{"SortPropertyA" "SortPropertyB" "SortPropertyC"}] + (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn") + _ (fs/writeFileSync cfg-path "{:output-format :json}") + create-result (run-cli ["graph" "create" "--graph" repo] data-dir cfg-path) + property-a-result (run-cli ["--graph" repo "upsert" "property" "--name" "SortPropertyA" "--type" "default"] data-dir cfg-path) + property-a-id (first-result-id (parse-json-output property-a-result)) + _ (run-cli ["--graph" repo "upsert" "property" "--name" "SortPropertyB" "--type" "default"] data-dir cfg-path) + _ (run-cli ["--graph" repo "upsert" "property" "--name" "SortPropertyC" "--type" "default"] data-dir cfg-path) + _ (p/delay 80) + update-result (run-cli ["--graph" repo "upsert" "property" "--id" (str property-a-id) "--type" "node"] data-dir cfg-path) + _ (p/delay 80) + default-result (run-cli ["--graph" repo "list" "property" "--user-only" "--fields" "title,id,updated-at"] data-dir cfg-path) + default-payload (parse-json-output default-result) + explicit-result (run-cli ["--graph" repo "list" "property" "--user-only" "--sort" "updated-at" "--fields" "title,id,updated-at"] data-dir cfg-path) + explicit-payload (parse-json-output explicit-result) + default-ids (ordered-item-ids-by-title-set default-payload titles) + explicit-ids (ordered-item-ids-by-title-set explicit-payload titles) + stop-payload (stop-repo! data-dir cfg-path repo)] + (is (= 0 (:exit-code create-result))) + (is (number? property-a-id)) + (is (= 0 (:exit-code update-result))) + (is (= "ok" (:status default-payload))) + (is (= "ok" (:status explicit-payload))) + (is (= 3 (count default-ids))) + (is (= explicit-ids default-ids)) + (is (= "ok" (:status stop-payload))) + (done)) + (p/catch (fn [e] + (is false (str "unexpected error: " e)) + (done))))))) + (deftest ^:long test-cli-list-outputs-include-id (async done (let [data-dir (node-helper/create-tmp-dir "db-worker")] From 9a9504b9b5d03f3e55887fe931c21bcc74b1da4d Mon Sep 17 00:00:00 2001 From: Gabriel Horner Date: Tue, 17 Mar 2026 11:56:03 -0400 Subject: [PATCH 166/375] enhance: cli completion for negating command boolean options with '--no-'. Don't do global options as most global options don't apply to all commands and are already turned off by default --- src/main/logseq/cli/completion_generator.cljs | 95 ++++++++++++------- .../logseq/cli/completion_generator_test.cljs | 26 ++++- 2 files changed, 86 insertions(+), 35 deletions(-) diff --git a/src/main/logseq/cli/completion_generator.cljs b/src/main/logseq/cli/completion_generator.cljs index 038cb402b5..231dcd7c9d 100644 --- a/src/main/logseq/cli/completion_generator.cljs +++ b/src/main/logseq/cli/completion_generator.cljs @@ -156,18 +156,23 @@ _logseq_queries() { (string/replace "'" "'\\''"))) (defn- zsh-token-for - "Generate a zsh _arguments token string for a spec token descriptor." - [{:keys [key type alias desc values complete]}] + "Generate a zsh _arguments token string for a spec token descriptor. + no-prefix-keys: set of keys that get --no-