012-logseq-cli-graph-storage.md

docs/agent-guide/003-db-worker-node-cli-orchestration.md

@@ -44,7 +44,7 @@ Key changes:
   - Reject `thread-api/create-or-open-db`, `thread-api/unsafe-unlink-db`, etc. when repo differs.
   - Return 409/400 with `:repo-mismatch` error shape.
 - **Lock file**:
-  - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-<graph>/db-worker.lock`).
+  - Location: inside repo dir (e.g. `~/.logseq/cli-graphs/<graph>/db-worker.lock`).
   - Content: JSON `{repo, pid, host, port, startedAt}`.
   - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”.
   - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM.
@@ -108,7 +108,7 @@ Implementation notes:
    - Answer: No --repo needed, using 'out-of-band access to data-dir' way
 2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions).
    - lockfile name:`db-worker.lock`,
-   - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-<graph>/db-worker.lock`).
+   - Location: inside repo dir (e.g. `~/.logseq/cli-graphs/<graph>/db-worker.lock`).
    - only consider linux/macos for now
 3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle.
 4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI?

docs/agent-guide/012-logseq-cli-graph-storage.md

@@ -0,0 +1,77 @@
+# Logseq CLI Graph Storage Plan
+
+## Context
+logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logseq/db-worker/` and use per-graph directories named like `.logseq-pool-<graph-name>/` (with partial encoding). This plan captures the non-functional updates requested:
+
+1) Rename `~/.logseq/db-worker/` to `~/.logseq/cli-graphs/` for CLI-managed graphs.
+2) Rename per-graph directory format from `.logseq-pool-<graph-name>/` to `<graph-name>/`.
+3) Ensure graph names that are not valid directory names are encoded, and decoding is symmetric when reading/listing.
+
+## Goals
+- Move CLI graph storage to `~/.logseq/cli-graphs` by default.
+- Use a clean per-graph directory name equal to the (encoded) graph name, without `.` prefix or `logseq-pool-` prefix.
+- Provide a reversible encode/decode for graph names so list/read operations reconstruct the original graph name.
+- CLI commands and outputs should hide the internal `logseq_db_` prefix; user-facing graph names strip that db prefix.
+- Maintain db-worker-node functionality (locks/logs/kv-store) with the new paths.
+
+## Non-goals
+- Changing Electron or browser-based db graph storage (`~/logseq/graphs`) or OPFS behavior.
+- Changing db schema or sqlite storage format.
+- Changing db-worker-node API semantics.
+
+## Current Behavior (Key References)
+- CLI default data dir is `~/.logseq/db-worker`: `src/main/logseq/cli/config.cljs`.
+- db-worker-node default data dir is `~/.logseq/db-worker`: `src/main/frontend/worker/db_worker_node.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`.
+- Per-graph directory currently `.logseq-pool-<sanitized>`:
+  - `frontend.worker-common.util/get-pool-name` returns `logseq-pool-<sanitized>`: `src/main/frontend/worker_common/util.cljc`.
+  - `repo-dir` uses `"." + pool-name` in CLI server, db-worker-node lock, and node platform: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`.
+- Current graph decoding in list operations reverses only `+3A+` and `++` (file-based graphs); other characters are not reversible: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/platform/node.cljs`.
+
+## Proposed Approach
+### 1) New default data dir
+- Change default data dir for CLI and db-worker-node from `~/.logseq/db-worker` to `~/.logseq/cli-graphs`.
+- Update help text and any user-facing docs mentioning the old default.
+
+### 2) New per-graph directory naming
+- Replace `.logseq-pool-<graph-name>/` with `<encoded-graph-name>/`.
+- Remove the leading dot and `logseq-pool-` prefix entirely for CLI-managed graphs.
+
+### 3) Reversible graph name encoding
+- Introduce a shared encode/decode pair for graph directory names that is bijective for all graph names.
+- The encoding must avoid path separators and other invalid characters (esp. `/`, `\`, `:` on Windows).
+- Suggested approach (reversible and simple):
+  - Encode: apply `encodeURIComponent` to the graph name, then replace `%` with a safe delimiter (e.g. `~`) to keep filenames readable and avoid `%` edge cases.
+  - Decode: reverse the delimiter replacement, then `decodeURIComponent`.
+- Provide helper functions in a shared place (e.g. `frontend.worker-common.util` or a new shared CLI/worker helper) so CLI server, db-worker-node lock, and node platform list all use the same encode/decode logic.
+
+## Implementation Steps
+1) Add encode/decode helpers
+   - Add new helpers for reversible graph name <-> directory name.
+   - Update `get-pool-name` or replace its usage for CLI/db-worker-node paths.
+   - Files: `src/main/frontend/worker_common/util.cljc`, potentially `deps/cli/src/logseq/cli/common/graph.cljs`.
+
+2) Update data dir defaults
+   - Change defaults to `~/.logseq/cli-graphs` in:
+     - `src/main/logseq/cli/config.cljs`
+     - `src/main/logseq/cli/server.cljs`
+     - `src/main/frontend/worker/db_worker_node_lock.cljs`
+     - `src/main/frontend/worker/db_worker_node.cljs` (help text)
+     - `src/main/frontend/worker/platform/node.cljs`
+   - Update any CLI docs/tests that reference `db-worker` as default.
+
+3) Update repo-dir/path derivation
+   - Replace `"." + pool-name` usage with new `<encoded-graph-name>` directory naming.
+   - Update list-graphs and list-servers to decode from new directory names.
+   - Files: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`.
+
+4) Tests & verification
+   - Update CLI integration tests that construct temp dirs named `db-worker*` to match new defaults or explicitly pass `--data-dir`.
+   - Update db-worker-node tests to use new naming and to validate encode/decode.
+   - Ensure `bb dev:lint-and-test` passes.
+
+## Open Questions
+- The new encoding is CLI and db-worker-node only (no Electron changes).
+
+## Rollout Notes
+- This is a filesystem layout change. Include release notes and ensure users can override via `--data-dir`.
+- Provide a one-time warning if old layout is detected and not migrated.

docs/agent-guide/task--db-worker-nodejs-compatible.md

@@ -129,7 +129,7 @@ The db-worker should be runnable as a standalone process for Node.js environment
 - Provide a CLI entry (example: `bin/logseq-db-worker` or `node dist/db-worker-node.js`).
 - CLI flags (suggested):
   - Binds to localhost on a random port and records it in the repo lock file.
-  - `--data-dir` (path for sqlite files, required or default to `~/.logseq/db-worker`)
+  - `--data-dir` (path for sqlite files, required or default to `~/.logseq/cli-graphs`)
   - `--repo` (optional: auto-open a repo on boot)
   - `--rtc-ws-url` (optional)
   - `--log-level` (default `info`)

src/main/frontend/worker/db_core.cljs

@@ -68,6 +68,12 @@
   []
   (= :node (platform/env-flag (platform/current) :runtime)))
 
+(defn- storage-pool-name
+  [graph]
+  (if (node-runtime?)
+    (worker-util/encode-graph-dir-name graph)
+    (worker-util/get-pool-name graph)))
+
 (defn- get-storage-pool
   [graph]
   (if (node-runtime?)
@@ -91,7 +97,7 @@
   (when-not @*publishing?
     (or (get-storage-pool graph)
         (p/let [storage (platform/storage (platform/current))
-                ^js pool ((:install-opfs-pool storage) @*sqlite (worker-util/get-pool-name graph))]
+                ^js pool ((:install-opfs-pool storage) @*sqlite (storage-pool-name graph))]
           (remember-storage-pool! graph pool)
           pool))))
 

src/main/frontend/worker/db_worker_node.cljs

@@ -241,7 +241,7 @@
 (defn- show-help!
   []
   (println "db-worker-node options:")
-  (println "  --data-dir <path>    (default ~/.logseq/db-worker)")
+  (println "  --data-dir <path>    (default ~/.logseq/cli-graphs)")
   (println "  --repo <name>        (required)")
   (println "  --rtc-ws-url <url>   (optional)")
   (println "  --log-level <level>  (default info)")

src/main/frontend/worker/db_worker_node_lock.cljs

@@ -16,12 +16,11 @@
 
 (defn resolve-data-dir
   [data-dir]
-  (expand-home (or data-dir "~/.logseq/db-worker")))
+  (expand-home (or data-dir "~/.logseq/cli-graphs")))
 
 (defn repo-dir
   [data-dir repo]
-  (let [pool-name (worker-util/get-pool-name repo)]
-    (node-path/join data-dir (str "." pool-name))))
+  (node-path/join data-dir (worker-util/encode-graph-dir-name repo)))
 
 (defn lock-path
   [data-dir repo]

src/main/frontend/worker/platform/node.cljs

@@ -30,7 +30,7 @@
 
 (defn- repo-dir
   [data-dir pool-name]
-  (node-path/join data-dir (str "." pool-name)))
+  (node-path/join data-dir pool-name))
 
 (defn- pool-path
   [^js pool path]
@@ -52,25 +52,20 @@
 
 (defn- list-graphs
   [data-dir]
-  (let [dir? #(and % (.isDirectory %))
-        db-dir-prefix ".logseq-pool-"]
+  (let [dir? #(and % (.isDirectory %))]
     (p/let [entries (fs/readdir data-dir #js {:withFileTypes true})
             db-dirs (->> entries
-                         (filter dir?)
-                         (filter (fn [dirent]
-                                   (string/starts-with? (.-name dirent) db-dir-prefix))))
+                         (filter dir?))
             graph-names (map (fn [dirent]
-                               (-> (.-name dirent)
-                                   (string/replace-first db-dir-prefix "")
-                                   ;; TODO: DRY
-                                   (string/replace "+3A+" ":")
-                                   (string/replace "++" "/")))
+                               (worker-util/decode-graph-dir-name (.-name dirent)))
                              db-dirs)]
-      (vec graph-names))))
+      (->> graph-names
+           (filter some?)
+           (vec)))))
 
 (defn- db-exists?
   [data-dir graph]
-  (p/let [pool-name (worker-util/get-pool-name graph)
+  (p/let [pool-name (worker-util/encode-graph-dir-name graph)
           db-path (node-path/join (repo-dir data-dir pool-name) "db.sqlite")]
     (-> (fs/stat db-path)
         (p/then (fn [_] true))
@@ -191,7 +186,7 @@
 
 (defn node-platform
   [{:keys [data-dir event-fn]}]
-  (let [data-dir (expand-home (or data-dir "~/.logseq/db-worker"))
+  (let [data-dir (expand-home (or data-dir "~/.logseq/cli-graphs"))
         kv (kv-store data-dir)]
     (p/do!
      (ensure-dir! data-dir)

src/main/frontend/worker_common/util.cljc

@@ -31,6 +31,21 @@
        (when-let [worker (or port js/self)]
          (.postMessage worker (ldb/write-transit-str [type data]))))
 
+     (defn encode-graph-dir-name
+       [graph-name]
+       (let [encoded (js/encodeURIComponent (or graph-name ""))]
+         (-> encoded
+             (string/replace "~" "%7E")
+             (string/replace "%" "~"))))
+
+     (defn decode-graph-dir-name
+       [dir-name]
+       (when (some? dir-name)
+         (try
+           (js/decodeURIComponent (string/replace dir-name "~" "%"))
+           (catch :default _
+             nil))))
+
      (defn get-pool-name
        [graph-name]
        (str "logseq-pool-" (common-sqlite/sanitize-db-name graph-name)))

src/main/logseq/cli/command/graph.cljs

@@ -135,7 +135,8 @@
 
 (defn execute-graph-list
   [_action config]
-  (let [graphs (cli-server/list-graphs config)]
+  (let [graphs (->> (cli-server/list-graphs config)
+                    (mapv core/repo->graph))]
     {:status :ok
      :data {:graphs graphs}}))
 

src/main/logseq/cli/config.cljs

@@ -80,7 +80,7 @@
   (let [defaults {:timeout-ms 10000
                   :retries 0
                   :output-format nil
-                  :data-dir "~/.logseq/db-worker"
+                  :data-dir "~/.logseq/cli-graphs"
                   :config-path (default-config-path)}
         env (env-config)
         config-path (or (:config-path opts)

src/main/logseq/cli/format.cljs

@@ -1,7 +1,8 @@
 (ns logseq.cli.format
   "Formatting helpers for CLI output."
   (:require [clojure.string :as string]
-            [clojure.walk :as walk]))
+            [clojure.walk :as walk]
+            [logseq.cli.command.core :as command-core]))
 
 (defn- normalize-json
   [value]
@@ -290,9 +291,23 @@
             (= status :ok) (assoc :data data)
             (= status :error) (assoc :error error))))
 
+(defn- normalize-graph-result
+  [result]
+  (walk/postwalk
+   (fn [entry]
+     (if (map? entry)
+       (cond-> entry
+         (string? (:repo entry)) (update :repo command-core/repo->graph)
+         (string? (:graph entry)) (update :graph command-core/repo->graph)
+         (seq (:graphs entry)) (update :graphs (fn [graphs]
+                                                 (mapv command-core/repo->graph graphs))))
+       entry))
+   result))
+
 (defn format-result
   [result {:keys [output-format] :as opts}]
-  (let [format (cond
+  (let [result (normalize-graph-result result)
+        format (cond
                  (= output-format :edn) :edn
                  (= output-format :json) :json
                  :else :human)]

src/main/logseq/cli/server.cljs

@@ -6,6 +6,7 @@
             ["os" :as os]
             ["path" :as node-path]
             [clojure.string :as string]
+            [logseq.cli.command.core :as command-core]
             [frontend.worker-common.util :as worker-util]
             [lambdaisland.glogi :as log]
             [promesa.core :as p]))
@@ -18,12 +19,11 @@
 
 (defn resolve-data-dir
   [config]
-  (expand-home (or (:data-dir config) "~/.logseq/db-worker")))
+  (expand-home (or (:data-dir config) "~/.logseq/cli-graphs")))
 
 (defn- repo-dir
   [data-dir repo]
-  (let [pool-name (worker-util/get-pool-name repo)]
-    (node-path/join data-dir (str "." pool-name))))
+  (node-path/join data-dir (worker-util/encode-graph-dir-name repo)))
 
 (defn lock-path
   [data-dir repo]
@@ -286,17 +286,12 @@
 (defn list-graphs
   [config]
   (let [data-dir (resolve-data-dir config)
-        db-dir-prefix ".logseq-pool-"
         entries (when (fs/existsSync data-dir)
                   (fs/readdirSync data-dir #js {:withFileTypes true}))]
     (->> entries
          (filter #(.isDirectory ^js %))
          (map (fn [^js dirent]
-                (.-name dirent)))
-         (filter #(string/starts-with? % db-dir-prefix))
-         (map (fn [dir-name]
-                (-> dir-name
-                    (string/replace-first db-dir-prefix "")
-                    (string/replace "+3A+" ":")
-                    (string/replace "++" "/"))))
+                (worker-util/decode-graph-dir-name (.-name dirent))))
+         (filter some?)
+         (map command-core/repo->graph)
          (vec))))

src/test/frontend/worker/db_worker_node_test.cljs

@@ -1,13 +1,11 @@
 (ns frontend.worker.db-worker-node-test
   (:require ["http" :as http]
             [cljs.test :refer [async deftest is]]
-            [clojure.string :as string]
             [frontend.test.node-helper :as node-helper]
             [frontend.worker-common.util :as worker-util]
             [frontend.worker.db-worker-node :as db-worker-node]
             [goog.object :as gobj]
             [logseq.db :as ldb]
-            [logseq.db.sqlite.util :as sqlite-util]
             [promesa.core :as p]
             ["fs" :as fs]
             ["path" :as node-path]))
@@ -75,8 +73,7 @@
 
 (defn- lock-path
   [data-dir repo]
-  (let [pool-name (worker-util/get-pool-name repo)
-        repo-dir (node-path/join data-dir (str "." pool-name))]
+  (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo))]
     (node-path/join repo-dir "db-worker.lock")))
 
 (defn- pad2
@@ -93,8 +90,7 @@
 
 (defn- log-path
   [data-dir repo]
-  (let [pool-name (worker-util/get-pool-name repo)
-        repo-dir (node-path/join data-dir (str "." pool-name))
+  (let [repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo))
         date-str (yyyymmdd (js/Date.))]
     (node-path/join repo-dir (str "db-worker-node-" date-str ".log"))))
 
@@ -144,8 +140,7 @@
   (let [enforce-log-retention! #'db-worker-node/enforce-log-retention!
         data-dir (node-helper/create-tmp-dir "db-worker-log-retention")
         repo (str "logseq_db_log_retention_" (subs (str (random-uuid)) 0 8))
-        pool-name (worker-util/get-pool-name repo)
-        repo-dir (node-path/join data-dir (str "." pool-name))
+        repo-dir (node-path/join data-dir (worker-util/encode-graph-dir-name repo))
         days ["20240101" "20240102" "20240103" "20240104" "20240105"
               "20240106" "20240107" "20240108" "20240109"]
         make-log (fn [day]
@@ -222,11 +217,7 @@
                   dbs (invoke host port "thread-api/list-db" [])
                   _ (do
                       (println "[db-worker-node-test] list-db" dbs)
-                      (let [prefix sqlite-util/db-version-prefix
-                            expected-name (if (string/starts-with? repo prefix)
-                                            (subs repo (count prefix))
-                                            repo)]
-                        (is (some #(= expected-name (:name %)) dbs))))
+                      (is (some #(= repo (:name %)) dbs)))
                   lock-file (lock-path data-dir repo)
                   _ (is (fs/existsSync lock-file))
                   lock-contents (js/JSON.parse (.toString (fs/readFileSync lock-file) "utf8"))

src/test/frontend/worker/worker_common_util_test.cljs

@@ -0,0 +1,20 @@
+(ns frontend.worker.worker-common-util-test
+  (:require [cljs.test :refer [deftest is]]
+            [clojure.string :as string]
+            [frontend.worker-common.util :as worker-util]))
+
+(deftest encode-decode-graph-dir-name-roundtrip
+  (let [names ["Demo"
+               "foo/bar"
+               "a:b"
+               "space name"
+               "100% legit"
+               "til~de"
+               "mix/ed:chars%~"]
+        encoded (map worker-util/encode-graph-dir-name names)]
+    (doseq [[name enc] (map vector names encoded)]
+      (is (= name (worker-util/decode-graph-dir-name enc))))
+    (doseq [enc encoded]
+      (is (not (string/includes? enc "/")))
+      (is (not (string/includes? enc "\\")))))
+  (is (nil? (worker-util/decode-graph-dir-name nil))))

src/test/logseq/cli/commands_test.cljs

@@ -871,3 +871,16 @@
                        (set! transport/read-input orig-read-input)
                        (set! transport/invoke orig-invoke)
                        (done)))))))
+
+(deftest test-execute-graph-list-strips-db-prefix
+  (async done
+    (let [orig-list-graphs cli-server/list-graphs]
+      (set! cli-server/list-graphs (fn [_] ["logseq_db_demo" "logseq_db_other"]))
+      (-> (p/let [result (commands/execute {:type :graph-list} {})]
+            (is (= :ok (:status result)))
+            (is (= ["demo" "other"] (get-in result [:data :graphs]))))
+          (p/catch (fn [e]
+                     (is false (str "unexpected error: " e))))
+          (p/finally (fn []
+                       (set! cli-server/list-graphs orig-list-graphs)
+                       (done)))))))

src/test/logseq/cli/format_test.cljs

@@ -150,7 +150,7 @@
   (testing "server status includes repo, status, host, port"
     (let [result (format/format-result {:status :ok
                                         :command :server-status
-                                        :data {:repo "demo-repo"
+                                        :data {:repo "logseq_db_demo-repo"
                                                :status :ready
                                                :host "127.0.0.1"
                                                :port 1234}}