021-logseq-cli-reference-uuid-rewrite.md

docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md

@@ -0,0 +1,87 @@
+# Logseq CLI Add Reference UUID Rewrite Plan
+
+Goal: For logseq-cli add block/page content, replace every `[[<reference-name>]]` with `[[block-uuid]]` before calling db-worker-node thread-api, creating missing pages when needed.
+
+Architecture: logseq-cli (`src/main/logseq/cli/command/add.cljs`) sends `:thread-api/apply-outliner-ops` calls through `logseq.cli.transport`. db-worker-node executes `outliner-op/apply-ops!` without normalizing refs. Frontend already normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI does not.
+
+Tech Stack: ClojureScript, logseq-cli, db-worker-node, Datascript.
+
+## Problem statement
+
+Today logseq-cli passes block content to db-worker-node as-is. If content includes `[[Page Name]]`, db-worker-node does not automatically resolve or create that page for outliner ops. We need to normalize content refs to uuid references up front so db-worker-node receives canonical refs and missing pages are created deterministically.
+
+## Current behavior summary
+
+- `logseq-cli add block` builds blocks and calls `:thread-api/apply-outliner-ops` with `:insert-blocks`.
+- `logseq-cli add page` calls `:thread-api/apply-outliner-ops` with `:create-page` only (no content normalization).
+- db-worker-node `:thread-api/apply-outliner-ops` applies ops verbatim and does not resolve page refs in block content.
+- Frontend editor normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI paths do not use it.
+
+## Requirements
+
+- For add block and add page content, replace every `[[<reference-name>]]` with `[[block-uuid]]` before invoking db-worker-node thread-api.
+- Page reference:
+  - If `<reference-name>` is not a UUID, treat it as a page title.
+  - If the page does not exist, create it first.
+  - Replace `[[Page Title]]` with `[[page-uuid]]` (case-insensitive).
+- Block reference:
+  - If `<reference-name>` is a UUID, treat it as a block ref.
+  - The block must exist; otherwise return a CLI error.
+- Do not change other syntax (e.g. `((uuid))` block refs, tags, or macros) unless they are inside `[[...]]`.
+
+## Non-goals
+
+- Do not alter how `((uuid))` block refs are parsed or stored.
+- Do not introduce automatic block creation for missing block UUIDs.
+- Do not change CLI command flags or output format.
+
+## Design decisions
+
+- Reuse `logseq.db.frontend.content/title-ref->id-ref` so CLI behavior matches frontend normalization rules.
+- Extract `[[...]]` refs using the existing page-ref regex from `logseq.common.util.page-ref` to avoid implementing a new parser.
+- Resolve refs once per CLI action, cache page-name -> uuid, and then update all affected blocks before the first `:thread-api/apply-outliner-ops` call.
+- Handle page creation with the existing `ensure-page!` helper in `src/main/logseq/cli/command/add.cljs` for consistent behavior.
+
+## Implementation plan
+
+### 1) Add reference extraction and resolution helpers
+
+- `src/main/logseq/cli/command/add.cljs`
+  - Add a helper to extract `[[...]]` tokens from a block title using `logseq.common.util.page-ref/page-ref-re`.
+  - Add a helper that partitions refs into:
+    - `uuid-refs`: `[[<uuid>]]` values
+    - `page-refs`: `[[<page-title>]]` values
+  - Add a resolver that:
+    - For `page-refs`, calls `ensure-page!` (once per unique title) and returns `{:block/uuid uuid :block/title title}`.
+    - For `uuid-refs`, pulls the entity by `[:block/uuid uuid]` and errors if missing.
+
+### 2) Normalize add block content before outliner ops
+
+- `src/main/logseq/cli/command/add.cljs`
+  - In `execute-add-block`, before building ops:
+    - Walk the `:blocks` tree (top-level and any nested `:block/children`) and collect all `[[...]]` refs.
+    - Resolve refs once per action using the resolver from step 1.
+    - For each block with `:block/title`, rewrite it with `db-content/title-ref->id-ref` using the resolved refs and `{:replace-tag? false}`.
+  - Use the rewritten blocks in the `:insert-blocks` op.
+
+### 3) Normalize add page content (when present)
+
+- `src/main/logseq/cli/command/add.cljs`
+  - If add page grows to accept initial blocks/content (or if `:create-page` options start supporting content), reuse the same ref normalization flow from step 2 before invoking `:create-page` or `:insert-blocks`.
+  - If no content is provided, no change is needed in the current implementation.
+
+### 4) Tests
+
+- `test/logseq/cli/integration_test.cljs`
+  - Add an integration test for `add block` with content `"See [[New Page]]"`:
+    - Assert the page is created.
+    - Pull the inserted block and verify its title contains `[[<page-uuid>]]` instead of `[[New Page]]`.
+  - Add an integration test for `add block` with content `"See [[<existing-block-uuid>]]"`:
+    - Assert the UUID is preserved and no new page is created.
+  - Add an integration test for `add block` with content `"See [[<missing-block-uuid>]]"`:
+    - Assert CLI returns an error with a clear message.
+
+## Notes
+
+- If we later decide to normalize tags (`#tag`) or macro-based refs, we can extend the resolver to call `db-content/title-ref->id-ref` with `:replace-tag? true` and add tests accordingly.
+- Keep all normalization in CLI to avoid changing db-worker-node semantics for other callers.

src/main/logseq/cli/command/add.cljs

@@ -8,8 +8,10 @@
             [logseq.cli.server :as cli-server]
             [logseq.cli.transport :as transport]
             [logseq.common.util :as common-util]
+            [logseq.common.util.page-ref :as page-ref]
             [logseq.common.util.date-time :as date-time-util]
             [logseq.common.uuid :as common-uuid]
+            [logseq.db.frontend.content :as db-content]
             [logseq.db.frontend.property :as db-property]
             [logseq.db.frontend.property.type :as db-property-type]
             [promesa.core :as p]))
@@ -46,14 +48,15 @@
 
 (defn- ensure-page!
   [config repo page-name]
-  (p/let [page (transport/invoke config :thread-api/pull false
-                                 [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]])]
+  (let [page-name-lc (common-util/page-name-sanity-lc page-name)]
+    (p/let [page (transport/invoke config :thread-api/pull false
+                                   [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]])]
     (if (:db/id page)
       page
       (p/let [_ (transport/invoke config :thread-api/apply-outliner-ops false
                                   [repo [[:create-page [page-name {}]]] {}])]
         (transport/invoke config :thread-api/pull false
-                          [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name]])))))
+                          [repo [:db/id :block/uuid :block/name :block/title] [:block/name page-name-lc]]))))))
 
 (def ^:private add-positions
   #{"first-child" "last-child" "sibling"})
@@ -109,6 +112,91 @@
               (assoc block :block/uuid (common-uuid/gen-uuid)))))
         blocks))
 
+(defn- extract-page-refs
+  [title]
+  (when (string? title)
+    (->> (re-seq page-ref/page-ref-re title)
+         (map second)
+         (remove string/blank?))))
+
+(defn- collect-page-refs
+  [blocks]
+  (->> blocks
+       (mapcat (fn walk [block]
+                 (let [refs (extract-page-refs (:block/title block))
+                       children (:block/children block)]
+                   (if (seq children)
+                     (concat refs (mapcat walk children))
+                     refs))))
+       (remove string/blank?)
+       vec))
+
+(defn- partition-ref-values
+  [refs]
+  (reduce
+   (fn [acc ref-value]
+     (let [value (string/trim ref-value)]
+       (cond
+         (string/blank? value)
+         acc
+
+         (common-util/uuid-string? value)
+         (update acc :uuid-refs conj value)
+
+         :else
+         (update acc :page-refs conj value))))
+   {:uuid-refs [] :page-refs []}
+   refs))
+
+(defn- resolve-page-ref-entities
+  [config repo page-refs]
+  (if (seq page-refs)
+    (let [unique (reduce (fn [acc ref-value]
+                           (let [value (string/trim ref-value)]
+                             (if (string/blank? value)
+                               acc
+                               (assoc acc (common-util/page-name-sanity-lc value) value))))
+                         {}
+                         page-refs)]
+      (p/let [resolved (p/all
+                        (map (fn [[_ page-name]]
+                               (p/let [page (ensure-page! config repo page-name)
+                                       page-uuid (:block/uuid page)]
+                                 (when-not page-uuid
+                                   (throw (ex-info "page not found"
+                                                   {:code :page-not-found
+                                                    :page page-name})))
+                                 {:block/uuid page-uuid
+                                  :block/title (or (:block/title page) page-name)}))
+                             unique))]
+        (vec resolved)))
+    (p/resolved nil)))
+
+(defn- ensure-block-refs-exist!
+  [config repo uuid-refs]
+  (when (seq uuid-refs)
+    (p/all
+     (map (fn [uuid-ref]
+            (p/let [entity (transport/invoke config :thread-api/pull false
+                                             [repo [:db/id :block/uuid] [:block/uuid (uuid uuid-ref)]])]
+              (when-not (:db/id entity)
+                (throw (ex-info (str "block ref not found: " uuid-ref)
+                                {:code :block-ref-not-found
+                                 :uuid uuid-ref})))))
+          (distinct uuid-refs)))))
+
+(defn- normalize-block-title-refs
+  [blocks refs]
+  (mapv (fn update-block [block]
+          (let [block' (if (string? (:block/title block))
+                         (update block :block/title
+                                 #(db-content/title-ref->id-ref % refs :replace-tag? false))
+                         block)]
+            (if (seq (:block/children block'))
+              (update block' :block/children #(normalize-block-title-refs % refs))
+              block')))
+        blocks))
+
 (defn- invalid-options-result
   [message]
   {:ok? false
@@ -729,6 +817,13 @@
   [action config]
   (-> (p/let [cfg (cli-server/ensure-server! config (:repo action))
               target-id (resolve-add-target cfg action)
+              ref-values (collect-page-refs (:blocks action))
+              {:keys [uuid-refs page-refs]} (partition-ref-values ref-values)
+              _ (ensure-block-refs-exist! cfg (:repo action) uuid-refs)
+              refs (or (resolve-page-ref-entities cfg (:repo action) page-refs) [])
+              blocks (if (seq refs)
+                       (normalize-block-title-refs (:blocks action) refs)
+                       (:blocks action))
               status (:status action)
               tags (resolve-tags cfg (:repo action) (:tags action))
               properties (resolve-properties cfg (:repo action) (:properties action))
@@ -741,11 +836,11 @@
               opts (cond-> opts
                      keep-uuid?
                      (assoc :keep-uuid? true))
-              ops [[:insert-blocks [(:blocks action)
+              ops [[:insert-blocks [blocks
                                     target-id
                                     (assoc opts :outliner-op :insert-blocks)]]]
               _ (transport/invoke cfg :thread-api/apply-outliner-ops false [(:repo action) ops {}])
-              block-ids (->> (:blocks action)
+              block-ids (->> blocks
                              (map :block/uuid)
                              (remove nil?)
                              vec)

src/test/logseq/cli/integration_test.cljs

@@ -8,6 +8,7 @@
             [frontend.test.node-helper :as node-helper]
             [logseq.cli.command.core :as command-core]
             [logseq.cli.main :as cli-main]
+            [logseq.common.util :as common-util]
             [logseq.db.frontend.property :as db-property]
             [promesa.core :as p]))
 
@@ -29,6 +30,16 @@
   [result]
   (js->clj (js/JSON.parse (:output result)) :keywordize-keys true))
 
+(defn- parse-json-output-safe
+  [result label]
+  (try
+    (parse-json-output result)
+    (catch :default e
+      (throw (ex-info (str "json parse failed: " label)
+                      {:label label
+                       :output (:output result)}
+                      e)))))
+
 (defn- parse-edn-output
   [result]
   (reader/read-string (:output result)))
@@ -220,6 +231,127 @@
                  (is (contains? (get-in show-payload [:data :root]) :uuid))
                  (is (= "ok" (:status remove-page-payload)))
                  (is (= "ok" (:status stop-payload)))
+                (done))
+              (p/catch (fn [e]
+                          (is false (str "unexpected error: " e))
+                          (done)))))))
+
+(deftest test-cli-add-block-rewrites-page-ref
+  (async done
+         (let [data-dir (node-helper/create-tmp-dir "db-worker-ref-rewrite")]
+           (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn")
+                       _ (fs/writeFileSync cfg-path "{:output-format :json}")
+                       _ (run-cli ["graph" "create" "--repo" "ref-rewrite-graph"] data-dir cfg-path)
+                       _ (run-cli ["--repo" "ref-rewrite-graph" "add" "page" "--page" "Home"] data-dir cfg-path)
+                       add-block-result (run-cli ["--repo" "ref-rewrite-graph"
+                                                  "add" "block"
+                                                  "--target-page-name" "Home"
+                                                  "--content" "See [[New Page]]"]
+                                                 data-dir cfg-path)
+                       add-block-payload (parse-json-output-safe add-block-result "add-block")
+                       _ (p/delay 100)
+                       list-page-result (run-cli ["--repo" "ref-rewrite-graph" "list" "page"] data-dir cfg-path)
+                       list-page-payload (parse-json-output-safe list-page-result "list-page")
+                       page-titles (->> (get-in list-page-payload [:data :items])
+                                        (map #(or (:block/title %) (:title %)))
+                                        set)
+                       query-payload (run-query data-dir cfg-path "ref-rewrite-graph"
+                                                "[:find ?title :in $ ?page-name :where [?p :block/name ?page-name] [?b :block/page ?p] [?b :block/title ?title]]"
+                                                (pr-str [(common-util/page-name-sanity-lc "Home")]))
+                       titles (map first (get-in query-payload [:data :result]))
+                       ref-title (some #(when (and (string? %)
+                                                   (string/includes? % "See [[")
+                                                   (string/includes? % "]]"))
+                                         %)
+                                       titles)
+                       ref-value (when ref-title
+                                   (second (first (re-seq #"\[\[(.*?)\]\]" ref-title))))
+                       stop-result (run-cli ["server" "stop" "--repo" "ref-rewrite-graph"] data-dir cfg-path)
+                       stop-payload (parse-json-output-safe stop-result "server-stop")]
+                 (is (= 0 (:exit-code add-block-result)))
+                 (is (= "ok" (:status add-block-payload)))
+                 (is (contains? page-titles "New Page"))
+                 (is (string? ref-value))
+                 (is (common-util/uuid-string? ref-value))
+                 (is (string? ref-title))
+                 (is (not (string/includes? ref-title "[[New Page]]")))
+                 (is (= "ok" (:status stop-payload)))
+                 (done))
+               (p/catch (fn [e]
+                          (is false (str "unexpected error: " e))
+                          (done)))))))
+
+(deftest test-cli-add-block-keeps-uuid-ref
+  (async done
+         (let [data-dir (node-helper/create-tmp-dir "db-worker-uuid-ref")]
+           (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn")
+                       _ (fs/writeFileSync cfg-path "{:output-format :json}")
+                       _ (run-cli ["graph" "create" "--repo" "uuid-ref-graph"] data-dir cfg-path)
+                       _ (run-cli ["--repo" "uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path)
+                       _ (run-cli ["--repo" "uuid-ref-graph"
+                                   "add" "block"
+                                   "--target-page-name" "Home"
+                                   "--content" "Target block"]
+                                  data-dir cfg-path)
+                       _ (p/delay 100)
+                       target-query-payload (run-query data-dir cfg-path "uuid-ref-graph"
+                                                       "[:find ?uuid :in $ ?title :where [?b :block/title ?title] [?b :block/uuid ?uuid]]"
+                                                       (pr-str ["Target block"]))
+                       target-uuid (first (first (get-in target-query-payload [:data :result])))
+                       add-block-result (run-cli ["--repo" "uuid-ref-graph"
+                                                  "add" "block"
+                                                  "--target-page-name" "Home"
+                                                  "--content" (str "See [[" target-uuid "]]")]
+                                                 data-dir cfg-path)
+                       add-block-payload (parse-json-output add-block-result)
+                       _ (p/delay 100)
+                       list-page-result (run-cli ["--repo" "uuid-ref-graph" "list" "page"] data-dir cfg-path)
+                       list-page-payload (parse-json-output list-page-result)
+                       page-titles (->> (get-in list-page-payload [:data :items])
+                                        (map #(or (:block/title %) (:title %)))
+                                        set)
+                       ref-query-payload (run-query data-dir cfg-path "uuid-ref-graph"
+                                                    "[:find ?title :in $ ?page-name :where [?p :block/name ?page-name] [?b :block/page ?p] [?b :block/title ?title]]"
+                                                    (pr-str [(common-util/page-name-sanity-lc "Home")]))
+                       titles (map first (get-in ref-query-payload [:data :result]))
+                       ref-title (some #(when (and (string? %)
+                                                   (string/includes? % (str "[[" target-uuid "]]")))
+                                         %)
+                                       titles)
+                       stop-result (run-cli ["server" "stop" "--repo" "uuid-ref-graph"] data-dir cfg-path)
+                       stop-payload (parse-json-output stop-result)]
+                 (is (string? target-uuid))
+                 (is (= 0 (:exit-code add-block-result)))
+                 (is (= "ok" (:status add-block-payload)))
+                 (is (not (contains? page-titles target-uuid)))
+                 (is (string? ref-title))
+                 (is (string/includes? ref-title (str "[[" target-uuid "]]")))
+                 (is (= "ok" (:status stop-payload)))
+                 (done))
+               (p/catch (fn [e]
+                          (is false (str "unexpected error: " e))
+                          (done)))))))
+
+(deftest test-cli-add-block-missing-uuid-ref-errors
+  (async done
+         (let [data-dir (node-helper/create-tmp-dir "db-worker-missing-uuid-ref")]
+           (-> (p/let [cfg-path (node-path/join (node-helper/create-tmp-dir "cli") "cli.edn")
+                       _ (fs/writeFileSync cfg-path "{:output-format :json}")
+                       _ (run-cli ["graph" "create" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path)
+                       _ (run-cli ["--repo" "missing-uuid-ref-graph" "add" "page" "--page" "Home"] data-dir cfg-path)
+                       missing-uuid (str (random-uuid))
+                       add-block-result (run-cli ["--repo" "missing-uuid-ref-graph"
+                                                  "add" "block"
+                                                  "--target-page-name" "Home"
+                                                  "--content" (str "See [[" missing-uuid "]]")]
+                                                 data-dir cfg-path)
+                       add-block-payload (parse-json-output add-block-result)
+                       stop-result (run-cli ["server" "stop" "--repo" "missing-uuid-ref-graph"] data-dir cfg-path)
+                       stop-payload (parse-json-output stop-result)]
+                 (is (= 1 (:exit-code add-block-result)))
+                 (is (= "error" (:status add-block-payload)))
+                 (is (string/includes? (get-in add-block-payload [:error :message]) missing-uuid))
+                 (is (= "ok" (:status stop-payload)))
                  (done))
                (p/catch (fn [e]
                           (is false (str "unexpected error: " e))