From 8de088e199ae84b1087bcc7262163f3a1d9a672e Mon Sep 17 00:00:00 2001 From: Tienson Qin Date: Tue, 5 May 2026 22:46:21 +0800 Subject: [PATCH] feat: sync markdown mirror edits Add Chokidar-backed Markdown Mirror file watching in the DB worker and start the watcher when Markdown Mirror is enabled. Regenerate mirror files on enable so external editors see current content immediately. Import a constrained set of mirror edits back into DB graphs: existing block content edits, new block insertion, block removal by omitted markers, page and journal creation from new files, page refs, and inline tags. Ignore mirror property edits and dangerous file delete/move events. Keep the importer strict: only Logseq '- ' list items are writable block boundaries, indented Markdown stays as block content, fenced code lines cannot create blocks, and unsupported top-level Markdown is rejected. Update ADR 0016 with this contract. Verified with: rtk bb dev:test -v frontend.worker.markdown-mirror-test; rtk bb dev:test -v frontend.worker.db-core-test/db-core-registers-all-thread-apis-test; Chokidar require smoke test from resources/. --- docs/adr/0016-markdown-mirror.md | 61 +- resources/package.json | 1 + resources/pnpm-lock.yaml | 17 + src/main/frontend/worker/db_core.cljs | 7 +- src/main/frontend/worker/markdown_mirror.cljs | 809 +++++++++++++++++- src/main/frontend/worker/platform/node.cljs | 2 + .../frontend/worker/markdown_mirror_test.cljs | 584 ++++++++++++- 7 files changed, 1425 insertions(+), 56 deletions(-) diff --git a/docs/adr/0016-markdown-mirror.md b/docs/adr/0016-markdown-mirror.md index 608d2b9793..cf08799419 100644 --- a/docs/adr/0016-markdown-mirror.md +++ b/docs/adr/0016-markdown-mirror.md @@ -28,9 +28,11 @@ builds do not have the same graph-directory filesystem guarantees. 4. For a graph at `~/logseq/graphs/graph-xxx`, mirror files are written under: - `~/logseq/graphs/graph-xxx/mirror/markdown/journals/` - `~/logseq/graphs/graph-xxx/mirror/markdown/pages/` -5. Markdown Mirror is derived output. The DB remains the source of truth. -6. Files under `mirror/markdown/**` must be ignored by graph import, file - watchers, and graph parsing so the mirror never feeds back into the graph. +5. Markdown Mirror is derived output. The DB remains the source of truth for + unsupported or ambiguous edits. +6. Files under `mirror/markdown/**` must be ignored by the normal graph import, + file watchers, and graph parsing. The only path from mirror files back into + the graph is the dedicated Markdown Mirror watcher described below. 7. The feature is not available in browser or mobile builds, even if a stale setting value exists. 8. Settings exposes an explicit "Regenerate full mirror" action that asks the @@ -196,16 +198,16 @@ builds do not have the same graph-directory filesystem guarantees. 8. Full regeneration is an explicit Settings action. The renderer only sends a worker request; page selection, rendering, and filesystem writes stay in the DB worker. -9. Enabling the setting starts incremental mirroring for subsequent page edits. - It does not implicitly run full regeneration. +9. Enabling the setting starts the Markdown Mirror watcher and runs a full + regeneration so external editors have current files immediately. ## Markdown Content 1. Reuse the existing page-to-Markdown export pipeline used by worker export APIs instead of introducing a separate renderer-side serializer. 2. The mirror output should match normal Markdown export semantics for page content. -3. Mirror files do not include Logseq-internal mirror metadata in the Markdown - body. +3. Mirror files include HTML comments for page and block identity. These + comments are the only supported identity markers for file-to-DB imports. 4. Mirror files include block and page property drawers, including user properties, with rendered property values. 5. Assets are referenced as normal exported Markdown references. This ADR does @@ -215,6 +217,36 @@ builds do not have the same graph-directory filesystem guarantees. limitation of Markdown Mirror. Do not rewrite page references to uuid-based links or relative Markdown links in this ADR. +## File-to-DB Import Contract +1. The mirror supports a constrained two-way sync path for DB graphs on Electron. + It is not a general Markdown importer. +2. The only writable structural unit is a Logseq block rendered as a `- ` list + item. New blocks must be inserted as `- ` items. +3. Indented non-list Markdown under a block belongs to that block's content. + This includes quotes, fenced code blocks, tables, and other Markdown + continuation lines. +4. Lines inside an indented fenced code block are always block content. A line + like `- not a block` inside the fence must not create a Logseq child block. +5. Unsupported top-level Markdown, for example a top-level quote or fenced code + block that is not under a `- ` item, is rejected. Rejecting is safer than + silently dropping it and then treating omitted block markers as deletes. +6. Property drawer edits from mirror files are ignored. The DB remains the + source of truth for page and block properties. +7. File deletes, moves, and directory deletes from the mirror watcher are + ignored. Page and block deletion from files is allowed only by removing block + marker/list-item pairs from an existing page file. +8. Creating a new page or journal from a new mirror file is supported only when + the path maps to `pages/.md` or `journals/YYYY_MM_DD.md` and the file + contains no page or block markers from another graph object. +9. `[[page]]` references in imported block content create missing pages and are + stored using internal id refs in the DB transaction. +10. Simple hashtag refs such as `#tag1` and page-ref hashtags such as + `#[[tag one]]` create missing tag/class pages and attach the tag to the + block. +11. The watcher suppresses Logseq's own mirror writes by comparing recent written + content, not by ignoring every file event for a path during a time window. + A real external edit that changes file content must import immediately. + ## Failure Handling 1. Filesystem and path errors fail the mirror job for the affected page. 2. Failures are logged with graph, page uuid, target path, and error details. @@ -226,8 +258,9 @@ builds do not have the same graph-directory filesystem guarantees. that graph until the graph is reopened with a valid directory. ## Non-goals -1. Markdown Mirror is not bidirectional sync. -2. Editing files in `mirror/markdown/` does not update the graph. +1. Markdown Mirror is not a general bidirectional Markdown sync engine. +2. Editing files in `mirror/markdown/` updates the graph only within the + constrained import contract above. 3. The mirror is not a backup format with guaranteed import fidelity. 4. The mirror does not replace existing graph export features. 5. The mirror does not support browser or mobile runtimes in this ADR. @@ -251,7 +284,8 @@ builds do not have the same graph-directory filesystem guarantees. - Duplicate page references such as `[[Foo]]` remain ambiguous in mirror output. - The first version does not backfill every existing page automatically when the setting is enabled; users run full regeneration explicitly. -- External edits to mirror files are overwritten by later Logseq edits. +- External edits outside the constrained import contract are rejected or + overwritten by later Logseq edits. - Property pages are intentionally absent from the mirror, so the output is not a complete DB export even though page and block property drawers are included. @@ -269,6 +303,13 @@ bb dev:test -v frontend.worker.markdown-mirror-test/same-title-pages-write-disti bb dev:test -v frontend.worker.markdown-mirror-test/page-references-remain-wiki-links-test bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-property-values-test bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-page-property-values-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-file-edit-can-change-generated-block-content-immediately-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-page-ref-edit-creates-page-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-inserted-block-page-ref-creates-page-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-hashtag-edit-creates-tag-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-preserves-continuation-markdown-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-inserted-block-preserves-continuation-markdown-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-top-level-markdown-without-block-is-rejected-test bb dev:test -v frontend.worker.markdown-mirror-test/full-regeneration-writes-existing-non-built-in-non-property-pages-test bb dev:test -v frontend.worker.markdown-mirror-test/invalid-filename-characters-are-normalized-test bb dev:test -v frontend.worker.markdown-mirror-test/windows-reserved-filename-fails-with-diagnostic-test diff --git a/resources/package.json b/resources/package.json index 0a0fe74196..bed95cd1ec 100644 --- a/resources/package.json +++ b/resources/package.json @@ -24,6 +24,7 @@ "@js-joda/core": "3.2.0", "@modelcontextprotocol/sdk": "^1.27.1", "abort-controller": "3.0.0", + "chokidar": "4.0.3", "command-exists": "1.2.9", "diff-match-patch": "1.0.5", "electron-dl": "4.0.0", diff --git a/resources/pnpm-lock.yaml b/resources/pnpm-lock.yaml index 3adc1946a0..295acaa653 100644 --- a/resources/pnpm-lock.yaml +++ b/resources/pnpm-lock.yaml @@ -23,6 +23,9 @@ importers: abort-controller: specifier: 3.0.0 version: 3.0.0 + chokidar: + specifier: 4.0.3 + version: 4.0.3 command-exists: specifier: 1.2.9 version: 1.2.9 @@ -469,6 +472,10 @@ packages: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} + chokidar@4.0.3: + resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} + engines: {node: '>= 14.16.0'} + chownr@1.1.4: resolution: {integrity: sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==} @@ -1701,6 +1708,10 @@ packages: resolution: {integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==} engines: {node: '>= 6'} + readdirp@4.1.2: + resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} + engines: {node: '>= 14.18.0'} + real-require@0.2.0: resolution: {integrity: sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==} engines: {node: '>= 12.13.0'} @@ -2739,6 +2750,10 @@ snapshots: ansi-styles: 4.3.0 supports-color: 7.2.0 + chokidar@4.0.3: + dependencies: + readdirp: 4.1.2 + chownr@1.1.4: {} chownr@3.0.0: {} @@ -4053,6 +4068,8 @@ snapshots: string_decoder: 1.3.0 util-deprecate: 1.0.2 + readdirp@4.1.2: {} + real-require@0.2.0: {} remove-accents@0.5.0: {} diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 8147dbd0c5..db5a51194c 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -1098,7 +1098,12 @@ (def-thread-api :thread-api/markdown-mirror-set-enabled [repo enabled?] (markdown-mirror/set-enabled! repo enabled?) - nil) + (if enabled? + (when-let [conn (worker-state/get-datascript-conn repo)] + (p/let [_ (markdown-mirror/\s*$") +(def ^:private block-marker-re #"^\s*\s*$") +(def ^:private markdown-block-re #"^(\s*)-\s?(.*)$") +(def ^:private property-line-re #"^\s*[^:\s][^:]*::\s?.*$") +(def ^:private ref-or-tag-re #"(#?)\[\[([^\[\]]+)\]\]") +(def ^:private simple-hashtag-re #"(?i)(^|\s)#([^\s#\[\]\(\),.;:'\"`]+)") +(def ^:private journal-relative-path-re #"^journals/(\d{4})_(\d{2})_(\d{2})\.md$") +(def ^:private page-relative-path-re #"^pages/(.+)\.md$") (defonce ^:private *repo->enabled? (atom {})) (defonce ^:private *repo->queued-page-jobs (atom {})) (defonce ^:private *repo->flush-timeout (atom {})) +(defonce ^:private *repo->file-watchers (atom {})) +(defonce ^:private *repo->recent-writes (atom {})) + +(declare path str (string/replace "\\" "/"))) + +(defn- markdown-relative-path? + [relative-path] + (boolean + (or (re-matches page-relative-path-re relative-path) + (re-matches journal-relative-path-re relative-path)))) + +(defn- watch-root-path + [platform* repo] + (let [relative-root (repo-mirror-dir repo)] + (if-let [f (get-in platform* [:storage :resolve-text-path])] + (f relative-root) + relative-root))) + +(defn- watcher-event-relative-path + [repo watch-root path] + (let [path' (normalize-watch-path path) + watch-root' (normalize-watch-path watch-root) + mirror-dir' (normalize-watch-path (repo-mirror-dir repo)) + prefix (str watch-root' "/") + mirror-prefix (str mirror-dir' "/") + relative-path (cond + (and watch-root' + (string/starts-with? path' prefix)) + (subs path' (count prefix)) + + (string/starts-with? path' mirror-prefix) + (subs path' (count mirror-prefix)) + + :else + path')] + (when (and relative-path (markdown-relative-path? relative-path)) + relative-path))) + +(defn- mark-recent-write! + [repo path content] + (swap! *repo->recent-writes assoc-in [repo path] {:time (common-util/time-ms) + :content content})) + +(defn- recent-written-content + [repo path ttl-ms] + (when-let [{:keys [time content]} (get-in @*repo->recent-writes [repo path])] + (if (< (- (common-util/time-ms) time) ttl-ms) + content + (do + (swap! *repo->recent-writes update repo dissoc path) + nil)))) + +(defn- parse-uuid-safe + [s] + (try + (uuid s) + (catch :default _e + nil))) + +(defn- line-level + [spaces] + (inc (quot (count spaces) 2))) + +(defn- strip-prefix-spaces + [line n] + (if (and (<= n (count line)) + (every? #(= \space %) (subs line 0 n))) + (subs line n) + line)) + +(defn- fenced-code-boundary? + [line] + (let [line (string/triml line)] + (or (string/starts-with? line "```") + (string/starts-with? line "~~~")))) + +(defn- append-block-line + [blocks idx line] + (update-in blocks [idx :title] #(str % "\n" line))) + +(defn- continuation-line + [line continuation-indent] + (let [prefix (apply str (repeat continuation-indent " "))] + (when (string/starts-with? line prefix) + (strip-prefix-spaces line continuation-indent)))) + +(defn- parse-mirror-content + [content] + (let [lines (string/split-lines (or content ""))] + (loop [[line & more] lines + pending-marker nil + page-uuid nil + stack [] + blocks [] + current-block-idx nil + continuation-indent nil + fenced-code? false] + (if (nil? line) + {:page-uuid page-uuid + :blocks blocks} + (if fenced-code? + (if (some? current-block-idx) + (let [line' (strip-prefix-spaces line continuation-indent) + fenced-code?' (if (fenced-code-boundary? line') + false + fenced-code?)] + (recur more pending-marker page-uuid stack + (append-block-line blocks current-block-idx line') + current-block-idx continuation-indent fenced-code?')) + {:error :unsupported-top-level-markdown}) + (if-let [[_ page-uuid-str] (re-matches page-marker-re line)] + (recur more nil (parse-uuid-safe page-uuid-str) stack blocks + current-block-idx continuation-indent fenced-code?) + (if-let [[_ block-uuid-str] (re-matches block-marker-re line)] + (recur more (parse-uuid-safe block-uuid-str) page-uuid stack blocks + current-block-idx continuation-indent fenced-code?) + (if-let [[_ spaces title] (re-matches markdown-block-re line)] + (let [current-level (line-level spaces) + idx (count blocks) + stack' (->> stack + (remove (fn [{:keys [level]}] (>= level current-level))) + vec) + parent-ref (:ref (peek stack')) + block {:uuid pending-marker + :idx idx + :title title + :level current-level + :parent-ref parent-ref} + stack'' (conj stack' {:level current-level + :ref (or (:uuid block) idx)}) + continuation-indent' (+ (count spaces) 2)] + (recur more nil page-uuid stack'' (conj blocks block) + idx continuation-indent' (fenced-code-boundary? title))) + (cond + (or (string/blank? line) + (re-matches property-line-re line)) + (recur more pending-marker page-uuid stack blocks + current-block-idx continuation-indent fenced-code?) + + (and (some? current-block-idx) + (some? continuation-indent) + (continuation-line line continuation-indent)) + (let [line' (continuation-line line continuation-indent)] + (recur more pending-marker page-uuid stack + (append-block-line blocks current-block-idx line') + current-block-idx continuation-indent (fenced-code-boundary? line'))) + + :else + {:error :unsupported-top-level-markdown}))))))))) + +(defn- relative-path->new-page + [relative-path] + (if-let [[_ y m d] (re-matches journal-relative-path-re relative-path)] + (let [journal-day (parse-long (str y m d))] + {:type :journal + :journal-day journal-day + :title (date-time-util/int->journal-title + journal-day + date-time-util/default-journal-title-formatter) + :uuid (common-uuid/gen-uuid :journal-page-uuid journal-day)}) + (when-let [[_ stem] (re-matches page-relative-path-re relative-path)] + (when-let [title (normalize-file-stem stem)] + {:type :page + :title title + :uuid (random-uuid)})))) + +(defn- existing-page-for-path + [db relative-path parsed-page-uuid] + (or (when-let [[_ y m d] (re-matches journal-relative-path-re relative-path)] + (let [journal-day (parse-long (str y m d))] + (first (map #(d/entity db (:e %)) + (d/datoms db :avet :block/journal-day journal-day))))) + (when-let [[_ stem] (re-matches page-relative-path-re relative-path)] + (ldb/get-page db stem)) + (when-let [page (and parsed-page-uuid + (d/entity db [:block/uuid parsed-page-uuid]))] + (when (= relative-path (page-relative-path db page)) + page)))) + +(defn- content-ref-targets + [title] + (let [title (or title "") + page-ref-targets (keep (fn [[_ prefix page-title]] + (when-not (common-util/uuid-string? page-title) + {:title page-title + :tag? (= "#" prefix)})) + (re-seq ref-or-tag-re title)) + simple-tag-targets (keep (fn [[_ _ tag-title]] + (when-not (common-util/uuid-string? tag-title) + {:title tag-title + :tag? true})) + (re-seq simple-hashtag-re title))] + (distinct (concat page-ref-targets simple-tag-targets)))) + +(defn- ensure-tag-txs + [db block-ref title page] + (cond-> [[:db/add block-ref :block/tags :logseq.class/Tag] + [:db/add block-ref :logseq.property.class/extends :logseq.class/Root] + [:db/retract block-ref :block/tags :logseq.class/Page]] + (nil? (:db/ident page)) + (conj [:db/add block-ref :db/ident (db-class/create-user-class-ident-from-name db title)]))) + +(defn- page-ref-plan + [db title planned-pages now] + (loop [[{target-title :title tag? :tag? :as target} & more] (content-ref-targets title) + planned-pages planned-pages + refs [] + tag-refs [] + page-txs []] + (if (nil? target) + {:planned-pages planned-pages + :refs refs + :page-txs page-txs + :tag-refs tag-refs + :db-title (db-content/title-ref->id-ref title refs {:replace-tag? true})} + (let [page-title target-title + page-name (common-util/page-name-sanity-lc page-title) + planned-page (get planned-pages page-name)] + (if planned-page + (let [planned-pages' (if (and tag? (not (:tag? planned-page))) + (assoc planned-pages page-name (assoc planned-page :tag? true)) + planned-pages) + page-txs' (cond-> page-txs + (and tag? (not (:tag? planned-page))) + (into (ensure-tag-txs db + [:block/uuid (:block/uuid planned-page)] + page-title + nil)))] + (recur more + planned-pages' + (conj refs planned-page) + (cond-> tag-refs tag? (conj planned-page)) + page-txs')) + (if-let [page (ldb/get-page db page-title)] + (let [ref {:block/title (:block/title page) + :block/uuid (:block/uuid page) + :tag? (or tag? + (some #(= :logseq.class/Tag (:db/ident %)) (:block/tags page)))} + page-txs' (cond-> page-txs + tag? + (into (ensure-tag-txs db (:db/id page) page-title page)))] + (recur more + (assoc planned-pages page-name ref) + (conj refs ref) + (cond-> tag-refs tag? (conj ref)) + page-txs')) + (let [page-uuid (random-uuid) + ref {:block/title page-title + :block/uuid page-uuid + :tag? tag?} + page-tx {:block/title page-title + :block/name page-name + :block/uuid page-uuid + :block/tags (if tag? + :logseq.class/Tag + :logseq.class/Page) + :block/created-at now + :block/updated-at now} + page-tx (cond-> page-tx + tag? + (assoc :db/ident (db-class/create-user-class-ident-from-name db page-title) + :logseq.property.class/extends :logseq.class/Root))] + (recur more + (assoc planned-pages page-name ref) + (conj refs ref) + (cond-> tag-refs tag? (conj ref)) + (conj page-txs page-tx))))))))) + +(defn- with-content-ref-plans + [db blocks now] + (loop [[block & more] blocks + planned-pages {} + blocks' [] + page-txs []] + (if (nil? block) + {:blocks blocks' + :page-txs page-txs} + (let [{planned-pages' :planned-pages + refs :refs + tag-refs :tag-refs + new-page-txs :page-txs + db-title :db-title} (page-ref-plan db (:title block) planned-pages now)] + (recur more + planned-pages' + (conj blocks' (assoc block + :db-title db-title + :content-ref-uuids (set (map :block/uuid refs)) + :tag-ref-uuids (set (map :block/uuid tag-refs)))) + (into page-txs new-page-txs)))))) + +(defn- page-root-blocks + [page] + (sort-by :block/order (:block/_parent page))) + +(defn- page-blocks-by-uuid + [db page] + (->> (mapcat #(ldb/get-block-and-children db (:block/uuid %)) + (page-root-blocks page)) + (filter :block/uuid) + (map (juxt :block/uuid identity)) + (into {}))) + +(defn- duplicate-marker? + [blocks] + (let [markers (keep :uuid blocks)] + (not= (count markers) (count (set markers))))) + +(defn- parsed-existing-markers-valid? + [blocks page-block-by-uuid] + (every? #(contains? page-block-by-uuid %) (keep :uuid blocks))) + +(defn- parsed-parent-valid? + [{:keys [parent-ref]} page-block-by-uuid new-uuid-by-index] + (or (nil? parent-ref) + (contains? page-block-by-uuid parent-ref) + (contains? new-uuid-by-index parent-ref))) + +(defn- new-block-tx + [page-id block page-block-by-uuid new-uuid-by-index now] + (let [block-uuid (or (:uuid block) + (get new-uuid-by-index (:idx block))) + parent-id (if-let [parent-ref (:parent-ref block)] + (if-let [parent (:db/id (get page-block-by-uuid parent-ref))] + parent + [:block/uuid (get new-uuid-by-index parent-ref)]) + page-id)] + {:block/uuid block-uuid + :block/title (or (:db-title block) (:title block)) + :block/page page-id + :block/parent parent-id + :block/order (db-order/gen-key) + :block/created-at now + :block/updated-at now})) + +(defn- top-level-delete-uuids + [db delete-uuids] + (let [delete-set (set delete-uuids)] + (->> delete-uuids + (remove (fn [block-uuid] + (some (fn [ancestor] + (and (not= block-uuid (:block/uuid ancestor)) + (contains? delete-set (:block/uuid ancestor)))) + (ldb/get-block-parents db block-uuid {:depth 100})))) + vec))) + +(defn- update-existing-block-tx + [block parsed-block now] + (let [title (or (:db-title parsed-block) (:title parsed-block))] + (when (not= (:block/title block) title) + {:db/id (:db/id block) + :block/title title + :block/updated-at now}))) + +(defn- title-content-ref-uuids + [title] + (if (string? title) + (set (db-content/get-matched-ids title)) + #{})) + +(defn- title-ref-uuids + [db title] + (->> (content-ref-targets title) + (keep (fn [{:keys [title]}] + (:block/uuid (ldb/get-page db title)))) + set)) + +(defn- title-tag-ref-uuids + [db title] + (->> (content-ref-targets title) + (keep (fn [{:keys [title tag?]}] + (when tag? + (:block/uuid (ldb/get-page db title))))) + set)) + +(defn- content-ref-add-txs + [block-ref block] + (map (fn [ref-uuid] + [:db/add block-ref :block/refs [:block/uuid ref-uuid]]) + (:content-ref-uuids block))) + +(defn- tag-ref-add-txs + [block-ref block] + (map (fn [tag-uuid] + [:db/add block-ref :block/tags [:block/uuid tag-uuid]]) + (:tag-ref-uuids block))) + +(defn- existing-block-content-ref-txs + [db block parsed-block] + (let [old-ref-uuids (set (concat (title-content-ref-uuids (:block/title block)) + (title-ref-uuids db (:block/title block)))) + new-ref-uuids (:content-ref-uuids parsed-block) + block-id (:db/id block)] + (concat + (map (fn [ref-uuid] + [:db/retract block-id :block/refs [:block/uuid ref-uuid]]) + (remove new-ref-uuids old-ref-uuids)) + (map (fn [ref-uuid] + [:db/add block-id :block/refs [:block/uuid ref-uuid]]) + (remove old-ref-uuids new-ref-uuids))))) + +(defn- existing-block-tag-ref-txs + [db block parsed-block] + (let [old-content-ref-uuids (set (concat (title-content-ref-uuids (:block/title block)) + (title-tag-ref-uuids db (:block/title block)))) + old-tag-uuids (->> (:block/tags block) + (keep (fn [tag] + (let [tag-uuid (:block/uuid tag)] + (when (contains? old-content-ref-uuids tag-uuid) + tag-uuid)))) + set) + new-tag-uuids (:tag-ref-uuids parsed-block) + block-id (:db/id block)] + (concat + (map (fn [tag-uuid] + [:db/retract block-id :block/tags [:block/uuid tag-uuid]]) + (remove new-tag-uuids old-tag-uuids)) + (map (fn [tag-uuid] + [:db/add block-id :block/tags [:block/uuid tag-uuid]]) + (remove old-tag-uuids new-tag-uuids))))) + +(defn- import-tx-meta + [relative-path outliner-ops] + {:outliner-op :markdown-mirror/import-page + :markdown-mirror/source :file + :markdown-mirror/path relative-path + :outliner-ops (vec outliner-ops)}) + +(defn- transact-import! + [conn tx-data relative-path outliner-ops] + (when (seq tx-data) + (ldb/transact! conn tx-data (import-tx-meta relative-path outliner-ops)))) + +(defn- materialize-markers-content + [db page] + (let [block-lines + (letfn [(render-block [block level] + (let [indent (apply str (repeat (dec level) " ")) + children (sort-by :block/order (:block/_parent block))] + (concat [(str "") + (str indent "- " (:block/title block))] + (mapcat #(render-block % (inc level)) children))))] + (mapcat #(render-block % 1) (page-root-blocks page)))] + (string/join "\n" + (cons (str "") + block-lines)))) + +(defn- block-marker-lines + [db page] + (->> (mapcat #(ldb/get-block-and-children db (:block/uuid %)) + (page-root-blocks page)) + (map (fn [block] {:uuid (:block/uuid block)})))) + +(defn- add-markers-to-rendered-content + [db page content] + (let [markers (block-marker-lines db page)] + (loop [[line & more] (string/split-lines (or content "")) + [marker & more-markers] markers + lines [(str "")]] + (if (nil? line) + (string/join "\n" lines) + (if (re-matches markdown-block-re line) + (recur more + more-markers + (cond-> lines + marker + (conj (str "")) + true + (conj line))) + (recur more + (cons marker more-markers) + (conj lines line))))))) + +(defn- new-page relative-path)] + (cond + (nil? page-plan) + (p/resolved {:status :error + :reason :unsupported-path}) + + (:page-uuid parsed) + (p/resolved {:status :error + :reason :new-file-has-page-marker}) + + (seq (keep :uuid (:blocks parsed))) + (p/resolved {:status :error + :reason :new-file-has-block-marker}) + + (existing-page-for-path db relative-path nil) + (p/resolved {:status :error + :reason :page-already-exists}) + + :else + (let [now (common-util/time-ms) + {parsed-blocks :blocks + ref-page-txs :page-txs} (with-content-ref-plans db (:blocks parsed) now) + page-uuid (:uuid page-plan) + page-tx (cond-> {:block/title (:title page-plan) + :block/name (common-util/page-name-sanity-lc (:title page-plan)) + :block/uuid page-uuid + :block/tags :logseq.class/Page + :block/created-at now + :block/updated-at now} + (= :journal (:type page-plan)) + (assoc :block/journal-day (:journal-day page-plan) + :block/tags :logseq.class/Journal)) + new-uuid-by-index (->> parsed-blocks + (map-indexed (fn [idx _] [idx (random-uuid)])) + (into {})) + block-txs (map-indexed + (fn [_idx block] + (new-block-tx [:block/uuid page-uuid] block {} new-uuid-by-index now)) + parsed-blocks) + block-ref-txs (mapcat (fn [block] + (content-ref-add-txs + [:block/uuid (get new-uuid-by-index (:idx block))] + block)) + parsed-blocks) + block-tag-txs (mapcat (fn [block] + (tag-ref-add-txs + [:block/uuid (get new-uuid-by-index (:idx block))] + block)) + parsed-blocks) + tx-data (vec (concat [page-tx] ref-page-txs block-txs block-ref-txs block-tag-txs)) + page-options (cond-> {:redirect? false + :uuid page-uuid} + (= :journal (:type page-plan)) + (assoc :journal? true + :journal-day (:journal-day page-plan))) + outliner-ops [[:create-page [(:title page-plan) page-options]] + [:insert-blocks [(vec block-txs) page-uuid {:sibling? false}]]]] + (transact-import! conn tx-data relative-path outliner-ops) + (p/let [_ (> (:blocks parsed) + (keep-indexed (fn [idx block] + (when (nil? (:uuid block)) + [idx (random-uuid)]))) + (into {})) + now (common-util/time-ms) + {parsed-blocks :blocks + ref-page-txs :page-txs} (with-content-ref-plans db (:blocks parsed) now)] + (if-not (every? #(parsed-parent-valid? % page-block-by-uuid new-uuids) + parsed-blocks) + (p/resolved {:status :error + :reason :unresolved-parent}) + (let [seen-markers (set (keep :uuid parsed-blocks)) + existing-uuids (set (keys page-block-by-uuid)) + delete-uuids (top-level-delete-uuids db (remove seen-markers existing-uuids)) + save-txs (keep (fn [block] + (when-let [uuid (:uuid block)] + (update-existing-block-tx (get page-block-by-uuid uuid) block now))) + parsed-blocks) + save-ref-txs (mapcat (fn [block] + (when-let [uuid (:uuid block)] + (existing-block-content-ref-txs + db + (get page-block-by-uuid uuid) + block))) + parsed-blocks) + save-tag-txs (mapcat (fn [block] + (when-let [uuid (:uuid block)] + (existing-block-tag-ref-txs + db + (get page-block-by-uuid uuid) + block))) + parsed-blocks) + new-blocks (keep-indexed (fn [idx block] + (when (nil? (:uuid block)) + (new-block-tx (:db/id page) + block + page-block-by-uuid + new-uuids + now))) + parsed-blocks) + new-block-ref-txs (mapcat (fn [block] + (when (nil? (:uuid block)) + (content-ref-add-txs + [:block/uuid (get new-uuids (:idx block))] + block))) + parsed-blocks) + new-block-tag-txs (mapcat (fn [block] + (when (nil? (:uuid block)) + (tag-ref-add-txs + [:block/uuid (get new-uuids (:idx block))] + block))) + parsed-blocks) + delete-txs (mapcat (fn [block-uuid] + [[:db/retractEntity (:db/id (get page-block-by-uuid block-uuid))]]) + delete-uuids) + tx-data (vec (concat ref-page-txs + save-txs + save-ref-txs + save-tag-txs + new-blocks + new-block-ref-txs + new-block-tag-txs + delete-txs)) + outliner-ops (cond-> [] + (seq save-txs) + (conj [:save-block [(first save-txs) {}]]) + (seq new-blocks) + (conj [:insert-blocks [(vec new-blocks) (:block/uuid page) {:sibling? false}]]) + (seq delete-uuids) + (conj [:delete-blocks [(vec delete-uuids) {}]]))] + (if (seq tx-data) + (do + (transact-import! conn tx-data relative-path outliner-ops) + (p/let [_ (when (or (seq new-blocks) (seq delete-uuids)) + (file-watchers repo)] + (when (.-close watcher) + (.close watcher))) + (swap! *repo->file-watchers dissoc repo) + (swap! *repo->recent-writes dissoc repo) + nil) + +(defn- chokidar-watch! + [watch-path opts] + (let [chokidar (js/require "chokidar")] + (.watch chokidar watch-path (clj->js opts)))) + +(defn- register-watch-handler! + [watcher event handler] + (.on ^js watcher event handler) + watcher) + +(defn file-watchers assoc repo watcher) + (letfn [(handle-result [relative-path event promise] + (-> promise + (p/catch (fn [error] + (log/error :markdown-mirror/import-file-event-failed + {:repo repo + :relative-path relative-path + :event event + :error error}))))) + (handle-path! [event path] + (when-let [relative-path (watcher-event-relative-path repo watch-root path)] + (let [storage-path (mirror-path repo relative-path) + ttl-ms (or ignored-recent-write-ms 1000)] + (if (= :changed event) + (handle-result + relative-path + event + (p/let [content (content + (add-markers-to-rendered-content db - (:block/uuid page) - {:include-page-properties? true} - {:export-bullet-indentation (or (:export-bullet-indentation options) " ") - :date-formatter (:date-formatter options)})) + page + (common-file/block->content + db + (:block/uuid page) + {:include-page-properties? true} + {:export-bullet-indentation (or (:export-bullet-indentation options) " ") + :date-formatter (:date-formatter options)}))) (defn- mirrorable-page? [page] @@ -195,13 +979,14 @@ (str (:block/uuid page))])))) (defn- page :block/_page first)) +(defn- block-by-title [db title] + (->> (d/datoms db :avet :block/title title) + (map #(d/entity db (:e %))) + (filter :block/page) + first)) + +(defn- block-title-includes? + [db block-uuid s] + (string/includes? (:block/title (d/entity db [:block/uuid block-uuid])) s)) + +(defn- page-marker [uuid] + (str "")) + +(defn- block-marker [uuid] + (str "")) + (defn- (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ ( (markdown-mirror/ (markdown-mirror/ (p/let [_ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (p/let [_ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ quoted\n" + " ```clojure\n" + " - not a child\n" + " ```")] + (-> (markdown-mirror/ quote")] + (-> (markdown-mirror/ quote"))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest two-way-top-level-markdown-without-block-is-rejected-test + (async done + (let [page-uuid #uuid "99999999-9999-4999-8999-999999999985" + block-uuid #uuid "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaa7" + conn (db-test/create-conn-with-blocks + {:pages-and-blocks [{:page {:block/title "Top Level Markdown" + :block/uuid page-uuid} + :blocks [{:block/title "existing" + :block/uuid block-uuid}]}]}) + content (str (page-marker page-uuid) "\n" + "> top-level quote")] + (-> (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (p/let [start-result (markdown-mirror/