diff --git a/docs/adr/0016-markdown-mirror.md b/docs/adr/0016-markdown-mirror.md index 608d2b9793..cf08799419 100644 --- a/docs/adr/0016-markdown-mirror.md +++ b/docs/adr/0016-markdown-mirror.md @@ -28,9 +28,11 @@ builds do not have the same graph-directory filesystem guarantees. 4. For a graph at `~/logseq/graphs/graph-xxx`, mirror files are written under: - `~/logseq/graphs/graph-xxx/mirror/markdown/journals/` - `~/logseq/graphs/graph-xxx/mirror/markdown/pages/` -5. Markdown Mirror is derived output. The DB remains the source of truth. -6. Files under `mirror/markdown/**` must be ignored by graph import, file - watchers, and graph parsing so the mirror never feeds back into the graph. +5. Markdown Mirror is derived output. The DB remains the source of truth for + unsupported or ambiguous edits. +6. Files under `mirror/markdown/**` must be ignored by the normal graph import, + file watchers, and graph parsing. The only path from mirror files back into + the graph is the dedicated Markdown Mirror watcher described below. 7. The feature is not available in browser or mobile builds, even if a stale setting value exists. 8. Settings exposes an explicit "Regenerate full mirror" action that asks the @@ -196,16 +198,16 @@ builds do not have the same graph-directory filesystem guarantees. 8. Full regeneration is an explicit Settings action. The renderer only sends a worker request; page selection, rendering, and filesystem writes stay in the DB worker. -9. Enabling the setting starts incremental mirroring for subsequent page edits. - It does not implicitly run full regeneration. +9. Enabling the setting starts the Markdown Mirror watcher and runs a full + regeneration so external editors have current files immediately. ## Markdown Content 1. Reuse the existing page-to-Markdown export pipeline used by worker export APIs instead of introducing a separate renderer-side serializer. 2. The mirror output should match normal Markdown export semantics for page content. -3. Mirror files do not include Logseq-internal mirror metadata in the Markdown - body. +3. Mirror files include HTML comments for page and block identity. These + comments are the only supported identity markers for file-to-DB imports. 4. Mirror files include block and page property drawers, including user properties, with rendered property values. 5. Assets are referenced as normal exported Markdown references. This ADR does @@ -215,6 +217,36 @@ builds do not have the same graph-directory filesystem guarantees. limitation of Markdown Mirror. Do not rewrite page references to uuid-based links or relative Markdown links in this ADR. +## File-to-DB Import Contract +1. The mirror supports a constrained two-way sync path for DB graphs on Electron. + It is not a general Markdown importer. +2. The only writable structural unit is a Logseq block rendered as a `- ` list + item. New blocks must be inserted as `- ` items. +3. Indented non-list Markdown under a block belongs to that block's content. + This includes quotes, fenced code blocks, tables, and other Markdown + continuation lines. +4. Lines inside an indented fenced code block are always block content. A line + like `- not a block` inside the fence must not create a Logseq child block. +5. Unsupported top-level Markdown, for example a top-level quote or fenced code + block that is not under a `- ` item, is rejected. Rejecting is safer than + silently dropping it and then treating omitted block markers as deletes. +6. Property drawer edits from mirror files are ignored. The DB remains the + source of truth for page and block properties. +7. File deletes, moves, and directory deletes from the mirror watcher are + ignored. Page and block deletion from files is allowed only by removing block + marker/list-item pairs from an existing page file. +8. Creating a new page or journal from a new mirror file is supported only when + the path maps to `pages/.md` or `journals/YYYY_MM_DD.md` and the file + contains no page or block markers from another graph object. +9. `[[page]]` references in imported block content create missing pages and are + stored using internal id refs in the DB transaction. +10. Simple hashtag refs such as `#tag1` and page-ref hashtags such as + `#[[tag one]]` create missing tag/class pages and attach the tag to the + block. +11. The watcher suppresses Logseq's own mirror writes by comparing recent written + content, not by ignoring every file event for a path during a time window. + A real external edit that changes file content must import immediately. + ## Failure Handling 1. Filesystem and path errors fail the mirror job for the affected page. 2. Failures are logged with graph, page uuid, target path, and error details. @@ -226,8 +258,9 @@ builds do not have the same graph-directory filesystem guarantees. that graph until the graph is reopened with a valid directory. ## Non-goals -1. Markdown Mirror is not bidirectional sync. -2. Editing files in `mirror/markdown/` does not update the graph. +1. Markdown Mirror is not a general bidirectional Markdown sync engine. +2. Editing files in `mirror/markdown/` updates the graph only within the + constrained import contract above. 3. The mirror is not a backup format with guaranteed import fidelity. 4. The mirror does not replace existing graph export features. 5. The mirror does not support browser or mobile runtimes in this ADR. @@ -251,7 +284,8 @@ builds do not have the same graph-directory filesystem guarantees. - Duplicate page references such as `[[Foo]]` remain ambiguous in mirror output. - The first version does not backfill every existing page automatically when the setting is enabled; users run full regeneration explicitly. -- External edits to mirror files are overwritten by later Logseq edits. +- External edits outside the constrained import contract are rejected or + overwritten by later Logseq edits. - Property pages are intentionally absent from the mirror, so the output is not a complete DB export even though page and block property drawers are included. @@ -269,6 +303,13 @@ bb dev:test -v frontend.worker.markdown-mirror-test/same-title-pages-write-disti bb dev:test -v frontend.worker.markdown-mirror-test/page-references-remain-wiki-links-test bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-property-values-test bb dev:test -v frontend.worker.markdown-mirror-test/page-mirror-exports-page-property-values-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-file-edit-can-change-generated-block-content-immediately-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-page-ref-edit-creates-page-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-inserted-block-page-ref-creates-page-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-hashtag-edit-creates-tag-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-existing-block-preserves-continuation-markdown-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-inserted-block-preserves-continuation-markdown-test +bb dev:test -v frontend.worker.markdown-mirror-test/two-way-top-level-markdown-without-block-is-rejected-test bb dev:test -v frontend.worker.markdown-mirror-test/full-regeneration-writes-existing-non-built-in-non-property-pages-test bb dev:test -v frontend.worker.markdown-mirror-test/invalid-filename-characters-are-normalized-test bb dev:test -v frontend.worker.markdown-mirror-test/windows-reserved-filename-fails-with-diagnostic-test diff --git a/resources/package.json b/resources/package.json index 0a0fe74196..bed95cd1ec 100644 --- a/resources/package.json +++ b/resources/package.json @@ -24,6 +24,7 @@ "@js-joda/core": "3.2.0", "@modelcontextprotocol/sdk": "^1.27.1", "abort-controller": "3.0.0", + "chokidar": "4.0.3", "command-exists": "1.2.9", "diff-match-patch": "1.0.5", "electron-dl": "4.0.0", diff --git a/resources/pnpm-lock.yaml b/resources/pnpm-lock.yaml index 3adc1946a0..295acaa653 100644 --- a/resources/pnpm-lock.yaml +++ b/resources/pnpm-lock.yaml @@ -23,6 +23,9 @@ importers: abort-controller: specifier: 3.0.0 version: 3.0.0 + chokidar: + specifier: 4.0.3 + version: 4.0.3 command-exists: specifier: 1.2.9 version: 1.2.9 @@ -469,6 +472,10 @@ packages: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} engines: {node: '>=10'} + chokidar@4.0.3: + resolution: {integrity: sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==} + engines: {node: '>= 14.16.0'} + chownr@1.1.4: resolution: {integrity: sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==} @@ -1701,6 +1708,10 @@ packages: resolution: {integrity: sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==} engines: {node: '>= 6'} + readdirp@4.1.2: + resolution: {integrity: sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==} + engines: {node: '>= 14.18.0'} + real-require@0.2.0: resolution: {integrity: sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==} engines: {node: '>= 12.13.0'} @@ -2739,6 +2750,10 @@ snapshots: ansi-styles: 4.3.0 supports-color: 7.2.0 + chokidar@4.0.3: + dependencies: + readdirp: 4.1.2 + chownr@1.1.4: {} chownr@3.0.0: {} @@ -4053,6 +4068,8 @@ snapshots: string_decoder: 1.3.0 util-deprecate: 1.0.2 + readdirp@4.1.2: {} + real-require@0.2.0: {} remove-accents@0.5.0: {} diff --git a/src/main/frontend/worker/db_core.cljs b/src/main/frontend/worker/db_core.cljs index 8147dbd0c5..db5a51194c 100644 --- a/src/main/frontend/worker/db_core.cljs +++ b/src/main/frontend/worker/db_core.cljs @@ -1098,7 +1098,12 @@ (def-thread-api :thread-api/markdown-mirror-set-enabled [repo enabled?] (markdown-mirror/set-enabled! repo enabled?) - nil) + (if enabled? + (when-let [conn (worker-state/get-datascript-conn repo)] + (p/let [_ (markdown-mirror/\s*$") +(def ^:private block-marker-re #"^\s*\s*$") +(def ^:private markdown-block-re #"^(\s*)-\s?(.*)$") +(def ^:private property-line-re #"^\s*[^:\s][^:]*::\s?.*$") +(def ^:private ref-or-tag-re #"(#?)\[\[([^\[\]]+)\]\]") +(def ^:private simple-hashtag-re #"(?i)(^|\s)#([^\s#\[\]\(\),.;:'\"`]+)") +(def ^:private journal-relative-path-re #"^journals/(\d{4})_(\d{2})_(\d{2})\.md$") +(def ^:private page-relative-path-re #"^pages/(.+)\.md$") (defonce ^:private *repo->enabled? (atom {})) (defonce ^:private *repo->queued-page-jobs (atom {})) (defonce ^:private *repo->flush-timeout (atom {})) +(defonce ^:private *repo->file-watchers (atom {})) +(defonce ^:private *repo->recent-writes (atom {})) + +(declare path str (string/replace "\\" "/"))) + +(defn- markdown-relative-path? + [relative-path] + (boolean + (or (re-matches page-relative-path-re relative-path) + (re-matches journal-relative-path-re relative-path)))) + +(defn- watch-root-path + [platform* repo] + (let [relative-root (repo-mirror-dir repo)] + (if-let [f (get-in platform* [:storage :resolve-text-path])] + (f relative-root) + relative-root))) + +(defn- watcher-event-relative-path + [repo watch-root path] + (let [path' (normalize-watch-path path) + watch-root' (normalize-watch-path watch-root) + mirror-dir' (normalize-watch-path (repo-mirror-dir repo)) + prefix (str watch-root' "/") + mirror-prefix (str mirror-dir' "/") + relative-path (cond + (and watch-root' + (string/starts-with? path' prefix)) + (subs path' (count prefix)) + + (string/starts-with? path' mirror-prefix) + (subs path' (count mirror-prefix)) + + :else + path')] + (when (and relative-path (markdown-relative-path? relative-path)) + relative-path))) + +(defn- mark-recent-write! + [repo path content] + (swap! *repo->recent-writes assoc-in [repo path] {:time (common-util/time-ms) + :content content})) + +(defn- recent-written-content + [repo path ttl-ms] + (when-let [{:keys [time content]} (get-in @*repo->recent-writes [repo path])] + (if (< (- (common-util/time-ms) time) ttl-ms) + content + (do + (swap! *repo->recent-writes update repo dissoc path) + nil)))) + +(defn- parse-uuid-safe + [s] + (try + (uuid s) + (catch :default _e + nil))) + +(defn- line-level + [spaces] + (inc (quot (count spaces) 2))) + +(defn- strip-prefix-spaces + [line n] + (if (and (<= n (count line)) + (every? #(= \space %) (subs line 0 n))) + (subs line n) + line)) + +(defn- fenced-code-boundary? + [line] + (let [line (string/triml line)] + (or (string/starts-with? line "```") + (string/starts-with? line "~~~")))) + +(defn- append-block-line + [blocks idx line] + (update-in blocks [idx :title] #(str % "\n" line))) + +(defn- continuation-line + [line continuation-indent] + (let [prefix (apply str (repeat continuation-indent " "))] + (when (string/starts-with? line prefix) + (strip-prefix-spaces line continuation-indent)))) + +(defn- parse-mirror-content + [content] + (let [lines (string/split-lines (or content ""))] + (loop [[line & more] lines + pending-marker nil + page-uuid nil + stack [] + blocks [] + current-block-idx nil + continuation-indent nil + fenced-code? false] + (if (nil? line) + {:page-uuid page-uuid + :blocks blocks} + (if fenced-code? + (if (some? current-block-idx) + (let [line' (strip-prefix-spaces line continuation-indent) + fenced-code?' (if (fenced-code-boundary? line') + false + fenced-code?)] + (recur more pending-marker page-uuid stack + (append-block-line blocks current-block-idx line') + current-block-idx continuation-indent fenced-code?')) + {:error :unsupported-top-level-markdown}) + (if-let [[_ page-uuid-str] (re-matches page-marker-re line)] + (recur more nil (parse-uuid-safe page-uuid-str) stack blocks + current-block-idx continuation-indent fenced-code?) + (if-let [[_ block-uuid-str] (re-matches block-marker-re line)] + (recur more (parse-uuid-safe block-uuid-str) page-uuid stack blocks + current-block-idx continuation-indent fenced-code?) + (if-let [[_ spaces title] (re-matches markdown-block-re line)] + (let [current-level (line-level spaces) + idx (count blocks) + stack' (->> stack + (remove (fn [{:keys [level]}] (>= level current-level))) + vec) + parent-ref (:ref (peek stack')) + block {:uuid pending-marker + :idx idx + :title title + :level current-level + :parent-ref parent-ref} + stack'' (conj stack' {:level current-level + :ref (or (:uuid block) idx)}) + continuation-indent' (+ (count spaces) 2)] + (recur more nil page-uuid stack'' (conj blocks block) + idx continuation-indent' (fenced-code-boundary? title))) + (cond + (or (string/blank? line) + (re-matches property-line-re line)) + (recur more pending-marker page-uuid stack blocks + current-block-idx continuation-indent fenced-code?) + + (and (some? current-block-idx) + (some? continuation-indent) + (continuation-line line continuation-indent)) + (let [line' (continuation-line line continuation-indent)] + (recur more pending-marker page-uuid stack + (append-block-line blocks current-block-idx line') + current-block-idx continuation-indent (fenced-code-boundary? line'))) + + :else + {:error :unsupported-top-level-markdown}))))))))) + +(defn- relative-path->new-page + [relative-path] + (if-let [[_ y m d] (re-matches journal-relative-path-re relative-path)] + (let [journal-day (parse-long (str y m d))] + {:type :journal + :journal-day journal-day + :title (date-time-util/int->journal-title + journal-day + date-time-util/default-journal-title-formatter) + :uuid (common-uuid/gen-uuid :journal-page-uuid journal-day)}) + (when-let [[_ stem] (re-matches page-relative-path-re relative-path)] + (when-let [title (normalize-file-stem stem)] + {:type :page + :title title + :uuid (random-uuid)})))) + +(defn- existing-page-for-path + [db relative-path parsed-page-uuid] + (or (when-let [[_ y m d] (re-matches journal-relative-path-re relative-path)] + (let [journal-day (parse-long (str y m d))] + (first (map #(d/entity db (:e %)) + (d/datoms db :avet :block/journal-day journal-day))))) + (when-let [[_ stem] (re-matches page-relative-path-re relative-path)] + (ldb/get-page db stem)) + (when-let [page (and parsed-page-uuid + (d/entity db [:block/uuid parsed-page-uuid]))] + (when (= relative-path (page-relative-path db page)) + page)))) + +(defn- content-ref-targets + [title] + (let [title (or title "") + page-ref-targets (keep (fn [[_ prefix page-title]] + (when-not (common-util/uuid-string? page-title) + {:title page-title + :tag? (= "#" prefix)})) + (re-seq ref-or-tag-re title)) + simple-tag-targets (keep (fn [[_ _ tag-title]] + (when-not (common-util/uuid-string? tag-title) + {:title tag-title + :tag? true})) + (re-seq simple-hashtag-re title))] + (distinct (concat page-ref-targets simple-tag-targets)))) + +(defn- ensure-tag-txs + [db block-ref title page] + (cond-> [[:db/add block-ref :block/tags :logseq.class/Tag] + [:db/add block-ref :logseq.property.class/extends :logseq.class/Root] + [:db/retract block-ref :block/tags :logseq.class/Page]] + (nil? (:db/ident page)) + (conj [:db/add block-ref :db/ident (db-class/create-user-class-ident-from-name db title)]))) + +(defn- page-ref-plan + [db title planned-pages now] + (loop [[{target-title :title tag? :tag? :as target} & more] (content-ref-targets title) + planned-pages planned-pages + refs [] + tag-refs [] + page-txs []] + (if (nil? target) + {:planned-pages planned-pages + :refs refs + :page-txs page-txs + :tag-refs tag-refs + :db-title (db-content/title-ref->id-ref title refs {:replace-tag? true})} + (let [page-title target-title + page-name (common-util/page-name-sanity-lc page-title) + planned-page (get planned-pages page-name)] + (if planned-page + (let [planned-pages' (if (and tag? (not (:tag? planned-page))) + (assoc planned-pages page-name (assoc planned-page :tag? true)) + planned-pages) + page-txs' (cond-> page-txs + (and tag? (not (:tag? planned-page))) + (into (ensure-tag-txs db + [:block/uuid (:block/uuid planned-page)] + page-title + nil)))] + (recur more + planned-pages' + (conj refs planned-page) + (cond-> tag-refs tag? (conj planned-page)) + page-txs')) + (if-let [page (ldb/get-page db page-title)] + (let [ref {:block/title (:block/title page) + :block/uuid (:block/uuid page) + :tag? (or tag? + (some #(= :logseq.class/Tag (:db/ident %)) (:block/tags page)))} + page-txs' (cond-> page-txs + tag? + (into (ensure-tag-txs db (:db/id page) page-title page)))] + (recur more + (assoc planned-pages page-name ref) + (conj refs ref) + (cond-> tag-refs tag? (conj ref)) + page-txs')) + (let [page-uuid (random-uuid) + ref {:block/title page-title + :block/uuid page-uuid + :tag? tag?} + page-tx {:block/title page-title + :block/name page-name + :block/uuid page-uuid + :block/tags (if tag? + :logseq.class/Tag + :logseq.class/Page) + :block/created-at now + :block/updated-at now} + page-tx (cond-> page-tx + tag? + (assoc :db/ident (db-class/create-user-class-ident-from-name db page-title) + :logseq.property.class/extends :logseq.class/Root))] + (recur more + (assoc planned-pages page-name ref) + (conj refs ref) + (cond-> tag-refs tag? (conj ref)) + (conj page-txs page-tx))))))))) + +(defn- with-content-ref-plans + [db blocks now] + (loop [[block & more] blocks + planned-pages {} + blocks' [] + page-txs []] + (if (nil? block) + {:blocks blocks' + :page-txs page-txs} + (let [{planned-pages' :planned-pages + refs :refs + tag-refs :tag-refs + new-page-txs :page-txs + db-title :db-title} (page-ref-plan db (:title block) planned-pages now)] + (recur more + planned-pages' + (conj blocks' (assoc block + :db-title db-title + :content-ref-uuids (set (map :block/uuid refs)) + :tag-ref-uuids (set (map :block/uuid tag-refs)))) + (into page-txs new-page-txs)))))) + +(defn- page-root-blocks + [page] + (sort-by :block/order (:block/_parent page))) + +(defn- page-blocks-by-uuid + [db page] + (->> (mapcat #(ldb/get-block-and-children db (:block/uuid %)) + (page-root-blocks page)) + (filter :block/uuid) + (map (juxt :block/uuid identity)) + (into {}))) + +(defn- duplicate-marker? + [blocks] + (let [markers (keep :uuid blocks)] + (not= (count markers) (count (set markers))))) + +(defn- parsed-existing-markers-valid? + [blocks page-block-by-uuid] + (every? #(contains? page-block-by-uuid %) (keep :uuid blocks))) + +(defn- parsed-parent-valid? + [{:keys [parent-ref]} page-block-by-uuid new-uuid-by-index] + (or (nil? parent-ref) + (contains? page-block-by-uuid parent-ref) + (contains? new-uuid-by-index parent-ref))) + +(defn- new-block-tx + [page-id block page-block-by-uuid new-uuid-by-index now] + (let [block-uuid (or (:uuid block) + (get new-uuid-by-index (:idx block))) + parent-id (if-let [parent-ref (:parent-ref block)] + (if-let [parent (:db/id (get page-block-by-uuid parent-ref))] + parent + [:block/uuid (get new-uuid-by-index parent-ref)]) + page-id)] + {:block/uuid block-uuid + :block/title (or (:db-title block) (:title block)) + :block/page page-id + :block/parent parent-id + :block/order (db-order/gen-key) + :block/created-at now + :block/updated-at now})) + +(defn- top-level-delete-uuids + [db delete-uuids] + (let [delete-set (set delete-uuids)] + (->> delete-uuids + (remove (fn [block-uuid] + (some (fn [ancestor] + (and (not= block-uuid (:block/uuid ancestor)) + (contains? delete-set (:block/uuid ancestor)))) + (ldb/get-block-parents db block-uuid {:depth 100})))) + vec))) + +(defn- update-existing-block-tx + [block parsed-block now] + (let [title (or (:db-title parsed-block) (:title parsed-block))] + (when (not= (:block/title block) title) + {:db/id (:db/id block) + :block/title title + :block/updated-at now}))) + +(defn- title-content-ref-uuids + [title] + (if (string? title) + (set (db-content/get-matched-ids title)) + #{})) + +(defn- title-ref-uuids + [db title] + (->> (content-ref-targets title) + (keep (fn [{:keys [title]}] + (:block/uuid (ldb/get-page db title)))) + set)) + +(defn- title-tag-ref-uuids + [db title] + (->> (content-ref-targets title) + (keep (fn [{:keys [title tag?]}] + (when tag? + (:block/uuid (ldb/get-page db title))))) + set)) + +(defn- content-ref-add-txs + [block-ref block] + (map (fn [ref-uuid] + [:db/add block-ref :block/refs [:block/uuid ref-uuid]]) + (:content-ref-uuids block))) + +(defn- tag-ref-add-txs + [block-ref block] + (map (fn [tag-uuid] + [:db/add block-ref :block/tags [:block/uuid tag-uuid]]) + (:tag-ref-uuids block))) + +(defn- existing-block-content-ref-txs + [db block parsed-block] + (let [old-ref-uuids (set (concat (title-content-ref-uuids (:block/title block)) + (title-ref-uuids db (:block/title block)))) + new-ref-uuids (:content-ref-uuids parsed-block) + block-id (:db/id block)] + (concat + (map (fn [ref-uuid] + [:db/retract block-id :block/refs [:block/uuid ref-uuid]]) + (remove new-ref-uuids old-ref-uuids)) + (map (fn [ref-uuid] + [:db/add block-id :block/refs [:block/uuid ref-uuid]]) + (remove old-ref-uuids new-ref-uuids))))) + +(defn- existing-block-tag-ref-txs + [db block parsed-block] + (let [old-content-ref-uuids (set (concat (title-content-ref-uuids (:block/title block)) + (title-tag-ref-uuids db (:block/title block)))) + old-tag-uuids (->> (:block/tags block) + (keep (fn [tag] + (let [tag-uuid (:block/uuid tag)] + (when (contains? old-content-ref-uuids tag-uuid) + tag-uuid)))) + set) + new-tag-uuids (:tag-ref-uuids parsed-block) + block-id (:db/id block)] + (concat + (map (fn [tag-uuid] + [:db/retract block-id :block/tags [:block/uuid tag-uuid]]) + (remove new-tag-uuids old-tag-uuids)) + (map (fn [tag-uuid] + [:db/add block-id :block/tags [:block/uuid tag-uuid]]) + (remove old-tag-uuids new-tag-uuids))))) + +(defn- import-tx-meta + [relative-path outliner-ops] + {:outliner-op :markdown-mirror/import-page + :markdown-mirror/source :file + :markdown-mirror/path relative-path + :outliner-ops (vec outliner-ops)}) + +(defn- transact-import! + [conn tx-data relative-path outliner-ops] + (when (seq tx-data) + (ldb/transact! conn tx-data (import-tx-meta relative-path outliner-ops)))) + +(defn- materialize-markers-content + [db page] + (let [block-lines + (letfn [(render-block [block level] + (let [indent (apply str (repeat (dec level) " ")) + children (sort-by :block/order (:block/_parent block))] + (concat [(str "") + (str indent "- " (:block/title block))] + (mapcat #(render-block % (inc level)) children))))] + (mapcat #(render-block % 1) (page-root-blocks page)))] + (string/join "\n" + (cons (str "") + block-lines)))) + +(defn- block-marker-lines + [db page] + (->> (mapcat #(ldb/get-block-and-children db (:block/uuid %)) + (page-root-blocks page)) + (map (fn [block] {:uuid (:block/uuid block)})))) + +(defn- add-markers-to-rendered-content + [db page content] + (let [markers (block-marker-lines db page)] + (loop [[line & more] (string/split-lines (or content "")) + [marker & more-markers] markers + lines [(str "")]] + (if (nil? line) + (string/join "\n" lines) + (if (re-matches markdown-block-re line) + (recur more + more-markers + (cond-> lines + marker + (conj (str "")) + true + (conj line))) + (recur more + (cons marker more-markers) + (conj lines line))))))) + +(defn- new-page relative-path)] + (cond + (nil? page-plan) + (p/resolved {:status :error + :reason :unsupported-path}) + + (:page-uuid parsed) + (p/resolved {:status :error + :reason :new-file-has-page-marker}) + + (seq (keep :uuid (:blocks parsed))) + (p/resolved {:status :error + :reason :new-file-has-block-marker}) + + (existing-page-for-path db relative-path nil) + (p/resolved {:status :error + :reason :page-already-exists}) + + :else + (let [now (common-util/time-ms) + {parsed-blocks :blocks + ref-page-txs :page-txs} (with-content-ref-plans db (:blocks parsed) now) + page-uuid (:uuid page-plan) + page-tx (cond-> {:block/title (:title page-plan) + :block/name (common-util/page-name-sanity-lc (:title page-plan)) + :block/uuid page-uuid + :block/tags :logseq.class/Page + :block/created-at now + :block/updated-at now} + (= :journal (:type page-plan)) + (assoc :block/journal-day (:journal-day page-plan) + :block/tags :logseq.class/Journal)) + new-uuid-by-index (->> parsed-blocks + (map-indexed (fn [idx _] [idx (random-uuid)])) + (into {})) + block-txs (map-indexed + (fn [_idx block] + (new-block-tx [:block/uuid page-uuid] block {} new-uuid-by-index now)) + parsed-blocks) + block-ref-txs (mapcat (fn [block] + (content-ref-add-txs + [:block/uuid (get new-uuid-by-index (:idx block))] + block)) + parsed-blocks) + block-tag-txs (mapcat (fn [block] + (tag-ref-add-txs + [:block/uuid (get new-uuid-by-index (:idx block))] + block)) + parsed-blocks) + tx-data (vec (concat [page-tx] ref-page-txs block-txs block-ref-txs block-tag-txs)) + page-options (cond-> {:redirect? false + :uuid page-uuid} + (= :journal (:type page-plan)) + (assoc :journal? true + :journal-day (:journal-day page-plan))) + outliner-ops [[:create-page [(:title page-plan) page-options]] + [:insert-blocks [(vec block-txs) page-uuid {:sibling? false}]]]] + (transact-import! conn tx-data relative-path outliner-ops) + (p/let [_ (> (:blocks parsed) + (keep-indexed (fn [idx block] + (when (nil? (:uuid block)) + [idx (random-uuid)]))) + (into {})) + now (common-util/time-ms) + {parsed-blocks :blocks + ref-page-txs :page-txs} (with-content-ref-plans db (:blocks parsed) now)] + (if-not (every? #(parsed-parent-valid? % page-block-by-uuid new-uuids) + parsed-blocks) + (p/resolved {:status :error + :reason :unresolved-parent}) + (let [seen-markers (set (keep :uuid parsed-blocks)) + existing-uuids (set (keys page-block-by-uuid)) + delete-uuids (top-level-delete-uuids db (remove seen-markers existing-uuids)) + save-txs (keep (fn [block] + (when-let [uuid (:uuid block)] + (update-existing-block-tx (get page-block-by-uuid uuid) block now))) + parsed-blocks) + save-ref-txs (mapcat (fn [block] + (when-let [uuid (:uuid block)] + (existing-block-content-ref-txs + db + (get page-block-by-uuid uuid) + block))) + parsed-blocks) + save-tag-txs (mapcat (fn [block] + (when-let [uuid (:uuid block)] + (existing-block-tag-ref-txs + db + (get page-block-by-uuid uuid) + block))) + parsed-blocks) + new-blocks (keep-indexed (fn [idx block] + (when (nil? (:uuid block)) + (new-block-tx (:db/id page) + block + page-block-by-uuid + new-uuids + now))) + parsed-blocks) + new-block-ref-txs (mapcat (fn [block] + (when (nil? (:uuid block)) + (content-ref-add-txs + [:block/uuid (get new-uuids (:idx block))] + block))) + parsed-blocks) + new-block-tag-txs (mapcat (fn [block] + (when (nil? (:uuid block)) + (tag-ref-add-txs + [:block/uuid (get new-uuids (:idx block))] + block))) + parsed-blocks) + delete-txs (mapcat (fn [block-uuid] + [[:db/retractEntity (:db/id (get page-block-by-uuid block-uuid))]]) + delete-uuids) + tx-data (vec (concat ref-page-txs + save-txs + save-ref-txs + save-tag-txs + new-blocks + new-block-ref-txs + new-block-tag-txs + delete-txs)) + outliner-ops (cond-> [] + (seq save-txs) + (conj [:save-block [(first save-txs) {}]]) + (seq new-blocks) + (conj [:insert-blocks [(vec new-blocks) (:block/uuid page) {:sibling? false}]]) + (seq delete-uuids) + (conj [:delete-blocks [(vec delete-uuids) {}]]))] + (if (seq tx-data) + (do + (transact-import! conn tx-data relative-path outliner-ops) + (p/let [_ (when (or (seq new-blocks) (seq delete-uuids)) + (file-watchers repo)] + (when (.-close watcher) + (.close watcher))) + (swap! *repo->file-watchers dissoc repo) + (swap! *repo->recent-writes dissoc repo) + nil) + +(defn- chokidar-watch! + [watch-path opts] + (let [chokidar (js/require "chokidar")] + (.watch chokidar watch-path (clj->js opts)))) + +(defn- register-watch-handler! + [watcher event handler] + (.on ^js watcher event handler) + watcher) + +(defn file-watchers assoc repo watcher) + (letfn [(handle-result [relative-path event promise] + (-> promise + (p/catch (fn [error] + (log/error :markdown-mirror/import-file-event-failed + {:repo repo + :relative-path relative-path + :event event + :error error}))))) + (handle-path! [event path] + (when-let [relative-path (watcher-event-relative-path repo watch-root path)] + (let [storage-path (mirror-path repo relative-path) + ttl-ms (or ignored-recent-write-ms 1000)] + (if (= :changed event) + (handle-result + relative-path + event + (p/let [content (content + (add-markers-to-rendered-content db - (:block/uuid page) - {:include-page-properties? true} - {:export-bullet-indentation (or (:export-bullet-indentation options) " ") - :date-formatter (:date-formatter options)})) + page + (common-file/block->content + db + (:block/uuid page) + {:include-page-properties? true} + {:export-bullet-indentation (or (:export-bullet-indentation options) " ") + :date-formatter (:date-formatter options)}))) (defn- mirrorable-page? [page] @@ -195,13 +979,14 @@ (str (:block/uuid page))])))) (defn- page :block/_page first)) +(defn- block-by-title [db title] + (->> (d/datoms db :avet :block/title title) + (map #(d/entity db (:e %))) + (filter :block/page) + first)) + +(defn- block-title-includes? + [db block-uuid s] + (string/includes? (:block/title (d/entity db [:block/uuid block-uuid])) s)) + +(defn- page-marker [uuid] + (str "")) + +(defn- block-marker [uuid] + (str "")) + (defn- (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ ( (markdown-mirror/ (markdown-mirror/ (p/let [_ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (p/let [_ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ quoted\n" + " ```clojure\n" + " - not a child\n" + " ```")] + (-> (markdown-mirror/ quote")] + (-> (markdown-mirror/ quote"))))) + (p/catch (fn [e] (is false (str "unexpected error: " e)))) + (p/finally done))))) + +(deftest two-way-top-level-markdown-without-block-is-rejected-test + (async done + (let [page-uuid #uuid "99999999-9999-4999-8999-999999999985" + block-uuid #uuid "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaa7" + conn (db-test/create-conn-with-blocks + {:pages-and-blocks [{:page {:block/title "Top Level Markdown" + :block/uuid page-uuid} + :blocks [{:block/title "existing" + :block/uuid block-uuid}]}]}) + content (str (page-marker page-uuid) "\n" + "> top-level quote")] + (-> (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (markdown-mirror/ (p/let [start-result (markdown-mirror/