Commit Graph

3732 Commits

Author SHA1 Message Date
Gabriel Horner
f28e001b5a enhance: ensure export EDN is idempotent across import and export
An EDN export should consistently produce the same EDN when imported
into a new graph and re-exported. This adds initial fn, cli option and
test to ensure this. There are some known TODOs. diff-graphs script was
removed as it was being used as a subset of the workflow provided by the
--roundtrip option
2026-02-18 15:23:41 -05:00
Gabriel Horner
1ff8a12d35 fix: export-edn shouldn't add empty :build/properties
This results in exports that aren't repeatable across exports a.k.a. roundtripable
2026-02-17 16:27:24 -05:00
Gabriel Horner
3d88cd1fd6 fix: import of multi word tags
Also fixes quirky editor bug for inline multi word tags. Both bugs
reported in https://github.com/logseq/db-test/issues/724
2026-02-16 14:35:53 -05:00
Gabriel Horner
6acf799b68 fix: UI import of example graph is invalid
With #12313, we introduced relative paths which fail to create
assets and then cause invalid blocks that are unable to reference those
assets. Any user graphs that have relative paths like this will also
have their invalid blocks fixed
2026-02-13 15:03:24 -05:00
Tienson Qin
ed240ac01e fix: today journal created-at/updated-at keep updating 2026-02-13 05:04:08 +08:00
Tienson Qin
daf7fc1475 enhance: p r to add emoji reaction 2026-02-12 23:03:28 +08:00
Tienson Qin
a895e33aeb fix: reaction validate 2026-02-12 22:50:32 +08:00
Tienson Qin
4a104ec0aa fix: error log 2026-02-12 20:33:00 +08:00
Tienson Qin
64b218896a code cleanup suggested by gabriel 2026-02-12 20:08:20 +08:00
Tienson Qin
084b5193c3 fix: sync doesn't work on node adapter 2026-02-12 19:31:24 +08:00
Gabriel Horner
0d1b997fa9 fix: reaction schema results in invalid nodes 2026-02-11 15:07:00 -05:00
Tienson Qin
adff65c6bc support unencrypted graph 2026-02-12 02:15:14 +08:00
Tienson Qin
603489e379 Merge branch 'master' into feat/worker-sync 2026-02-12 00:43:42 +08:00
Gabriel Horner
e5c0cd036b fix: graph-parser cljs tests 2026-02-11 11:36:32 -05:00
Gabriel Horner
e7b5c9a6b6 fix: CLI import results in invalid nodes
Had to add <get-file-stat w/ changes in #12313
2026-02-11 11:30:58 -05:00
Tienson Qin
5d515a8e28 fix: don't throw error for exp/iss not found 2026-02-11 21:11:04 +08:00
Tienson Qin
2341fb9b7f Merge branch 'master' into feat/worker-sync 2026-02-11 18:12:22 +08:00
megayu
40c9c86129 Fix issues related to importing Zotero (#12313)
* Fix import Zotero not recognized as asset

* Fix the inconsistency between imported Zotero label and actual filename

* Import support for zotero linked file like using zotfile

* Fix zotero highlighting block error after importing from file

* compatible with old configuration

* fix zotero picture highlight block does not consider as asset after importing

* Reconstructed to meet bb requirements

* empty datascript to avoid Conflicting upsert error after importing file to db graph

* use external-file-name instead of adding new property

* fix zotero link file can't open in asset page

* compatible with commit 33db791

* compatible with windows path & support zotero path can be relative for ci testing

* add zotero importing test

* remove prn log

* remove useless line

* Revert commit 45ebb9e

Future imports will be performed in the worker, will no longer encounter these issues

* refactor: update today page check to use async block retrieval

* Revert "refactor: update today page check to use async block retrieval"

This reverts commit 6750333df1.
2026-02-11 17:43:04 +08:00
Tienson Qin
84122a6171 fix: ensure worker state synced before asking for user rsa keys 2026-02-11 17:22:34 +08:00
Tienson Qin
f95600e595 fix: emoji reaction sync 2026-02-11 15:29:25 +08:00
Tienson Qin
dbef84fbb4 fix: auto-reconnect after long sleep by adding ws health check 2026-02-11 15:15:40 +08:00
Tienson Qin
10a8e3c491 fix: integrate large graph fixes from ulcaefna
Discord link: https://discord.com/channels/725182569297215569/1400122537744007198/1470853605753094288

1. Added snapshot streaming endpoint on worker:

  - deps/db-sync/src/logseq/db_sync/worker/routes/sync.cljs
  - deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs
      - Added :sync/snapshot-stream
      - Increased snapshot-download-batch-size to 5000
      - Added schema-init fallback probing (kvs, tx_log, sync_meta)
      - Improved HTTP error responses with debug fields

2. Switched frontend graph download from R2 URL flow to direct stream:

  - src/main/frontend/handler/db_based/sync.cljs
      - Removed /snapshot/download + cleanup DELETE flow
      - Uses direct GET /snapshot/stream

3. Applied large-graph upload stability fixes:

  - src/main/frontend/worker/sync.cljs
      - upload-kvs-batch-size changed 2000 -> 500
      - Reworked offload-large-titles-in-datoms to avoid deep promise chains
      - Fixed upload progress message to use updated loaded count
  - src/main/frontend/worker/sync/crypt.cljs
      - Batched <encrypt-datoms in chunks of 5000
  - src/main/frontend/worker/sync/client_op.cljs
      - update-graph-uuid now retracts old :graph-uuid datoms before add

4. Applied worker-side error visibility and catch hardening:

  - deps/db-sync/src/logseq/db_sync/worker.cljs
  - deps/db-sync/src/logseq/db_sync/worker/dispatch.cljs
  - deps/db-sync/src/logseq/db_sync/worker/coerce.cljs
  - deps/db-sync/src/logseq/db_sync/worker/http.cljs
  - deps/db-sync/src/logseq/db_sync/worker/handler/index.cljs

5. Added/updated tests for these behaviors:
2026-02-11 14:41:26 +08:00
Gabriel Horner
4cc8ef3ce4 fix: importing property value that changes
from :date to :node. Should fix
https://github.com/logseq/db-test/issues/748
2026-02-10 14:10:04 -05:00
Gabriel Horner
cd7c1ef50e enhance(dev): add linters for db-sync
Fix minor kondo errors. Also removed obvious dead code found by carve linter
2026-02-09 12:25:50 -05:00
Gabriel Horner
8bf63e9715 chore: docs on running wrangler locally 2026-02-09 11:31:41 -05:00
Tienson Qin
54844ff95f no need to run db-sync tests in build CI
because it's already handled by its own dep ci test
2026-02-06 19:08:52 +08:00
Tienson Qin
e428d52ee3 fix: can't upsert user in node adapter 2026-02-06 18:13:25 +08:00
Tienson Qin
7a915a31b1 remove static auth because signin/up doesn't support custom login 2026-02-06 17:35:30 +08:00
Tienson Qin
8bbc0fede8 enhance: gzip download graph 2026-02-06 16:30:01 +08:00
Tienson Qin
2ea618b76f fix: don't notify source client with presence update 2026-02-06 15:55:55 +08:00
Gabriel Horner
adf9770da0 fix: lint and docstring 2026-02-05 13:13:46 -05:00
Gabriel Horner
1a1896ba5b fix: internal-ident?
Also fix db test
2026-02-05 09:01:04 -05:00
Gabriel Horner
4fbd147f08 Merge branch 'master' into feat/worker-sync 2026-02-04 16:54:56 -05:00
Gabriel Horner
c1ca626b36 enhance(dev): add workflow to run db-sync's tests
Also disable test ns that has hanging tests. Was able to pinpoint
where tests are hanging and left logging there to fix it in the future
2026-02-04 16:36:16 -05:00
Gabriel Horner
5f34b04faa chore: add test for / property page fix from #12353 2026-02-02 11:23:32 -05:00
Tienson Qin
ef17ce59c3 add sentry to collect errors 2026-02-01 23:39:25 +08:00
Tienson Qin
ad2328d234 fix: lint 2026-02-01 21:55:15 +08:00
Tienson Qin
c7f2699edb add migration for reactions 2026-02-01 21:53:55 +08:00
Tienson Qin
05be455371 feat: reactions 2026-02-01 21:45:31 +08:00
Tienson Qin
267587a551 enhance(ux): add progress bar when exporting zip 2026-02-01 16:29:28 +08:00
Tienson Qin
f26f984b07 fix: latency 2026-01-31 07:32:13 +08:00
Tienson Qin
8f76e4dbb0 don't initialize schema at request time 2026-01-31 07:01:30 +08:00
Tienson Qin
0fa673978f debug latency 2026-01-31 06:44:01 +08:00
Tienson Qin
a9c7a35d77 disable smart placement 2026-01-31 05:38:47 +08:00
Tienson Qin
ddb4db21c6 add session api to use read replica 2026-01-31 05:14:48 +08:00
Tienson Qin
328d8e2bf3 fix: use prod d1 db 2026-01-31 04:23:17 +08:00
Tienson Qin
b8445a6395 node.js adapter 2026-01-31 00:51:35 +08:00
Tienson Qin
3adff65c90 add prod env for sync 2026-01-30 20:55:55 +08:00
Tienson Qin
d169b587f9 fix: graph parser tests 2026-01-30 20:45:02 +08:00
Tienson Qin
e7bb0963d4 enhance: upload large block title to r2 2026-01-30 20:04:29 +08:00