53 Commits

Author SHA1 Message Date
Gabriel Horner
b4f3e364e6 fix: duplication of exporter config leads to bugs
in script and test. Also fix <export-file not being waited on and new docs
after #12260
2025-12-15 12:15:48 -05:00
Gabriel Horner
8fc56bfaa3 refactor: Update importer to use validate-local-db
Also update validation in DB scripts:
* replace alternative validate-db impl with standard CLI one
* Add validation to import script
* Choose to decouple other DB scripts from validate-db as it may go away
  later
2025-12-04 10:30:56 -05:00
Gabriel Horner
2500fe888a fix: dev import can't import
All CLI imports were failing with "Maximum call stack size exceeded".
2025-10-22 10:37:25 -04:00
Tienson Qin
999618a406 refactor: compute pipeline tx-data before d/transact! 2025-10-15 15:31:42 +08:00
Tienson Qin
7ed7fb4f31 enhance: prefer ldb/transact! than d/transact! 2025-10-13 19:03:11 +08:00
Gabriel Horner
846f9307c3 enhance: notify users when assets exceed max size
so they are aware why asset was ignored. Also if asset ids are not
present we should fail loudly like we everywhere else in the importer.
Users should know when data is missing instead of hiding it in the
console. Also refactor confusing usage of :block/uuid to :asset-created?
as it was only used to track the first time an asset is built to be
transacted
2025-09-29 17:37:04 -04:00
Tienson Qin
971cec54bc fix(import): remove asset byte-array usage to prevent OOM
This commit also combine read-asset and copy-asset to read-and-copy-asset
2025-09-29 17:37:04 -04:00
Gabriel Horner
536f39cbbd enhance: import basic pdf text annotation
Only read edn for now. Works in UI and CLI.
Also correctly keeps annotation edn from being copied since
it's no longer used
2025-07-22 08:57:21 -04:00
Tienson Qin
7b46076170 Merge branch 'master' into refactor/page-parent 2025-06-17 19:44:34 +08:00
Gabriel Horner
f318d68530 enhance: import assets handles
unused assets by still copying them by name and handles
existing assets
2025-06-17 19:36:05 +08:00
Gabriel Horner
bb31c4e6f1 fix: importing assets in subdirectories
Also fix lint
2025-06-17 19:36:05 +08:00
Gabriel Horner
4524fc8370 enhance: import asset metadata
Also handle multiple assets per block and displaying that to user
2025-06-17 19:36:05 +08:00
Gabriel Horner
90c7fc5775 enhance: UI imports assets as #Asset
Also made importer and db asset creation use same helper fns
2025-06-17 19:36:05 +08:00
Gabriel Horner
38734bde40 enhance: CLI imports assets as #Asset
Had to split assets into two steps now that asset naming is dependent
on uuids. Also add explicit asserts in exporter to catch silent
failure in tests
2025-06-17 19:36:05 +08:00
Tienson Qin
92ec65d61d Merge branch 'master' into refactor/page-parent 2025-06-15 08:27:33 +08:00
Gabriel Horner
3fa32ae990 fix(dev): CLI import not copying assets to correct location 2025-06-09 12:38:23 -04:00
Gabriel Horner
38e05aa6b8 enhance: new graph from script saves git sha in :kv/value
Also for
https://test.logseq.com/#/page/6835e455-69f6-4bbd-9865-b76fa57ce457
2025-05-27 12:16:15 -04:00
Gabriel Horner
3fe790d4d7 enhance(dev): scripts can read or write db graphs as full path files
This allows for easier use of debugging graphs as they are exported by
users as a file. Also DRYed up duplicated helper
2025-05-23 13:59:02 -04:00
Gabriel Horner
fecb42f49d fix(dev): deps scripts not handling relative and absolute paths for graphs
transact and create didn't handle relative paths. All scripts except
import didn't handle absolute paths
2025-03-17 11:09:44 -04:00
Gabriel Horner
7917afaa45 chore: update nbb scripts to be repl friendly
With https://github.com/babashka/nbb/issues/95 done, this allows
all nbb script namespaces to required or used from a repl
2025-03-07 11:49:28 -05:00
Tienson Qin
926d05c185 feat: table columns pinning (#11693)
* Add property :logseq.property.table/pinned-columns

* feat: table column pinning

* enhance: hide :id column by default

---------

Co-authored-by: charlie <xyhp915@qq.com>
2025-01-24 11:17:09 +08:00
Gabriel Horner
6c58af2605 enhance(dev): importing cli debug option prints all tx
related to a failing path. Helpfully provides more context
and also doesn't misleadingly print tx when it's not related to an
import failure with another path
2025-01-23 16:15:53 -05:00
Tienson Qin
2e12918d5c Merge branch 'feat/db' into refactor/block-schema 2025-01-14 14:17:24 +08:00
Gabriel Horner
63eaacdbe1 enhance(dev): import cli and tests print original error's ex-data 2025-01-13 13:27:39 -05:00
Gabriel Horner
1ab621069d enhance: importer displays one notification about ignored files
rather than one notification per file. Also ignore pdf highlight pages
as user graphs shouldn't fail hard on features that aren't imported yet.
This allowed the user graph in
https://github.com/logseq/db-test/issues/191 to import without errors
2025-01-13 11:44:02 -05:00
Tienson Qin
e318e0d6c1 Merge branch 'feat/db' into refactor/block-schema 2025-01-12 16:48:19 +08:00
Gabriel Horner
2782eea315 enhance(dev): CLI import prints last tx
since most import failures are due to a failing tx
2025-01-10 15:22:37 -05:00
Gabriel Horner
8e81d7d855 enhance(dev): task to import multiple file graphs easily
Also add a --continue option to see all errors on a CLI import
2024-11-21 10:30:58 -05:00
Gabriel Horner
5541205674 enhance: default to importing all tags
Fixes LOG-3235. Still default to not importing all tags in tests and
scripts to simplify their workflows
2024-11-20 13:31:42 -05:00
Gabriel Horner
3154b849dd enhance(dev): add latest import options to script 2024-11-12 10:56:05 -05:00
Gabriel Horner
0fcbe9fe89 fix(dev): import script's user options 2024-11-06 15:22:45 -05:00
Gabriel Horner
8b9d2a601e enhance(dev): print ignored properties for CLI import 2024-09-12 17:25:42 -04:00
Gabriel Horner
4b365da70c fix: display more helpful notifications for invalid
properties and classes that can't build a db-ident. Part of LOG-3176
2024-07-24 16:35:05 -04:00
Gabriel Horner
33785dca66 fix: db-import of :query-sort-by property
Also fix warning notifications printing empty error logs.
Part of LOG-3176
2024-07-11 18:00:21 -04:00
Gabriel Horner
f0ecf03c4a fix: db import of tagged page with pageTags
Part of LOG-3176
2024-07-02 16:50:00 -04:00
Gabriel Horner
9ee09372aa fix: cleanup inconsistent and incorrect import error handling
Some notified errors didn't include error object. Import cli didn't fail
on failure and stack was reversed. Error notifications sometimes
included full error which makes errors unreadable in CLI and possibly
confusing for users. Also fixed missing classpath. Part of
LOG-3176
2024-06-28 11:16:22 -04:00
Gabriel Horner
e619874dc4 enhance: add acceptance test for file to db graph import
Part of LOG-3092
2024-06-26 15:59:09 -04:00
Gabriel Horner
5520bc1d33 fix: import single file from CLI didn't print error
Also improved CLI error reporting so sci stack is readable and
not filled with stack fn docstrings
2024-06-26 15:59:09 -04:00
Gabriel Horner
95db811f55 fix: frontend tests shouldn't depend on sqlite3
frontend.test.helper was accidentally coupled to sqlite3 via
logseq.outliner.db-pipeline. Split out sqlite3 dependent fns to
logseq.outliner.cli so it's clear this shouldn't be coupled to frontend
2024-06-14 14:41:24 -04:00
Gabriel Horner
0e1ada2ef6 enhance: bb task for creating graphs from EDN file
Converted inferred graph to an EDN file now that this task exists. Also
merge last of tasks.create-graph to relevant ns so that external users
can also create such tasks
2024-06-11 10:09:37 -04:00
Gabriel Horner
e08e52acd3 enhance: db graph importer supports optional parent class property 2024-03-29 10:37:09 -04:00
Gabriel Horner
345fd96e2a enhance: full file graph import works from CLI
Also cleaned up naming, added docstrings and fixed a bug that
failed import hard when user didn't input any tag-classes or
property-classes
2024-03-13 16:23:49 -04:00
Gabriel Horner
2f599919fb fix: favorites not importing
Also made underlying fns reusable from UI and nbb compatible.
Fixes LOG-3031
2024-03-12 17:57:47 -04:00
Gabriel Horner
09816b6427 enhance: assets can be imported on CLI
Also add individual asset error handling, like is done for doc files
2024-03-05 17:05:18 -05:00
Gabriel Horner
9669755cab enhance: detect and add properties to imported classes
fixes LOG-3069
2024-03-04 23:05:06 -05:00
Gabriel Horner
0b0299b28f fix: more import fixes after testing more graphs
- collapsed:: false was failing an import
- ignore more attributes for existing pages
- also fixed import script not handling macros and absolute paths
  correctly
2024-02-29 15:06:49 -05:00
Gabriel Horner
49cc17dea5 fix: update property values for any type to :default change
that occur within same file. Also made importing behavior reproducible
across runs and made import script script less verbose by default.
Part of LOG-2985
2024-02-29 13:01:09 -05:00
Gabriel Horner
b32eb7e9be enhance: automatically handle properties that change to :default
docs graph :platforms property now imports correctly rather than
ignored 30 times. Also allow import script to import multiple files
for testing purposes. Part of LOG-2985
2024-02-28 17:16:09 -05:00
Gabriel Horner
44d60808c1 fix: correctly detect property type
for property values containing text and refs.
Also allow import script to only import a file for testing.
Part of LOG-2985
2024-02-27 16:27:26 -05:00
Gabriel Horner
13439ed699 enhance: port two more import steps to be script compatible
also fix save-file! not waiting on async transact before post-save
2024-02-27 12:07:25 -05:00