logseq/docs/agent-guide/036-db-worker-node-ncc-bundling.md

db-worker-node ncc Standalone Bundle Implementation Plan

Goal: Build db-worker-node.js with @vercel/ncc so the runtime can run without node_modules present next to the executable.

Architecture: Keep shadow-cljs as the source compiler for :db-worker-node, then run ncc on the generated entry and publish a single runtime artifact in dist/ that is used by CLI daemon orchestration. Architecture: Preserve local development ergonomics by keeping static/db-worker-node.js for fast dev loops, while production and package paths resolve to the ncc artifact first.

Tech Stack: ClojureScript, shadow-cljs :node-script, @vercel/ncc, Node.js 22, yarn scripts in package.json, existing CLI daemon and doctor checks.

Related: Builds on docs/agent-guide/031-logseq-cli-doctor-command.md. Related: Relates to docs/agent-guide/033-desktop-db-worker-node-backend.md. Related: Relates to docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md.

Problem statement

/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js is currently generated by shadow-cljs, and runtime behavior assumes dependencies are available from node_modules.

The CLI server startup path in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs points to ../dist/db-worker-node.js, which is currently a thin wrapper that forwards to ../static/db-worker-node.js.

This wrapper model keeps runtime coupled to workspace layout and to node_modules, so the daemon script is not independently portable.

We need a deterministic packaging path that produces one runnable artifact plus copied native assets, and we need to verify that artifact works when node_modules is absent.

Electron release packaging also needs to include the db-worker standalone bundle step, so yarn release-electron always ships the same packaged runtime artifact.

The solution must keep existing CLI and Electron daemon orchestration behavior unchanged, including lock-file semantics, owner-source semantics, and health endpoint behavior.

Current packaging map

Area	Current behavior	Limitation
Build output	yarn db-worker-node:compile writes /Users/rcmerci/gh-repos/logseq/static/db-worker-node.js.	Output is not a standalone distribution artifact.
Dist entry	/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js only requires ../static/db-worker-node.js.	Runtime still depends on static output and installed dependencies.
Daemon spawn	/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs spawns ../dist/db-worker-node.js.	Spawn path is stable, but executable is not standalone.
Doctor check	/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs checks ../static/db-worker-node.js by default.	Diagnostic target does not match the intended distributable runtime.
Package manifest	/Users/rcmerci/gh-repos/logseq/package.json includes static/db-worker-node.js in files.	Published package does not guarantee standalone daemon artifact contract.
Electron release	yarn release-electron does not guarantee db-worker bundle refresh before packaging.	Desktop release artifact can drift from standalone db-worker bundle contract.

Target packaging map

Area	Target behavior	Verification signal
Bundle output	ncc emits a standalone db-worker-node runtime in /Users/rcmerci/gh-repos/logseq/dist/ with required runtime assets copied adjacent to entrypoint.	Daemon starts and serves /healthz and /readyz without node_modules.
Spawn path	CLI server keeps spawning /Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js as canonical runtime.	Existing logseq.cli.server-test assertions remain green with updated contract.
Doctor check	Doctor defaults to the same packaged runtime path used for spawn, and does not auto-fallback to static runtime.	Doctor check path matches runtime path in tests and manual runs.
Dev flow	Fast local dev command remains available using static/db-worker-node.js for watch and debug workflows.	yarn db-worker-node:compile and node ./static/db-worker-node.js still work during development.
Publish flow	Package files include standalone runtime assets required by ncc output.	Installed package can execute daemon without extra dependency install.
Electron release	yarn release-electron runs db-worker bundle build before Electron packaging steps.	Electron release artifact includes the same standalone db-worker runtime contract.

Integration sketch

shadow-cljs (:db-worker-node)
  -> /static/db-worker-node.js
  -> ncc build step
      -> /dist/db-worker-node.js
      -> /dist/<ncc runtime assets, including native binaries if emitted>

logseq-cli runtime
  -> logseq.cli.server/spawn-server!
  -> /dist/db-worker-node.js
  -> db-worker daemon HTTP + SSE API

Testing Plan

I will follow @test-driven-development and add failing tests before implementation changes in each phase.

I will add behavior tests for runtime path resolution so spawn and doctor point to the same canonical bundle target.

I will add a standalone smoke test that launches the bundled daemon from a temporary directory without node_modules and verifies /healthz, /readyz, and shutdown behavior.

I will keep existing daemon lifecycle tests green to ensure no regression in lock cleanup, owner checks, and timeout error semantics.

I will run focused tests first, then full validation with yarn cljs:lint && yarn test, and if any unexpected failures appear I will use @clojure-debug before changing behavior.

NOTE: I will write all tests before I add any implementation behavior.

Implementation plan

Phase 1: RED for runtime artifact contract.

1. Add a failing test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs asserting canonical db-worker runtime path resolves to dist/db-worker-node.js as the production target.
2. Add a failing test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs asserting default doctor script check points to the same canonical runtime target as server spawn.
3. Add a failing test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs asserting optional dev-mode check can still validate static/db-worker-node.js when explicitly requested.
4. Add a new failing bundle smoke test file at /Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs that expects daemon startup success from a bundle-only temp directory.
5. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test' and confirm failures occur on the new path-contract assertions.
6. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test' and confirm failures occur on the new default-path expectations.
7. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' and confirm standalone smoke test fails before implementation.

Phase 2: Add ncc build pipeline.

8. Add @vercel/ncc to /Users/rcmerci/gh-repos/logseq/package.json as a dev dependency.
9. Add a dedicated script in /Users/rcmerci/gh-repos/logseq/package.json to build :db-worker-node in release mode before bundling.
10. Add a dedicated script in /Users/rcmerci/gh-repos/logseq/package.json to run ncc against /Users/rcmerci/gh-repos/logseq/static/db-worker-node.js.
11. Add a dedicated script in /Users/rcmerci/gh-repos/logseq/package.json to normalize ncc output into /Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js with adjacent assets preserved.
12. If script complexity is non-trivial, add /Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs to encapsulate output normalization and deterministic cleanup.
13. Add or update yarn scripts in /Users/rcmerci/gh-repos/logseq/package.json for one-command bundle build and optional local run of the bundled artifact.
14. Update yarn release-electron in /Users/rcmerci/gh-repos/logseq/package.json so it includes db-worker-node:release:bundle before Electron packaging.
15. Run yarn db-worker-node:release:bundle and verify dist/db-worker-node.js is regenerated with executable permissions preserved.

Phase 3: Align runtime path and diagnostics.

15. Refactor runtime path helpers in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs so there is one canonical function for packaged runtime and one explicit dev fallback function.
16. Keep spawn-server! in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs bound to the packaged runtime path to avoid ambiguity.
17. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs to default-check the same packaged runtime path used by spawn.
18. Add an explicit action option in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs for fallback static-path diagnostics used only for development troubleshooting.
19. Ensure doctor failure codes remain stable as :doctor-script-missing and :doctor-script-unreadable.
20. Re-run yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test' and yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test' to make the new path contract green.

Phase 4: Package manifest and docs alignment.

22. Update files in /Users/rcmerci/gh-repos/logseq/package.json so packaged runtime includes dist/db-worker-node.js and ncc-emitted adjacent assets.
23. Keep static/db-worker-node.js inclusion only if required for development workflows, and document that distinction explicitly.
24. Update /Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md build instructions to include the ncc bundle command and standalone runtime expectations.
25. Update daemon runtime notes in /Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md so doctor references packaged runtime as primary target.
26. Add troubleshooting notes for native module asset copy behavior from ncc output.

Phase 5: Standalone runtime behavior verification.

27. Implement bundle smoke test setup in /Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs to copy only bundle artifacts into a temp directory with no node_modules.
28. In that test, spawn node ./db-worker-node.js --repo <repo> --data-dir <tmp> from the bundle-only directory.
29. In that test, poll /healthz and /readyz and assert both return HTTP 200 after startup.
30. In that test, invoke /v1/shutdown and assert process exits and lock file is cleaned or becomes stale-removable.
31. In that test, assert failure output is actionable if native binary asset is missing, to guard accidental packaging regressions.
32. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' and make it green.

Phase 6: Final validation and review checklist.

33. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test' and confirm zero failures and zero errors.
34. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test' and confirm zero failures and zero errors.
35. Run yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' and confirm zero failures and zero errors.
36. Run yarn db-worker-node:compile and verify static/db-worker-node.js remains valid for local dev flow.
37. Run yarn db-worker-node:release:bundle and verify dist/db-worker-node.js starts successfully with node dist/db-worker-node.js --help.
38. Run yarn release-electron and verify the script execution includes db-worker-node:release:bundle before Electron packaging steps.
39. Run yarn cljs:lint && yarn test and confirm repository review checklist passes.
40. Validate changed code against @prompts/review.md before merge.

Edge cases to validate during implementation

Scenario	Expected behavior
ncc emits native .node assets for better-sqlite3.	Bundle output keeps those assets adjacent to entrypoint and runtime loads without node_modules.
Bundle is copied to another directory without static files.	Daemon still starts because packaged runtime no longer requires ../static/db-worker-node.js.
Developer runs doctor in source workspace before bundle build.	Doctor reports missing packaged artifact by default, and only checks static runtime when explicitly requested.
dist/db-worker-node.js exists but is not readable or is a directory.	Doctor returns :doctor-script-unreadable with path detail.
Bundle build is run twice.	Build output remains deterministic and stale ncc artifacts are cleaned safely.
yarn release-electron is run directly.	Release flow still builds db-worker standalone bundle before Electron packaging artifacts are produced.
CLI and Electron share lock for same repo under bundled runtime.	Existing ownership and lock semantics remain unchanged from current behavior.

Verification commands and expected outputs

yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.server-test'
yarn cljs:test && yarn cljs:run-test -v 'logseq.cli.command.doctor-test'
yarn cljs:test && yarn cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'
yarn db-worker-node:compile
yarn db-worker-node:release:bundle
yarn release-electron
node dist/db-worker-node.js --help
yarn cljs:lint && yarn test

All test commands should finish with 0 failures, 0 errors.

The bundle command should finish without MODULE_NOT_FOUND for runtime dependencies.

node dist/db-worker-node.js --help should print daemon help text and exit with code 0.

Testing Details

Behavior-focused tests will validate that runtime path resolution, doctor diagnostics, and daemon startup behavior match user-visible expectations.

The standalone smoke test will verify real process startup and HTTP readiness in a bundle-only filesystem layout, rather than asserting internal helper calls.

Regression safety is provided by existing CLI server and doctor tests to ensure lock lifecycle and error-code contracts remain stable.

Implementation Details

• Keep packaged runtime path as one canonical helper shared by spawn and doctor.
• Keep dev runtime path explicit and opt-in for local diagnostics only.
• Introduce ncc build scripts with deterministic output normalization into dist/.
• Preserve local shadow-cljs development flow and avoid slowing watch mode.
• Add a dedicated standalone bundle smoke test namespace for runtime validation.
• Keep error code contracts stable for doctor and server command callers.
• Ensure package files include all ncc runtime assets required at execution time.
• Document build and troubleshooting steps in CLI docs for contributors and release workflows.
• Use @test-driven-development for every behavior change and follow @clojure-debug for unexpected failures.
• Finish with full lint and test validation and checklist review from @prompts/review.md.

Question

Resolved: choose option 1.

doctor defaults to strict packaged-runtime validation only.

doctor does not auto-fallback to static runtime without an explicit flag.

---