diff --git a/.gitignore b/.gitignore index 4d22d6e9e8..44c09b6fab 100644 --- a/.gitignore +++ b/.gitignore @@ -82,3 +82,4 @@ clj-e2e/e2e-dump .dir-locals.el .projectile deps/db-sync/data +*.map diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md new file mode 100644 index 0000000000..27e650ff31 --- /dev/null +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -0,0 +1,115 @@ +# db-worker-node and logseq-cli Orchestration Plan + +Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, refactor db-worker-node to be repo-bound with locking, make logseq-cli fully manage db-worker-node lifecycle, and add server subcommands for management. + +## Background and Current State (from existing code) + +- `db-worker-node` currently accepts `--repo` but it is optional; it can open/switch graphs via `thread-api/create-or-open-db` at runtime. Entrypoint: `src/main/frontend/worker/db_worker_node.cljs`. +- `logseq-cli` connects to an existing db-worker-node via `base-url`; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`. +- Tests explicitly start db-worker-node (`src/test/logseq/cli/integration_test.cljs`, `src/test/frontend/worker/db_worker_node_test.cljs`). + +## Requirements + +1. Refactor `db-worker-node`: startup must require `--repo`; on startup it must open or create that graph; it must not switch graphs at runtime; it must create a lock file so a graph can be served by only one db-worker-node instance; it only needs to bind to localhost. +2. In `logseq-cli`, all commands requiring `--repo` or any graph operations must connect to or create the corresponding db-worker-node server. +3. db-worker-node server must not be started manually; logseq-cli is fully responsible. +4. Add `server` subcommand(s) to logseq-cli for managing db-worker-node servers. + +## Scope / Non-goals + +- In scope: db-worker-node startup and lifecycle, repo binding enforcement, lock files, CLI server orchestration, management commands, tests/docs. +- Out of scope: changing db-worker core query/write protocol; changing db-worker-node HTTP API semantics beyond repo binding constraints. + +## Proposed Design + +- **Repo-bound server**: db-worker-node opens a single repo at startup and refuses repo changes for the lifetime of the process. It only listens on localhost. +- **Lock file**: each repo directory has a lock file to ensure one server per graph. Lock contains metadata for status/cleanup; db-worker-node handles it by default, and logseq-cli handles cases db-worker-node cannot. +- **CLI orchestration**: logseq-cli discovers/starts/reuses db-worker-node servers per repo. It is the only entrypoint for starting servers. +- **Server subcommands**: add `server list|status|start|stop|restart` (or similar) to manage servers explicitly. + +## Detailed Design + +### 1) db-worker-node: required repo, repo binding, lock file + +Files: +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/platform/node.cljs` (for data-dir / repo dir resolution) +- Optional new helper: `src/main/frontend/worker/db_worker_node_lock.cljs` + +Key changes: +- **Argument parsing**: `--repo` becomes required. If missing, print help and exit non-zero. Host binding is restricted to localhost (e.g., `127.0.0.1`) regardless of input. +- **Startup flow**: replace `/db-worker.lock`). + - Content: JSON `{repo, pid, host, port, startedAt}`. + - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”. + - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM. + - Stale lock: if lock exists but pid is dead, allow replacement (db-worker-node first; CLI can repair when server cannot). + +### 2) logseq-cli: auto start/reuse db-worker-node per repo + +Files: +- `src/main/logseq/cli/commands.cljs` +- `src/main/logseq/cli/main.cljs` +- `src/main/logseq/cli/config.cljs` +- New: `src/main/logseq/cli/server.cljs` (process management + lock handling) + +Key changes: +- **Repo resolution**: for all graph/content commands, require `--repo` or resolved repo from config; otherwise error. +- **Ensure server** (new helper `ensure-server!`): + 1. Derive data-dir, repo dir, and lock file path from repo. + 2. If lock file exists, read host/port/pid; probe `/healthz` + `/readyz`. + 3. If healthy, reuse existing server; set `config :base-url` dynamically. + 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. + 5. Spawn via `child_process.spawn`: `node ./static/db-worker-node.js --repo --data-dir <...> --host 127.0.0.1 --port 0`. + 6. Resolve actual port from lock file or server output. +- **base-url usage**: dynamically set based on repo server; no longer required from user. If `--base-url` is provided, decide if it is ignored or overrides orchestration (see Compatibility section). + +### 3) CLI `server` subcommands + +Suggested command group: +- `server list`: list servers from lock files (repo, pid, port, status). +- `server start --repo `: start server for repo. +- `server stop --repo `: stop server (SIGTERM or `/v1/shutdown`). +- `server restart --repo `: stop + start. + +Implementation notes: +- `start|stop|restart` require `--repo`. +- `list` scans data-dir for repo directories, reads lock files, and verifies status. +- Consider adding `/v1/shutdown` in db-worker-node for graceful stop. + +## Compatibility / Migration + +- No need to preserve compatibility for existing env vars, config keys, or flags related to db-worker-node or CLI server connectivity; remove them if they are no longer needed. + +## Test Plan + +- **Unit tests**: + - CLI: repo resolution, server orchestration logic, lock parsing, error codes (`src/test/logseq/cli/*`). + - db-worker-node: repo required, repo mismatch rejection, lock file create/cleanup (`src/test/frontend/worker/db_worker_node_test.cljs`). +- **Integration tests**: + - CLI runs graph/content commands without manual server startup (`src/test/logseq/cli/integration_test.cljs`). + - Concurrent startup of same repo must fail due to lock. + +## Milestones + +1. **db-worker-node binding & lock file**: repo required + repo enforcement + lock creation/cleanup. +2. **CLI server module**: `ensure-server!` with lock/health checks and spawning. +3. **CLI command updates**: graph/content commands require repo and auto-start server; add `server` subcommands. +4. **Tests + docs**: update/extend tests and adjust CLI docs (`docs/cli/logseq-cli.md`). + +## Open Questions + +1. Should `graph list` require `--repo`? If not, define a “global” server or out-of-band access to data-dir. + - Answer: No --repo needed, using 'out-of-band access to data-dir' way +2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). + - lockfile name:`db-worker.lock`, + - Location: inside repo dir (e.g. `~/.logseq/db-worker/.logseq-pool-/db-worker.lock`). + - only consider linux/macos for now +3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle. +4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI? +5. db-worker-node servers should keep running unless `logseq-cli server stop` is invoked or the process exits unexpectedly; in the latter case, CLI should handle lockfile cleanup on restart.