diff --git a/.agents/skills/logseq-cli-maintenance/SKILL.md b/.agents/skills/logseq-cli-maintenance/SKILL.md new file mode 100644 index 0000000000..29e4a53556 --- /dev/null +++ b/.agents/skills/logseq-cli-maintenance/SKILL.md @@ -0,0 +1,139 @@ +--- +name: logseq-cli-maintenance +description: Improve readability, consistency, and long-term maintainability of Logseq CLI-related code, command flows, and tests. +--- + +# logseq-cli-maintenance + +## Purpose + +Use this skill when working on Logseq CLI-related code and you want to improve: + +- Readability +- Maintainability +- Consistency across command modules +- Long-term development ergonomics + +This skill focuses on practical refactoring and guardrails, without changing behavior unless explicitly requested. + +--- + +## When to use + +Trigger this skill when tasks include any of the following: + +1. “Refactor Logseq CLI code” +2. “Clean up command implementation” +3. “Improve readability/structure of CLI logic” +4. “Make CLI easier to extend and test” +5. “Reduce duplication in argument parsing/output handling” + +Do **not** use this skill for pure feature delivery unless the request also asks for code quality improvements. + +--- + +## Core maintenance principles + +### 1) Separate concerns clearly + +Keep these responsibilities isolated: + +- **Input parsing** (args/options/env) +- **Validation** (schema/rules/errors) +- **Execution** (domain logic) +- **Presentation** (stdout/stderr formatting, exit codes) + +A command handler should orchestrate these steps, not mix all logic in one large function. + +### 2) Prefer small, composable functions + +- Keep functions single-purpose. +- Use descriptive names that explain intent. +- Extract repeated logic into shared helpers. +- Avoid deep nesting; use early returns for invalid states. + +### 3) Keep command contracts explicit + +For each command/subcommand, make explicit: + +- Required and optional args +- Defaults +- Validation constraints +- Output shape and error format +- Exit code semantics + +### 4) Make errors actionable + +- Show concise error messages with next-step hints. +- Keep error wording consistent across commands. +- Distinguish user errors (invalid input) from internal errors. + +### 5) Standardize CLI output + +- Use one style for success, warning, and error output. +- Ensure machine-readable mode (if supported) is stable and documented. +- Avoid hidden format drift across subcommands. + +### 6) Preserve behavior while refactoring + +When request is maintenance-focused, avoid feature changes. +If behavior must change, call it out explicitly and add tests. + +--- + +## Suggested workflow + +1. **Map current command flow** + - Locate parse → validate → execute → print boundaries. +2. **Identify maintenance hotspots** + - Long functions, duplicated parsing/output logic, hidden side effects. +3. **Refactor in small, reviewable steps** + - One concern per change. +4. **Add/update tests around behavior contracts** + - Especially for error cases and output format. +5. **Run relevant CLI tests and linters** + - Confirm no regressions. +6. **Run relevant cmds in logseq-cli** + - Use logseq-cli skill + - Confirm no regressions. +7. **Document extension points** + - Show where future subcommands/options should be added. + +--- + +## Refactoring checklist + +Use this checklist before finishing: + +- [ ] Command entrypoint is concise and easy to scan. +- [ ] Parsing/validation/execution/output are separated. +- [ ] Repeated logic is extracted to shared helpers. +- [ ] Names are intention-revealing. +- [ ] Error messages are consistent and actionable. +- [ ] Output and exit code behavior are tested. +- [ ] No behavior changes were introduced unintentionally. +- [ ] `bb lint:large-vars` passes for touched vars. +- [ ] Existing `^:large-vars/cleanup-todo` annotations were removed when possible. +- [ ] New structure is easy for future contributors to extend. + +--- + +## Common anti-patterns to remove + +- God-function command handlers +- Implicit defaults scattered across files +- Inconsistent option names/aliases +- Silent failures or ambiguous exit codes +- Copy-pasted output formatting logic +- Mixing domain logic directly with terminal I/O + +--- + +## Deliverable expectations + +When applying this skill, produce: + +1. Cleaner structure with equivalent behavior (unless requested otherwise) +2. Focused tests for command contracts +3. Brief rationale for key refactors +4. Clear future extension path for new CLI commands/options diff --git a/.agents/skills/logseq-cli/SKILL.md b/.agents/skills/logseq-cli/SKILL.md new file mode 100644 index 0000000000..51536dcf31 --- /dev/null +++ b/.agents/skills/logseq-cli/SKILL.md @@ -0,0 +1,87 @@ +--- +name: logseq-cli +description: Operate the current Logseq command-line interface to inspect or modify graphs, pages, blocks, tags, and properties; run Datascript queries; show page/block trees; manage graphs; and manage db-worker-node servers. Use when a request involves running `logseq` commands or interpreting CLI output. +--- + +# Logseq CLI + +## Overview + +Use `logseq` to inspect and edit graph entities, run Datascript queries, and control graph/server lifecycle. + +## Quick start + +- Run `logseq --help` to see top-level commands and global flags. +- Run `logseq --help` to see command-specific options. +- Use `--graph` to target a specific graph. +- Omit `--output` for human output. Set `--output json` or `--output edn` only when machine-readable output is required. + +## Command groups (from `logseq --help`) + +- Graph inspect/edit: +- `list node`, `list page`, `list tag`, `list property`, `list task`, `list asset` +- `upsert block`, `upsert page`, `upsert tag`, `upsert property`, `upsert task`, `upsert assert` +- `remove block`, `remove page`, `remove tag`, `remove property` +- `query`, `query list`, `show`, `search` +- Graph management: `graph list|create|switch|remove|validate|info|export|import|backup` +- Server management: `server list|cleanup|start|stop|restart` +- Diagnostics: `doctor`, `debug` + +## Global options + +- `--config` Path to `cli.edn` (default `~/logseq/cli.edn`) +- `--graph` Graph name +- `--data-dir` Path to db-worker data dir (default `~/logseq/graphs`) +- `--timeout-ms` Request timeout in ms (default `10000`) +- `--output` Output format (`human`, `json`, `edn`) +- `--verbose` Enable verbose debug logging to stderr + +## Command option policy + +- Do not memorize or hardcode command options in this skill. +- Before running any command, always check live options with: +- `logseq --help` +- `logseq --help` + +## Examples policy + +- Do not maintain long static command examples in this skill. +- Use `logseq example` as the source of truth for runnable examples. +- Before proposing runnable commands, always inspect live examples with: + - `logseq example` + - `logseq example ` + - `logseq example --help` +- Prefer exact selectors when possible (for example, `logseq example upsert page`). +- Use prefix selectors when grouped examples are needed (for example, `logseq example upsert`). +- Replace placeholder ids/uuids in retrieved examples with real entities from the target graph. +- Use `logseq list ...`, `logseq show ...`, or `logseq query ...` first to discover valid ids/uuids. +- For graph transfer flows, keep `graph export --file` and `graph import --input` paths consistent. + + +## Tag association semantics + +- For block or page tag association, prefer explicit CLI tag options such as `--update-tags` and `--remove-tags`. +- Do not treat writing `#TagName` inside `--content` as equivalent guidance to explicit tag association. +- `upsert block` supports `--update-tags` in both create mode and update mode. +- `--update-tags` expects an EDN vector. +- Tag values may be tag title/name strings, db/id, UUID, or `:db/ident` values. +- String tag values may include a leading `#`, but they should still be passed inside `--update-tags` rather than embedded in content as a substitute for association. +- If the user asks to tag a block or page, prefer explicit tag association over embedding hashtags in content. +- Tags must already exist and be public. If needed, create the tag first with `upsert tag --name ""`. + +## Pitfalls + +- `--content "Summary #AI-GENERATED"` is not the same guidance as `--update-tags '["AI-GENERATED"]'`. +- Do not pass `--update-tags` as a comma-separated string. Use an EDN vector. +- Do not assume a hashtag in block text will replace the need for explicit tag association when the user asks for a tagged block. +- If tag association fails, verify the tag exists and is public before retrying. + +## Tips + +- `query list` returns both built-ins and `custom-queries` from `cli.edn`. +- `show --id` accepts either one db/id or an EDN vector of ids. +- `remove block --id` also accepts one db/id or an EDN vector. +- `upsert block` enters update mode when `--id` or `--uuid` is provided. +- Always verify command flags with `logseq --help` and `logseq <...> --help` before execution. +- If `logseq` reports that it doesn’t have read/write permission for data-dir, then add read/write permission for data-dir in the agent’s config. +- In sandboxed environments, `graph create` may print a process-scan warning to stderr; if command status is `ok`, the graph is still created. \ No newline at end of file diff --git a/.agents/skills/logseq-debug-workflow/SKILL.md b/.agents/skills/logseq-debug-workflow/SKILL.md new file mode 100644 index 0000000000..ce581a5a9d --- /dev/null +++ b/.agents/skills/logseq-debug-workflow/SKILL.md @@ -0,0 +1,108 @@ +--- +name: logseq-debug-workflow +description: Debug Logseq bugs with the right runtime, concrete before/after evidence, and end-to-end reproduction steps. +--- + +# Logseq Debug Workflow + +Use for any Logseq bug investigation. + +## Core rule + +Treat this as a debugging workflow, not a code-only change. + +Before claiming a fix, you must: + +1. Choose the correct *runtime repl* and say why: + - `:app` for renderer, DOM, UI, frontend state + - `:electron` for main-process, BrowserWindow, IPC, app config + - `:db-worker-node` for worker DB behavior, worker IPC, queries, transactions + - CLI for `logseq` command behavior +2. Reproduce the bug in *runtime REPL* before editing code. +3. Capture concrete evidence from that runtime: REPL output, CLI output, logs, or a failing test that matches the real runtime path. +4. Check relevant logs early. +5. Apply the smallest justified fix. +6. Re-run the same reproduction flow after the fix and capture evidence again. +7. Include restart/reload/reopen verification when the bug involves settings, startup, persistence, window creation, or cross-process behavior. + +## Do not conclude early + +Do **not** say the bug is fixed if any of these is missing: + +- pre-fix reproduction evidence +- relevant log evidence, or an explicit statement that checked logs had nothing useful +- post-fix evidence from the same runtime/path +- required end-to-end lifecycle verification + +Unit tests alone are **not** enough when this skill applies, unless the bug is truly unit-level and you explicitly justify that. + +If the environment blocks full verification, report: + +- the blocker +- what you tried +- which evidence is still missing +- the strongest partial evidence you have + +## Debugging tools + +### General + +- Add labeled `prn` checkpoints when helpful. +- Use small REPL/eval checks. +- Use targeted tests to confirm behavior. +- Inspect only relevant inputs, branches, transformed values, outputs, and errors. +- For async flows, inspect both sides of the async boundary. + +### Logseq REPL + +check `logseq-repl` skill. + +### Logs + +Logs are evidence. Check them early for Electron, CLI, worker, IPC, async, or persistence issues. + +Common locations: +- `tmp/desktop-app-repl/desktop-electron.log` (logseq-repl skill) +- `tmp/logseq-repl/shared-shadow-watch.log` (logseq-repl skill) +- `~/Library/Logs/Logseq/main.log` +- `~/Library/Logs/Logseq/main.old.log` +- `~/Library/Application Support/Logseq/configs.edn` +- graph-local `db-worker-node-.log` + +### Logseq CLI + +Before using `logseq`, load `logseq-cli` skill. + +## Required final output + +The final response must include these sections or an equivalent structure: + +1. **Runtime chosen** — which *runtime REPL* you used and why +2. **Pre-fix reproduction** — reproduce bug in *runtime REPL*, exact steps and evidence +3. **Root cause** — concrete cause and relevant files/flow +4. **Fix applied** — short description of the change +5. **Post-fix verification** — same steps again with new evidence +6. **Additional verification** — tests/checks run, and what was not verified +7. **Gaps or blockers** — any missing evidence and why + +## Quick checklist + +Before ending, make sure the answer is yes to all: + +- Did I reproduce the bug before fixing it? +- Did I show evidence, not just claim reproduction? +- Did I inspect relevant logs? +- Did I verify in the correct runtime? +- Did I rerun the same scenario after the fix? +- Did I include both before and after evidence in the final output? +- Did I avoid claiming completion if required evidence is missing? + +## Verification reminders + +- Never run tests, lint, build, or E2E verification in the background. +- Check logs before trusting REPL/CLI output alone. +- For performance bugs, compare `--profile` before/after on the same graph and command. +- For CLI bugs, reuse the same `--graph`, `--data-dir`, and output mode. +- For REPL debugging, verify against the intended runtime, not a stale one. + +Common checks: `bb dev:test -v `, `bb dev:lint-and-test`, `bb dev:cli-e2e`. diff --git a/.agents/skills/logseq-i18n/SKILL.md b/.agents/skills/logseq-i18n/SKILL.md index a0c3fff2b8..fdffe6e494 100644 --- a/.agents/skills/logseq-i18n/SKILL.md +++ b/.agents/skills/logseq-i18n/SKILL.md @@ -74,7 +74,10 @@ If the English text matches but the meaning differs, create a new key and follow ### Rule 3: English source lives in `en.edn` - Add new English source text to `src/resources/dicts/en.edn`. -- Add non-English entries only when you are also providing actual translations. +- **When introducing a new key for the first time, you must also add the + Simplified Chinese (`zh-CN`) translation in the same change.** English and + `zh-CN` are the two required locales for any new key. +- Add other non-English entries only when you are also providing actual translations. - When renaming or removing keys, update affected locale files so stale keys do not remain behind. - Do not copy English into non-English locale files just to fill gaps. Tongue diff --git a/.agents/skills/logseq-repl/SKILL.md b/.agents/skills/logseq-repl/SKILL.md new file mode 100644 index 0000000000..1b0a587898 --- /dev/null +++ b/.agents/skills/logseq-repl/SKILL.md @@ -0,0 +1,205 @@ +--- +name: logseq-repl +description: Start and coordinate Logseq development REPL workflows for the Desktop renderer `:app`, Electron main-process `:electron`, and `:db-worker-node` runtimes through one unified workflow. +--- + +# Logseq REPL Workflow + +Use this skill when the user needs a Logseq development REPL for: + +- Desktop renderer `:app` +- Electron main process `:electron` +- `:db-worker-node` +- any combination of those runtimes + +The workflow uses one shared state directory: `/tmp/logseq-repl/`. + +## Scripts + +Start everything with the default Logseq data root (`$LOGSEQ_CLI_ROOT_DIR` or `~/logseq`): + +```bash +./scripts/start-repl.sh --repo demo +``` + +Start everything with an explicit Logseq data root: + +```bash +./scripts/start-repl.sh --repo demo --root-dir ~/logseq +``` + +Clean up everything: + +```bash +./scripts/cleanup-repl.sh +``` + +Verify all REPL targets after startup: + +```bash +./scripts/verify-repls.sh +``` + +`start-repl.sh` starts: + +1. shared `pnpm watch` +2. Desktop dev app via `pnpm dev-electron-app` +3. `db-worker-node` via `node ./static/db-worker-node.js --repo --root-dir --owner-source cli` + +`start-repl.sh` is a small shell wrapper around `start-repl.py`. The Python script verifies that `:app`, `:electron`, and `:db-worker-node` runtimes are live, runs `verify-repls.sh` to connect to each target REPL and print a small result, then prints attach commands. It does not attach an interactive REPL by itself and exits after verification. + +`cleanup-repl.sh` stops all workflow-managed processes without requiring a target selection. It also removes legacy state files from older split workflows and attempts to stop repo-owned listeners on the standard REPL ports. + +`verify-repls.sh` connects to `app`, `electron`, and `db-worker-node` with `pnpm exec shadow-cljs cljs-repl`, evaluates one target-specific expression in each REPL, and prints the output. + +## Standard Flow + +Before starting or attaching: + +```bash +./scripts/cleanup-repl.sh +``` + +Then start all runtimes: + +```bash +./scripts/start-repl.sh --repo demo +``` + +Use `--root-dir ` if the target graph lives outside `$LOGSEQ_CLI_ROOT_DIR` or `~/logseq`. + +Attach only to the target you need: + +```bash +pnpm exec shadow-cljs cljs-repl app +pnpm exec shadow-cljs cljs-repl electron +pnpm exec shadow-cljs cljs-repl db-worker-node +``` + +## Runtime Selection + +- Need DOM, UI, renderer state, or page rendering? Use `:app`. +- Need Electron main process APIs, menus, window lifecycle, or main-process filesystem behavior? Use `:electron`. +- Need Node worker behavior or db-worker-node code paths? Use `:db-worker-node`. + +Runtime reminders: + +- `:app` = Electron renderer +- `:electron` = Electron main process +- `:db-worker` = browser worker +- `:db-worker-node` = Node worker + +## Readiness Model + +Keep these separate: + +1. watch alive: a `shadow-cljs` server or `pnpm watch` process exists +2. build ready: the target build completed successfully +3. runtime attached: a live JS runtime is connected for `:app`, `:electron`, or `:db-worker-node` + +If runtime count is `0`, do not attach yet. Fix runtime startup first. + +Check runtime counts: + +```bash +pnpm exec shadow-cljs clj-eval "(do (require '[shadow.cljs.devtools.api :as api]) (println {:app (count (api/repl-runtimes :app)) :electron (count (api/repl-runtimes :electron)) :db-worker-node (count (api/repl-runtimes :db-worker-node))}))" +``` + +Interpretation: + +- `:app > 0` means a Desktop renderer runtime is attached +- `:electron > 0` means an Electron main-process runtime is attached +- `:db-worker-node > 0` means a worker-node runtime is attached +- `0` means not ready, even if watch/build logs look healthy + +## Logs + +Look here first: + +- `/tmp/logseq-repl/shared-shadow-watch.log` +- `/tmp/logseq-repl/desktop-electron.log` +- `/tmp/logseq-repl/db-worker-node.log` + +## Port Audit + +After cleanup, verify standard ports if startup still reports conflicts: + +```bash +lsof -nP -iTCP:8701 -sTCP:LISTEN +lsof -nP -iTCP:3001 -sTCP:LISTEN +lsof -nP -iTCP:3002 -sTCP:LISTEN +lsof -nP -iTCP:9630 -sTCP:LISTEN +lsof -nP -iTCP:9631 -sTCP:LISTEN +``` + +Interpretation: + +- no listeners: clean enough to continue +- listeners after cleanup: resolve the external conflict first +- listeners only after startup: expected if owned by the workflow + +## Non-Interactive Verification Examples + +Desktop `:app`: + +```bash +cat <<'EOF' | pnpm exec shadow-cljs cljs-repl app +(prn {:runtime :app :document? (some? js/document) :title (.-title js/document)}) +:cljs/quit +EOF +``` + +Electron `:electron`: + +```bash +cat <<'EOF' | pnpm exec shadow-cljs cljs-repl electron +(prn {:runtime :electron :process? (some? js/process) :type (.-type js/process)}) +:cljs/quit +EOF +``` + +`db-worker-node`: + +```bash +cat <<'EOF' | pnpm exec shadow-cljs cljs-repl db-worker-node +(prn {:runtime :db-worker-node :process? (some? js/process) :platform (.-platform js/process)}) +:cljs/quit +EOF +``` + +## Editor Attach Helpers + +```clojure +(shadow.user/cljs-repl) +(shadow.user/electron-repl) +(shadow.user/worker-node-repl) +``` + +## Troubleshooting + +Failure triage order: + +1. inspect `tmp/logseq-repl/shared-shadow-watch.log` +2. inspect `tmp/logseq-repl/desktop-electron.log` +3. inspect `tmp/logseq-repl/db-worker-node.log` +4. inspect standard port listeners with `lsof` +5. inspect runtime counts with `shadow.cljs.devtools.api/repl-runtimes` + +Common cases: + +- `No available JS runtime`: the build may be ready, but the runtime has not connected. Check runtime counts before retrying attach. +- multiple `:app` runtimes: close browser dev app instances so only the Desktop renderer remains. +- ports already in use after cleanup: another Logseq/shadow-cljs dev session is still running. +- `db-worker-node` repo mismatch: rerun `start-repl.sh --repo `; it restarts the worker runtime for the requested repo. + +## Recommended Response Pattern + +When helping a user connect to a REPL: + +1. identify whether they need `:app`, `:electron`, `:db-worker-node`, or a combination +2. run `cleanup-repl.sh` +3. if standard ports remain occupied, resolve that conflict first +4. run `start-repl.sh --repo ` +5. verify runtime counts if attach fails +6. attach to the matching build or helper +7. run `cleanup-repl.sh` when finished diff --git a/.agents/skills/logseq-repl/scripts/cleanup-repl.sh b/.agents/skills/logseq-repl/scripts/cleanup-repl.sh new file mode 100755 index 0000000000..b627603bcc --- /dev/null +++ b/.agents/skills/logseq-repl/scripts/cleanup-repl.sh @@ -0,0 +1,176 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEFAULT_REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)" +REPO_ROOT="${REPO_ROOT:-$DEFAULT_REPO_ROOT}" +FORCE_KILL=0 + +usage() { + cat <<'EOF' +Stop all processes started by start-repl.sh. + +Usage: + cleanup-repl.sh [options] + +Options: + --repo-root Logseq repository root (default: auto-detect from script location) + --force Use SIGKILL if a process does not stop gracefully + -h, --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --repo-root) + shift + REPO_ROOT="${1:?missing value for --repo-root}" + ;; + --force) + FORCE_KILL=1 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + usage + exit 1 + ;; + esac + shift +done + +LOG_DIR="$REPO_ROOT/tmp/logseq-repl" +LEGACY_DESKTOP_LOG_DIR="$REPO_ROOT/tmp/desktop-app-repl" +LEGACY_DB_LOG_DIR="$REPO_ROOT/tmp/db-worker-node-repl" + +is_running_pid() { + local pid="$1" + [[ "$pid" =~ ^[0-9]+$ ]] && kill -0 "$pid" 2>/dev/null +} + +read_pid() { + local file="$1" + if [[ -f "$file" ]]; then + tr -d '[:space:]' < "$file" + fi +} + +process_group_id() { + local pid="$1" + ps -o pgid= -p "$pid" 2>/dev/null | tr -d '[:space:]' +} + +current_process_group_id() { + process_group_id "$$" +} + +signal_process_group() { + local signal="$1" + local pid="$2" + + local current_pgid pgid + pgid="$(process_group_id "$pid" || true)" + current_pgid="$(current_process_group_id || true)" + + if [[ -n "${pgid:-}" && "$pgid" =~ ^[0-9]+$ && "$pgid" != "${current_pgid:-}" ]]; then + kill "-$signal" "-$pgid" 2>/dev/null || true + fi + + kill "-$signal" "$pid" 2>/dev/null || true +} + +stop_by_pid_file() { + local pid_file="$1" + local label="$2" + + local pid + pid="$(read_pid "$pid_file" || true)" + + if [[ -z "${pid:-}" ]]; then + rm -f "$pid_file" + return 0 + fi + + if ! is_running_pid "$pid"; then + echo "$label: process already stopped (pid=$pid)" + rm -f "$pid_file" + return 0 + fi + + echo "$label: stopping pid=$pid" + signal_process_group TERM "$pid" + + local i + for ((i=0; i<10; i++)); do + if ! is_running_pid "$pid"; then + echo "$label: stopped" + rm -f "$pid_file" + return 0 + fi + sleep 1 + done + + if [[ "$FORCE_KILL" -eq 1 ]]; then + echo "$label: force killing pid=$pid" + signal_process_group KILL "$pid" + sleep 1 + fi + + if is_running_pid "$pid"; then + echo "$label: failed to stop pid=$pid" >&2 + return 1 + fi + + echo "$label: stopped" + rm -f "$pid_file" +} + +repo_owns_pid() { + local pid="$1" + local cwd + cwd="$(lsof -nP -a -p "$pid" -d cwd 2>/dev/null | awk 'NR > 1 {print $NF; exit}' || true)" + [[ "$cwd" == "$REPO_ROOT" ]] +} + +stop_repo_port_listener() { + local port="$1" + local pid + + while read -r pid; do + [[ -n "${pid:-}" ]] || continue + if is_running_pid "$pid" && repo_owns_pid "$pid"; then + echo "port $port listener: stopping pid=$pid" + signal_process_group TERM "$pid" + fi + done < <(lsof -nP -tiTCP:"$port" -sTCP:LISTEN 2>/dev/null || true) +} + +if [[ ! -d "$LOG_DIR" && ! -d "$LEGACY_DESKTOP_LOG_DIR" && ! -d "$LEGACY_DB_LOG_DIR" ]]; then + echo "State directory not found: $LOG_DIR" + echo "Nothing to clean up." + exit 0 +fi + +stop_by_pid_file "$LOG_DIR/db-worker-node.pid" "db-worker-node" +stop_by_pid_file "$LOG_DIR/desktop-electron.pid" "desktop-electron" +stop_by_pid_file "$LOG_DIR/shared-shadow-watch.pid" "shadow-cljs watch" + +stop_by_pid_file "$LEGACY_DB_LOG_DIR/db-worker-node.pid" "legacy db-worker-node" +stop_by_pid_file "$LEGACY_DB_LOG_DIR/shadow-db-worker-node.pid" "legacy shadow-cljs watch" +stop_by_pid_file "$LEGACY_DESKTOP_LOG_DIR/desktop-electron.pid" "legacy desktop-electron" +stop_by_pid_file "$LEGACY_DESKTOP_LOG_DIR/shadow-watch.pid" "legacy shadow-cljs watch" + +rm -f "$LOG_DIR/db-worker-node.repo" \ + "$LOG_DIR/db-worker-node.options.json" \ + "$LEGACY_DB_LOG_DIR/db-worker-node.repo" + +for port in 8701 3001 3002 9630 9631; do + stop_repo_port_listener "$port" +done + +rm -f "$LOG_DIR"/*.pid "$LEGACY_DB_LOG_DIR"/*.pid "$LEGACY_DESKTOP_LOG_DIR"/*.pid 2>/dev/null || true + +echo "Cleanup done." diff --git a/.agents/skills/logseq-repl/scripts/common.sh b/.agents/skills/logseq-repl/scripts/common.sh new file mode 100644 index 0000000000..c8598c57b3 --- /dev/null +++ b/.agents/skills/logseq-repl/scripts/common.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash + +logseq_repl_is_running_pid() { + local pid="$1" + [[ "$pid" =~ ^[0-9]+$ ]] && kill -0 "$pid" 2>/dev/null +} + +logseq_repl_read_pid() { + local file="$1" + if [[ -f "$file" ]]; then + tr -d '[:space:]' < "$file" + fi +} + +logseq_repl_port_check_line() { + local port="$1" + lsof -nP -iTCP:"$port" -sTCP:LISTEN 2>/dev/null || true +} + +logseq_repl_audit_standard_ports() { + local had_conflict=0 + local ports=(8701 3001 3002 9630 9631) + local port output + + for port in "${ports[@]}"; do + output="$(logseq_repl_port_check_line "$port")" + if [[ -n "$output" ]]; then + had_conflict=1 + echo "Port $port is already listening:" >&2 + echo "$output" >&2 + echo >&2 + fi + done + + return "$had_conflict" +} + +logseq_repl_require_clean_standard_ports() { + if logseq_repl_audit_standard_ports; then + return 0 + fi + + echo "Error: standard Logseq REPL ports are still occupied after cleanup." >&2 + echo "Resolve the external conflict first, then retry." >&2 + return 1 +} + +logseq_repl_runtime_count() { + local repo_root="$1" + local build_name="$2" + local output + + pushd "$repo_root" >/dev/null + if ! output="$(pnpm exec shadow-cljs clj-eval "(do (require '[shadow.cljs.devtools.api :as api]) (println (count (api/repl-runtimes :$build_name))))" 2>&1)"; then + popd >/dev/null + echo "Error: failed to inspect :$build_name runtimes." >&2 + echo "--- clj-eval output ---" >&2 + echo "$output" >&2 + echo "-----------------------" >&2 + return 1 + fi + popd >/dev/null + + local runtime_count + runtime_count="$(printf '%s\n' "$output" | awk '/^[0-9]+$/{n=$0} END{if (n != "") print n; else exit 1}')" || { + echo "Error: could not parse :$build_name runtime count." >&2 + echo "--- clj-eval output ---" >&2 + echo "$output" >&2 + echo "-----------------------" >&2 + return 1 + } + + printf '%s\n' "$runtime_count" +} diff --git a/.agents/skills/logseq-repl/scripts/start-repl.py b/.agents/skills/logseq-repl/scripts/start-repl.py new file mode 100755 index 0000000000..5df4b522b1 --- /dev/null +++ b/.agents/skills/logseq-repl/scripts/start-repl.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +import argparse +import json +import os +import re +import signal +import subprocess +import sys +import time +from pathlib import Path + + +SCRIPT_DIR = Path(__file__).resolve().parent +DEFAULT_REPO_ROOT = SCRIPT_DIR.parents[3] +STANDARD_PORTS = (8701, 3001, 3002, 9630, 9631) +if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(line_buffering=True) + sys.stderr.reconfigure(line_buffering=True) + + +def parse_args(): + parser = argparse.ArgumentParser( + prog="start-repl.sh", + description="Start the unified Logseq REPL workflow.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "This starts shared pnpm watch, the Desktop dev app, and the\n" + "db-worker-node runtime, verifies the REPL targets, and exits." + ), + ) + parser.add_argument("--repo", default=os.environ.get("DB_REPO", "demo"), + help="Graph repo name passed to db-worker-node (default: demo)") + parser.add_argument("--root-dir", default=os.environ.get("LOGSEQ_CLI_ROOT_DIR", str(Path.home() / "logseq")), + help="Logseq data root passed to db-worker-node (default: $LOGSEQ_CLI_ROOT_DIR or ~/logseq)") + parser.add_argument("--repo-root", default=os.environ.get("REPO_ROOT", str(DEFAULT_REPO_ROOT)), + help="Logseq repository root (default: auto-detect from script location)") + parser.add_argument("extra_node_args", nargs=argparse.REMAINDER, + help="Arguments after -- are passed to db-worker-node") + args = parser.parse_args() + if args.extra_node_args and args.extra_node_args[0] == "--": + args.extra_node_args = args.extra_node_args[1:] + return args + + +def require_command(name): + if subprocess.run(["/usr/bin/env", "sh", "-c", f"command -v {name} >/dev/null 2>&1"]).returncode != 0: + raise SystemExit(f"Error: {name} not found in PATH") + + +def read_pid(path): + try: + text = path.read_text().strip() + except FileNotFoundError: + return None + return int(text) if re.fullmatch(r"\d+", text) else None + + +def is_running(pid): + if not pid: + return False + try: + os.kill(pid, 0) + return True + except OSError: + return False + + +def write_pid(path, pid): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"{pid}\n") + + +def wait_for_patterns(path, timeout, patterns, all_required=True): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if path.exists(): + text = path.read_text(errors="replace") + if all_required and all(pattern in text for pattern in patterns): + return True + if not all_required and any(pattern in text for pattern in patterns): + return True + time.sleep(1) + return False + + +def process_group_kwargs(): + if os.name == "nt": + return {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP} + return {"start_new_session": True} + + +def start_process(repo_root, log_path, command): + log_path.parent.mkdir(parents=True, exist_ok=True) + log_file = log_path.open("wb") + return subprocess.Popen( + command, + cwd=repo_root, + stdin=subprocess.DEVNULL, + stdout=log_file, + stderr=subprocess.STDOUT, + **process_group_kwargs(), + ) + + +def port_listener_pid(port): + try: + output = subprocess.check_output( + ["lsof", "-nP", f"-iTCP:{port}", "-sTCP:LISTEN", "-t"], + text=True, + stderr=subprocess.DEVNULL, + ) + except (FileNotFoundError, subprocess.CalledProcessError): + return None + for line in output.splitlines(): + if line.strip().isdigit(): + return int(line.strip()) + return None + + +def has_managed_processes(paths): + return any(is_running(read_pid(path)) for path in paths) + + +def read_json(path): + try: + return json.loads(path.read_text()) + except (FileNotFoundError, json.JSONDecodeError): + return None + + +def write_json(path, payload): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, sort_keys=True) + "\n") + + +def require_clean_ports(): + had_conflict = False + for port in STANDARD_PORTS: + pid = port_listener_pid(port) + if pid: + had_conflict = True + print(f"Port {port} is already listening (pid={pid})", file=sys.stderr) + if had_conflict: + print("Error: standard Logseq REPL ports are still occupied after cleanup.", file=sys.stderr) + print("Resolve the external conflict first, then retry.", file=sys.stderr) + return False + return True + + +def ensure_shadow_watch(repo_root, shadow_pid_file, shadow_log): + pid = read_pid(shadow_pid_file) + if is_running(pid): + print(f"Reusing shared shadow-cljs watch (pid={pid})") + return pid + + print("Starting shared shadow-cljs watch via pnpm watch ...") + process = start_process(repo_root, shadow_log, ["pnpm", "watch"]) + write_pid(shadow_pid_file, process.pid) + time.sleep(1) + + if process.poll() is not None: + raise SystemExit(f"Error: pnpm watch exited early. Check {shadow_log}") + + if not wait_for_patterns(shadow_log, 180, [ + "[:electron] Build completed.", + "[:app] Build completed.", + "[:db-worker-node] Build completed.", + ]): + raise SystemExit(f"Error: pnpm watch did not finish the expected builds in time. Check {shadow_log}") + + listener_pid = port_listener_pid(8701) + if listener_pid and is_running(listener_pid): + write_pid(shadow_pid_file, listener_pid) + pid = listener_pid + else: + pid = process.pid + + print("shadow-cljs watch builds are ready") + return pid + + +def ensure_desktop_app(repo_root, desktop_pid_file, desktop_log): + pid = read_pid(desktop_pid_file) + if is_running(pid): + print(f"Reusing Desktop dev app (pid={pid})") + return pid + + print("Starting Desktop dev app via pnpm dev-electron-app ...") + process = start_process(repo_root, desktop_log, ["pnpm", "dev-electron-app"]) + write_pid(desktop_pid_file, process.pid) + time.sleep(1) + + if process.poll() is not None: + raise SystemExit(f"Error: pnpm dev-electron-app exited early. Check {desktop_log}") + + if not wait_for_patterns(desktop_log, 120, ["shadow-cljs - #", "Logseq App("], all_required=False): + raise SystemExit(f"Error: Desktop dev app did not report startup in time. Check {desktop_log}") + + print(f"Desktop dev app is running (pid={process.pid})") + return process.pid + + +def ensure_db_worker_node(repo_root, db_pid_file, db_options_file, db_log, repo, root_dir, extra_args): + pid = read_pid(db_pid_file) + startup_options = { + "repo": repo, + "root_dir": str(root_dir), + "owner_source": "cli", + "extra_args": list(extra_args), + } + existing_options = read_json(db_options_file) + + if is_running(pid): + if existing_options == startup_options: + print(f"Reusing db-worker-node runtime (pid={pid}, repo={repo}, root-dir={root_dir})") + return pid + print(f"Stopping existing db-worker-node (pid={pid}) due to startup option mismatch") + terminate_pid(pid) + time.sleep(1) + + print(f"Starting db-worker-node (repo={repo}, root-dir={root_dir}) ...") + process = start_process( + repo_root, + db_log, + [ + "node", + "./static/db-worker-node.js", + "--repo", + repo, + "--root-dir", + str(root_dir), + "--owner-source", + "cli", + *extra_args, + ], + ) + write_pid(db_pid_file, process.pid) + write_json(db_options_file, startup_options) + time.sleep(1) + + if process.poll() is not None: + raise SystemExit(f"Error: db-worker-node exited early. Check {db_log}") + + print(f"db-worker-node is running (pid={process.pid}, repo={repo}, root-dir={root_dir})") + return process.pid + + +def runtime_count(repo_root, build_name): + form = f"(do (require '[shadow.cljs.devtools.api :as api]) (println (count (api/repl-runtimes :{build_name}))))" + proc = subprocess.run( + ["pnpm", "exec", "shadow-cljs", "clj-eval", form], + cwd=repo_root, + text=True, + capture_output=True, + ) + if proc.returncode != 0: + raise SystemExit( + f"Error: failed to inspect :{build_name} runtimes.\n" + f"--- clj-eval output ---\n{proc.stdout}{proc.stderr}\n-----------------------" + ) + matches = re.findall(r"^(\d+)$", proc.stdout + proc.stderr, flags=re.MULTILINE) + if not matches: + raise SystemExit( + f"Error: could not parse :{build_name} runtime count.\n" + f"--- clj-eval output ---\n{proc.stdout}{proc.stderr}\n-----------------------" + ) + return int(matches[-1]) + + +def wait_for_runtime_count(repo_root, build_name, expected, timeout): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + count = runtime_count(repo_root, build_name) + if expected == "exactly-one": + if count == 1: + print(f"Detected exactly one live :{build_name} runtime") + return + if count != 0: + raise SystemExit(f"Error: Expected exactly one live :{build_name} runtime, found {count}.") + elif expected == "nonzero" and count != 0: + print(f"Detected live :{build_name} runtime count: {count}") + return + time.sleep(1) + + if expected == "exactly-one": + raise SystemExit(f"Error: Expected exactly one live :{build_name} runtime, found 0 after waiting.") + raise SystemExit(f"Error: expected a live :{build_name} runtime, but runtime count stayed 0.") + + +def verify_repls(repo_root): + subprocess.run([str(SCRIPT_DIR / "verify-repls.sh"), "--repo-root", str(repo_root)], check=True) + + +def print_summary(shadow_log, desktop_log, db_log, shadow_pid_file, desktop_pid_file, db_pid_file): + print() + print("Logs:") + print(f" shared shadow-cljs: {shadow_log}") + print(f" desktop-app: {desktop_log}") + print(f" db-worker-node: {db_log}") + print("PID files:") + print(f" {shadow_pid_file}") + print(f" {desktop_pid_file}") + print(f" {db_pid_file}") + print() + print("Attach commands:") + print(" pnpm exec shadow-cljs cljs-repl app") + print(" pnpm exec shadow-cljs cljs-repl electron") + print(" pnpm exec shadow-cljs cljs-repl db-worker-node") + print() + print("Startup complete. Attach to the needed REPL manually.") + + +def terminate_pid(pid): + if not pid: + return + try: + if os.name == "nt": + os.kill(pid, signal.CTRL_BREAK_EVENT) + else: + os.kill(pid, signal.SIGTERM) + except OSError: + pass + + +def main(): + args = parse_args() + repo_root = Path(args.repo_root).resolve() + db_root_dir = Path(args.root_dir).expanduser().resolve() + if not repo_root.is_dir(): + raise SystemExit(f"Error: repo root not found: {repo_root}") + + require_command("pnpm") + require_command("node") + + log_dir = repo_root / "tmp" / "logseq-repl" + legacy_desktop_dir = repo_root / "tmp" / "desktop-app-repl" + legacy_db_dir = repo_root / "tmp" / "db-worker-node-repl" + log_dir.mkdir(parents=True, exist_ok=True) + + shadow_pid_file = log_dir / "shared-shadow-watch.pid" + desktop_pid_file = log_dir / "desktop-electron.pid" + db_pid_file = log_dir / "db-worker-node.pid" + db_options_file = log_dir / "db-worker-node.options.json" + shadow_log = log_dir / "shared-shadow-watch.log" + desktop_log = log_dir / "desktop-electron.log" + db_log = log_dir / "db-worker-node.log" + + managed_pid_files = [ + shadow_pid_file, + desktop_pid_file, + db_pid_file, + legacy_desktop_dir / "shadow-watch.pid", + legacy_desktop_dir / "desktop-electron.pid", + legacy_db_dir / "shadow-db-worker-node.pid", + legacy_db_dir / "db-worker-node.pid", + ] + + if not has_managed_processes(managed_pid_files) and not require_clean_ports(): + return 1 + + ensure_shadow_watch(repo_root, shadow_pid_file, shadow_log) + ensure_desktop_app(repo_root, desktop_pid_file, desktop_log) + ensure_db_worker_node(repo_root, db_pid_file, db_options_file, db_log, args.repo, db_root_dir, args.extra_node_args) + wait_for_runtime_count(repo_root, "app", "exactly-one", 60) + wait_for_runtime_count(repo_root, "electron", "nonzero", 60) + wait_for_runtime_count(repo_root, "db-worker-node", "nonzero", 60) + verify_repls(repo_root) + print_summary(shadow_log, desktop_log, db_log, shadow_pid_file, desktop_pid_file, db_pid_file) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/logseq-repl/scripts/start-repl.sh b/.agents/skills/logseq-repl/scripts/start-repl.sh new file mode 100755 index 0000000000..6047d810a0 --- /dev/null +++ b/.agents/skills/logseq-repl/scripts/start-repl.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +exec python3 "$SCRIPT_DIR/start-repl.py" "$@" diff --git a/.agents/skills/logseq-repl/scripts/verify-repls.sh b/.agents/skills/logseq-repl/scripts/verify-repls.sh new file mode 100755 index 0000000000..4f84abf51a --- /dev/null +++ b/.agents/skills/logseq-repl/scripts/verify-repls.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEFAULT_REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)" +REPO_ROOT="${REPO_ROOT:-$DEFAULT_REPO_ROOT}" + +usage() { + cat <<'EOF' +Verify that all Logseq CLJS REPL targets are usable. + +Usage: + verify-repls.sh [options] + +Options: + --repo-root Logseq repository root (default: auto-detect from script location) + -h, --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --repo-root) + shift + REPO_ROOT="${1:?missing value for --repo-root}" + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + usage + exit 1 + ;; + esac + shift +done + +if [[ ! -d "$REPO_ROOT" ]]; then + echo "Error: repo root not found: $REPO_ROOT" >&2 + exit 1 +fi + +if ! command -v pnpm >/dev/null 2>&1; then + echo "Error: pnpm not found in PATH" >&2 + exit 1 +fi + +verify_target() { + local target="$1" + local form="$2" + + echo "Checking :$target ..." + + local repl_output + pushd "$REPO_ROOT" >/dev/null + if ! repl_output="$(printf '%s\n' "$form" | pnpm exec shadow-cljs cljs-repl "$target" 2>&1)"; then + popd >/dev/null + echo "Error: REPL verification failed for :$target." >&2 + echo "--- :$target output ---" >&2 + echo "$repl_output" >&2 + echo "-----------------------" >&2 + return 1 + fi + popd >/dev/null + + if [[ "$repl_output" != *"shadow-cljs - connected to server"* ]]; then + echo "Error: REPL verification did not connect for :$target." >&2 + echo "--- :$target output ---" >&2 + echo "$repl_output" >&2 + echo "-----------------------" >&2 + return 1 + fi + + echo "--- :$target result ---" + echo "$repl_output" + echo "-----------------------" + echo "REPL verification passed for :$target" +} + +echo "Verifying CLJS REPL targets ..." +verify_target app "(prn {:runtime :app :document? (some? js/document)})" +verify_target electron "(prn {:runtime :electron :process? (some? js/process) :type (some-> js/process .-type)})" +verify_target db-worker-node "(prn {:runtime :db-worker-node :process? (some? js/process) :platform (some-> js/process .-platform)})" +echo "All CLJS REPL targets verified." diff --git a/.agents/skills/logseq-repl/tests/test-lib.sh b/.agents/skills/logseq-repl/tests/test-lib.sh new file mode 100644 index 0000000000..27be12e615 --- /dev/null +++ b/.agents/skills/logseq-repl/tests/test-lib.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash + +fail() { + echo "FAIL: $*" >&2 + return 1 +} + +assert_file_exists() { + local path="$1" + [[ -e "$path" ]] || fail "expected file to exist: $path" +} + +assert_not_exists() { + local path="$1" + [[ ! -e "$path" ]] || fail "expected path to be absent: $path" +} + +assert_contains() { + local needle="$1" + local haystack_file="$2" + if ! grep -Fq "$needle" "$haystack_file"; then + echo "Expected to find: $needle" >&2 + echo "--- file: $haystack_file ---" >&2 + cat "$haystack_file" >&2 + echo "----------------------------" >&2 + return 1 + fi +} + +assert_not_contains_text() { + local needle="$1" + local file="$2" + if grep -Fq "$needle" "$file"; then + echo "Did not expect to find: $needle" >&2 + echo "--- file: $file ---" >&2 + cat "$file" >&2 + echo "--------------------" >&2 + return 1 + fi +} + +assert_not_matches() { + local pattern="$1" + local file="$2" + if grep -Eq "$pattern" "$file"; then + echo "Did not expect regex match: $pattern" >&2 + echo "--- file: $file ---" >&2 + cat "$file" >&2 + echo "--------------------" >&2 + return 1 + fi +} + +assert_equals() { + local expected="$1" + local actual="$2" + [[ "$expected" == "$actual" ]] || fail "expected '$expected' but got '$actual'" +} + +portable_path_pattern() { + local slash='/' + local windows_drive='[A-Za-z]:\\[^[:space:]]+' + printf '%s' "${slash}Users${slash}[^[:space:]]+|${slash}home${slash}[^[:space:]]+|${windows_drive}" +} + +run_test() { + local name="$1" + local fn="$2" + + local status + set +e + (set -e; "$fn") + status=$? + set -e + + if [[ "$status" -eq 0 ]]; then + echo "PASS: $name" + PASS_COUNT=$((PASS_COUNT + 1)) + else + echo "FAIL: $name" >&2 + FAIL_COUNT=$((FAIL_COUNT + 1)) + fi +} diff --git a/.agents/skills/logseq-repl/tests/test-logseq-repl.sh b/.agents/skills/logseq-repl/tests/test-logseq-repl.sh new file mode 100755 index 0000000000..e521c26769 --- /dev/null +++ b/.agents/skills/logseq-repl/tests/test-logseq-repl.sh @@ -0,0 +1,438 @@ +#!/usr/bin/env bash +set -euo pipefail + +TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SKILL_DIR="$(cd "$TEST_DIR/.." && pwd)" +START_SCRIPT="$SKILL_DIR/scripts/start-repl.sh" +START_PY_SCRIPT="$SKILL_DIR/scripts/start-repl.py" +CLEANUP_SCRIPT="$SKILL_DIR/scripts/cleanup-repl.sh" +VERIFY_SCRIPT="$SKILL_DIR/scripts/verify-repls.sh" +SKILL_FILE="$SKILL_DIR/SKILL.md" +ORIGINAL_PATH="$PATH" + +# shellcheck source=./test-lib.sh +source "$TEST_DIR/test-lib.sh" + +PASS_COUNT=0 +FAIL_COUNT=0 + +create_fake_env() { + TEST_ROOT="$(mktemp -d)" + REPO_ROOT="$TEST_ROOT/repo" + DB_ROOT_DIR="$TEST_ROOT/logseq-root" + BIN_DIR="$TEST_ROOT/bin" + CMD_LOG="$TEST_ROOT/commands.log" + + mkdir -p "$REPO_ROOT/static" "$DB_ROOT_DIR" "$BIN_DIR" + DB_ROOT_DIR="$(cd "$DB_ROOT_DIR" && pwd -P)" + : > "$CMD_LOG" + + cat > "$BIN_DIR/pnpm" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail + +echo "pnpm $*" >> "$FAKE_CMD_LOG" + +case "${1:-}" in + watch) + echo "shadow-cljs - nREPL server started on port 8701" + echo "[:electron] Build completed." + echo "[:app] Build completed." + echo "[:db-worker-node] Build completed." + while true; do sleep 1; done + ;; + dev-electron-app) + echo "17:12:00.841 Logseq App(2.0.1) Starting..." + echo "shadow-cljs - #6 ready!" + while true; do sleep 1; done + ;; + exec) + shift + if [[ "${1:-}" != "shadow-cljs" ]]; then + echo "Unexpected pnpm exec command: $*" >&2 + exit 1 + fi + shift + + input="" + if [[ "${1:-}" == "cljs-repl" ]] && [[ -p /dev/stdin ]]; then + input="$(cat)" + fi + + if [[ "${1:-}" == "clj-eval" ]]; then + echo "pnpm-exec-clj-eval shadow-cljs $*" >> "$FAKE_CMD_LOG" + echo "shadow-cljs - connected to server" + case "$*" in + *"repl-runtimes :app"*) echo "${FAKE_APP_RUNTIME_COUNT:-1}" ;; + *"repl-runtimes :electron"*) echo "${FAKE_ELECTRON_RUNTIME_COUNT:-1}" ;; + *"repl-runtimes :db-worker-node"*) echo "${FAKE_DB_RUNTIME_COUNT:-1}" ;; + *) echo "0" ;; + esac + exit 0 + fi + + if [[ "${1:-}" == "cljs-repl" ]]; then + echo "pnpm-exec-repl ${2:-missing}" >> "$FAKE_CMD_LOG" + echo "shadow-cljs - connected to server" + + if [[ "$input" == *":cljs/quit"* ]]; then + echo "verification must not send :cljs/quit" >&2 + exit 1 + fi + + case "${2:-}" in + app) + if [[ "$input" != *":runtime :app"* ]]; then + echo "unexpected app verification form" >&2 + exit 1 + fi + echo 'cljs.user=> {:runtime :app, :document? true}' + ;; + electron) + if [[ "$input" != *":runtime :electron"* ]]; then + echo "unexpected electron verification form" >&2 + exit 1 + fi + echo 'cljs.user=> {:runtime :electron, :process? true}' + ;; + db-worker-node) + if [[ "$input" != *":runtime :db-worker-node"* ]]; then + echo "unexpected db-worker-node verification form" >&2 + exit 1 + fi + echo 'cljs.user=> {:runtime :db-worker-node, :process? true}' + ;; + *) + echo "Unexpected cljs-repl target: ${2:-}" >&2 + exit 1 + ;; + esac + + echo "cljs.user=>" + exit 0 + fi + + echo "Unexpected shadow-cljs command: $*" >&2 + exit 1 + ;; + *) + echo "Unexpected pnpm command: $*" >&2 + exit 1 + ;; +esac +EOF + + cat > "$BIN_DIR/node" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail + +echo "node $*" >> "$FAKE_CMD_LOG" + +repo="" +root_dir="" +owner_source="" +while [[ $# -gt 0 ]]; do + case "$1" in + ./static/db-worker-node.js) + shift + ;; + --repo) + repo="${2:-}" + shift 2 + ;; + --root-dir) + root_dir="${2:-}" + shift 2 + ;; + --owner-source) + owner_source="${2:-}" + shift 2 + ;; + *) + shift + ;; + esac +done + +if [[ -z "$repo" ]]; then + echo "repo is required" >&2 + exit 1 +fi + +if [[ -z "$root_dir" ]]; then + echo "root-dir is required" >&2 + exit 1 +fi + +if [[ "$owner_source" != "cli" ]]; then + echo "owner-source cli is required" >&2 + exit 1 +fi + +echo "shadow-cljs - #6 ready!" +while true; do sleep 1; done +EOF + + cat > "$BIN_DIR/lsof" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail + +exit 1 +EOF + + cat > "$BIN_DIR/python3" </dev/null || true + fi + done + fi + + PATH="$ORIGINAL_PATH" + unset FAKE_CMD_LOG FAKE_APP_RUNTIME_COUNT FAKE_ELECTRON_RUNTIME_COUNT FAKE_DB_RUNTIME_COUNT || true + + if [[ -n "${TEST_ROOT:-}" && -d "$TEST_ROOT" ]]; then + rm -rf "$TEST_ROOT" + fi +} + +scripts_exist_test() { + assert_file_exists "$START_SCRIPT" + assert_file_exists "$START_PY_SCRIPT" + assert_file_exists "$CLEANUP_SCRIPT" + assert_file_exists "$VERIFY_SCRIPT" +} + +start_launches_all_repl_processes_without_attaching_test() { + create_fake_env + trap cleanup_fake_env RETURN + + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1 + + assert_contains "Verifying CLJS REPL targets ..." "$TEST_ROOT/start.log" + assert_contains "REPL verification passed for :app" "$TEST_ROOT/start.log" + assert_contains "REPL verification passed for :electron" "$TEST_ROOT/start.log" + assert_contains "REPL verification passed for :db-worker-node" "$TEST_ROOT/start.log" + assert_contains "Startup complete. Attach to the needed REPL manually." "$TEST_ROOT/start.log" + assert_contains "pnpm exec shadow-cljs cljs-repl app" "$TEST_ROOT/start.log" + assert_contains "pnpm exec shadow-cljs cljs-repl electron" "$TEST_ROOT/start.log" + assert_contains "pnpm exec shadow-cljs cljs-repl db-worker-node" "$TEST_ROOT/start.log" + assert_contains "pnpm watch" "$CMD_LOG" + assert_contains "pnpm dev-electron-app" "$CMD_LOG" + assert_contains "node ./static/db-worker-node.js --repo demo --root-dir $DB_ROOT_DIR --owner-source cli" "$CMD_LOG" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.log" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.log" + assert_file_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.log" +} + +verify_script_checks_all_targets_test() { + create_fake_env + trap cleanup_fake_env RETURN + + bash "$VERIFY_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/verify.log" 2>&1 + + assert_contains "Verifying CLJS REPL targets ..." "$TEST_ROOT/verify.log" + assert_contains "REPL verification passed for :app" "$TEST_ROOT/verify.log" + assert_contains "REPL verification passed for :electron" "$TEST_ROOT/verify.log" + assert_contains "REPL verification passed for :db-worker-node" "$TEST_ROOT/verify.log" + assert_contains "All CLJS REPL targets verified." "$TEST_ROOT/verify.log" + assert_contains "pnpm-exec-repl app" "$CMD_LOG" + assert_contains "pnpm-exec-repl electron" "$CMD_LOG" + assert_contains "pnpm-exec-repl db-worker-node" "$CMD_LOG" +} + +verify_script_fails_when_target_repl_fails_test() { + create_fake_env + trap cleanup_fake_env RETURN + + cat > "$BIN_DIR/pnpm" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail + +echo "pnpm $*" >> "$FAKE_CMD_LOG" + +if [[ "${1:-}" == "exec" && "${2:-}" == "shadow-cljs" && "${3:-}" == "cljs-repl" && "${4:-}" == "electron" ]]; then + echo "No available JS runtime" >&2 + exit 1 +fi + +echo "shadow-cljs - connected to server" +echo "cljs.user=> {:ok true}" +EOF + chmod +x "$BIN_DIR/pnpm" + + if bash "$VERIFY_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/verify.log" 2>&1; then + fail "expected verify script to fail when one target REPL fails" + fi + + assert_contains "Error: REPL verification failed for :electron." "$TEST_ROOT/verify.log" + assert_contains "No available JS runtime" "$TEST_ROOT/verify.log" +} + +start_reuses_all_running_processes_test() { + create_fake_env + trap cleanup_fake_env RETURN + + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/first.log" 2>&1 + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/second.log" 2>&1 + + assert_equals "1" "$(grep -c '^pnpm watch$' "$CMD_LOG")" + assert_equals "1" "$(grep -c '^pnpm dev-electron-app$' "$CMD_LOG")" + assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $DB_ROOT_DIR --owner-source cli" "$CMD_LOG")" + assert_contains "Reusing shared shadow-cljs watch" "$TEST_ROOT/second.log" + assert_contains "Reusing Desktop dev app" "$TEST_ROOT/second.log" + assert_contains "Reusing db-worker-node runtime" "$TEST_ROOT/second.log" +} + +start_restarts_db_worker_when_root_dir_changes_test() { + create_fake_env + trap cleanup_fake_env RETURN + + local first_root second_root + first_root="$TEST_ROOT/logseq-root-a" + second_root="$TEST_ROOT/logseq-root-b" + mkdir -p "$first_root" "$second_root" + first_root="$(cd "$first_root" && pwd -P)" + second_root="$(cd "$second_root" && pwd -P)" + + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$first_root" --repo demo > "$TEST_ROOT/first.log" 2>&1 + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$second_root" --repo demo > "$TEST_ROOT/second.log" 2>&1 + + assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $first_root --owner-source cli" "$CMD_LOG")" + assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $second_root --owner-source cli" "$CMD_LOG")" + assert_contains "Stopping existing db-worker-node" "$TEST_ROOT/second.log" +} + +start_fails_when_app_runtime_is_ambiguous_test() { + create_fake_env + trap cleanup_fake_env RETURN + export FAKE_APP_RUNTIME_COUNT=2 + + if bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1; then + fail "expected start script to fail when more than one :app runtime exists" + fi + + assert_contains "Expected exactly one live :app runtime" "$TEST_ROOT/start.log" +} + +cleanup_stops_all_repl_processes_test() { + create_fake_env + trap cleanup_fake_env RETURN + + bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1 + + local watch_pid desktop_pid db_pid + watch_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid")" + desktop_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid")" + db_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid")" + + bash "$CLEANUP_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/cleanup.log" 2>&1 + + if kill -0 "$watch_pid" 2>/dev/null; then + fail "expected shared watch to stop" + fi + + if kill -0 "$desktop_pid" 2>/dev/null; then + fail "expected Desktop dev app to stop" + fi + + if kill -0 "$db_pid" 2>/dev/null; then + fail "expected db-worker-node to stop" + fi + + assert_not_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid" + assert_not_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid" + assert_not_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid" + assert_contains "Cleanup done." "$TEST_ROOT/cleanup.log" +} + +cleanup_removes_legacy_state_files_test() { + create_fake_env + trap cleanup_fake_env RETURN + + mkdir -p "$REPO_ROOT/tmp/desktop-app-repl" "$REPO_ROOT/tmp/db-worker-node-repl" "$REPO_ROOT/tmp/logseq-repl" + sleep 30 & + local legacy_desktop_pid=$! + sleep 30 & + local legacy_db_pid=$! + echo "$legacy_desktop_pid" > "$REPO_ROOT/tmp/desktop-app-repl/desktop-electron.pid" + echo "$legacy_db_pid" > "$REPO_ROOT/tmp/db-worker-node-repl/db-worker-node.pid" + echo "$legacy_db_pid" > "$REPO_ROOT/tmp/db-worker-node-repl/shadow-db-worker-node.pid" + + bash "$CLEANUP_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/cleanup.log" 2>&1 + + if kill -0 "$legacy_desktop_pid" 2>/dev/null; then + fail "expected legacy desktop pid to stop" + fi + + if kill -0 "$legacy_db_pid" 2>/dev/null; then + fail "expected legacy db-worker pid to stop" + fi + + assert_not_exists "$REPO_ROOT/tmp/desktop-app-repl/desktop-electron.pid" + assert_not_exists "$REPO_ROOT/tmp/db-worker-node-repl/db-worker-node.pid" + assert_not_exists "$REPO_ROOT/tmp/db-worker-node-repl/shadow-db-worker-node.pid" +} + +help_and_docs_describe_unified_scripts_test() { + local temp_dir start_help cleanup_help + temp_dir="$(mktemp -d)" + start_help="$temp_dir/start-help.txt" + cleanup_help="$temp_dir/cleanup-help.txt" + + bash "$START_SCRIPT" --help > "$start_help" + bash "$CLEANUP_SCRIPT" --help > "$cleanup_help" + + assert_contains "start-repl.sh" "$start_help" + assert_contains "start-repl.py" "$SKILL_FILE" + assert_contains "cleanup-repl.sh" "$cleanup_help" + assert_contains "start-repl.sh" "$SKILL_FILE" + assert_contains "cleanup-repl.sh" "$SKILL_FILE" + assert_contains "verify-repls.sh" "$SKILL_FILE" + assert_not_contains_text "start-desktop-app-repl.sh" "$SKILL_FILE" + assert_not_contains_text "start-db-worker-node-repl.sh" "$SKILL_FILE" + assert_not_contains_text "cleanup-desktop-app-repl.sh" "$SKILL_FILE" + assert_not_contains_text "cleanup-db-worker-node-repl.sh" "$SKILL_FILE" + + rm -rf "$temp_dir" +} + +run_test "scripts exist" scripts_exist_test +run_test "start launches all REPL processes without attaching" start_launches_all_repl_processes_without_attaching_test +run_test "verify script checks all targets" verify_script_checks_all_targets_test +run_test "verify script fails when target REPL fails" verify_script_fails_when_target_repl_fails_test +run_test "start reuses all running processes" start_reuses_all_running_processes_test +run_test "start restarts db-worker when root-dir changes" start_restarts_db_worker_when_root_dir_changes_test +run_test "start fails when app runtime is ambiguous" start_fails_when_app_runtime_is_ambiguous_test +run_test "cleanup stops all REPL processes" cleanup_stops_all_repl_processes_test +run_test "cleanup removes legacy state files" cleanup_removes_legacy_state_files_test +run_test "help and docs describe unified scripts" help_and_docs_describe_unified_scripts_test + +echo +if [[ "$FAIL_COUNT" -gt 0 ]]; then + echo "$FAIL_COUNT test(s) failed; $PASS_COUNT passed." >&2 + exit 1 +fi + +echo "All $PASS_COUNT test(s) passed." diff --git a/.carve/ignore b/.carve/ignore index 7145230ed3..0bf7dec926 100644 --- a/.carve/ignore +++ b/.carve/ignore @@ -60,6 +60,10 @@ frontend.ui/_emoji-init-data frontend.worker.rtc.op-mem-layer/_sync-loop-canceler ;; Used by shadow.cljs frontend.worker.db-worker/init +;; Used by shadow.cljs (node entrypoint) +frontend.worker.db-worker-node/main +;; CLI entrypoint (shadow-cljs :node-script) +logseq.cli.main/main ;; Future use? frontend.worker.rtc.hash/hash-blocks ;; Repl fn diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index f2d0a3cb57..d66d42db48 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -62,6 +62,7 @@ jobs: uses: DeLaGuardo/setup-clojure@13.5 with: cli: ${{ env.CLOJURE_VERSION }} + bb: ${{ env.BABASHKA_VERSION }} - name: Clojure cache uses: actions/cache@v4 @@ -84,11 +85,21 @@ jobs: run: clojure -M:test compile test - name: Run ClojureScript query tests against basic query type - run: DB_QUERY_TYPE=basic node static/tests.js -r frontend.db.query-dsl-test + run: DB_QUERY_TYPE=basic pnpm cljs:run-test -r frontend.db.query-dsl-test - name: Run ClojureScript tests run: pnpm cljs:run-test + - name: Setup CLI integration test + # logseq-cli.js needed just for test-cli-query-human-output-pipes-to-show + run: clojure -M:cljs compile logseq-cli && pnpm db-worker-node:release:bundle + + - name: Run ClojureScript integration tests + run: pnpm cljs:run-integration-test + + - name: Run CLI E2E tests + run: bb dev:cli-e2e --verbose --skip-build + lint: runs-on: ubuntu-22.04 diff --git a/.github/workflows/deps-graph-parser.yml b/.github/workflows/deps-graph-parser.yml index e98d7ea2d5..ad8fb53636 100644 --- a/.github/workflows/deps-graph-parser.yml +++ b/.github/workflows/deps-graph-parser.yml @@ -83,9 +83,6 @@ jobs: - name: Fetch pnpm deps run: pnpm install --frozen-lockfile - - name: Run ClojureScript tests - run: clojure -M:test - - name: Run nbb-logseq tests run: pnpm test diff --git a/.gitignore b/.gitignore index d4eeb375e7..5e58de6737 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ node_modules/ static/** tmp cljs-test-runner-out +.tmp/ .cpcache/ /src/gen @@ -44,6 +45,7 @@ resources/electron.js /libs/dist/ charlie/ .vscode +/.claude /.preprocessor-cljs docker android/app/src/main/assets/capacitor.plugin.json @@ -79,3 +81,12 @@ clj-e2e/e2e-dump .dir-locals.el .projectile deps/db-sync/data +*.map +/dist/db-worker-node.js +/dist/build/ +/dist/db-worker-node-assets.json +/dist/*.wasm +/dist/cljs-runtime/ +/.agent-shell/ + +clj-e2e/.clj-kondo/imports \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 94ee64e1f0..c7b3d66e2d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,20 +1,20 @@ # Repository Guidelines -## Project Structure & Module Organization -- `src/` is the main codebase. - - `src/main/` contains core application logic. - - `src/main/mobile/` is the mobile app code. - - `src/main/frontend/components/` houses UI components. - - `src/main/frontend/worker/` holds webworker code, including RTC in `src/main/frontend/worker/rtc/`. -- `src/electron/` is Electron-specific code. -- `src/test/` contains unit tests. -- `deps/` contains internal dependencies/modules. -- `clj-e2e/` contains end-to-end tests. - ## Build, Test, and Development Commands - `bb dev:lint-and-test` runs linters and unit tests. - `bb dev:test -v ` runs a single unit test (example: `bb dev:test -v logseq.some-test/foo`). -- E2E tests live in `clj-e2e/`; run them from that directory if needed. +- App E2E tests live in `clj-e2e/`; run from that directory with `bb test` (or `bb -f clj-e2e/bb.edn test` from repo root). +- CLI E2E tests live in `cli-e2e/`; run with `bb -f cli-e2e/bb.edn test --skip-build` (or `bb -f cli-e2e/bb.edn build` first when needed). +- If a request says only “e2e”, clarify whether it targets `clj-e2e/` or `cli-e2e/` before planning changes. + +## Error handling and compatibility +- When modifying code, first consider removing compatibility layers rather than extending them. +- Prefer fail-fast over fallback. +- Do not add backward compatibility unless explicitly requested. +- Do not introduce default values to mask invalid state. +- Do not silently recover from programmer errors. +- Keep one clear code path whenever possible. +- Internal code may assume well-formed inputs from controlled callers. ## Coding Style & Naming Conventions - ClojureScript keywords are defined via `logseq.common.defkeywords/defkeyword`; use existing keywords and add new ones in the shared definitions. @@ -29,12 +29,16 @@ - Name tests after their namespaces; use `-v` to target a specific test case. - Run lint/tests before submitting PRs; keep changes green. +## *IMPORTANT*: Always respect directory-specific AGENTS.md based on file path +- when editing code in a specific directory, you must recursively read `AGENTS.md` files up the directory tree, and `AGENTS.md` in subdirectories takes precedence over the root-level one + ## Commit & Pull Request Guidelines - Commit subjects are short and imperative; optional scope prefixes appear (e.g., `fix:` or `enhance(rtc):`). - PRs should describe the behavior change, link relevant issues, and note any test coverage added or skipped. ## Agent-Specific Notes -- Project-specific skills live under `.agents/skills/`; load `.agents/skills/logseq-i18n/SKILL.md` for i18n/localization/hardcoded UI text tasks. +- Use repo-local skills discovered under `.agents/skills/`; load the matching `SKILL.md` before editing files or proposing changes. +- **i18n (mandatory)**: Always load `.agents/skills/logseq-i18n/SKILL.md` before any change that adds, edits, or removes user-facing UI text, regardless of whether other skills also apply. - Review notes live in `prompts/review.md`; check them when preparing changes. - DB-sync feature guide for AI agents: `docs/agent-guide/db-sync/db-sync-guide.md`. - DB-sync protocol reference: `docs/agent-guide/db-sync/protocol.md`. diff --git a/README.md b/README.md index d5f8c0ba89..04e75f892e 100644 --- a/README.md +++ b/README.md @@ -169,6 +169,10 @@ If you want to set up a development environment for the Logseq web or desktop ap In addition to these guides, you can also find other helpful resources in the [docs/](docs/) folder, such as the [Guide for Contributing to Translations](docs/contributing-to-translations.md), the [Docker Web App Guide](docs/docker-web-app-guide.md) and the [mobile development guide](docs/develop-logseq-on-mobile.md) +### 🧰 Logseq CLI (Node) + +Logseq CLI documentation is maintained in `docs/cli/logseq-cli.md`. + ## ✨ Inspiration Logseq is inspired by several unique tools and projects, including [Roam Research](https://roamresearch.com/), [Org Mode](https://orgmode.org/), [TiddlyWiki](https://tiddlywiki.com/), [Workflowy](https://workflowy.com/), and [Cuekeeper](https://github.com/talex5/cuekeeper). diff --git a/bb.edn b/bb.edn index 71f92b5491..2d447a6429 100644 --- a/bb.edn +++ b/bb.edn @@ -68,6 +68,7 @@ (run '-dev:publishing-dev {:parallel true}) (run '-dev:publishing-release))} + ;; legacy cli dev:cli {:doc "Run CLI with current deps/db code. Commands with JS deps are not usable e.g. mcp-server" :task (apply shell {:dir "deps/db"} @@ -192,6 +193,24 @@ dev:gen-malli-kondo-config logseq.tasks.dev/gen-malli-kondo-config + dev:db-worker-node + {:doc "Compile and start db-worker-node (pass-through args forwarded to node)" + :task (do + (shell "clojure" "-M:cljs" "compile" "db-worker-node") + (apply shell "node" "./static/db-worker-node.js" *command-line-args*))} + + dev:cli-e2e + {:doc "Run shell-first CLI end-to-end tests" + :task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "test" *command-line-args*)} + + dev:cli-e2e-cleanup + {:doc "Run shell-first CLI end-to-end cleanup" + :task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "cleanup" *command-line-args*)} + + dev:cli-e2e-sync + {:doc "Run shell-first CLI sync end-to-end tests" + :task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "test-sync" *command-line-args*)} + lint:dev logseq.tasks.dev.lint/dev diff --git a/cli-e2e/AGENTS.md b/cli-e2e/AGENTS.md new file mode 100644 index 0000000000..e872a40c81 --- /dev/null +++ b/cli-e2e/AGENTS.md @@ -0,0 +1,38 @@ +# cli-e2e + +Shell-first end-to-end tests for logseq CLI. + +## Test cli-e2e itself +- Run internal cli-e2e harness unit tests: `bb unit-test` + +## cleanup first +- Execute cleanup: terminate stale db-worker-node processes, terminate stale db-sync listeners on port `18080`, and remove cli-e2e temp roots: `bb cleanup` +- Preview cleanup actions only (no kill/delete): `bb cleanup --dry-run` + +## Run non-sync suite +- List declared non-sync case ids: `bb list-cases` +- Run non-sync cases with build preflight unless `--skip-build` is provided: `bb test` + - `bb test --help` for options + - Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test --skip-build --jobs 4` + - Parallelism is case-scoped only; each case still runs setup/main/cleanup sequentially in the existing ephemeral shell model + +## Run sync suite +- List declared sync case ids: `bb list-sync-cases` +- Run sync cases with build preflight by default: `bb test-sync` + - Add `--skip-build` only when you intentionally want to reuse existing artifacts + - `bb test-sync --help` for options + - Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test-sync --jobs 4` + - Parallelism is case-scoped only; each sync case still runs its own setup/main/cleanup sequentially + - The local db-sync server is shared per suite and starts once before cases begin, then stops once after all cases complete + - Each sync case gets its own isolated temp root, data directories, home directory, auth copy, and generated config files + - Configure sync E2EE password: `--e2ee-password ` (default: `11111`) + - Run only sync MVP case: `bb test-sync --case sync-upload-download-mvp` + +### Sync suite prerequisites +- CLI auth file must exist at `~/logseq/auth.json` (generated by `logseq login`). +- Sync suite starts/stops one shared local db-sync server per suite run. +- Required build artifact for local db-sync server: + - `deps/db-sync/worker/dist/node-adapter.js` +- If artifact is missing, build it before running sync suite: + - `pnpm --dir deps/db-sync build:node-adapter` + diff --git a/cli-e2e/README.md b/cli-e2e/README.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/cli-e2e/README.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/cli-e2e/bb.edn b/cli-e2e/bb.edn new file mode 100644 index 0000000000..8011c614e5 --- /dev/null +++ b/cli-e2e/bb.edn @@ -0,0 +1,47 @@ +{:paths ["src" "test"] + :deps {cheshire/cheshire {:mvn/version "5.12.0"}} + :tasks + {:requires ([babashka.cli :as cli] + [logseq.cli.e2e.main :as main] + [logseq.cli.e2e.test-runner :as test-runner]) + :init (def cli-opts + (cli/parse-opts + *command-line-args* + {:alias {:i :include + :h :help} + :spec {:jobs {:default 4}} + :coerce {:include [] + :help :boolean + :dry-run :boolean + :skip-build :boolean + :verbose :boolean + :timings :boolean + :jobs :long}})) + + unit-test + {:doc "Run internal cli-e2e harness unit tests" + :task (test-runner/run! cli-opts)} + + build + {:doc "Compile required CLI artifacts and verify expected outputs exist" + :task (main/build! cli-opts)} + + list-cases + {:doc "List declared non-sync cli-e2e case ids" + :task (main/list-cases! cli-opts)} + + list-sync-cases + {:doc "List declared sync cli-e2e case ids" + :task (main/list-sync-cases! cli-opts)} + + test + {:doc "Run non-sync cli-e2e cases with build preflight unless --skip-build is provided" + :task (main/test! cli-opts)} + + test-sync + {:doc "Run sync cli-e2e cases with build preflight unless --skip-build is provided; use --jobs for case-level parallelism" + :task (main/test-sync! cli-opts)} + + cleanup + {:doc "Terminate cli-e2e db-worker-node processes, terminate stale db-sync listeners, and remove cli-e2e temp roots" + :task (main/cleanup! cli-opts)}}} diff --git a/cli-e2e/scripts/compare_graph_queries.py b/cli-e2e/scripts/compare_graph_queries.py new file mode 100644 index 0000000000..b5d95c1e35 --- /dev/null +++ b/cli-e2e/scripts/compare_graph_queries.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Compare normalized query payloads between two cli graph contexts.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any, Dict + +IGNORED_KEYS = { + "db/id", + "db/created-at", + "db/updated-at", + "block/uuid", + "block/updated-at", + "block/created-at", +} + + +def fail(message: str, **context: object) -> None: + payload = {"status": "error", "message": message} + if context: + payload["context"] = context + print(json.dumps(payload), file=sys.stderr) + raise SystemExit(1) + + +def normalize(value: Any) -> Any: + if isinstance(value, dict): + normalized = {} + for key in sorted(value.keys(), key=str): + key_str = str(key) + if key_str in IGNORED_KEYS: + continue + normalized[key_str] = normalize(value[key]) + return normalized + if isinstance(value, list): + normalized_list = [normalize(item) for item in value] + try: + return sorted(normalized_list, key=lambda item: json.dumps(item, sort_keys=True)) + except TypeError: + return normalized_list + return value + + +def run_query(cli_path: Path, config_path: Path, root_dir: Path, graph: str, query: str) -> Dict[str, Any]: + command = [ + "node", + str(cli_path), + "--root-dir", + str(root_dir), + "--config", + str(config_path), + "--output", + "json", + "query", + "--graph", + graph, + "--query", + query, + ] + + result = subprocess.run(command, capture_output=True, text=True) + if result.returncode != 0: + fail( + "query command failed", + command=command, + exit=result.returncode, + stdout=result.stdout, + stderr=result.stderr, + ) + + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as error: + fail( + "query command did not return valid JSON", + command=command, + stdout=result.stdout, + stderr=result.stderr, + detail=str(error), + ) + + if payload.get("status") != "ok": + fail("query command returned non-ok status", command=command, payload=payload) + + data = payload.get("data") or {} + return { + "payload": payload, + "result": data.get("result"), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Compare normalized query payloads between two cli contexts") + parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js") + parser.add_argument("--graph", required=True) + parser.add_argument("--query", required=True, action="append") + parser.add_argument("--config-a", required=True) + parser.add_argument("--root-dir-a", required=True) + parser.add_argument("--config-b", required=True) + parser.add_argument("--root-dir-b", required=True) + parser.add_argument("--require-result", action="store_true") + return parser.parse_args() + + + +def main() -> None: + args = parse_args() + + cli_path = Path(args.cli).expanduser().resolve() + if not cli_path.exists(): + fail("cli entry file does not exist", cli=str(cli_path)) + + queries = args.query + left_config = Path(args.config_a).expanduser().resolve() + left_root_dir = Path(args.root_dir_a).expanduser().resolve() + right_config = Path(args.config_b).expanduser().resolve() + right_root_dir = Path(args.root_dir_b).expanduser().resolve() + + normalized_results = {} + + for query in queries: + left = run_query( + cli_path, + left_config, + left_root_dir, + args.graph, + query, + ) + right = run_query( + cli_path, + right_config, + right_root_dir, + args.graph, + query, + ) + + left_result = left["result"] + right_result = right["result"] + + if args.require_result and (left_result is None or right_result is None): + fail( + "query result is empty", + query=query, + left_result=left_result, + right_result=right_result, + ) + + left_normalized = normalize(left_result) + right_normalized = normalize(right_result) + + if left_normalized != right_normalized: + fail( + "normalized query results differ", + query=query, + left_result=left_normalized, + right_result=right_normalized, + left_payload=left["payload"], + right_payload=right["payload"], + ) + + normalized_results[query] = left_normalized + + payload_key = "result" if len(normalized_results) == 1 else "results" + payload_value = next(iter(normalized_results.values())) if len(normalized_results) == 1 else normalized_results + print( + json.dumps( + { + "status": "ok", + payload_key: payload_value, + } + ) + ) + + +if __name__ == "__main__": + main() diff --git a/cli-e2e/scripts/db_sync_server.py b/cli-e2e/scripts/db_sync_server.py new file mode 100644 index 0000000000..fe041a4f92 --- /dev/null +++ b/cli-e2e/scripts/db_sync_server.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +"""Manage local db-sync server process for cli-e2e sync suite.""" + +from __future__ import annotations + +import argparse +import base64 +import ctypes +import json +import os +import signal +import socket +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any, Dict, Optional + +DEFAULT_COGNITO_ISSUER = "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_kAqZcxIeM" +DEFAULT_COGNITO_CLIENT_ID = "1qi1uijg8b6ra70nejvbptis0q" + + +def fail(message: str, **context: object) -> None: + payload = {"status": "error", "message": message} + if context: + payload["context"] = context + print(json.dumps(payload), file=sys.stderr) + raise SystemExit(1) + + +def load_pid(pid_file: Path) -> Optional[int]: + if not pid_file.exists(): + return None + raw = pid_file.read_text(encoding="utf-8").strip() + if not raw: + return None + try: + return int(raw) + except ValueError: + return None + + +def process_running(pid: int) -> bool: + if pid <= 0: + return False + if os.name == "nt": + kernel32 = ctypes.windll.kernel32 + access = 0x00100000 | 0x1000 # SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION + handle = kernel32.OpenProcess(access, False, pid) + if handle: + kernel32.CloseHandle(handle) + return True + return ctypes.get_last_error() == 5 + try: + os.kill(pid, 0) + except ProcessLookupError: + return False + except PermissionError: + return True + else: + return True + + +def terminate_process(pid: int, force: bool) -> bool: + if pid <= 0: + return True + if os.name == "nt": + cmd = ["taskkill", "/PID", str(pid), "/T"] + if force: + cmd.append("/F") + result = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False) + return result.returncode == 0 or not process_running(pid) + try: + os.kill(pid, signal.SIGKILL if force else signal.SIGTERM) + except ProcessLookupError: + return True + except PermissionError: + return False + return True + + +def parse_json_file(path: Path) -> Dict[str, Any]: + if not path.exists(): + fail("sync auth file is missing", auth_path=str(path), hint="Run `logseq login` first.") + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + fail("sync auth file is invalid JSON", auth_path=str(path), detail=str(error)) + if not isinstance(payload, dict): + fail("sync auth file must be a JSON object", auth_path=str(path)) + return payload + + +def decode_jwt_claims(token: str) -> Optional[Dict[str, Any]]: + if not isinstance(token, str): + return None + parts = token.split(".") + if len(parts) != 3: + return None + payload = parts[1] + padded = payload + "=" * ((4 - (len(payload) % 4)) % 4) + try: + decoded = base64.urlsafe_b64decode(padded.encode("utf-8")).decode("utf-8") + claims = json.loads(decoded) + except (ValueError, UnicodeDecodeError, json.JSONDecodeError): + return None + return claims if isinstance(claims, dict) else None + + +def auth_cognito_from_auth_file(auth_path: Path) -> Dict[str, str]: + payload = parse_json_file(auth_path) + token = payload.get("id-token") or payload.get("access-token") + claims = decode_jwt_claims(token) if isinstance(token, str) else None + + issuer = claims.get("iss") if isinstance(claims, dict) else None + client_id = None + if isinstance(claims, dict): + client_id = claims.get("aud") or claims.get("client_id") + + return { + "issuer": issuer if isinstance(issuer, str) and issuer else "", + "client_id": client_id if isinstance(client_id, str) and client_id else "", + } + + +def wait_health(base_url: str, timeout_s: float, interval_s: float) -> bool: + deadline = time.time() + timeout_s + url = base_url.rstrip("/") + "/health" + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + while time.time() < deadline: + try: + with opener.open(url, timeout=2) as response: + if response.status == 200: + return True + except (urllib.error.URLError, TimeoutError, socket.timeout): + pass + time.sleep(interval_s) + return False + + +def start_server(args: argparse.Namespace) -> None: + repo_root = Path(args.repo_root).expanduser().resolve() + entry = repo_root / "deps" / "db-sync" / "worker" / "dist" / "node-adapter.js" + if not entry.exists(): + fail("db-sync node adapter build artifact is missing", entry=str(entry), hint="Run: yarn --cwd deps/db-sync build:node-adapter") + + pid_file = Path(args.pid_file).expanduser().resolve() + log_file = Path(args.log_file).expanduser().resolve() + data_dir = Path(args.data_dir).expanduser().resolve() + + pid_file.parent.mkdir(parents=True, exist_ok=True) + log_file.parent.mkdir(parents=True, exist_ok=True) + data_dir.mkdir(parents=True, exist_ok=True) + + existing_pid = load_pid(pid_file) + if existing_pid and process_running(existing_pid): + fail("db-sync server already running", pid=existing_pid, pid_file=str(pid_file)) + + auth_path = Path(args.auth_path).expanduser().resolve() if args.auth_path else None + auth_derived = auth_cognito_from_auth_file(auth_path) if auth_path else {"issuer": "", "client_id": ""} + + issuer = args.cognito_issuer or auth_derived.get("issuer") or DEFAULT_COGNITO_ISSUER + client_id = args.cognito_client_id or auth_derived.get("client_id") or DEFAULT_COGNITO_CLIENT_ID + jwks_url = args.cognito_jwks_url or f"{issuer}/.well-known/jwks.json" + + env = os.environ.copy() + env.update( + { + "DB_SYNC_PORT": str(args.port), + "DB_SYNC_DATA_DIR": str(data_dir), + "COGNITO_ISSUER": issuer, + "COGNITO_CLIENT_ID": client_id, + "COGNITO_JWKS_URL": jwks_url, + # CLI e2e sync suite should remain runnable without outbound internet. + "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS": "true", + } + ) + + with log_file.open("a", encoding="utf-8") as stream: + stream.write(f"\n=== cli-e2e db-sync start {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n") + stream.flush() + process = subprocess.Popen( + ["node", str(entry)], + stdout=stream, + stderr=stream, + cwd=str(repo_root), + env=env, + start_new_session=True, + ) + + base_url = f"http://{args.host}:{args.port}" + if not wait_health(base_url, args.startup_timeout_s, args.poll_interval_s): + terminate_process(process.pid, force=False) + fail( + "db-sync server failed health check before timeout", + base_url=base_url, + pid=process.pid, + log_file=str(log_file), + ) + + pid_file.write_text(f"{process.pid}\n", encoding="utf-8") + print( + json.dumps( + { + "status": "ok", + "pid": process.pid, + "pid_file": str(pid_file), + "log_file": str(log_file), + "base_url": base_url, + "data_dir": str(data_dir), + "cognito_issuer": issuer, + "cognito_client_id": client_id, + "auth_path": str(auth_path) if auth_path else None, + } + ) + ) + + +def stop_server(args: argparse.Namespace) -> None: + pid_file = Path(args.pid_file).expanduser().resolve() + pid = load_pid(pid_file) + + if pid is None: + print(json.dumps({"status": "ok", "stopped": False, "reason": "pid-file-missing-or-empty", "pid_file": str(pid_file)})) + return + + if not process_running(pid): + try: + pid_file.unlink(missing_ok=True) + except OSError: + pass + print(json.dumps({"status": "ok", "stopped": False, "reason": "process-not-running", "pid": pid, "pid_file": str(pid_file)})) + return + + if not terminate_process(pid, force=False): + fail("db-sync server stop failed", pid=pid, signal="SIGTERM", pid_file=str(pid_file)) + deadline = time.time() + args.shutdown_timeout_s + while time.time() < deadline: + if not process_running(pid): + pid_file.unlink(missing_ok=True) + print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGTERM", "pid": pid, "pid_file": str(pid_file)})) + return + time.sleep(args.poll_interval_s) + + if not terminate_process(pid, force=True): + fail("db-sync server force stop failed", pid=pid, signal="SIGKILL", pid_file=str(pid_file)) + pid_file.unlink(missing_ok=True) + print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGKILL", "pid": pid, "pid_file": str(pid_file)})) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Manage db-sync local server for cli-e2e sync tests") + subparsers = parser.add_subparsers(dest="command", required=True) + + start = subparsers.add_parser("start", help="Start server and wait for /health") + start.add_argument("--repo-root", required=True) + start.add_argument("--pid-file", required=True) + start.add_argument("--log-file", required=True) + start.add_argument("--data-dir", required=True) + start.add_argument("--host", default="127.0.0.1") + start.add_argument("--port", type=int, default=8080) + start.add_argument("--startup-timeout-s", type=float, default=25.0) + start.add_argument("--poll-interval-s", type=float, default=0.5) + start.add_argument("--auth-path", default="~/logseq/auth.json") + start.add_argument("--cognito-issuer") + start.add_argument("--cognito-client-id") + start.add_argument("--cognito-jwks-url") + + stop = subparsers.add_parser("stop", help="Stop server if running") + stop.add_argument("--pid-file", required=True) + stop.add_argument("--shutdown-timeout-s", type=float, default=10.0) + stop.add_argument("--poll-interval-s", type=float, default=0.25) + + return parser + + +def main() -> None: + args = build_parser().parse_args() + if args.command == "start": + start_server(args) + elif args.command == "stop": + stop_server(args) + else: + fail("unknown command", command=args.command) + + +if __name__ == "__main__": + main() diff --git a/cli-e2e/scripts/prepare_sync_config.py b/cli-e2e/scripts/prepare_sync_config.py new file mode 100644 index 0000000000..9b750f6ae6 --- /dev/null +++ b/cli-e2e/scripts/prepare_sync_config.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +"""Prepare per-case CLI config for sync e2e tests.""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +def fail(message: str, **context: object) -> None: + payload = {"status": "error", "message": message} + if context: + payload["context"] = context + print(json.dumps(payload), file=sys.stderr) + raise SystemExit(1) + + +def read_auth(auth_path: Path) -> dict: + if not auth_path.exists(): + fail("sync auth file is missing", auth_path=str(auth_path), hint="Run `logseq login` first.") + try: + payload = json.loads(auth_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as error: + fail("sync auth file is invalid JSON", auth_path=str(auth_path), detail=str(error)) + + has_token = any(payload.get(key) for key in ("refresh-token", "id-token", "access-token")) + if not has_token: + fail( + "sync auth file does not contain usable tokens", + auth_path=str(auth_path), + required_any_of=["refresh-token", "id-token", "access-token"], + ) + return payload + + +def write_config(output_path: Path, http_base: str, ws_url: str) -> None: + output_path.parent.mkdir(parents=True, exist_ok=True) + payload = "\n".join( + [ + "{", + " :output-format :json", + f' :http-base "{http_base}"', + f' :ws-url "{ws_url}"', + "}", + "", + ] + ) + output_path.write_text(payload, encoding="utf-8") + + +def main() -> None: + parser = argparse.ArgumentParser(description="Prepare cli.edn for sync e2e") + parser.add_argument("--output", required=True) + parser.add_argument("--auth-path", default="~/logseq/auth.json") + parser.add_argument("--http-base", required=True) + parser.add_argument("--ws-url", required=True) + args = parser.parse_args() + + auth_path = Path(args.auth_path).expanduser().resolve() + _auth = read_auth(auth_path) + + output_path = Path(args.output).expanduser().resolve() + write_config(output_path, args.http_base, args.ws_url) + + print( + json.dumps( + { + "status": "ok", + "auth_path": str(auth_path), + "config_path": str(output_path), + "http_base": args.http_base, + "ws_url": args.ws_url, + } + ) + ) + + +if __name__ == "__main__": + main() diff --git a/cli-e2e/scripts/random_bidirectional_block_ops.py b/cli-e2e/scripts/random_bidirectional_block_ops.py new file mode 100644 index 0000000000..d68c1ef5cd --- /dev/null +++ b/cli-e2e/scripts/random_bidirectional_block_ops.py @@ -0,0 +1,480 @@ +#!/usr/bin/env python3 +"""Run randomized bidirectional block operations on two synced graph peers.""" + +from __future__ import annotations + +import argparse +import json +import random +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, List + + +class CliCommandError(RuntimeError): + """Raised when a CLI command does not complete successfully.""" + + def __init__(self, message: str, *, context: Dict[str, Any]) -> None: + super().__init__(message) + self.context = context + + +@dataclass(frozen=True) +class ClientContext: + name: str + config: Path + root_dir: Path + + +def fail(message: str, **context: object) -> None: + payload = {"status": "error", "message": message} + if context: + payload["context"] = context + print(json.dumps(payload), file=sys.stderr) + raise SystemExit(1) + + +def run_cli_json( + *, + cli_path: Path, + graph: str, + client: ClientContext, + args: List[str], +) -> Dict[str, Any]: + command = [ + "node", + str(cli_path), + "--root-dir", + str(client.root_dir), + "--config", + str(client.config), + "--output", + "json", + *args, + "--graph", + graph, + ] + result = subprocess.run(command, capture_output=True, text=True) + if result.returncode != 0: + raise CliCommandError( + "cli command exited with non-zero status", + context={ + "client": client.name, + "command": command, + "exit": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + }, + ) + + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as error: + raise CliCommandError( + "cli command did not return valid json", + context={ + "client": client.name, + "command": command, + "stdout": result.stdout, + "stderr": result.stderr, + "detail": str(error), + }, + ) from error + + if payload.get("status") != "ok": + raise CliCommandError( + "cli command returned non-ok status", + context={ + "client": client.name, + "command": command, + "payload": payload, + }, + ) + return payload + + +def page_block_ids( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, +) -> List[int]: + query = ( + "[:find [?e ...] " + ":where " + f"[?p :block/title {json.dumps(page_title)}] " + "[?e :block/page ?p] " + "[?e :block/uuid]]" + ) + payload = run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=["query", "--query", query], + ) + result = (payload.get("data") or {}).get("result") + if not isinstance(result, list): + return [] + output: List[int] = [] + for item in result: + try: + output.append(int(item)) + except (TypeError, ValueError): + continue + return output + + +def upsert_page( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, +) -> None: + run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=["upsert", "page", "--page", page_title], + ) + + +def create_block( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, + content: str, + ids: List[int], + rng: random.Random, +) -> None: + if ids and rng.random() < 0.6: + target_id = str(rng.choice(ids)) + run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=[ + "upsert", + "block", + "--target-id", + target_id, + "--pos", + "first-child", + "--content", + content, + ], + ) + return + + run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=[ + "upsert", + "block", + "--target-page", + page_title, + "--content", + content, + ], + ) + + +def move_block( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, + ids: List[int], + rng: random.Random, +) -> bool: + if not ids: + return False + source_id = str(rng.choice(ids)) + run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=[ + "upsert", + "block", + "--id", + source_id, + "--target-page", + page_title, + "--pos", + "last-child", + ], + ) + return True + + +def delete_block( + *, + cli_path: Path, + graph: str, + client: ClientContext, + ids: List[int], + rng: random.Random, +) -> bool: + if len(ids) <= 2: + return False + source_id = str(rng.choice(ids)) + run_cli_json( + cli_path=cli_path, + graph=graph, + client=client, + args=["remove", "block", "--id", source_id], + ) + return True + + +def feasible(operation: str, ids: List[int]) -> bool: + if operation == "create": + return True + if operation == "move": + return len(ids) >= 1 + if operation == "delete": + return len(ids) > 2 + return False + + +def choose_operation(op_counts: Dict[str, int], ids: List[int], rng: random.Random) -> str: + for required in ("create", "move", "delete"): + if op_counts.get(required, 0) == 0 and feasible(required, ids): + return required + + candidates = ["create", "create", "move"] + if feasible("delete", ids): + candidates.append("delete") + if feasible("move", ids): + candidates.append("move") + return rng.choice(candidates) + + +def apply_operation( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, + op_counts: Dict[str, int], + round_index: int, + rng: random.Random, +) -> None: + last_error: Dict[str, Any] | None = None + for attempt in range(1, 7): + ids = page_block_ids( + cli_path=cli_path, + graph=graph, + client=client, + page_title=page_title, + ) + operation = choose_operation(op_counts, ids, rng) + content = f"{client.name}-rnd-{round_index:03d}-{rng.randint(100000, 999999)}" + + try: + if operation == "create": + create_block( + cli_path=cli_path, + graph=graph, + client=client, + page_title=page_title, + content=content, + ids=ids, + rng=rng, + ) + op_counts["create"] = op_counts.get("create", 0) + 1 + return + + if operation == "move": + moved = move_block( + cli_path=cli_path, + graph=graph, + client=client, + page_title=page_title, + ids=ids, + rng=rng, + ) + if moved: + op_counts["move"] = op_counts.get("move", 0) + 1 + return + continue + + if operation == "delete": + deleted = delete_block( + cli_path=cli_path, + graph=graph, + client=client, + ids=ids, + rng=rng, + ) + if deleted: + op_counts["delete"] = op_counts.get("delete", 0) + 1 + return + continue + except CliCommandError as error: + last_error = { + "attempt": attempt, + "operation": operation, + "context": error.context, + } + continue + + fail( + "failed to apply random operation after retries", + client=client.name, + round_index=round_index, + last_error=last_error, + ) + + +def ensure_non_empty_page( + *, + cli_path: Path, + graph: str, + client: ClientContext, + page_title: str, + rng: random.Random, +) -> None: + ids = page_block_ids( + cli_path=cli_path, + graph=graph, + client=client, + page_title=page_title, + ) + if ids: + return + content = f"{client.name}-reseed-{rng.randint(100000, 999999)}" + create_block( + cli_path=cli_path, + graph=graph, + client=client, + page_title=page_title, + content=content, + ids=[], + rng=rng, + ) + + +PROFILE_DEFAULT_ROUNDS = { + "default": 40, + "high-stress": 100, +} + + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Run randomized bidirectional block operations on two synced graph clients" + ) + parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js") + parser.add_argument("--graph", required=True) + parser.add_argument("--config-a", required=True) + parser.add_argument("--root-dir-a", required=True) + parser.add_argument("--config-b", required=True) + parser.add_argument("--root-dir-b", required=True) + parser.add_argument("--page", required=True) + parser.add_argument( + "--profile", + choices=sorted(PROFILE_DEFAULT_ROUNDS.keys()), + default="default", + help="Execution profile controlling default stress level", + ) + parser.add_argument("--rounds-per-client", type=int, default=None) + parser.add_argument("--seed", type=int, default=424242) + args = parser.parse_args() + if args.rounds_per_client is None: + args.rounds_per_client = PROFILE_DEFAULT_ROUNDS[args.profile] + return args + + +def main() -> None: + args = parse_args() + cli_path = Path(args.cli).expanduser().resolve() + if not cli_path.exists(): + fail("cli path does not exist", cli=str(cli_path)) + + rng = random.Random(args.seed) + client_a = ClientContext( + name="a", + config=Path(args.config_a).expanduser().resolve(), + root_dir=Path(args.root_dir_a).expanduser().resolve(), + ) + client_b = ClientContext( + name="b", + config=Path(args.config_b).expanduser().resolve(), + root_dir=Path(args.root_dir_b).expanduser().resolve(), + ) + clients = [client_a, client_b] + + for client in clients: + upsert_page( + cli_path=cli_path, + graph=args.graph, + client=client, + page_title=args.page, + ) + + # Seed both peers so move/delete have available targets from the beginning. + for seed_round in range(3): + for client in clients: + create_block( + cli_path=cli_path, + graph=args.graph, + client=client, + page_title=args.page, + content=f"{client.name}-seed-{seed_round}-{rng.randint(100000, 999999)}", + ids=[], + rng=rng, + ) + + op_stats: Dict[str, Dict[str, int]] = { + client.name: {"create": 0, "move": 0, "delete": 0} for client in clients + } + + for round_index in range(args.rounds_per_client): + for client in clients: + apply_operation( + cli_path=cli_path, + graph=args.graph, + client=client, + page_title=args.page, + op_counts=op_stats[client.name], + round_index=round_index, + rng=rng, + ) + + for client in clients: + ensure_non_empty_page( + cli_path=cli_path, + graph=args.graph, + client=client, + page_title=args.page, + rng=rng, + ) + + print( + json.dumps( + { + "status": "ok", + "graph": args.graph, + "page": args.page, + "rounds_per_client": args.rounds_per_client, + "seed": args.seed, + "stats": op_stats, + "total_operations": args.rounds_per_client * len(clients), + } + ) + ) + + +if __name__ == "__main__": + main() diff --git a/cli-e2e/scripts/wait_sync_status.py b/cli-e2e/scripts/wait_sync_status.py new file mode 100644 index 0000000000..0c2804112d --- /dev/null +++ b/cli-e2e/scripts/wait_sync_status.py @@ -0,0 +1,302 @@ +#!/usr/bin/env python3 +"""Poll `logseq sync status` until queues settle and tx converges.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Dict + + +def fail(message: str, **context: object) -> None: + payload = {"status": "error", "message": message} + if context: + payload["context"] = context + print(json.dumps(payload), file=sys.stderr) + raise SystemExit(1) + + +def parse_int(value: Any) -> int: + if value is None: + return 0 + if isinstance(value, bool): + return int(value) + if isinstance(value, (int, float)): + return int(value) + try: + return int(str(value)) + except (TypeError, ValueError): + return 0 + + +def status_command(args: argparse.Namespace) -> list[str]: + existing = getattr(args, "status_command", None) + if existing is not None: + return existing + + command = [ + "node", + str(Path(args.cli).expanduser().resolve()), + "--root-dir", + str(Path(args.root_dir).expanduser().resolve()), + "--config", + str(Path(args.config).expanduser().resolve()), + "--output", + "json", + "sync", + "status", + "--graph", + args.graph, + ] + args.status_command = command + return command + + + +def run_status(args: argparse.Namespace) -> Dict[str, Any]: + command = status_command(args) + + result = subprocess.run(command, capture_output=True, text=True) + if result.returncode != 0: + fail( + "sync status command failed", + command=command, + exit=result.returncode, + stdout=result.stdout, + stderr=result.stderr, + ) + + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as error: + fail( + "sync status command did not return valid JSON", + command=command, + stdout=result.stdout, + stderr=result.stderr, + detail=str(error), + ) + + if payload.get("status") != "ok": + fail("sync status returned non-ok status", payload=payload) + + return payload + + +def pending_counts(status_payload: Dict[str, Any]) -> Dict[str, int]: + data = status_payload.get("data") or {} + return { + "pending-local": parse_int(data.get("pending-local")), + "pending-asset": parse_int(data.get("pending-asset")), + "pending-server": parse_int(data.get("pending-server")), + } + + +def all_settled(counts: Dict[str, int]) -> bool: + required_keys = ("pending-local", "pending-asset", "pending-server") + return all(counts.get(key, 0) == 0 for key in required_keys) + + +def parse_required_int(value: Any) -> int | None: + if value is None: + return None + if isinstance(value, str) and value.strip() == "": + return None + if isinstance(value, bool): + return int(value) + if isinstance(value, (int, float)): + return int(value) + try: + return int(str(value)) + except (TypeError, ValueError): + return None + + +def tx_sync_status(status_payload: Dict[str, Any]) -> Dict[str, Any]: + data = status_payload.get("data") or {} + local_tx = parse_required_int(data.get("local-tx")) + remote_tx = parse_required_int(data.get("remote-tx")) + synced = ( + local_tx is not None + and remote_tx is not None + and local_tx == remote_tx + ) + return { + "local-tx": local_tx, + "remote-tx": remote_tx, + "synced": synced, + } + + +def tx_deltas( + tx_status: Dict[str, Any], + baseline_tx: Dict[str, Any], +) -> Dict[str, int | None]: + local_tx = tx_status.get("local-tx") + remote_tx = tx_status.get("remote-tx") + baseline_local_tx = baseline_tx.get("local-tx") + baseline_remote_tx = baseline_tx.get("remote-tx") + + local_tx_delta = ( + None + if local_tx is None or baseline_local_tx is None + else local_tx - baseline_local_tx + ) + remote_tx_delta = ( + None + if remote_tx is None or baseline_remote_tx is None + else remote_tx - baseline_remote_tx + ) + + return { + "local-tx-delta": local_tx_delta, + "remote-tx-delta": remote_tx_delta, + } + + +def min_tx_delta_reached( + tx_status: Dict[str, Any], + baseline_tx: Dict[str, Any], + min_tx_delta: int, +) -> bool: + if min_tx_delta <= 0: + return True + + deltas = tx_deltas(tx_status, baseline_tx) + local_tx_delta = deltas.get("local-tx-delta") + remote_tx_delta = deltas.get("remote-tx-delta") + if local_tx_delta is None or remote_tx_delta is None: + return False + + return local_tx_delta >= min_tx_delta and remote_tx_delta >= min_tx_delta + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Wait for sync status to settle" + ) + parser.add_argument( + "--cli", + required=True, + help="Path to static/logseq-cli.js", + ) + parser.add_argument("--root-dir", required=True) + parser.add_argument("--config", required=True) + parser.add_argument("--graph", required=True) + parser.add_argument("--timeout-s", type=float, default=120.0) + parser.add_argument("--interval-s", type=float, default=1.0) + parser.add_argument( + "--min-tx-delta", + type=int, + default=0, + help=( + "Require synced tx delta to be >= this value: " + "(new-tx - old-tx) >= min-tx-delta " + "(default: 0)" + ), + ) + parser.add_argument( + "--baseline-tx", + type=int, + default=None, + help=( + "Optional explicit baseline tx used as old-tx for both local " + "and remote; if omitted, first observed tx values " + "are used" + ), + ) + args = parser.parse_args() + + if args.min_tx_delta < 0: + invalid_min_tx_delta = args.min_tx_delta + fail( + "min-tx-delta must be non-negative", + min_tx_delta=invalid_min_tx_delta, + ) + + started = time.time() + deadline = started + args.timeout_s + last_payload: Dict[str, Any] | None = None + baseline_tx: Dict[str, Any] | None = ( + { + "local-tx": args.baseline_tx, + "remote-tx": args.baseline_tx, + } + if args.baseline_tx is not None + else None + ) + + while time.time() < deadline: + payload = run_status(args) + last_payload = payload + data = payload.get("data") or {} + last_error = data.get("last-error") + + if last_error is not None: + fail("sync status reports last-error", payload=payload) + + counts = pending_counts(payload) + tx_status = tx_sync_status(payload) + if baseline_tx is None: + baseline_tx = { + "local-tx": tx_status.get("local-tx"), + "remote-tx": tx_status.get("remote-tx"), + } + + deltas = tx_deltas(tx_status, baseline_tx) + if ( + all_settled(counts) + and tx_status["synced"] + and min_tx_delta_reached( + tx_status, + baseline_tx, + args.min_tx_delta, + ) + ): + print( + json.dumps( + { + "status": "ok", + "elapsed_s": round(time.time() - started, 3), + "counts": counts, + "tx": { + "local-tx": tx_status["local-tx"], + "remote-tx": tx_status["remote-tx"], + }, + "tx-delta": deltas, + "baseline-tx": baseline_tx, + "min_tx_delta": args.min_tx_delta, + "payload": payload, + } + ) + ) + return + + time.sleep(max(args.interval_s, 0.0)) + + last_tx_status = tx_sync_status(last_payload or {}) + last_baseline_tx = baseline_tx or {"local-tx": None, "remote-tx": None} + fail( + ( + "sync status polling timed out before queues settled, tx synced, " + "and min tx delta reached" + ), + timeout_s=args.timeout_s, + min_tx_delta=args.min_tx_delta, + baseline_tx=last_baseline_tx, + last_payload=last_payload, + last_counts=pending_counts(last_payload or {}), + last_tx=last_tx_status, + last_tx_delta=tx_deltas( + last_tx_status, + last_baseline_tx, + ), + ) + + +if __name__ == "__main__": + main() diff --git a/cli-e2e/spec/non_sync_cases.edn b/cli-e2e/spec/non_sync_cases.edn new file mode 100644 index 0000000000..130892edf0 --- /dev/null +++ b/cli-e2e/spec/non_sync_cases.edn @@ -0,0 +1,1100 @@ +{:templates + {:non-sync/graph-json-env + {:setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"], + :cleanup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"]}, + :non-sync/graph-json-secondary-env + {:extends :non-sync/graph-json-env, + :vars + {:secondary-graph "graph-two", :secondary-graph-arg "graph-two"}, + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{secondary-graph-arg}} >/dev/null"], + :cleanup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{secondary-graph-arg}}"]}, + :non-sync/source-graph-json-env + {:extends :non-sync/graph-json-secondary-env, + :vars + {:secondary-graph "source-graph", + :secondary-graph-arg "source-graph"}}, + :non-sync/example-command + {:tags [:example], + :covers {:options {:global ["--output"]}}, + :expect {:exit 0}, + :vars {:example-output "json"}, + :cmds + ["{{cli}} --output {{example-output}} example {{example-selector}}"]}, + :non-sync/example-human-command + {:extends :non-sync/example-command, + :vars {:example-output "human"}}, + :non-sync/example-json-command + {:extends :non-sync/example-command, + :expect {:stdout-json-paths {[:status] "ok"}}}, + :non-sync/example-edn-command + {:extends :non-sync/example-command, + :vars {:example-output "edn"}, + :expect {:stdout-edn-paths {[:status] :ok}}}, + :non-sync/completion-command + {:tags [:completion], + :covers {:commands ["completion"]}, + :expect {:exit 0}, + :cmds ["{{cli}} completion {{completion-selector}}"]}}, + :cases + [{:id "global-help", + :cmds ["{{cli}} --help"], + :expect + {:exit 0, + :stdout-contains + ["Usage: logseq [options]" "graph create" "completion"]}, + :covers {:options {:global ["--help"]}}, + :tags [:global :smoke]} + {:id "global-version", + :cmds ["{{cli}} --version"], + :expect {:exit 0, :stdout-contains ["Build time:" "Revision:"]}, + :covers {:options {:global ["--version"]}}, + :tags [:global :smoke]} + {:id "verbose-graph-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json --verbose graph list"], + :expect + {:exit 0, + :stderr-contains [":cli/parsed-options"], + :stdout-json-paths {[:status] "ok"}}, + :covers + {:commands ["graph list"], + :options + {:global ["--config" "--root-dir" "--output" "--verbose"]}}, + :tags [:global :graph]} + {:id "graph-create-and-info-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph info --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok", [:data :graph] "{{graph}}"}}, + :covers + {:commands ["graph create" "graph info"], + :options {:global ["--config" "--graph" "--root-dir" "--output"]}}, + :cleanup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"], + :tags [:graph :smoke]} + {:id "graph-list-human", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human graph list"], + :expect {:exit 0, :stdout-contains ["{{graph}}"]}, + :covers + {:commands ["graph list"], + :options {:global ["--config" "--root-dir" "--output"]}}, + :tags [:graph], + :extends :non-sync/graph-json-env} + {:id "graph-list-edn", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output edn graph list"], + :expect + {:exit 0, + :stdout-edn-paths {[:status] :ok, [:data :graphs] ["{{graph}}"]}}, + :covers + {:commands ["graph list"], + :options {:global ["--config" "--root-dir" "--output"]}}, + :tags [:graph], + :extends :non-sync/graph-json-env} + {:tags [:graph], + :extends :non-sync/graph-json-secondary-env, + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :message] "switched to {{secondary-graph}}"}}, + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph switch --graph {{secondary-graph-arg}}"], + :id "graph-switch-json", + :covers + {:commands ["graph switch"], + :options {:global ["--config" "--graph" "--root-dir" "--output"]}}} + {:id "graph-validate-fix-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph validate --graph {{graph-arg}} --fix"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok", [:data :result :errors] nil}}, + :covers + {:commands ["graph validate"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :graph ["--fix"]}}, + :tags [:graph], + :extends :non-sync/graph-json-env} + {:id "graph-export-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page ExportHome >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph export --graph {{graph-arg}} --type edn --file {{export-path-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["{{export-path}}"]}, + :covers + {:commands ["graph export"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :graph ["--type" "--file"]}}, + :tags [:graph], + :extends :non-sync/graph-json-env} + {:tags [:graph], + :extends :non-sync/source-graph-json-env, + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["Imported edn from" "{{export-path}}"]}, + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{secondary-graph-arg}} --page ImportSeed >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph export --graph {{secondary-graph-arg}} --type edn --file {{export-path-arg}} >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph import --graph {{graph-arg}} --type edn --input {{export-path-arg}}"], + :id "graph-import-json", + :covers + {:commands ["graph import"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :graph ["--type" "--input"]}}} + {:tags [:graph], + :extends :non-sync/graph-json-env, + :expect {:exit 0, :stdout-json-paths {[:status] "ok"}}, + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page BackupSeed >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup create --graph {{graph-arg}} --name nightly >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup list" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup restore --src \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup list | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"backups\"][0][\"name\"])')\" --dst {{restore-graph-arg}} >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup remove --src \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph backup list | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"backups\"][0][\"name\"])')\""], + :id "graph-backup-lifecycle-json", + :covers + {:commands + ["graph backup create" + "graph backup list" + "graph backup restore" + "graph backup remove"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :graph ["--name" "--src" "--dst"]}}, + :vars + {:restore-graph "backup-restore", + :restore-graph-arg "backup-restore"}, + :cleanup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{restore-graph-arg}}"]} + {:id "page-upsert-and-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list page --graph {{graph-arg}} --fields title,id --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :items 0 :block/title] "Home"}}, + :covers + {:commands ["upsert page" "list page"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--page"], + :list ["--fields" "--limit" "--sort" "--order"]}}, + :tags [:upsert :list :smoke], + :extends :non-sync/graph-json-env} + {:id "task-upsert-and-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert task --graph {{graph-arg}} --page TaskHome --status todo --priority high --scheduled '2026-02-10T08:00:00.000Z' --deadline '2026-02-12T18:00:00.000Z' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list task --graph {{graph-arg}} --status todo --priority high --content task --fields id,title,status,priority,scheduled,deadline --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :items 0 :block/title] "TaskHome", + [:data :items 0 :logseq.property/status] + "logseq.property/status.todo", + [:data :items 0 :logseq.property/priority] + "logseq.property/priority.high", + [:data :items 0 :logseq.property/scheduled] 1770710400000, + [:data :items 0 :logseq.property/deadline] 1770919200000}}, + :covers + {:commands ["upsert task" "list task"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert + ["--page" "--status" "--priority" "--scheduled" "--deadline"], + :list + ["--status" + "--priority" + "--content" + "--fields" + "--limit" + "--sort" + "--order"]}}, + :tags [:upsert :list :smoke], + :extends :non-sync/graph-json-env} + {:id "task-upsert-clear-properties-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert task --graph {{graph-arg}} --page TaskHome --status todo --priority high --scheduled '2026-02-10T08:00:00.000Z' --deadline '2026-02-12T18:00:00.000Z' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert task --graph {{graph-arg}} --page TaskHome --no-status --no-priority --no-scheduled --no-deadline >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list task --graph {{graph-arg}} --content taskhome --fields id,title,status,priority,scheduled,deadline --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :items 0 :block/title] "TaskHome"}, + :stdout-not-contains + ["logseq.property/priority.high" + "2026-02-10T08:00:00.000Z" + "2026-02-12T18:00:00.000Z"]}, + :covers + {:commands ["upsert task" "list task"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert + ["--page" + "--no-status" + "--no-priority" + "--no-scheduled" + "--no-deadline"], + :list ["--content" "--fields" "--limit" "--sort" "--order"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "task-upsert-set-clear-conflict-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert task --graph {{graph-arg}} --page TaskHome --status todo --no-status"], + :expect + {:exit 1, + :stdout-contains + ["invalid-options" + "--status and --no-status are mutually exclusive"]}, + :covers + {:commands ["upsert task"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--page" "--status" "--no-status"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "page-upsert-invalid-update-properties-atomic-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name AtomicTag >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page AtomicPage --update-tags '[\"AtomicTag\"]' --update-properties '{:missing-prop \"value\"}'"], + :expect + {:exit 1, + :stdout-json-paths + {[:status] "error", + [:error :code] "property-not-found", + [:error :option] "--update-properties", + [:error :phase] "resolve-options"}, + :stdout-contains ["missing-prop"]}, + :covers + {:commands ["upsert page"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--page" "--update-tags" "--update-properties"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "node-list-by-tags-properties-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name NodeTag >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name node-prop >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page NodeHome --update-tags '[\"NodeTag\"]' --update-properties '{:node-prop \"v\"}' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list node --graph {{graph-arg}} --tags NodeTag --properties node-prop --fields id,title,type --sort updated-at --order desc --limit 20"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :items 0 :block/title] "NodeHome", + [:data :items 0 :node/type] "page"}}, + :covers + {:commands + ["upsert tag" "upsert property" "upsert page" "list node"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--name" "--page" "--update-tags" "--update-properties"], + :list + ["--tags" + "--properties" + "--fields" + "--limit" + "--sort" + "--order"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "asset-upsert-and-list-json", + :setup ["printf 'asset-content' > {{export-path-arg}}.png"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert asset --graph {{graph-arg}} --path {{export-path-arg}}.png --content 'Asset One' --target-page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list asset --graph {{graph-arg}} --fields id,title,asset-type,size --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :items 0 :block/title] "Asset One", + [:data :items 0 :logseq.property.asset/type] "png", + [:data :items 0 :logseq.property.asset/size] 13}}, + :covers + {:commands ["upsert asset" "list asset"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--path" "--content" "--target-page"], + :list ["--fields" "--limit" "--sort" "--order"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "asset-upsert-update-json", + :setup + ["printf 'asset-content' > {{export-path-arg}}.png" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert asset --graph {{graph-arg}} --path {{export-path-arg}}.png --content 'Asset Original' --target-page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert asset --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Asset Original\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --content 'Asset Updated' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list asset --graph {{graph-arg}} --fields id,title --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :items 0 :block/title] "Asset Updated"}}, + :covers + {:commands ["upsert asset" "list asset"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--id" "--content"], + :list ["--fields" "--limit" "--sort" "--order"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "block-upsert-and-show-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json show --graph {{graph-arg}} --page Home --level 2 --linked-references false"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :root :block/children 0 :block/title] "Alpha block"}}, + :covers + {:commands ["upsert block" "show"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--target-page" "--content"], + :show ["--page" "--level" "--linked-references"]}}, + :tags [:upsert :show :smoke], + :extends :non-sync/graph-json-env} + {:id "tag-upsert-and-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name TagOne >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list tag --graph {{graph-arg}} --with-properties --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :items 0 :block/title] "TagOne"}}, + :covers + {:commands ["upsert tag" "list tag"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--name"], + :list ["--with-properties" "--sort" "--order" "--limit"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "property-upsert-and-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name score --type number --cardinality one >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list property --graph {{graph-arg}} --with-type --sort updated-at --order desc --limit 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :items 0 :block/title] "score", + [:data :items 0 :logseq.property/type] "number", + [:data :items 0 :db/cardinality] "db.cardinality/one"}}, + :covers + {:commands ["upsert property" "list property"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--name" "--type" "--cardinality"], + :list ["--with-type" "--sort" "--order" "--limit"]}}, + :tags [:upsert :list], + :extends :non-sync/graph-json-env} + {:id "list-tag-expand-offset-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name TagA >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name TagB >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list tag --graph {{graph-arg}} --expand --with-extends --sort updated-at --order desc --offset 0 --limit 20"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["TagA" "TagB"]}, + :covers + {:commands ["list tag"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :list + ["--expand" + "--with-extends" + "--offset" + "--sort" + "--order" + "--limit"]}}, + :tags [:list], + :extends :non-sync/graph-json-env} + {:id "list-property-classes-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name alpha --type number --cardinality one >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name beta --type number --cardinality one >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list property --graph {{graph-arg}} --with-classes --sort updated-at --order desc --offset 0 --limit 20"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["alpha" "beta"]}, + :covers + {:commands ["list property"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :list + ["--with-classes" "--sort" "--order" "--offset" "--limit"]}}, + :tags [:list], + :extends :non-sync/graph-json-env} + {:id "list-page-journal-only-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list page --graph {{graph-arg}} --journal-only"], + :expect + {:exit 0, :stdout-json-paths {[:status] "ok", [:data :items] []}}, + :covers + {:commands ["list page"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :list ["--journal-only"]}}, + :tags [:list], + :extends :non-sync/graph-json-env} + {:id "search-block-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SearchBlockTarget >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} search block --content blocktarget --output json --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["SearchBlockTarget"]}, + :covers + {:commands ["search block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :search ["--content"]}}, + :tags [:search :smoke], + :extends :non-sync/graph-json-env} + {:id "search-page-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SearchPageTarget >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json search page --content target --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["SearchPageTarget"]}, + :covers + {:commands ["search page"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :search ["--content"]}}, + :tags [:search], + :extends :non-sync/graph-json-env} + {:id "search-property-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name SearchOwner --type default >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json search property --content owner --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["SearchOwner"]}, + :covers + {:commands ["search property"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :search ["--content"]}}, + :tags [:search], + :extends :non-sync/graph-json-env} + {:id "search-tag-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name SearchTagTarget >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json search tag --content target --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-contains ["SearchTagTarget"]}, + :covers + {:commands ["search tag"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :search ["--content"]}}, + :tags [:search], + :extends :non-sync/graph-json-env} + {:id "block-upsert-blocks-file-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Anchor block' >/dev/null" + "printf '[{:block/title \"Inserted from file\"}]' > {{export-path-arg}}"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human upsert block --graph {{graph-arg}} --target-id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Anchor block\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --blocks-file {{export-path-arg}} --pos sibling >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --page Home"], + :expect {:exit 0, :stdout-contains ["Inserted from file"]}, + :covers + {:commands ["upsert block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--blocks-file" "--target-id" "--pos"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "block-upsert-target-uuid-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Anchor block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human upsert block --graph {{graph-arg}} --target-uuid \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?uuid . :where [?e :block/title \"Anchor block\"] [?e :block/uuid ?uuid]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --content 'Inserted by uuid' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --page Home"], + :expect {:exit 0, :stdout-contains ["Inserted by uuid"]}, + :covers + {:commands ["upsert block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--target-uuid"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "block-upsert-update-id-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human upsert block --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Alpha block\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --content 'Updated by id' --update-tags '[:logseq.class/Quote-block]' --update-properties '{:logseq.property/publishing-public? true}' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --page Home"], + :expect {:exit 0, :stdout-contains ["Updated by id"]}, + :covers + {:commands ["upsert block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--id" "--update-tags" "--update-properties"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "block-upsert-update-id-custom-many-property-json", + :vars {:prop-name "Reproducible steps"}, + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name '{{prop-name}}' --type default --cardinality many --public true >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Alpha block\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --update-properties '{\"{{prop-name}}\" [\"Step 1\" \"Step 2\" \"Step 3\"]}' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Alpha block\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\""], + :expect + {:exit 0, + :stdout-contains ["{{prop-name}}" "Step 1" "Step 2" "Step 3"]}, + :covers + {:commands ["upsert block" "show"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert + ["--id" "--target-page" "--content" "--update-properties"], + :show ["--id"]}}, + :tags [:upsert :show]} + {:id "block-upsert-update-uuid-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' --update-tags '[:logseq.class/Quote-block]' --update-properties '{:logseq.property/publishing-public? true}' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human upsert block --graph {{graph-arg}} --uuid \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?uuid . :where [?e :block/title \"Alpha block\"] [?e :block/uuid ?uuid]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" --content 'Updated by uuid' --remove-tags '[:logseq.class/Quote-block]' --remove-properties '[:logseq.property/publishing-public?]' >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --page Home"], + :expect {:exit 0, :stdout-contains ["Updated by uuid"]}, + :covers + {:commands ["upsert block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--uuid" "--remove-tags" "--remove-properties"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "property-upsert-update-id-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name score --type number --cardinality one >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list property --graph {{graph-arg}} --fields title,id --limit 200 --sort updated-at --order desc | python3 -c 'import sys,json; print(next(item[\"db/id\"] for item in json.load(sys.stdin)[\"data\"][\"items\"] if item[\"block/title\"] == \"score\"))')\" --hide true --public false"], + :expect {:exit 0, :stdout-json-paths {[:status] "ok"}}, + :covers + {:commands ["upsert property"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :upsert ["--id" "--hide" "--public"]}}, + :tags [:upsert], + :extends :non-sync/graph-json-env} + {:id "query-custom-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?title . :where [?b :block/title ?title] [(= ?title \"Alpha block\")]]'"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :result] "Alpha block"}}, + :covers + {:commands ["query"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :query ["--query"]}}, + :tags [:query], + :extends :non-sync/graph-json-env} + {:id "query-named-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --name recent-updated --inputs '[1]'"], + :expect {:exit 0, :stdout-json-paths {[:status] "ok"}}, + :covers + {:commands ["query"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :query ["--name" "--inputs"]}}, + :tags [:query], + :extends :non-sync/graph-json-env} + {:id "query-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query list --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :queries 0 :name] "list-priority"}}, + :covers + {:commands ["query list"], + :options {:global ["--config" "--graph" "--root-dir" "--output"]}}, + :tags [:query], + :extends :non-sync/graph-json-env} + {:id "example-show-human", + :expect + {:exit 0, + :stdout-contains + ["Selector: show" "Matched commands:" "show" "Examples:"]}, + :covers {:commands ["example show"]}, + :tags [:example], + :extends :non-sync/example-human-command, + :vars {:example-selector "show"}} + {:id "example-upsert-page-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "upsert page", + [:data :matched-commands 0] "upsert page"}}, + :covers {:commands ["example upsert page"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "upsert page"}} + {:id "example-upsert-prefix-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "upsert", + [:data :matched-commands 0] "upsert block"}}, + :covers {:commands ["example upsert"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "upsert"}} + {:id "example-list-prefix-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "list", + [:data :matched-commands 0] "list page"}}, + :covers {:commands ["example list"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "list"}} + {:id "example-list-page-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "list page", + [:data :matched-commands] ["list page"]}}, + :covers {:commands ["example list page"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "list page"}} + {:id "example-query-prefix-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "query", [:data :matched-commands 0] "query"}}, + :covers {:commands ["example query"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "query"}} + {:id "example-query-list-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "query list", + [:data :matched-commands] ["query list"]}}, + :covers {:commands ["example query list"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "query list"}} + {:id "example-remove-prefix-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "remove", + [:data :matched-commands 0] "remove block"}}, + :covers {:commands ["example remove"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "remove"}} + {:id "example-remove-page-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "remove page", + [:data :matched-commands] ["remove page"]}}, + :covers {:commands ["example remove page"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "remove page"}} + {:id "example-search-prefix-json", + :expect + {:exit 0, + :stdout-json-paths + {[:data :selector] "search", + [:data :matched-commands 0] "search block"}}, + :covers {:commands ["example search"]}, + :tags [:example], + :extends :non-sync/example-json-command, + :vars {:example-selector "search"}} + {:id "example-search-block-edn", + :expect + {:exit 0, + :stdout-edn-paths + {[:data :selector] "search block", + [:data :matched-commands] ["search block"]}}, + :covers {:commands ["example search block"]}, + :tags [:example], + :extends :non-sync/example-edn-command, + :vars {:example-selector "search block"}} + {:id "show-id-human", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output human show --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output human query --query '[:find ?e . :where [?e :block/title \"Alpha block\"]]')\""], + :expect {:exit 0, :stdout-contains ["Alpha block"]}, + :covers + {:commands ["show"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :show ["--id"]}}, + :tags [:show], + :extends :non-sync/graph-json-env} + {:id "show-uuid-human", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output human show --uuid \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output json query --query '[:find ?uuid . :where [?e :block/title \"Alpha block\"] [?e :block/uuid ?uuid]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\""], + :expect {:exit 0, :stdout-contains ["Alpha block"]}, + :covers + {:commands ["show"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :show ["--uuid"]}}, + :tags [:show], + :extends :non-sync/graph-json-env} + {:id "show-stdin-id-human", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output human query --query '[:find [?e ...] :in $ ?q :where [?e :block/title ?title] [(clojure.string/includes? ?title ?q)]]' --inputs '[\"Alpha\"]' | {{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --graph {{graph-arg}} --output human show --id"], + :expect {:exit 0, :stdout-contains ["Alpha block"]}, + :covers + {:commands ["show"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :show ["stdin:--id"]}}, + :tags [:show :pipe], + :extends :non-sync/graph-json-env} + {:id "debug-pull-id-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json debug pull --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Home\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\""], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :entity :block/title] "Home"}}, + :covers + {:commands ["debug pull"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :debug ["--id"]}}, + :tags [:debug], + :extends :non-sync/graph-json-env} + {:id "debug-pull-uuid-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json debug pull --graph {{graph-arg}} --uuid \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?uuid . :where [?e :block/title \"Home\"] [?e :block/uuid ?uuid]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\""], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :entity :block/title] "Home"}}, + :covers + {:commands ["debug pull"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :debug ["--uuid"]}}, + :tags [:debug], + :extends :non-sync/graph-json-env} + {:id "debug-pull-ident-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json debug pull --graph {{graph-arg}} --ident :logseq.class/Tag"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :entity :db/ident] "logseq.class/Tag"}}, + :covers + {:commands ["debug pull"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :debug ["--ident"]}}, + :tags [:debug], + :extends :non-sync/graph-json-env} + {:id "debug-pull-current-graph-fallback-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json debug pull --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?e . :where [?e :block/title \"Home\"]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\""], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :entity :block/title] "Home"}}, + :covers + {:commands ["debug pull"], + :options + {:global ["--config" "--root-dir" "--output"], :debug ["--id"]}}, + :tags [:debug], + :extends :non-sync/graph-json-env} + {:id "remove-page-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json remove page --graph {{graph-arg}} --page Home"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok", [:data :result] true}}, + :covers + {:commands ["remove page"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :remove ["--page"]}}, + :tags [:remove], + :extends :non-sync/graph-json-env} + {:id "graph-remove-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}} >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph remove --graph {{graph-arg}}" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph list"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok"}, + :stdout-not-contains ["{{graph}}"]}, + :covers + {:commands ["graph remove" "graph list"], + :options {:global ["--config" "--graph" "--root-dir" "--output"]}}, + :tags [:graph :remove]} + {:id "remove-block-uuid-human", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page Home >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page Home --content 'Alpha block' >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human remove block --graph {{graph-arg}} --uuid \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json query --graph {{graph-arg}} --query '[:find ?uuid . :where [?e :block/title \"Alpha block\"] [?e :block/uuid ?uuid]]' | python3 -c 'import sys,json; print(json.load(sys.stdin)[\"data\"][\"result\"])')\" >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output human show --graph {{graph-arg}} --page Home --linked-references false"], + :expect {:exit 0, :stdout-not-contains ["Alpha block"]}, + :covers + {:commands ["remove block"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :remove ["--uuid"]}}, + :tags [:remove], + :extends :non-sync/graph-json-env} + {:id "remove-tag-name-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert tag --graph {{graph-arg}} --name TagOne >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json remove tag --graph {{graph-arg}} --name TagOne >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list tag --graph {{graph-arg}} --fields title,id"], + :expect {:exit 0, :stdout-not-contains ["TagOne"]}, + :covers + {:commands ["remove tag"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :remove ["--name"]}}, + :tags [:remove], + :extends :non-sync/graph-json-env} + {:id "remove-property-id-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert property --graph {{graph-arg}} --name score --type number --cardinality one >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json remove property --graph {{graph-arg}} --id \"$({{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list property --graph {{graph-arg}} --fields title,id --limit 200 --sort updated-at --order desc | python3 -c 'import sys,json; print(next(item[\"db/id\"] for item in json.load(sys.stdin)[\"data\"][\"items\"] if item[\"block/title\"] == \"score\"))')\" >/dev/null" + "{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json list property --graph {{graph-arg}} --fields title,id"], + :expect {:exit 0, :stdout-not-contains ["score"]}, + :covers + {:commands ["remove property"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :remove ["--id"]}}, + :tags [:remove], + :extends :non-sync/graph-json-env} + {:id "server-cleanup-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server cleanup"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :mismatched] 0, + [:data :eligible] 0, + [:data :skipped-owner] 0}}, + :covers + {:commands ["server cleanup"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :server []}}, + :tags [:server], + :extends :non-sync/graph-json-env} + {:id "server-list-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server list"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :servers 0 :repo] "{{graph}}"}}, + :covers + {:commands ["server list"], + :options {:global ["--config" "--root-dir" "--output"]}}, + :tags [:server], + :extends :non-sync/graph-json-env} + {:id "server-restart-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server restart --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :repo] "{{graph}}", [:data :owned?] true}}, + :covers + {:commands ["server restart"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :server ["--graph"]}}, + :tags [:server], + :extends :non-sync/graph-json-env} + {:id "server-stop-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths {[:status] "ok", [:data :repo] "{{graph}}"}}, + :covers + {:commands ["server stop"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :server ["--graph"]}}, + :tags [:server]} + {:id "server-start-json", + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}} >/dev/null"], + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server start --graph {{graph-arg}}"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", [:data :repo] "{{graph}}", [:data :owned?] true}}, + :covers + {:commands ["server start"], + :options + {:global ["--config" "--graph" "--root-dir" "--output"], + :server ["--graph"]}}, + :tags [:server], + :extends :non-sync/graph-json-env} + {:id "skill-show-human", + :cmds ["{{cli}} skill show"], + :expect + {:exit 0, :stdout-contains ["name: logseq-cli" "# Logseq CLI"]}, + :covers {:commands ["skill show"], :options {:skill []}}, + :tags [:skill :smoke]} + {:id "skill-show-json-still-raw-markdown", + :cmds ["{{cli}} --output json skill show"], + :expect + {:exit 0, + :stdout-contains ["name: logseq-cli" "# Logseq CLI"], + :stdout-not-contains ["\"status\"" "\"data\""]}, + :covers {:commands ["skill show"], :options {:global ["--output"]}}, + :tags [:skill]} + ;; Repo root already contains tracked `.agents/skills/logseq-cli/SKILL.md`, + ;; so this local-install case must run from an isolated cwd. + {:id "skill-install-local", + :setup + ["rm -rf ./tmp/cli-e2e-skill-install-local" + "mkdir -p ./tmp/cli-e2e-skill-install-local"], + :cmds + ["cd ./tmp/cli-e2e-skill-install-local && {{cli}} skill install >/dev/null" + "python3 -c 'import pathlib; p=pathlib.Path(\"./tmp/cli-e2e-skill-install-local/.agents/skills/logseq-cli/SKILL.md\"); print(\"installed\" if p.exists() else \"missing\")'"], + :expect {:exit 0, :stdout-contains ["installed"]}, + :covers {:commands ["skill install"], :options {:skill []}}, + :cleanup ["rm -rf ./tmp/cli-e2e-skill-install-local"], + :tags [:skill]} + {:id "skill-install-global-preserves-other-skills", + :setup + ["rm -rf ./tmp/cli-e2e-skill-home" + "mkdir -p ./tmp/cli-e2e-skill-home/.agents/skills/existing-skill" + "python3 -c 'import pathlib; pathlib.Path(\"./tmp/cli-e2e-skill-home/.agents/skills/existing-skill/SKILL.md\").write_text(\"keep\", encoding=\"utf8\")'"], + :cmds + ["HOME=\"$(pwd)/tmp/cli-e2e-skill-home\" {{cli}} skill install --global >/dev/null" + "python3 -c 'import pathlib; home=pathlib.Path(\"./tmp/cli-e2e-skill-home\"); existing=(home / \".agents/skills/existing-skill/SKILL.md\").read_text(encoding=\"utf8\"); installed=(home / \".agents/skills/logseq-cli/SKILL.md\").exists(); print(\"ok\" if existing==\"keep\" and installed else \"bad\")'"], + :expect {:exit 0, :stdout-contains ["ok"]}, + :covers + {:commands ["skill install"], :options {:skill ["--global"]}}, + :cleanup ["rm -rf ./tmp/cli-e2e-skill-home"], + :tags [:skill]} + {:id "completion-zsh", + :expect + {:exit 0, + :stdout-contains + ["#compdef logseq" "Auto-generated by `logseq completion zsh`"]}, + :covers {:options {:completion ["zsh"]}}, + :tags [:smoke], + :extends :non-sync/completion-command, + :vars {:completion-selector "zsh"}} + {:id "completion-bash-flag", + :expect + {:exit 0, + :stdout-contains + ["Auto-generated by `logseq completion bash`" + "_logseq_json_names_bash"]}, + :covers {:options {:completion ["--shell" "bash"]}}, + :extends :non-sync/completion-command, + :vars {:completion-selector "--shell bash"}} + {:id "completion-invalid-shell", + :expect {:exit 1, :stdout-contains ["unsupported shell: fish"]}, + :extends :non-sync/completion-command, + :vars {:completion-selector "fish"}} + {:id "doctor-dev-script-json", + :cmds + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json doctor --dev-script"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :status] "ok", + [:data :checks 0 :id] "db-worker-script"}, + :stdout-contains ["static/db-worker-node.js"]}, + :covers + {:commands ["doctor"], + :options + {:global ["--config" "--root-dir" "--output"], + :doctor ["--dev-script"]}}, + :tags [:doctor :smoke]}]} diff --git a/cli-e2e/spec/non_sync_inventory.edn b/cli-e2e/spec/non_sync_inventory.edn new file mode 100644 index 0000000000..01c70a2168 --- /dev/null +++ b/cli-e2e/spec/non_sync_inventory.edn @@ -0,0 +1,165 @@ +{:excluded-command-prefixes ["sync" "login" "logout"] + :scopes + {:global + {:options ["--help" + "--version" + "--config" + "--graph" + "--root-dir" + "--output" + "--verbose"]} + + :graph + {:commands ["graph list" + "graph create" + "graph switch" + "graph remove" + "graph validate" + "graph info" + "graph export" + "graph import" + "graph backup list" + "graph backup create" + "graph backup restore" + "graph backup remove"] + :options ["--fix" + "--type" + "--file" + "--input" + "--name" + "--src" + "--dst"]} + + :list + {:commands ["list page" + "list tag" + "list property" + "list task" + "list node" + "list asset"] + :options ["--expand" + "--fields" + "--limit" + "--offset" + "--sort" + "--order" + "--status" + "--priority" + "--content" + "--tags" + "--properties" + "--journal-only" + "--with-properties" + "--with-extends" + "--with-classes" + "--with-type"]} + + :upsert + {:commands ["upsert block" + "upsert page" + "upsert task" + "upsert asset" + "upsert tag" + "upsert property"] + :options ["--id" + "--uuid" + "--target-id" + "--target-uuid" + "--target-page" + "--pos" + "--content" + "--path" + "--blocks-file" + "--status" + "--priority" + "--scheduled" + "--deadline" + "--no-status" + "--no-priority" + "--no-scheduled" + "--no-deadline" + "--update-tags" + "--update-properties" + "--remove-tags" + "--remove-properties" + "--page" + "--name" + "--type" + "--cardinality" + "--hide" + "--public"]} + + :remove + {:commands ["remove block" + "remove page" + "remove tag" + "remove property"] + :options ["--id" + "--uuid" + "--name"]} + + :query + {:commands ["query" + "query list"] + :options ["--query" + "--name" + "--inputs"]} + + :search + {:commands ["search block" + "search page" + "search property" + "search tag"] + :options ["--content"]} + + :show + {:commands ["show"] + :options ["--id" + "--uuid" + "--page" + "--linked-references" + "--level" + "stdin:--id"]} + + :debug + {:commands ["debug pull"] + :options ["--id" + "--uuid" + "--ident"]} + + :example + {:commands ["example list" + "example list page" + "example upsert" + "example upsert page" + "example remove" + "example remove page" + "example query" + "example query list" + "example search" + "example search block" + "example show"] + :options []} + + :server + {:commands ["server list" + "server cleanup" + "server start" + "server stop" + "server restart"] + :options ["--graph"]} + + :doctor + {:commands ["doctor"] + :options ["--dev-script"]} + + :completion + {:commands ["completion"] + :options ["bash" + "zsh" + "--shell"]} + + :skill + {:commands ["skill show" + "skill install"] + :options ["--global"]}}} diff --git a/cli-e2e/spec/sync_cases.edn b/cli-e2e/spec/sync_cases.edn new file mode 100644 index 0000000000..3b3e369bc1 --- /dev/null +++ b/cli-e2e/spec/sync_cases.edn @@ -0,0 +1,171 @@ +{:templates + {:sync/base + {:cleanup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}" + "{{cli}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json server stop --graph {{graph-arg}}"], + :tags [:sync], + :vars + {:sync-port "18080", + :sync-http-base "http://127.0.0.1:18080", + :sync-ws-url "ws://127.0.0.1:18080/sync/%s", + :home-dir "{{tmp-dir}}/home", + :auth-path "{{tmp-dir}}/home/logseq/auth.json", + :cli-home "HOME='{{tmp-dir}}/home' {{cli}}"}, + :setup + ["mkdir -p '{{tmp-dir}}/graphs-b'" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"]}, + :sync/common + {:extends :sync/base, + :cmds + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync download --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"], + :expect + {:exit 0, + :stdout-json-paths + {[:status] "ok", + [:data :pending-local] 0, + [:data :pending-asset] 0, + [:data :pending-server] 0, + [:data :last-error] nil}}, + :covers + {:commands ["sync upload" "sync download" "sync status"], + :options {:global ["--config" "--graph" "--root-dir" "--output"]}}}} + :cases + [{:tags [:happy-path :bootstrap :a-to-b], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncBootstrapHome >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncBootstrapHome --content '{{marker-content}}' >/dev/null"], + :cmds + ["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-bootstrap-upload-download-a-to-b", + :graph "sync-e2e-bootstrap-a-to-b", + :vars {:marker-content "sync-happy-bootstrap-a-to-b-marker"}} + {:tags [:happy-path :incremental :a-to-b], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncIncrementalHome >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{seed-marker}}' >/dev/null"], + :cmds + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{incremental-marker}}' >/dev/null" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1" + "python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{incremental-marker}}\")] ]' --require-result" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-incremental-update-a-to-b", + :graph "sync-e2e-incremental-a-to-b", + :vars + {:seed-marker "sync-happy-incremental-seed-marker", + :incremental-marker "sync-happy-incremental-update-marker"}} + {:tags [:happy-path :bidirectional :roundtrip], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncRoundtripAHome >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripAHome --content '{{marker-a}}' >/dev/null"], + :cmds + ["{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert page --graph {{graph-arg}} --page SyncRoundtripBHome >/dev/null" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripBHome --content '{{marker-b}}' >/dev/null" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1" + "python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-b}}\")] ]' --require-result" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync status --graph {{graph-arg}}"], + :id "sync-bidirectional-roundtrip", + :graph "sync-e2e-bidirectional-roundtrip", + :vars + {:marker-a "sync-happy-roundtrip-a-seed-marker", + :marker-b "sync-happy-roundtrip-b-origin-marker"}} + {:tags [:happy-path :multi-batch :a-to-b], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchHome >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchHome --content '{{seed-marker}}' >/dev/null"], + :cmds + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchOne >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchOne --content '{{batch-marker-1}}' >/dev/null" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchTwo >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchTwo --content '{{batch-marker-2}}' >/dev/null" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchThree >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchThree --content '{{batch-marker-3}}' >/dev/null" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1" + "python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-1}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-2}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-3}}\")] ]' --require-result" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-multi-batch-operations", + :graph "sync-e2e-multi-batch-operations", + :vars + {:seed-marker "sync-happy-multi-batch-seed-marker", + :batch-marker-3 "sync-happy-multi-batch-marker-3", + :batch-marker-1 "sync-happy-multi-batch-marker-1", + :batch-marker-2 "sync-happy-multi-batch-marker-2"}} + {:tags [:negative :upload :duplicate], + :extends :sync/base, + :setup + ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}} >/dev/null"], + :cmds + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}"], + :expect + {:exit 1, + :stdout-json-paths + {[:status] "error", + [:error :code] "graph-already-exists", + [:error :context :graph-name] "{{graph}}"}, + :stdout-contains ["delete it before uploading again"]}, + :id "sync-upload-rejects-duplicate-remote-graph", + :graph "sync-e2e-upload-rejects-duplicate-remote-graph"} + {:tags [:happy-path :steady-state :status], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncSteadyStateHome >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncSteadyStateHome --content '{{marker-content}}' >/dev/null"], + :cmds + ["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-status-steady-state", + :graph "sync-e2e-status-steady-state", + :vars {:marker-content "sync-happy-steady-state-marker"}} + {:tags [:stress :bidirectional :random :block-ops], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null"], + :cmds + ["python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile default --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0" + "python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-random-bidirectional-block-ops", + :graph "sync-e2e-random-bidirectional-block-ops", + :vars + {:random-seed "424242", + :random-page "SyncRandomOpsHome", + :rounds-per-client "40"}} + {:tags [:stress :offline :bidirectional :random :block-ops], + :extends :sync/common, + :setup + ["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page '{{random-page}}' --content '{{seed-marker}}' >/dev/null"], + :cmds + ["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{seed-marker}}\")] ]' --require-result" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync stop --graph {{graph-arg}}" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync stop --graph {{graph-arg}}" + "python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile high-stress --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0" + "python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0" + "python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result" + "{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"], + :id "sync-offline-random-bidirectional-block-ops", + :graph "sync-e2e-offline-random-bidirectional-block-ops", + :vars + {:seed-marker "sync-offline-random-seed-marker", + :random-seed "989898", + :random-page "SyncOfflineRandomOpsHome", + :rounds-per-client "40"}}]} diff --git a/cli-e2e/spec/sync_inventory.edn b/cli-e2e/spec/sync_inventory.edn new file mode 100644 index 0000000000..7a664eb15f --- /dev/null +++ b/cli-e2e/spec/sync_inventory.edn @@ -0,0 +1,13 @@ +{:excluded-command-prefixes ["login" "logout"] + :scopes + {:global + {:options ["--config" + "--graph" + "--root-dir" + "--output"]} + + :sync + {:commands ["sync upload" + "sync download" + "sync status"] + :options []}}} diff --git a/cli-e2e/src/logseq/cli/e2e/cleanup.clj b/cli-e2e/src/logseq/cli/e2e/cleanup.clj new file mode 100644 index 0000000000..906f69d5f1 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/cleanup.clj @@ -0,0 +1,184 @@ +(ns logseq.cli.e2e.cleanup + (:require [babashka.fs :as fs] + [clojure.java.shell :as java-shell] + [clojure.string :as string])) + +(def ^:private cli-e2e-temp-prefix "logseq-cli-e2e-") +(def ^:private db-sync-default-port 18080) + +(defn- parse-ps-line + [line] + (when-let [[_ pid-str command] (re-matches #"\s*(\d+)\s+(.*)" (or line ""))] + {:pid (Long/parseLong pid-str) + :command command})) + +(defn- cli-e2e-db-worker-command? + [command] + (and (re-find #"db-worker-node(?:\.js)?\b" command) + (re-find #"logseq-cli-e2e-" command))) + +(defn list-cli-e2e-db-worker-pids + ([] + (list-cli-e2e-db-worker-pids {})) + ([{:keys [shell-fn] + :or {shell-fn java-shell/sh}}] + (let [{:keys [exit out err]} (shell-fn "ps" "-ax" "-o" "pid=" "-o" "command=")] + (when-not (zero? exit) + (throw (ex-info "Unable to scan processes" + {:exit exit + :err err}))) + (->> (string/split-lines (or out "")) + (keep parse-ps-line) + (filter (fn [{:keys [command]}] + (cli-e2e-db-worker-command? command))) + (mapv :pid))))) + +(defn- pid-alive? + [pid] + (zero? (:exit (java-shell/sh "kill" "-0" (str pid))))) + +(defn- kill-pid! + [pid] + (try + (if-not (pid-alive? pid) + :not-found + (do + (java-shell/sh "kill" "-TERM" (str pid)) + (Thread/sleep 100) + (if-not (pid-alive? pid) + :killed + (do + (java-shell/sh "kill" "-KILL" (str pid)) + (Thread/sleep 50) + (if (pid-alive? pid) + :failed + :killed))))) + (catch Exception _ + :failed))) + +(defn cleanup-db-worker-processes! + ([] + (cleanup-db-worker-processes! {})) + ([{:keys [dry-run list-pids-fn kill-pid-fn]}] + (let [list-pids-fn (or list-pids-fn list-cli-e2e-db-worker-pids) + kill-pid-fn (or kill-pid-fn kill-pid!) + found-pids (vec (list-pids-fn))] + (if dry-run + {:dry-run? true + :found-pids found-pids + :would-kill-pids found-pids + :killed-pids [] + :failed-pids []} + (let [{:keys [killed-pids failed-pids]} + (reduce (fn [acc pid] + (if (= :failed (kill-pid-fn pid)) + (update acc :failed-pids conj pid) + (update acc :killed-pids conj pid))) + {:killed-pids [] + :failed-pids []} + found-pids)] + {:dry-run? false + :found-pids found-pids + :would-kill-pids [] + :killed-pids killed-pids + :failed-pids failed-pids}))))) + +(defn- parse-long-safe + [value] + (try + (Long/parseLong value) + (catch Exception _ + nil))) + +(defn list-cli-e2e-db-sync-port-pids + ([] + (list-cli-e2e-db-sync-port-pids {})) + ([{:keys [shell-fn port] + :or {shell-fn java-shell/sh + port db-sync-default-port}}] + (let [{:keys [exit out err]} (shell-fn "lsof" "-nP" (str "-iTCP:" port) "-sTCP:LISTEN") + out-lines (->> (string/split-lines (or out "")) + (map string/trim) + (remove string/blank?) + vec)] + (when (and (not (zero? exit)) + (or (not (string/blank? err)) + (seq out-lines))) + (throw (ex-info "Unable to scan db-sync server port listeners" + {:exit exit + :err err + :port port}))) + (->> out-lines + (filter #(re-find (re-pattern (str ":" port "\\b")) %)) + (keep (fn [line] + (some-> (string/split line #"\s+") + (nth 1 nil) + parse-long-safe))) + distinct + vec)))) + +(defn cleanup-db-sync-port-processes! + ([] + (cleanup-db-sync-port-processes! {})) + ([{:keys [dry-run list-pids-fn kill-pid-fn] + :as _opts}] + (let [list-pids-fn (or list-pids-fn list-cli-e2e-db-sync-port-pids) + kill-pid-fn (or kill-pid-fn kill-pid!) + found-pids (vec (list-pids-fn))] + (if dry-run + {:dry-run? true + :found-pids found-pids + :would-kill-pids found-pids + :killed-pids [] + :failed-pids []} + (let [{:keys [killed-pids failed-pids]} + (reduce (fn [acc pid] + (if (= :failed (kill-pid-fn pid)) + (update acc :failed-pids conj pid) + (update acc :killed-pids conj pid))) + {:killed-pids [] + :failed-pids []} + found-pids)] + {:dry-run? false + :found-pids found-pids + :would-kill-pids [] + :killed-pids killed-pids + :failed-pids failed-pids}))))) + +(defn- list-cli-e2e-temp-roots + [tmp-root] + (->> (fs/list-dir tmp-root) + (filter fs/directory?) + (filter #(string/starts-with? (fs/file-name %) cli-e2e-temp-prefix)) + (mapv str))) + +(defn cleanup-temp-roots! + ([] + (cleanup-temp-roots! {})) + ([{:keys [dry-run tmp-root list-dirs-fn delete-dir-fn] + :or {tmp-root (System/getProperty "java.io.tmpdir") + delete-dir-fn fs/delete-tree}}] + (let [list-dirs-fn (or list-dirs-fn + #(list-cli-e2e-temp-roots tmp-root)) + found-dirs (vec (list-dirs-fn))] + (if dry-run + {:dry-run? true + :found-dirs found-dirs + :would-remove-dirs found-dirs + :removed-dirs [] + :failed-dirs []} + (let [{:keys [removed-dirs failed-dirs]} + (reduce (fn [acc dir] + (try + (delete-dir-fn dir) + (update acc :removed-dirs conj dir) + (catch Exception _ + (update acc :failed-dirs conj dir)))) + {:removed-dirs [] + :failed-dirs []} + found-dirs)] + {:dry-run? false + :found-dirs found-dirs + :would-remove-dirs [] + :removed-dirs removed-dirs + :failed-dirs failed-dirs}))))) diff --git a/cli-e2e/src/logseq/cli/e2e/coverage.clj b/cli-e2e/src/logseq/cli/e2e/coverage.clj new file mode 100644 index 0000000000..2b5e13fb2b --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/coverage.clj @@ -0,0 +1,120 @@ +(ns logseq.cli.e2e.coverage + (:require [clojure.set :as set] + [clojure.string :as string])) + +(defn- normalize-commands + [commands] + (->> commands + (map str) + (remove string/blank?) + distinct + sort + vec)) + +(defn- normalize-options + [options] + (->> options + (map str) + (remove string/blank?) + distinct + sort + vec)) + +(defn- excluded-command? + [excluded-prefixes command] + (let [head (first (string/split (str command) #" "))] + (contains? (set excluded-prefixes) head))) + +(defn validate-inventory! + [{:keys [excluded-command-prefixes scopes] :as inventory}] + (let [invalid-commands (->> scopes + vals + (mapcat :commands) + (filter #(excluded-command? excluded-command-prefixes %)) + normalize-commands) + invalid-scopes (->> scopes + (keep (fn [[scope {:keys [commands]}]] + (when (some #(excluded-command? excluded-command-prefixes %) commands) + scope))) + sort + vec)] + (when (seq invalid-commands) + (throw (ex-info "Excluded commands cannot appear in cli-e2e inventory" + {:invalid-commands invalid-commands + :invalid-scopes invalid-scopes + :inventory inventory}))) + inventory)) + +(defn validate-cases! + [{:keys [excluded-command-prefixes] :as inventory} cases] + (let [invalid-cases (->> cases + (filter (fn [{:keys [covers]}] + (some #(excluded-command? excluded-command-prefixes %) + (get covers :commands [])))) + (mapv :id))] + (when (seq invalid-cases) + (throw (ex-info "Excluded commands cannot be covered by cli-e2e cases" + {:invalid-case-ids invalid-cases + :inventory inventory}))) + cases)) + +(defn command->scope + [{:keys [scopes]}] + (into {} + (mapcat (fn [[scope {:keys [commands]}]] + (map (fn [command] + [command scope]) + commands))) + scopes)) + +(defn required-commands + [{:keys [scopes]}] + (->> scopes + vals + (mapcat :commands) + normalize-commands)) + +(defn covered-commands + [cases] + (->> cases + (mapcat #(get-in % [:covers :commands])) + normalize-commands)) + +(defn required-options-by-scope + [{:keys [scopes]}] + (into {} + (map (fn [[scope {:keys [options]}]] + [scope (normalize-options options)])) + scopes)) + +(defn covered-options-by-scope + [cases] + (reduce (fn [acc {:keys [covers]}] + (reduce-kv (fn [acc' scope options] + (update acc' scope (fnil into #{}) options)) + acc + (get covers :options {}))) + {} + cases)) + +(defn coverage-report + [inventory cases] + (validate-inventory! inventory) + (validate-cases! inventory cases) + (let [required-commands* (set (required-commands inventory)) + covered-commands* (set (covered-commands cases)) + covered-options* (covered-options-by-scope cases) + required-options* (required-options-by-scope inventory)] + {:missing-commands (->> (set/difference required-commands* covered-commands*) + normalize-commands) + :missing-options (into {} + (map (fn [[scope options]] + [scope (->> (set/difference (set options) + (get covered-options* scope #{})) + normalize-options)])) + required-options*)})) + +(defn complete? + [{:keys [missing-commands missing-options]}] + (and (empty? missing-commands) + (every? empty? (vals missing-options)))) diff --git a/cli-e2e/src/logseq/cli/e2e/main.clj b/cli-e2e/src/logseq/cli/e2e/main.clj new file mode 100644 index 0000000000..87f5daf0f4 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/main.clj @@ -0,0 +1,467 @@ +(ns logseq.cli.e2e.main + (:require [clojure.set :as set] + [logseq.cli.e2e.cleanup :as cleanup] + [logseq.cli.e2e.coverage :as coverage] + [logseq.cli.e2e.manifests :as manifests] + [logseq.cli.e2e.preflight :as preflight] + [logseq.cli.e2e.report :as report] + [logseq.cli.e2e.runner :as runner] + [logseq.cli.e2e.shell :as shell] + [logseq.cli.e2e.sync-fixture :as sync-fixture]) + (:import (java.util.concurrent Executors LinkedBlockingQueue TimeUnit))) + +(defn select-cases + [cases {:keys [case include]}] + (cond + case + (filterv #(= case (:id %)) cases) + + (seq include) + (let [include-tags (set (map keyword include))] + (filterv (fn [{:keys [tags]}] + (not-empty (set/intersection include-tags (set tags)))) + cases)) + + :else + (vec cases))) + +(def default-suite :non-sync) +(def default-jobs 1) +(def default-cli-jobs 4) + +(defn- suite-from-opts + [opts] + (or (:suite opts) default-suite)) + +(defn- elapsed-ms + [started-at] + (long (/ (- (System/nanoTime) started-at) 1000000))) + +(defn- format-duration + [started-at] + (format "%.2fs" (/ (double (- (System/nanoTime) started-at)) 1000000000.0))) + +(defn- positive-jobs + [jobs] + (let [jobs (or jobs default-jobs)] + (when-not (and (integer? jobs) (pos? jobs)) + (throw (ex-info "--jobs must be a positive integer" + {:jobs jobs}))) + jobs)) + +(defn run-selected-cases! + [selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}] + (let [total (count selected-cases) + _ (positive-jobs jobs)] + (reduce (fn [acc [idx case]] + (let [index (inc idx) + started-at (System/nanoTime)] + (when on-case-start + (on-case-start {:index index + :total total + :case case})) + (try + (let [result (run-case case {:run-command run-command + :detailed-log? detailed-log? + :timings? timings?}) + payload {:index index + :total total + :case case + :result result + :elapsed-ms (elapsed-ms started-at)}] + (when on-case-success + (on-case-success payload)) + (conj acc result)) + (catch Exception error + (when on-case-failure + (on-case-failure {:index index + :total total + :case case + :error error + :elapsed-ms (elapsed-ms started-at)})) + (throw error))))) + [] + (map-indexed vector selected-cases)))) + +(defn run-selected-cases-in-parallel! + [selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}] + (let [total (count selected-cases) + jobs (positive-jobs jobs) + executor (Executors/newFixedThreadPool jobs) + completions (LinkedBlockingQueue.)] + (try + (doseq [[idx case] (map-indexed vector selected-cases)] + (let [index (inc idx)] + (when on-case-start + (on-case-start {:index index + :total total + :case case})) + (.submit executor + ^Runnable + (fn [] + (let [started-at (System/nanoTime)] + (.put completions + (try + (let [result (run-case case {:run-command run-command + :detailed-log? detailed-log? + :timings? timings?})] + {:index index + :total total + :case case + :result result + :elapsed-ms (elapsed-ms started-at)}) + (catch Exception error + {:index index + :total total + :case case + :error error + :elapsed-ms (elapsed-ms started-at)})))))))) + (loop [remaining total + results [] + failure nil] + (if (zero? remaining) + (do + (when failure + (throw failure)) + (->> results + (sort-by :index) + (mapv :result))) + (let [payload (.take completions)] + (if-let [error (:error payload)] + (do + (when on-case-failure + (on-case-failure payload)) + (recur (dec remaining) results (or failure error))) + (do + (when on-case-success + (on-case-success payload)) + (recur (dec remaining) (conj results payload) failure)))))) + (finally + (.shutdown executor) + (.awaitTermination executor 1 TimeUnit/MINUTES))))) + +(defn run! + [{:keys [inventory cases skip-build run-command jobs] + :as opts}] + (let [run-command (or run-command shell/run!) + run-case (or (:run-case opts) runner/run-case!) + suite (suite-from-opts opts) + sync-suite? (= suite :sync) + jobs (positive-jobs jobs) + targeted-run? (or (:case opts) (seq (:include opts))) + on-preflight-start (:on-preflight-start opts) + on-preflight-complete (:on-preflight-complete opts) + on-cases-ready (:on-cases-ready opts)] + (when on-preflight-start + (on-preflight-start {:skip-build skip-build})) + (let [preflight-result (preflight/run! {:skip-build skip-build + :run-command run-command}) + inventory (or inventory (manifests/load-inventory suite)) + cases (or cases (manifests/load-cases suite))] + (when on-preflight-complete + (on-preflight-complete preflight-result)) + (let [selected-cases (select-cases cases opts) + coverage-result (when-not targeted-run? + (coverage/coverage-report inventory selected-cases))] + (when (and coverage-result + (not (coverage/complete? coverage-result))) + (throw (ex-info "Missing coverage" + {:coverage coverage-result + :message (report/format-missing-coverage coverage-result)}))) + (when on-cases-ready + (on-cases-ready {:total (count selected-cases) + :targeted-run? targeted-run?})) + (let [suite-context (when sync-suite? + (sync-fixture/before-suite! {:run-command run-command})) + sync-context (if suite-context + (assoc suite-context :e2ee-password (:e2ee-password opts)) + suite-context) + run-case* (if sync-suite? + (fn [case case-opts] + (run-case (sync-fixture/prepare-case case sync-context) + case-opts)) + run-case)] + (try + {:status :ok + :cases selected-cases + :coverage coverage-result + :results ((if (> jobs 1) + run-selected-cases-in-parallel! + run-selected-cases!) + selected-cases + run-case* + run-command + (assoc opts :jobs jobs))} + (finally + (when suite-context + (sync-fixture/after-suite! suite-context {:run-command run-command}))))))))) +(defn build! + [opts] + (preflight/run! opts)) + +(defn list-cases! + [opts] + (doseq [{:keys [id]} (manifests/load-cases (suite-from-opts opts))] + (println id))) + +(defn list-sync-cases! + [opts] + (list-cases! (assoc opts :suite :sync))) + +(defn- print-cleanup-help! + [] + (println "Usage: bb -f cli-e2e/bb.edn cleanup [options]") + (println) + (println "Options:") + (println " -h, --help Show this help and exit") + (println " --dry-run Scan and report only; do not kill/delete") + (println) + (println "Cleanups performed:") + (println " - Terminate cli-e2e db-worker-node processes") + (println " - Terminate db-sync server listeners on port 18080") + (println " - Remove cli-e2e temp roots") + (flush)) + +(defn cleanup! + [opts] + (if (:help opts) + (do + (print-cleanup-help!) + {:status :help}) + (let [dry-run? (boolean (:dry-run opts)) + cleanup-opts (cond-> {} + dry-run? (assoc :dry-run true)) + processes (cleanup/cleanup-db-worker-processes! cleanup-opts) + db-sync-port-processes (cleanup/cleanup-db-sync-port-processes! cleanup-opts) + temp-roots (cleanup/cleanup-temp-roots! cleanup-opts)] + (println "==> Running cli-e2e cleanup") + (if dry-run? + (do + (println (format "[dry-run] db-worker-node processes: found %d, would kill %d" + (count (:found-pids processes)) + (count (:would-kill-pids processes)))) + (println (format "[dry-run] db-sync server processes (port 18080): found %d, would kill %d" + (count (:found-pids db-sync-port-processes)) + (count (:would-kill-pids db-sync-port-processes)))) + (println (format "[dry-run] temp roots: found %d, would remove %d" + (count (:found-dirs temp-roots)) + (count (:would-remove-dirs temp-roots))))) + (do + (println (format "db-worker-node processes: found %d, killed %d, failed %d" + (count (:found-pids processes)) + (count (:killed-pids processes)) + (count (:failed-pids processes)))) + (println (format "db-sync server processes (port 18080): found %d, killed %d, failed %d" + (count (:found-pids db-sync-port-processes)) + (count (:killed-pids db-sync-port-processes)) + (count (:failed-pids db-sync-port-processes)))) + (println (format "temp roots: found %d, removed %d, failed %d" + (count (:found-dirs temp-roots)) + (count (:removed-dirs temp-roots)) + (count (:failed-dirs temp-roots)))))) + (flush) + {:status :ok + :dry-run? dry-run? + :processes processes + :db-sync-port-processes db-sync-port-processes + :temp-roots temp-roots}))) + +(defn- print-failure-details! + [error] + (let [data (ex-data error)] + (println (str " reason: " (.getMessage error))) + (when-let [cmd (:cmd data)] + (println (str " cmd: " cmd))) + (when-let [stream (:stream data)] + (println (str " stream: " stream))) + (when-let [snippet (:snippet data)] + (println (str " snippet: " snippet))) + (flush))) + +(defn- format-step-label + [{:keys [phase step-index step-total]}] + (let [phase-name (name (or phase :command))] + (format "%s %d/%d" phase-name (or step-index 1) (or step-total 1)))) + +(defn- print-case-timings! + [timings] + (when (seq timings) + (println " step timings:") + (doseq [{:keys [elapsed-ms status cmd] :as timing} timings] + (println (format " - [%-12s] %6dms (%s) $ %s" + (format-step-label timing) + elapsed-ms + (name (or status :ok)) + cmd))) + (flush))) + +(defn- print-slow-steps! + [all-step-timings] + (when (seq all-step-timings) + (println "Slow steps (top 10):") + (doseq [{:keys [case-id elapsed-ms cmd] :as timing} + (->> all-step-timings + (sort-by :elapsed-ms >) + (take 10))] + (println (format " - %-45s [%-12s] %6dms $ %s" + case-id + (format-step-label timing) + elapsed-ms + cmd))) + (flush))) + +(defn- progress-prefix + [{:keys [parallel? index total]} symbol] + (if parallel? + (str symbol " ") + (format "[%d/%d] %s " index total symbol))) + +(defn- print-test-help! + [command-name suite] + (let [sync-suite? (= suite :sync)] + (println (str "Usage: bb -f cli-e2e/bb.edn " command-name " [options]")) + (println) + (println "Options:") + (println " -h, --help Show this help and exit") + (println " --skip-build Skip build preflight steps") + (println " -i, --include TAG Run only cases with matching tag (repeatable)") + (println " --case ID Run a single case by id") + (println (format " --jobs N %s (Default: %d)" + (if sync-suite? + "Run up to N sync cases in parallel" + "Run up to N non-sync cases in parallel") + default-cli-jobs)) + (when sync-suite? + (println " --e2ee-password VALUE E2EE password for sync commands (Default: 11111)")) + (println " --verbose Enable verbose output") + (println " --timings Print per-step timings and slow-step summary") + (println) + (println "Examples:") + (if sync-suite? + (println (str " bb -f cli-e2e/bb.edn " command-name)) + (println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build"))) + (println (str " bb -f cli-e2e/bb.edn " command-name + (if sync-suite? + " --jobs 4" + " --skip-build --jobs 4"))) + (println (str " bb -f cli-e2e/bb.edn " command-name " -i smoke")) + (if sync-suite? + (println (str " bb -f cli-e2e/bb.edn " command-name " --case sync-upload-download-mvp")) + (println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build --case global-help"))) + (when sync-suite? + (println (str " bb -f cli-e2e/bb.edn " command-name " --e2ee-password 'my-secret'"))) + (flush))) + +(defn- test-suite! + [opts {:keys [suite command-name] + :or {suite default-suite + command-name "test"}}] + (let [suite (or suite default-suite) + opts (assoc opts :suite suite)] + (if (:help opts) + (do + (print-test-help! command-name suite) + {:status :help}) + (let [started-at (System/nanoTime) + passed (atom 0) + failed (atom 0) + total-count (atom 0) + timings? (boolean (:timings opts)) + all-step-timings (atom []) + detailed-case-log? (some? (:case opts)) + parallel? (> (positive-jobs (:jobs opts)) 1) + base-run-command (or (:run-command opts) shell/run!) + run-command (if detailed-case-log? + (fn [{:keys [cmd phase step-index step-total] :as command-opts}] + (let [prefix (case phase + :setup (format " [setup %d/%d]" step-index step-total) + :main (format " [main %d/%d]" (or step-index 1) (or step-total 1)) + :cleanup (format " [cleanup %d/%d]" step-index step-total) + " [command]")] + (println (str prefix " $ " cmd)) + (flush) + (base-run-command (assoc command-opts :stream-output? true)))) + base-run-command)] + (println "==> Running cli-e2e cases") + (when detailed-case-log? + (println (format "==> Detailed case logging enabled (--case %s)" (:case opts)))) + (when timings? + (println "==> Step timing enabled (--timings)")) + (flush) + (try + (run! (assoc opts + :run-command run-command + :detailed-log? detailed-case-log? + :timings? timings? + :on-preflight-start (fn [_] + (println "==> Build preflight: running...") + (flush)) + :on-preflight-complete (fn [{:keys [status]}] + (println (case status + :skipped "==> Build preflight: skipped (--skip-build)" + "==> Build preflight: completed")) + (flush)) + :on-cases-ready (fn [{:keys [total]}] + (reset! total-count total) + (println (format "==> Prepared %d case(s), starting execution" total)) + (flush)) + :on-case-start (fn [{:keys [index total case]}] + (println (str (progress-prefix {:parallel? parallel? + :index index + :total total} + "▶") + (:id case))) + (flush)) + :on-case-success (fn [{:keys [index total result elapsed-ms]}] + (swap! passed inc) + (println (format "%s%s (%dms)" + (progress-prefix {:parallel? parallel? + :index index + :total total} + "✓") + (:id result) + elapsed-ms)) + (when timings? + (let [case-timings (vec (:timings result))] + (swap! all-step-timings into + (map #(assoc % :case-id (:id result)) case-timings)) + (print-case-timings! case-timings))) + (flush)) + :on-case-failure (fn [{:keys [index total case error elapsed-ms]}] + (swap! failed inc) + (println (format "%s%s (%dms)" + (progress-prefix {:parallel? parallel? + :index index + :total total} + "✗") + (:id case) + elapsed-ms)) + (print-failure-details! error) + (when timings? + (let [case-timings (vec (:timings (ex-data error)))] + (swap! all-step-timings into + (map #(assoc % :case-id (:id case)) case-timings)) + (print-case-timings! case-timings)))))) + (println (format "Summary: %d passed, %d failed" @passed @failed)) + (println (str "Selected cases: " @total-count)) + (println (str "Duration: " (format-duration started-at))) + (when timings? + (print-slow-steps! @all-step-timings)) + (catch Exception error + (let [failed-count (max 1 @failed)] + (println (format "Summary: %d passed, %d failed" @passed failed-count)) + (println (str "Selected cases: " (max @total-count failed-count))) + (println (str "Duration: " (format-duration started-at))) + (when timings? + (print-slow-steps! @all-step-timings))) + (throw error))))))) + +(defn test! + [opts] + (test-suite! opts {:suite :non-sync + :command-name "test"})) + +(defn test-sync! + [opts] + (test-suite! opts {:suite :sync + :command-name "test-sync"})) diff --git a/cli-e2e/src/logseq/cli/e2e/manifests.clj b/cli-e2e/src/logseq/cli/e2e/manifests.clj new file mode 100644 index 0000000000..3032052f5d --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/manifests.clj @@ -0,0 +1,237 @@ +(ns logseq.cli.e2e.manifests + (:require [clojure.edn :as edn] + [logseq.cli.e2e.paths :as paths])) + +(def suite->manifest-files + {:non-sync {:inventory "non_sync_inventory.edn" + :cases "non_sync_cases.edn"} + :sync {:inventory "sync_inventory.edn" + :cases "sync_cases.edn"}}) + +(def default-suite :non-sync) + +(def ^:private append-merge-keys + #{:setup :cmds :cleanup :tags}) + +(def ^:private deep-merge-keys + #{:vars :covers :expect}) + +(defn read-edn-file + [path] + (edn/read-string (slurp path))) + +(defn- normalize-suite + [suite] + (let [suite' (cond + (nil? suite) default-suite + (keyword? suite) suite + (string? suite) (keyword suite) + :else suite)] + (when-not (contains? suite->manifest-files suite') + (throw (ex-info "Unknown cli-e2e suite" + {:suite suite + :known-suites (sort (keys suite->manifest-files))}))) + suite')) + +(defn- manifest-file + [suite kind] + (get-in suite->manifest-files [(normalize-suite suite) kind])) + +(defn load-inventory + ([] + (load-inventory nil)) + ([suite] + (read-edn-file (paths/spec-path (manifest-file suite :inventory))))) + +(defn- normalize-extends + [extends] + (let [extends' (cond + (nil? extends) [] + (keyword? extends) [extends] + (vector? extends) extends + :else + (throw (ex-info "Invalid :extends value in cli-e2e manifest" + {:extends extends + :expected "keyword | vector | nil"}))) + invalid-entries (remove keyword? extends')] + (when (seq invalid-entries) + (throw (ex-info "Invalid :extends entries in cli-e2e manifest" + {:extends extends + :invalid-entries (vec invalid-entries) + :expected "keyword | vector | nil"}))) + extends')) + +(defn- parse-manifest + [manifest-data] + (if-not (map? manifest-data) + (throw (ex-info "Invalid cli-e2e manifest format" + {:manifest-type (type manifest-data) + :expected "{:templates {...} :cases [...]}"})) + (let [templates (or (:templates manifest-data) {}) + cases (:cases manifest-data)] + (when-not (map? templates) + (throw (ex-info "Invalid cli-e2e manifest :templates format" + {:templates templates + :expected "map"}))) + (when-not (vector? cases) + (throw (ex-info "Invalid cli-e2e manifest :cases format" + {:cases cases + :expected "vector"}))) + (doseq [case cases] + (when-not (map? case) + (throw (ex-info "Invalid cli-e2e case format" + {:case case + :expected "map"})))) + {:templates templates + :cases cases}))) + +(defn- reachable-template-ids + [templates roots] + (loop [stack (vec roots) + visited #{}] + (if-let [template-id (peek stack)] + (if (contains? visited template-id) + (recur (pop stack) visited) + (let [template (get templates template-id) + parent-ids (if template + (normalize-extends (:extends template)) + [])] + (recur (into (pop stack) parent-ids) + (conj visited template-id)))) + visited))) + +(defn- lint-manifest! + [{:keys [templates cases]}] + (let [template-ids (set (keys templates)) + template-refs (mapcat (fn [[template-id template]] + (map (fn [target] + {:type :invalid-extends + :source-type :template + :source template-id + :target target}) + (normalize-extends (:extends template)))) + templates) + case-refs (mapcat (fn [[index case]] + (map (fn [target] + {:type :invalid-extends + :source-type :case + :source (or (:id case) + (str "case#" (inc index))) + :target target}) + (normalize-extends (:extends case)))) + (map-indexed vector cases)) + invalid-extends-issues (->> (concat template-refs case-refs) + (filter (fn [{:keys [target]}] + (not (contains? template-ids target))))) + duplicate-id-issues (->> cases + (keep :id) + frequencies + (filter (fn [[_ count]] (> count 1))) + (sort-by first) + (map (fn [[case-id count]] + {:type :duplicate-case-id + :id case-id + :count count}))) + roots (->> cases + (mapcat #(normalize-extends (:extends %))) + distinct) + used-template-ids (reachable-template-ids templates roots) + unused-template-issues (->> (keys templates) + (remove used-template-ids) + sort + (map (fn [template-id] + {:type :unused-template + :template template-id}))) + issues (vec (concat invalid-extends-issues + duplicate-id-issues + unused-template-issues))] + (when (seq issues) + (throw (ex-info "cli-e2e manifest lint failed" + {:issues issues}))))) + +(defn- as-seq + [value] + (cond + (nil? value) [] + (sequential? value) value + :else [value])) + +(defn- deep-merge-maps + [left right] + (merge-with (fn [left-val right-val] + (if (and (map? left-val) + (map? right-val)) + (deep-merge-maps left-val right-val) + right-val)) + (or left {}) + (or right {}))) + +(defn- merge-entry + [parent child] + (let [all-keys (set (concat (keys parent) (keys child)))] + (reduce (fn [acc key] + (let [parent-val (get parent key) + child-val (get child key)] + (assoc acc + key + (cond + (contains? append-merge-keys key) + (if (contains? child key) + (vec (concat (as-seq parent-val) + (as-seq child-val))) + (vec (as-seq parent-val))) + + (contains? deep-merge-keys key) + (if (contains? child key) + (deep-merge-maps parent-val child-val) + parent-val) + + (contains? child key) + child-val + + :else + parent-val)))) + {} + all-keys))) + +(defn- resolve-template + [templates template-id stack] + (when (some #{template-id} stack) + (throw (ex-info "Circular template inheritance detected in cli-e2e manifest" + {:template template-id + :cycle (conj (vec stack) template-id)}))) + (let [template (get templates template-id)] + (when-not template + (throw (ex-info "Unknown template in cli-e2e manifest" + {:template template-id + :known-templates (sort (keys templates))}))) + (let [parent-ids (normalize-extends (:extends template)) + parent-values (map #(resolve-template templates % (conj stack template-id)) + parent-ids)] + (reduce merge-entry + {} + (concat parent-values + [(dissoc template :extends)]))))) + +(defn- expand-manifest-cases + [{:keys [templates cases]}] + (mapv (fn [case] + (let [parent-ids (normalize-extends (:extends case)) + parent-values (map #(resolve-template templates % []) + parent-ids)] + (reduce merge-entry + {} + (concat parent-values + [(dissoc case :extends)])))) + cases)) + +(defn load-cases + ([] + (load-cases nil)) + ([suite] + (let [manifest-data (-> (manifest-file suite :cases) + paths/spec-path + read-edn-file + parse-manifest)] + (lint-manifest! manifest-data) + (expand-manifest-cases manifest-data)))) diff --git a/cli-e2e/src/logseq/cli/e2e/paths.clj b/cli-e2e/src/logseq/cli/e2e/paths.clj new file mode 100644 index 0000000000..7fcac572b6 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/paths.clj @@ -0,0 +1,48 @@ +(ns logseq.cli.e2e.paths + (:require [babashka.fs :as fs])) + +(defn- ancestors + [path] + (take-while some? (iterate fs/parent path))) + +(defn- find-parent-named + [path name] + (some (fn [candidate] + (when (= name (fs/file-name candidate)) + candidate)) + (ancestors (fs/canonicalize path)))) + +(def ^:private cli-e2e-root-path + (or (some-> *file* + (find-parent-named "cli-e2e") + str) + (throw (ex-info "Unable to locate cli-e2e root" + {:file *file*})))) + +(defn cli-e2e-root + [] + cli-e2e-root-path) + +(defn repo-root + [] + (str (fs/parent cli-e2e-root-path))) + +(defn repo-path + [& segments] + (str (apply fs/path (repo-root) segments))) + +(defn cli-e2e-path + [& segments] + (str (apply fs/path (cli-e2e-root) segments))) + +(defn spec-path + [filename] + (cli-e2e-path "spec" filename)) + +(defn required-artifacts + [] + [(repo-path "static" "logseq-cli.js") + (repo-path "static" "db-worker-node.js") + (repo-path "dist" "db-worker-node.js") + (repo-path "dist" "db-worker-node-assets.json") + (repo-path "deps" "db-sync" "worker" "dist" "node-adapter.js")]) diff --git a/cli-e2e/src/logseq/cli/e2e/preflight.clj b/cli-e2e/src/logseq/cli/e2e/preflight.clj new file mode 100644 index 0000000000..8e48e3f375 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/preflight.clj @@ -0,0 +1,37 @@ +(ns logseq.cli.e2e.preflight + (:require [babashka.fs :as fs] + [logseq.cli.e2e.paths :as paths] + [logseq.cli.e2e.shell :as shell])) + +(def build-plan + [{:cmd "clojure -M:cljs compile logseq-cli"} + {:cmd "pnpm db-worker-node:compile:bundle"} + {:cmd "pnpm --dir deps/db-sync build:node-adapter"}]) + +(defn missing-artifacts + ([] + (missing-artifacts (paths/required-artifacts) fs/exists?)) + ([artifacts file-exists?] + (->> artifacts + (remove file-exists?) + vec))) + +(defn run! + [{:keys [skip-build run-command file-exists?] + :or {run-command shell/run! + file-exists? fs/exists?}}] + (if skip-build + {:status :skipped + :commands [] + :missing-artifacts []} + (let [artifacts (paths/required-artifacts)] + (doseq [{:keys [cmd]} build-plan] + (run-command {:cmd cmd + :dir (paths/repo-root)})) + (let [missing-after-build (missing-artifacts artifacts file-exists?)] + (when (seq missing-after-build) + (throw (ex-info "Build preflight completed but required artifacts are missing" + {:missing-artifacts missing-after-build}))) + {:status :ok + :commands build-plan + :missing-artifacts []})))) diff --git a/cli-e2e/src/logseq/cli/e2e/report.clj b/cli-e2e/src/logseq/cli/e2e/report.clj new file mode 100644 index 0000000000..33c217e0ec --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/report.clj @@ -0,0 +1,18 @@ +(ns logseq.cli.e2e.report + (:require [clojure.string :as string])) + +(defn format-missing-coverage + [{:keys [missing-commands missing-options]}] + (str + "Missing coverage\n" + (when (seq missing-commands) + (str "Commands:\n" + (string/join "\n" (map #(str "- " %) missing-commands)) + "\n")) + (string/join + "\n" + (keep (fn [[scope options]] + (when (seq options) + (str (name scope) " options:\n" + (string/join "\n" (map #(str "- " %) options))))) + missing-options)))) diff --git a/cli-e2e/src/logseq/cli/e2e/runner.clj b/cli-e2e/src/logseq/cli/e2e/runner.clj new file mode 100644 index 0000000000..42c619b959 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/runner.clj @@ -0,0 +1,280 @@ +(ns logseq.cli.e2e.runner + (:require [babashka.fs :as fs] + [cheshire.core :as json] + [clojure.edn :as edn] + [clojure.string :as string] + [logseq.cli.e2e.paths :as paths] + [logseq.cli.e2e.shell :as shell])) + +(defn shell-escape + [value] + (let [text (str value)] + (if (re-find #"[^\w@%+=:,./-]" text) + (str "'" (string/replace text #"'" "'\"'\"'") "'") + text))) + +(def template-pattern #"\{\{([^}]+)\}\}") +(def ^:private e2e-env {"CLI_E2E_TEST" "1" + "NO_COLOR" "1"}) + +(defn- render-string + [template context] + (string/replace template + template-pattern + (fn [[_ key]] + (let [lookup-key (keyword (string/trim key))] + (when-not (contains? context lookup-key) + (throw (ex-info "Missing case template value" + {:template template + :key lookup-key + :context context}))) + (str (get context lookup-key)))))) + +(defn- render-value + [value context] + (cond + (string? value) (render-string value context) + (map? value) (into {} + (map (fn [[k v]] + [k (render-value v context)])) + value) + (vector? value) (mapv #(render-value % context) value) + (seq? value) (doall (map #(render-value % context) value)) + :else value)) + +(defn render-case + [case context] + (render-value case context)) + +(defn- ensure-contains! + [id cmd output snippets stream] + (doseq [snippet (or snippets [])] + (let [normalize-pathish-text (fn [text] + (-> (str text) + (string/replace "\\\\" "\\") + (string/replace "\\" "/") + (string/replace "\r" ""))) + matches? (or (string/includes? output snippet) + (string/includes? (normalize-pathish-text output) + (normalize-pathish-text snippet)))] + (when-not matches? + (throw (ex-info (str stream " did not contain expected text") + {:id id + :cmd cmd + :snippet snippet + :output output + :stream stream})))))) + +(defn- ensure-not-contains! + [id cmd output snippets stream] + (doseq [snippet (or snippets [])] + (when (string/includes? output snippet) + (throw (ex-info (str stream " contained forbidden text") + {:id id + :cmd cmd + :snippet snippet + :output output + :stream stream}))))) + +(defn- ensure-equals! + [id cmd actual expected label] + (when (and (some? expected) + (not= expected actual)) + (throw (ex-info (str label " did not match") + {:id id + :cmd cmd + :expected expected + :actual actual})))) + +(defn- ensure-json-paths! + [id cmd output expected-paths] + (when expected-paths + (let [payload (json/parse-string output true)] + (doseq [[path expected] expected-paths] + (let [actual (get-in payload path)] + (when-not (= expected actual) + (throw (ex-info "stdout JSON path did not match" + {:id id + :cmd cmd + :path path + :expected expected + :actual actual + :payload payload})))))))) + +(defn- ensure-edn-paths! + [id cmd output expected-paths] + (when expected-paths + (let [payload (edn/read-string output)] + (doseq [[path expected] expected-paths] + (let [actual (get-in payload path)] + (when-not (= expected actual) + (throw (ex-info "stdout EDN path did not match" + {:id id + :cmd cmd + :path path + :expected expected + :actual actual + :payload payload})))))))) + +(defn assert-result! + [{:keys [id expect]} {:keys [cmd exit out err] :as result}] + (when-let [expected-exit (:exit expect)] + (when-not (= expected-exit exit) + (throw (ex-info "Unexpected exit code" + {:id id + :cmd cmd + :expected expected-exit + :actual exit + :result result})))) + (ensure-equals! id cmd out (:stdout-equals expect) "stdout") + (ensure-equals! id cmd err (:stderr-equals expect) "stderr") + (ensure-contains! id cmd out (:stdout-contains expect) "stdout") + (ensure-contains! id cmd err (:stderr-contains expect) "stderr") + (ensure-not-contains! id cmd out (:stdout-not-contains expect) "stdout") + (ensure-not-contains! id cmd err (:stderr-not-contains expect) "stderr") + (ensure-json-paths! id cmd out (:stdout-json-paths expect)) + (ensure-edn-paths! id cmd out (:stdout-edn-paths expect)) + nil) + +(defn- default-context + [case context] + (let [tmp-dir (or (:tmp-dir context) + (str (fs/create-temp-dir {:prefix (str "logseq-cli-e2e-" (:id case) "-")}))) + root-dir (or (:root-dir context) + tmp-dir) + config-path (or (:config-path context) + (str (fs/path root-dir "cli.edn"))) + export-path (or (:export-path context) + (str (fs/path tmp-dir "graph-export.edn"))) + graph (or (:graph context) + (:graph case) + (str "cli-e2e-" (:id case)))] + (fs/create-dirs (fs/path root-dir "graphs")) + (spit config-path "{:output-format :json}\n") + {:tmp-dir tmp-dir + :tmp-dir-arg (shell-escape tmp-dir) + :root-dir root-dir + :root-dir-arg (shell-escape root-dir) + :config-path config-path + :config-path-arg (shell-escape config-path) + :export-path export-path + :export-path-arg (shell-escape export-path) + :repo-root (paths/repo-root) + :repo-root-arg (shell-escape (paths/repo-root)) + :cli (str "node " (shell-escape (paths/repo-path "static" "logseq-cli.js"))) + :graph graph + :graph-arg (shell-escape graph)})) + +(defn- case-context + [case {:keys [context]}] + (merge (default-context case context) + context + (:vars case))) + +(defn- run-command! + [command context {:keys [run-command stdin allow-failure phase step-index step-total case-id]}] + (run-command {:cmd (render-string command context) + :dir (paths/repo-root) + :env e2e-env + :stdin (some-> stdin (render-string context)) + :phase phase + :step-index step-index + :step-total step-total + :case-id case-id + :throw? (not allow-failure)})) + +(defn- elapsed-ms + [started-at] + (long (/ (- (System/nanoTime) started-at) 1000000))) + +(defn- run-step! + [timings command context {:keys [timings? phase step-index step-total] + :as run-opts}] + (if-not timings? + (run-command! command context run-opts) + (let [started-at (System/nanoTime)] + (try + (let [result (run-command! command context run-opts)] + (swap! timings conj {:phase phase + :step-index step-index + :step-total step-total + :cmd (:cmd result) + :elapsed-ms (elapsed-ms started-at) + :status :ok}) + result) + (catch Exception error + (swap! timings conj {:phase phase + :step-index step-index + :step-total step-total + :cmd (or (:cmd (ex-data error)) + (render-string command context)) + :elapsed-ms (elapsed-ms started-at) + :status :failed}) + (throw error)))))) + +(defn run-case! + [case {:keys [run-command detailed-log? timings?] + :or {run-command shell/run!} + :as opts}] + (let [context (case-context case opts) + rendered (render-case case context) + case-id (:id rendered) + timings? (boolean timings?) + timings (atom []) + cleanup-commands (vec (:cleanup rendered)) + setup-commands (vec (:setup rendered)) + main-commands (vec (:cmds rendered)) + cleanup! (fn [] + (doseq [[idx command] (map-indexed vector cleanup-commands)] + (try + (run-step! timings command context {:run-command run-command + :timings? timings? + :allow-failure true + :phase (when (or detailed-log? timings?) :cleanup) + :step-index (inc idx) + :step-total (count cleanup-commands) + :case-id case-id}) + (catch Exception _ + nil))))] + (try + (doseq [[idx command] (map-indexed vector setup-commands)] + (run-step! timings command context {:run-command run-command + :timings? timings? + :phase (when (or detailed-log? timings?) :setup) + :step-index (inc idx) + :step-total (count setup-commands) + :case-id case-id})) + (let [main-total (count main-commands) + _ (when (zero? main-total) + (throw (ex-info "Missing case commands" + {:id case-id + :case rendered}))) + result (reduce (fn [_ [idx command]] + (let [last-step? (= idx (dec main-total))] + (run-step! timings command context {:run-command run-command + :timings? timings? + :stdin (when last-step? (:stdin rendered)) + :allow-failure last-step? + :phase (when (or detailed-log? timings?) :main) + :step-index (inc idx) + :step-total main-total + :case-id case-id}))) + nil + (map-indexed vector main-commands))] + (assert-result! rendered result) + (cleanup!) + (cond-> {:id case-id + :status :ok + :cmd (:cmd result) + :result result + :context context} + timings? (assoc :timings @timings))) + (catch Exception error + (cleanup!) + (if timings? + (throw (ex-info (.getMessage error) + (assoc (or (ex-data error) {}) + :timings @timings + :case-id case-id) + error)) + (throw error)))))) diff --git a/cli-e2e/src/logseq/cli/e2e/shell.clj b/cli-e2e/src/logseq/cli/e2e/shell.clj new file mode 100644 index 0000000000..4cbc052e8d --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/shell.clj @@ -0,0 +1,67 @@ +(ns logseq.cli.e2e.shell + (:require [babashka.process :as process]) + (:import [java.io ByteArrayOutputStream OutputStream])) + +(defn- utf8-string + [^ByteArrayOutputStream payload] + (.toString payload "UTF-8")) + +(defn- capture-stream + [target stream-output?] + (let [payload (ByteArrayOutputStream.)] + {:payload payload + :stream (if stream-output? + (proxy [OutputStream] [] + (write + ([byte] + (.write payload byte) + (.write target byte) + (.flush target)) + ([bytes offset length] + (.write payload bytes offset length) + (.write target bytes offset length) + (.flush target))) + (flush [] + (.flush target)) + (close [] + (.flush target))) + payload)})) + +(defn- default-executor + [cmd {:keys [dir extra-env in stream-output?]}] + (let [{out-payload :payload + out-stream :stream} (capture-stream System/out stream-output?) + {err-payload :payload + err-stream :stream} (capture-stream System/err stream-output?) + result @(process/process {:continue true + :dir dir + :extra-env extra-env + :in in + :out out-stream + :err err-stream} + "bash" + "-c" + cmd)] + {:exit (:exit result 0) + :out (utf8-string out-payload) + :err (utf8-string err-payload)})) + +(defn run! + [{:keys [cmd dir env stdin executor throw? stream-output?] + :or {throw? true}}] + (let [runner (or executor default-executor) + result (runner cmd {:dir dir + :extra-env env + :in stdin + :stream-output? stream-output?}) + exit (:exit result 0) + payload {:cmd cmd + :dir dir + :exit exit + :out (:out result "") + :err (:err result "")}] + (when (and throw? + (not (zero? exit))) + (throw (ex-info "Shell command failed" + payload))) + payload)) diff --git a/cli-e2e/src/logseq/cli/e2e/sync_fixture.clj b/cli-e2e/src/logseq/cli/e2e/sync_fixture.clj new file mode 100644 index 0000000000..6d8b1cc6e9 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/sync_fixture.clj @@ -0,0 +1,132 @@ +(ns logseq.cli.e2e.sync-fixture + (:require [babashka.fs :as fs] + [clojure.string :as string] + [logseq.cli.e2e.paths :as paths] + [logseq.cli.e2e.runner :as runner] + [logseq.cli.e2e.shell :as shell])) + +(def default-sync-port "18080") +(def default-e2ee-password "11111") + +(def ^:private heavy-setup-patterns + [#"^mkdir -p '\{\{tmp-dir\}\}/home/logseq'$" + #"^cp ~/logseq/auth\.json\b" + #"prepare_sync_config\.py" + #"db_sync_server\.py'? start"]) + +(def ^:private heavy-cleanup-patterns + [#"db_sync_server\.py'? stop"]) + +(defn- shell-quote + [value] + (runner/shell-escape value)) + +(defn- heavy-command? + [command patterns] + (boolean (some #(re-find % command) patterns))) + +(defn- case-local-setup-prefix + [] + ["mkdir -p '{{tmp-dir}}/home/logseq'" + "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"]) + +(def ^:private case-local-resource-markers + ["{{cli-home}}" + "{{config-path}}" + "{{config-path-arg}}" + "{{tmp-dir}}/cli-b.edn" + "{{auth-path}}" + "{{home-dir}}"]) + +(defn- requires-case-local-resources? + [command] + (boolean (some #(string/includes? command %) case-local-resource-markers))) + +(defn- suite-user-keys-bootstrap-command + [suite-tmp-dir] + (let [lock-dir (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.lock"))) + done-file (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.ready")))] + (format (str "LOCK_DIR=%s; DONE_FILE=%s; " + "if [ -f \"$DONE_FILE\" ]; then exit 0; fi; " + "while ! mkdir \"$LOCK_DIR\" 2>/dev/null; do [ -f \"$DONE_FILE\" ] && exit 0; sleep 0.1; done; " + "trap 'rmdir \"$LOCK_DIR\" 2>/dev/null || true' EXIT; " + "if [ ! -f \"$DONE_FILE\" ]; then " + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}} --upload-keys >/dev/null && touch \"$DONE_FILE\"; " + "fi") + lock-dir + done-file))) + +(defn before-suite! + [{:keys [run-command sync-port] + :or {run-command shell/run! + sync-port default-sync-port}}] + (let [sync-port (str sync-port) + suite-tmp-dir (str (fs/create-temp-dir {:prefix "logseq-cli-e2e-sync-suite-"})) + db-sync-pid-file (str (fs/path suite-tmp-dir "db-sync-server.pid")) + db-sync-log-file (str (fs/path suite-tmp-dir "db-sync-server.log")) + db-sync-root-dir (str (fs/path suite-tmp-dir "db-sync-server-data")) + sync-http-base (str "http://127.0.0.1:" sync-port) + sync-ws-url (str "ws://127.0.0.1:" sync-port "/sync/%s") + auth-path (str (fs/path (System/getProperty "user.home") "logseq" "auth.json")) + start-db-sync-cmd (format "python3 %s start --repo-root %s --pid-file %s --log-file %s --data-dir %s --port %s --startup-timeout-s 60 --auth-path %s" + (shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py")) + (shell-quote (paths/repo-root)) + (shell-quote db-sync-pid-file) + (shell-quote db-sync-log-file) + (shell-quote db-sync-root-dir) + sync-port + (shell-quote auth-path))] + (run-command {:cmd start-db-sync-cmd + :dir (paths/repo-root)}) + {:suite-tmp-dir suite-tmp-dir + :db-sync-pid-file db-sync-pid-file + :db-sync-log-file db-sync-log-file + :db-sync-root-dir db-sync-root-dir + :sync-port sync-port + :sync-http-base sync-http-base + :sync-ws-url sync-ws-url})) + +(defn prepare-case + [case {:keys [suite-tmp-dir sync-port sync-http-base sync-ws-url e2ee-password]}] + (let [e2ee-password (or e2ee-password default-e2ee-password) + setup-commands (vec (:setup case)) + insertion-point? (fn [command] + (or (heavy-command? command heavy-setup-patterns) + (requires-case-local-resources? command))) + leading-setup (->> setup-commands + (take-while #(not (insertion-point? %))) + vec) + trailing-setup (->> setup-commands + (drop (count leading-setup)) + (remove #(heavy-command? % heavy-setup-patterns)) + vec) + bootstrap-setup (when suite-tmp-dir + [(suite-user-keys-bootstrap-command suite-tmp-dir)]) + cleanup' (->> (:cleanup case) + (remove #(heavy-command? % heavy-cleanup-patterns)) + vec)] + (-> case + (update :vars merge {:sync-port sync-port + :sync-http-base sync-http-base + :sync-ws-url sync-ws-url + :e2ee-password e2ee-password + :e2ee-password-arg (shell-quote e2ee-password)}) + (assoc :setup (vec (concat leading-setup + (case-local-setup-prefix) + bootstrap-setup + trailing-setup))) + (assoc :cleanup cleanup')))) + +(defn after-suite! + [{:keys [db-sync-pid-file]} + {:keys [run-command] + :or {run-command shell/run!}}] + (when (and (string? db-sync-pid-file) + (not (string/blank? db-sync-pid-file))) + (run-command {:cmd (format "python3 %s stop --pid-file %s" + (shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py")) + (shell-quote db-sync-pid-file)) + :dir (paths/repo-root) + :throw? false}))) diff --git a/cli-e2e/src/logseq/cli/e2e/test_runner.clj b/cli-e2e/src/logseq/cli/e2e/test_runner.clj new file mode 100644 index 0000000000..447015b3c4 --- /dev/null +++ b/cli-e2e/src/logseq/cli/e2e/test_runner.clj @@ -0,0 +1,23 @@ +(ns logseq.cli.e2e.test-runner + (:require [clojure.test :as test])) + +(def test-namespaces + '[logseq.cli.e2e.coverage-test + logseq.cli.e2e.preflight-test + logseq.cli.e2e.paths-test + logseq.cli.e2e.shell-test + logseq.cli.e2e.runner-test + logseq.cli.e2e.cleanup-test + logseq.cli.e2e.manifests-test + logseq.cli.e2e.sync-fixture-test + logseq.cli.e2e.main-test]) + +(defn run! + [_opts] + (doseq [ns-sym test-namespaces] + (require ns-sym)) + (let [{:keys [fail error]} (apply test/run-tests test-namespaces)] + (when (pos? (+ fail error)) + (throw (ex-info "cli-e2e unit tests failed" + {:fail fail + :error error}))))) diff --git a/cli-e2e/test/logseq/cli/e2e/cleanup_test.clj b/cli-e2e/test/logseq/cli/e2e/cleanup_test.clj new file mode 100644 index 0000000000..2f61fa460b --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/cleanup_test.clj @@ -0,0 +1,149 @@ +(ns logseq.cli.e2e.cleanup-test + (:require [babashka.fs :as fs] + [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.cleanup :as cleanup])) + +(deftest list-cli-e2e-db-worker-pids-filters-processes + (let [shell-fn (fn [& _] + {:exit 0 + :out (str " 101 node /repo/dist/db-worker-node.js --repo graph-a --root-dir /tmp/logseq-cli-e2e-graph-a-123/graphs --owner-source cli\n" + " 202 node /repo/dist/db-worker-node.js --repo production --root-dir /tmp/production-graphs --owner-source cli\n" + " 303 node /repo/static/logseq-cli.js graph list\n" + " 404 /usr/bin/python3 background.py\n" + " 505 node /repo/static/db-worker-node.js --repo graph-b --root-dir /private/tmp/logseq-cli-e2e-graph-b-999/graphs --owner-source cli\n") + :err ""})] + (is (= [101 505] + (cleanup/list-cli-e2e-db-worker-pids {:shell-fn shell-fn})))) + + (testing "throws when ps invocation fails" + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Unable to scan processes" + (cleanup/list-cli-e2e-db-worker-pids {:shell-fn (fn [& _] + {:exit 1 + :out "" + :err "permission denied"})}))))) + +(deftest cleanup-db-worker-processes-separates-killed-and-failed + (let [killed (atom []) + result (cleanup/cleanup-db-worker-processes! {:list-pids-fn (fn [] [11 22 33]) + :kill-pid-fn (fn [pid] + (swap! killed conj pid) + (if (= pid 22) + :failed + :killed))})] + (is (= [11 22 33] (:found-pids result))) + (is (= [11 33] (:killed-pids result))) + (is (= [22] (:failed-pids result))) + (is (= [11 22 33] @killed)))) + +(deftest cleanup-db-worker-processes-dry-run-does-not-kill + (let [killed (atom []) + result (cleanup/cleanup-db-worker-processes! {:dry-run true + :list-pids-fn (fn [] [11 22]) + :kill-pid-fn (fn [pid] + (swap! killed conj pid) + :killed)})] + (is (= [11 22] (:found-pids result))) + (is (true? (:dry-run? result))) + (is (= [11 22] (:would-kill-pids result))) + (is (= [] (:killed-pids result))) + (is (= [] (:failed-pids result))) + (is (= [] @killed)))) + +(deftest list-cli-e2e-db-sync-port-pids-filters-port-18080 + (let [shell-fn (fn [& _] + {:exit 0 + :out (str "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n" + "node 111 me 13u IPv6 0x0 0t0 TCP *:18080 (LISTEN)\n" + "python 222 me 13u IPv4 0x0 0t0 TCP 127.0.0.1:18080 (LISTEN)\n" + "node 333 me 13u IPv6 0x0 0t0 TCP *:18081 (LISTEN)\n") + :err ""})] + (is (= [111 222] + (cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn shell-fn})))) + + (testing "returns empty when no listener exists" + (is (= [] + (cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _] + {:exit 1 + :out "" + :err ""})})))) + + (testing "throws when lsof invocation fails" + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Unable to scan db-sync server port listeners" + (cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _] + {:exit 1 + :out "" + :err "permission denied"})}))))) + +(deftest cleanup-db-sync-port-processes-separates-killed-and-failed + (let [killed (atom []) + result (cleanup/cleanup-db-sync-port-processes! {:list-pids-fn (fn [] [44 55]) + :kill-pid-fn (fn [pid] + (swap! killed conj pid) + (if (= pid 55) + :failed + :killed))})] + (is (= [44 55] (:found-pids result))) + (is (= [44] (:killed-pids result))) + (is (= [55] (:failed-pids result))) + (is (= [44 55] @killed)))) + +(deftest cleanup-db-sync-port-processes-dry-run-does-not-kill + (let [killed (atom []) + result (cleanup/cleanup-db-sync-port-processes! {:dry-run true + :list-pids-fn (fn [] [44]) + :kill-pid-fn (fn [pid] + (swap! killed conj pid) + :killed)})] + (is (= [44] (:found-pids result))) + (is (true? (:dry-run? result))) + (is (= [44] (:would-kill-pids result))) + (is (= [] (:killed-pids result))) + (is (= [] (:failed-pids result))) + (is (= [] @killed)))) + +(deftest cleanup-temp-roots-removes-only-cli-e2e-temp-roots + (let [tmp-root (fs/create-temp-dir {:prefix "cleanup-e2e-test-"}) + matching-root (fs/path tmp-root "logseq-cli-e2e-case-a-123") + another-matching-root (fs/path tmp-root "logseq-cli-e2e-case-b-456") + non-matching-root (fs/path tmp-root "other-case")] + (try + (fs/create-dirs (fs/path matching-root "graphs")) + (fs/create-dirs (fs/path matching-root "home" "logseq")) + (fs/create-dirs (fs/path another-matching-root "graphs")) + (fs/create-dirs (fs/path non-matching-root "graphs")) + + (let [result (cleanup/cleanup-temp-roots! {:tmp-root (str tmp-root)}) + expected-dirs (sort [(str matching-root) + (str another-matching-root)])] + (is (= expected-dirs + (sort (:found-dirs result)))) + (is (= expected-dirs + (sort (:removed-dirs result)))) + (is (empty? (:failed-dirs result))) + (is (false? (fs/exists? matching-root))) + (is (false? (fs/exists? another-matching-root))) + (is (true? (fs/exists? non-matching-root)))) + (finally + (fs/delete-tree tmp-root))))) + +(deftest cleanup-temp-roots-dry-run-does-not-delete + (let [deleted (atom []) + result (cleanup/cleanup-temp-roots! {:dry-run true + :list-dirs-fn (fn [] ["/tmp/logseq-cli-e2e-a" + "/tmp/logseq-cli-e2e-b"]) + :delete-dir-fn (fn [dir] + (swap! deleted conj dir))})] + (is (= ["/tmp/logseq-cli-e2e-a" + "/tmp/logseq-cli-e2e-b"] + (:found-dirs result))) + (is (true? (:dry-run? result))) + (is (= ["/tmp/logseq-cli-e2e-a" + "/tmp/logseq-cli-e2e-b"] + (:would-remove-dirs result))) + (is (= [] (:removed-dirs result))) + (is (= [] (:failed-dirs result))) + (is (= [] @deleted)))) diff --git a/cli-e2e/test/logseq/cli/e2e/compare_graph_queries_test.py b/cli-e2e/test/logseq/cli/e2e/compare_graph_queries_test.py new file mode 100644 index 0000000000..cbea01104a --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/compare_graph_queries_test.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + + +MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "compare_graph_queries.py" +spec = importlib.util.spec_from_file_location("compare_graph_queries", MODULE_PATH) +compare_graph_queries = importlib.util.module_from_spec(spec) +assert spec.loader is not None +spec.loader.exec_module(compare_graph_queries) + + +def test_parse_args_supports_repeated_queries(monkeypatch) -> None: + monkeypatch.setattr( + compare_graph_queries.sys, + "argv", + [ + "compare_graph_queries.py", + "--cli", + "/tmp/logseq-cli.js", + "--graph", + "demo", + "--config-a", + "/tmp/a.edn", + "--root-dir-a", + "/tmp/a", + "--config-b", + "/tmp/b.edn", + "--root-dir-b", + "/tmp/b", + "--query", + "[:find ?x]", + "--query", + "[:find ?y]", + ], + ) + + args = compare_graph_queries.parse_args() + + assert args.root_dir_a == "/tmp/a" + assert args.root_dir_b == "/tmp/b" + assert args.query == ["[:find ?x]", "[:find ?y]"] + + +def test_main_batches_multiple_queries_in_one_process(tmp_path: Path) -> None: + left_queries = [] + right_queries = [] + printed = [] + cli_path = tmp_path / "logseq-cli.js" + cli_path.write_text("// mock cli\n") + + def fake_run_query(cli_path, config_path, root_dir, graph, query): + record = { + "cli_path": str(cli_path), + "config_path": str(config_path), + "root_dir": str(root_dir), + "graph": graph, + "query": query, + } + if str(config_path).endswith("a.edn"): + left_queries.append(record) + else: + right_queries.append(record) + return { + "payload": {"status": "ok"}, + "result": [{"title": query}], + } + + compare_graph_queries.run_query = fake_run_query + compare_graph_queries.parse_args = lambda: SimpleNamespace( + cli=str(cli_path), + graph="demo", + query=["[:find ?x]", "[:find ?y]"], + config_a="/tmp/a.edn", + root_dir_a="/tmp/a", + config_b="/tmp/b.edn", + root_dir_b="/tmp/b", + require_result=False, + ) + compare_graph_queries.print = lambda value, **kwargs: printed.append(json.loads(value)) + + compare_graph_queries.main() + + assert [item["query"] for item in left_queries] == ["[:find ?x]", "[:find ?y]"] + assert [item["query"] for item in right_queries] == ["[:find ?x]", "[:find ?y]"] + assert printed == [ + { + "status": "ok", + "results": { + "[:find ?x]": [{"title": "[:find ?x]"}], + "[:find ?y]": [{"title": "[:find ?y]"}], + }, + } + ] diff --git a/cli-e2e/test/logseq/cli/e2e/coverage_test.clj b/cli-e2e/test/logseq/cli/e2e/coverage_test.clj new file mode 100644 index 0000000000..883a27edef --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/coverage_test.clj @@ -0,0 +1,58 @@ +(ns logseq.cli.e2e.coverage-test + (:require [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.coverage :as coverage] + [logseq.cli.e2e.manifests :as manifests])) + +(def sample-inventory + {:excluded-command-prefixes ["sync" "login" "logout"] + :scopes {:global {:options ["--help" "--version"]} + :graph {:commands ["graph list" "graph create"] + :options ["--file" "--type"]}}}) + +(deftest reports-missing-commands-and-options + (let [cases [{:id "global-help" + :covers {:options {:global ["--help"]}}} + {:id "graph-list" + :covers {:commands ["graph list"] + :options {:graph ["--file"]}}}] + report (coverage/coverage-report sample-inventory cases)] + (is (= ["graph create"] (:missing-commands report))) + (is (= ["--version"] (get-in report [:missing-options :global]))) + (is (= ["--type"] (get-in report [:missing-options :graph]))))) + +(deftest rejects-excluded-commands-in-inventory + (let [inventory (assoc-in sample-inventory [:scopes :bad :commands] ["login"])] + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Excluded commands" + (coverage/validate-inventory! inventory))))) + +(deftest rejects-excluded-commands-in-case-covers + (let [cases [{:id "bad-login" + :covers {:commands ["logout"]}}]] + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Excluded commands" + (coverage/validate-cases! sample-inventory cases))))) + +(deftest complete-coverage-is-recognized + (let [cases [{:id "global" + :covers {:options {:global ["--help" "--version"]}}} + {:id "graph-create" + :covers {:commands ["graph create"] + :options {:graph ["--type"]}}} + {:id "graph-list" + :covers {:commands ["graph list"] + :options {:graph ["--file"]}}}] + report (coverage/coverage-report sample-inventory cases)] + (is (coverage/complete? report)))) + +(deftest sync-suite-manifests-cover-mvp-commands + (let [inventory (manifests/load-inventory :sync) + cases (manifests/load-cases :sync) + covered-commands (set (mapcat #(get-in % [:covers :commands]) cases)) + report (coverage/coverage-report inventory cases)] + (is (contains? covered-commands "sync upload")) + (is (contains? covered-commands "sync download")) + (is (contains? covered-commands "sync status")) + (is (coverage/complete? report)))) diff --git a/cli-e2e/test/logseq/cli/e2e/main_test.clj b/cli-e2e/test/logseq/cli/e2e/main_test.clj new file mode 100644 index 0000000000..977d6523af --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/main_test.clj @@ -0,0 +1,812 @@ +(ns logseq.cli.e2e.main-test + (:require [babashka.cli :as cli] + [clojure.string :as string] + [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.cleanup :as cleanup] + [logseq.cli.e2e.main :as main] + [logseq.cli.e2e.manifests :as manifests] + [logseq.cli.e2e.sync-fixture :as sync-fixture])) + +(def sample-cases + [{:id "global-help" + :cmds ["node static/logseq-cli.js --help"] + :covers {:options {:global ["--help"]}} + :tags [:global :smoke]} + {:id "graph-create" + :cmds ["node static/logseq-cli.js graph create --graph demo"] + :covers {:commands ["graph create"] + :options {:graph ["--type"]}} + :tags [:graph]} + {:id "graph-list" + :cmds ["node static/logseq-cli.js graph list"] + :covers {:commands ["graph list"] + :options {:graph ["--file"]}} + :tags [:graph :smoke]}]) + +(def complete-inventory + {:excluded-command-prefixes ["sync" "login" "logout"] + :scopes {:global {:options ["--help"]} + :graph {:commands ["graph create" "graph list"] + :options ["--type" "--file"]}}}) + +(def cli-parse-config + {:alias {:i :include + :h :help} + :spec {:jobs {:default 4}} + :coerce {:include [] + :help :boolean + :dry-run :boolean + :skip-build :boolean + :verbose :boolean + :timings :boolean + :jobs :long}}) + +(deftest cli-opts-parses-jobs-as-integer + (is (= 3 + (:jobs (cli/parse-opts ["--jobs" "3"] cli-parse-config))))) + +(deftest cli-opts-defaults-jobs-to-four + (is (= 4 + (:jobs (cli/parse-opts [] cli-parse-config))))) + +(deftest cli-opts-rejects-non-integer-jobs + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Coerce failure" + (cli/parse-opts ["--jobs" "nope"] cli-parse-config)))) + +(deftest run-rejects-jobs-less-than-one + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"--jobs must be a positive integer" + (main/run! {:inventory complete-inventory + :cases sample-cases + :skip-build true + :jobs 0 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})))) + +(deftest run-non-sync-uses-parallel-runner-when-jobs-greater-than-one + (let [parallel-call (atom nil) + serial-called? (atom false)] + (with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts] + (reset! parallel-call {:case-ids (mapv :id selected-cases) + :run-case run-case + :run-command run-command + :jobs (:jobs opts)}) + [{:id "global-help" :status :ok} + {:id "graph-list" :status :ok}]) + main/run-selected-cases! (fn [& _] + (reset! serial-called? true) + (throw (ex-info "serial runner should not be used" {})))] + (let [result (main/run! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :jobs 2 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))) + (is (= ["global-help" "graph-list"] (:case-ids @parallel-call))) + (is (= 2 (:jobs @parallel-call))) + (is (false? @serial-called?)))))) + +(deftest run-sync-suite-uses-parallel-runner-when-jobs-greater-than-one + (let [parallel-call (atom nil) + serial-called? (atom false) + sync-inventory {:excluded-command-prefixes ["login" "logout"] + :scopes {:sync {:commands ["sync upload" "sync status"] + :options []}}} + sync-cases [{:id "sync-upload" + :cmds ["node static/logseq-cli.js sync upload"] + :covers {:commands ["sync upload"]}} + {:id "sync-status" + :cmds ["node static/logseq-cli.js sync status"] + :covers {:commands ["sync status"]}}]] + (with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts] + (reset! parallel-call {:case-ids (mapv :id selected-cases) + :run-case run-case + :run-command run-command + :jobs (:jobs opts)}) + (mapv (fn [case] + {:id (:id case) + :status :ok}) + selected-cases)) + main/run-selected-cases! (fn [& _] + (reset! serial-called? true) + (throw (ex-info "serial runner should not be used" {})))] + (let [result (main/run! {:suite :sync + :inventory sync-inventory + :cases sync-cases + :skip-build true + :jobs 4 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))) + (is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call))) + (is (= 4 (:jobs @parallel-call))) + (is (false? @serial-called?)))))) + +(deftest run-jobs-one-keeps-serial-runner + (let [serial-call (atom nil) + parallel-called? (atom false)] + (with-redefs [main/run-selected-cases! (fn [selected-cases run-case run-command opts] + (reset! serial-call {:case-ids (mapv :id selected-cases) + :run-case run-case + :run-command run-command + :jobs (:jobs opts)}) + (mapv (fn [case] + {:id (:id case) + :status :ok}) + selected-cases)) + main/run-selected-cases-in-parallel! (fn [& _] + (reset! parallel-called? true) + (throw (ex-info "parallel runner should not be used" {})))] + (let [result (main/run! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :jobs 1 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))) + (is (= ["global-help" "graph-list"] (:case-ids @serial-call))) + (is (= 1 (:jobs @serial-call))) + (is (false? @parallel-called?)))))) + +(deftest parallel-runner-collects-completions-before-rethrowing-failure + (let [started (atom []) + finished (atom []) + started-latch (java.util.concurrent.CountDownLatch. 2) + release-success (promise) + cases [{:id "global-help"} + {:id "graph-list"}] + error (try + (main/run-selected-cases-in-parallel! + cases + (fn [case _opts] + (swap! started conj (:id case)) + (.countDown started-latch) + (.await started-latch) + (if (= "graph-list" (:id case)) + (do + (swap! finished conj [:failed (:id case)]) + (deliver release-success true) + (throw (ex-info "boom" {:id (:id case)}))) + (do + @release-success + (swap! finished conj [:ok (:id case)]) + {:id (:id case) + :status :ok}))) + (fn [_] + {:exit 0 + :out "" + :err ""}) + {:jobs 2}) + nil + (catch Exception ex + ex))] + (is (instance? Exception error)) + (is (= #{"global-help" "graph-list"} + (set @started))) + (is (= #{[:ok "global-help"] + [:failed "graph-list"]} + (set @finished))))) + +(deftest parallel-runner-elapsed-ms-starts-when-case-begins-running + (let [events (atom [])] + (main/run-selected-cases-in-parallel! + [{:id "slow-1"} + {:id "slow-2"} + {:id "fast-after-queue"}] + (fn [test-case _opts] + (clojure.core/case (:id test-case) + "slow-1" (Thread/sleep 180) + "slow-2" (Thread/sleep 180) + "fast-after-queue" (Thread/sleep 10)) + {:id (:id test-case) + :status :ok}) + (fn [_] + {:exit 0 + :out "" + :err ""}) + {:jobs 2 + :on-case-success (fn [payload] + (swap! events conj [(:id (:result payload)) (:elapsed-ms payload)]))}) + (let [elapsed-map (into {} @events)] + (is (< (get elapsed-map "fast-after-queue" 1000) 120) + "queued case should measure only its own execution time, not time spent waiting in the pool") + (is (>= (get elapsed-map "slow-1" 0) 150)) + (is (>= (get elapsed-map "slow-2" 0) 150))))) + +(deftest select-cases-supports-case-id + (is (= ["graph-create"] + (mapv :id (main/select-cases sample-cases {:case "graph-create"}))))) + +(deftest select-cases-supports-include-tags + (is (= ["global-help" "graph-list"] + (mapv :id (main/select-cases sample-cases {:include ["smoke"]}))))) + +(deftest run-fails-on-missing-coverage-before-case-execution + (let [ran? (atom false)] + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"Missing coverage" + (main/run! {:inventory complete-inventory + :cases [{:id "global-help" + :cmds ["node static/logseq-cli.js --help"] + :covers {:options {:global ["--help"]}}}] + :skip-build true + :run-command (fn [_] + (reset! ran? true) + {:exit 0 + :out "" + :err ""})}))) + (is (false? @ran?)))) + +(deftest run-succeeds-when-coverage-is-complete + (let [result (main/run! {:inventory complete-inventory + :cases [{:id "global-help" + :cmds ["node static/logseq-cli.js --help"] + :covers {:options {:global ["--help"]}}} + {:id "graph-create" + :cmds ["node static/logseq-cli.js graph create --type markdown"] + :covers {:commands ["graph create"] + :options {:graph ["--type"]}}} + {:id "graph-list" + :cmds ["node static/logseq-cli.js graph list --file demo.edn"] + :covers {:commands ["graph list"] + :options {:graph ["--file"]}}}] + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""})})] + (is (= :ok (:status result))) + (is (= 3 (count (:cases result)))))) + +(deftest targeted-run-skips-full-coverage-check + (let [executed (atom [])] + (let [result (main/run! {:inventory complete-inventory + :cases [{:id "global-help" + :cmds ["node static/logseq-cli.js --help"] + :covers {:options {:global ["--help"]}}} + {:id "graph-create" + :cmds ["node static/logseq-cli.js graph create --graph demo"] + :covers {:commands ["graph create"] + :options {:graph ["--type"]}}}] + :case "global-help" + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + (swap! executed conj (:id case)) + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))) + (is (= ["global-help"] @executed))))) + +(deftest run-invokes-case-progress-hooks + (let [events (atom [])] + (main/run! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok}) + :on-case-start (fn [{:keys [index total case]}] + (swap! events conj [:start index total (:id case)])) + :on-case-success (fn [{:keys [index total result]}] + (swap! events conj [:ok index total (:id result)]))}) + (is (= [[:start 1 2 "global-help"] + [:ok 1 2 "global-help"] + [:start 2 2 "graph-list"] + [:ok 2 2 "graph-list"]] + @events)))) + +(deftest run-loads-non-sync-suite-by-default + (let [suite-calls (atom [])] + (with-redefs [manifests/load-inventory (fn [suite] + (swap! suite-calls conj [:inventory suite]) + complete-inventory) + manifests/load-cases (fn [suite] + (swap! suite-calls conj [:cases suite]) + sample-cases)] + (let [result (main/run! {:skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result)))) + (is (= [[:inventory :non-sync] + [:cases :non-sync]] + @suite-calls))))) + +(deftest run-loads-sync-suite-when-explicit + (let [suite-calls (atom []) + sync-inventory {:excluded-command-prefixes ["login" "logout"] + :scopes {:sync {:commands ["sync upload"] + :options []}}} + sync-cases [{:id "sync-upload" + :cmds ["node static/logseq-cli.js sync upload"] + :covers {:commands ["sync upload"]}}]] + (with-redefs [manifests/load-inventory (fn [suite] + (swap! suite-calls conj [:inventory suite]) + sync-inventory) + manifests/load-cases (fn [suite] + (swap! suite-calls conj [:cases suite]) + sync-cases)] + (let [result (main/run! {:suite :sync + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result)))) + (is (= [[:inventory :sync] + [:cases :sync]] + @suite-calls))))) + +(deftest run-sync-suite-uses-suite-fixture-once-with-parallel-runner + (let [before-called (atom 0) + after-called (atom 0) + prepared-case-ids (atom []) + run-case-seen (atom []) + parallel-call (atom nil) + sync-inventory {:excluded-command-prefixes ["login" "logout"] + :scopes {:sync {:commands ["sync upload" "sync status"] + :options []}}} + sync-cases [{:id "sync-upload" + :cmds ["node static/logseq-cli.js sync upload"] + :covers {:commands ["sync upload"]}} + {:id "sync-status" + :cmds ["node static/logseq-cli.js sync status"] + :covers {:commands ["sync status"]}}]] + (with-redefs [sync-fixture/before-suite! (fn [_] + (swap! before-called inc) + {:suite :sync}) + sync-fixture/prepare-case (fn [case _suite-context] + (swap! prepared-case-ids conj (:id case)) + (assoc case :prepared? true)) + sync-fixture/after-suite! (fn [_ _] + (swap! after-called inc)) + main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts] + (reset! parallel-call {:case-ids (mapv :id selected-cases) + :jobs (:jobs opts)}) + (mapv (fn [case] + (run-case case {:run-command run-command})) + selected-cases))] + (let [result (main/run! {:suite :sync + :inventory sync-inventory + :cases sync-cases + :skip-build true + :jobs 2 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + (swap! run-case-seen conj [(:id case) (:prepared? case)]) + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))))) + (is (= 1 @before-called)) + (is (= 1 @after-called)) + (is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call))) + (is (= 2 (:jobs @parallel-call))) + (is (= ["sync-upload" "sync-status"] @prepared-case-ids)) + (is (= [["sync-upload" true] + ["sync-status" true]] + @run-case-seen)))) + +(deftest run-sync-suite-forwards-e2ee-password-to-prepare-case + (let [seen-passwords (atom []) + sync-inventory {:excluded-command-prefixes ["login" "logout"] + :scopes {:sync {:commands ["sync status"] + :options []}}} + sync-cases [{:id "sync-status" + :cmds ["node static/logseq-cli.js sync status"] + :covers {:commands ["sync status"]}}]] + (with-redefs [sync-fixture/before-suite! (fn [_] + {:suite :sync}) + sync-fixture/prepare-case (fn [case suite-context] + (swap! seen-passwords conj (:e2ee-password suite-context)) + case) + sync-fixture/after-suite! (fn [_ _] nil)] + (main/run! {:suite :sync + :inventory sync-inventory + :cases sync-cases + :skip-build true + :e2ee-password "abc 123" + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})) + (is (= ["abc 123"] @seen-passwords)))) + +(deftest list-cases-defaults-to-non-sync + (let [selected-suite (atom nil) + output (with-out-str + (with-redefs [manifests/load-cases (fn [suite] + (reset! selected-suite suite) + [{:id "non-sync-case"}])] + (main/list-cases! {})))] + (is (= :non-sync @selected-suite)) + (is (string/includes? output "non-sync-case")))) + +(deftest list-sync-cases-uses-sync-suite + (let [selected-suite (atom nil) + output (with-out-str + (with-redefs [manifests/load-cases (fn [suite] + (reset! selected-suite suite) + [{:id "sync-case"}])] + (main/list-sync-cases! {})))] + (is (= :sync @selected-suite)) + (is (string/includes? output "sync-case")))) + +(deftest test-prints-progress-and-summary + (let [output (with-out-str + (main/test! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok + :cmd (last (:cmds case))})}))] + (is (string/includes? output "==> Running cli-e2e cases")) + (is (string/includes? output "==> Build preflight: running...")) + (is (string/includes? output "==> Build preflight: skipped (--skip-build)")) + (is (string/includes? output "==> Prepared 2 case(s), starting execution")) + (is (string/includes? output "[1/2] ▶ global-help")) + (is (string/includes? output "[1/2] ✓ global-help")) + (is (string/includes? output "[2/2] ▶ graph-list")) + (is (string/includes? output "[2/2] ✓ graph-list")) + (is (string/includes? output "Summary: 2 passed, 0 failed")))) + +(deftest test-parallel-output-omits-meaningless-index-prefixes + (let [output (with-out-str + (main/test! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :jobs 2 + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + (Thread/sleep (if (= "global-help" (:id case)) 25 5)) + {:id (:id case) + :status :ok})}))] + (is (string/includes? output "▶ global-help")) + (is (string/includes? output "✓ global-help")) + (is (string/includes? output "✓ graph-list")) + (is (not (string/includes? output "[1/2]"))) + (is (not (string/includes? output "[2/2]"))))) + +(deftest test-timings-prints-step-details-and-slow-summary + (let [output (with-out-str + (main/test! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :timings true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + (if (= "global-help" (:id case)) + {:id "global-help" + :status :ok + :timings [{:phase :setup + :step-index 1 + :step-total 1 + :elapsed-ms 12 + :status :ok + :cmd "setup-global"} + {:phase :main + :step-index 1 + :step-total 1 + :elapsed-ms 55 + :status :ok + :cmd "main-global"}]} + {:id "graph-list" + :status :ok + :timings [{:phase :main + :step-index 1 + :step-total 1 + :elapsed-ms 210 + :status :ok + :cmd "main-graph-list"}]}))}))] + (is (string/includes? output "==> Step timing enabled (--timings)")) + (is (string/includes? output "step timings:")) + (is (string/includes? output "Slow steps (top 10):")) + (is (string/includes? output "main-graph-list")))) + +(deftest test-without-timings-keeps-output-concise + (let [output (with-out-str + (main/test! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok + :timings [{:phase :main + :step-index 1 + :step-total 1 + :elapsed-ms 88 + :status :ok + :cmd "hidden-step"}]})}))] + (is (not (string/includes? output "==> Step timing enabled (--timings)"))) + (is (not (string/includes? output "step timings:"))) + (is (not (string/includes? output "Slow steps (top 10):"))))) + +(deftest test-sync-timings-prints-step-details-and-slow-summary + (let [sync-inventory {:excluded-command-prefixes ["login" "logout"] + :scopes {:sync {:commands ["sync status"] + :options []}}} + sync-cases [{:id "sync-status-case" + :cmds ["node static/logseq-cli.js sync status"] + :covers {:commands ["sync status"]}}] + output (with-out-str + (main/test-sync! {:inventory sync-inventory + :cases sync-cases + :skip-build true + :timings true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok + :timings [{:phase :setup + :step-index 1 + :step-total 1 + :elapsed-ms 10 + :status :ok + :cmd "sync-setup"} + {:phase :main + :step-index 1 + :step-total 1 + :elapsed-ms 150 + :status :ok + :cmd "sync-main"} + {:phase :cleanup + :step-index 1 + :step-total 1 + :elapsed-ms 20 + :status :ok + :cmd "sync-cleanup"}]})}))] + (is (string/includes? output "==> Running cli-e2e cases")) + (is (string/includes? output "==> Step timing enabled (--timings)")) + (is (string/includes? output "step timings:")) + (is (string/includes? output "Slow steps (top 10):")) + (is (string/includes? output "sync-main")))) + +(deftest test-help-prints-usage-and-skips-execution + (let [ran? (atom false) + result (atom nil) + output (with-out-str + (reset! result + (main/test! {:help true + :run-command (fn [_] + (reset! ran? true) + {:exit 0 + :out "" + :err ""}) + :run-case (fn [_ _] + (reset! ran? true) + {:id "unexpected" + :status :ok})}))) ] + (is (= :help (:status @result))) + (is (false? @ran?)) + (is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test [options]")) + (is (string/includes? output "--skip-build")) + (is (not (string/includes? output "--force-build"))) + (is (string/includes? output "--include TAG")) + (is (string/includes? output "--case ID")) + (is (string/includes? output "--jobs N")) + (is (string/includes? output "Run up to N non-sync cases in parallel")) + (is (string/includes? output "--skip-build --jobs 4")) + (is (string/includes? output "Default: 4")) + (is (string/includes? output "--timings")) + (is (not (string/includes? output "--e2ee-password"))))) + +(deftest test-sync-help-prints-usage-and-skips-execution + (let [ran? (atom false) + result (atom nil) + output (with-out-str + (reset! result + (main/test-sync! {:help true + :run-command (fn [_] + (reset! ran? true) + {:exit 0 + :out "" + :err ""}) + :run-case (fn [_ _] + (reset! ran? true) + {:id "unexpected" + :status :ok})})))] + (is (= :help (:status @result))) + (is (false? @ran?)) + (is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test-sync [options]")) + (is (string/includes? output "--skip-build")) + (is (not (string/includes? output "--force-build"))) + (is (string/includes? output "--include TAG")) + (is (string/includes? output "--case ID")) + (is (string/includes? output "--jobs N")) + (is (string/includes? output "Run up to N sync cases in parallel")) + (is (string/includes? output "--jobs 4")) + (is (string/includes? output "bb -f cli-e2e/bb.edn test-sync")) + (is (string/includes? output "--case sync-upload-download-mvp")) + (is (not (string/includes? output "--skip-build --case sync-upload-download-mvp"))) + (is (string/includes? output "Default: 4")) + (is (string/includes? output "--timings")) + (is (string/includes? output "--e2ee-password VALUE")) + (is (string/includes? output "Default: 11111")))) + +(deftest run-does-not-pass-force-build-to-preflight + (let [preflight-call (atom nil)] + (with-redefs [logseq.cli.e2e.preflight/run! (fn [opts] + (reset! preflight-call opts) + {:status :ok + :commands [] + :missing-artifacts []})] + (let [result (main/run! {:inventory complete-inventory + :cases sample-cases + :include ["smoke"] + :skip-build false + :force-build true + :run-command (fn [_] + {:exit 0 + :out "" + :err ""}) + :run-case (fn [case _opts] + {:id (:id case) + :status :ok})})] + (is (= :ok (:status result))) + (is (not (contains? @preflight-call :force-build))))))) + +(deftest test-single-case-enables-detailed-command-logging + (let [command-opts (atom nil) + output (with-out-str + (main/test! {:inventory complete-inventory + :cases [{:id "global-help" + :cmds ["node static/logseq-cli.js --help"] + :covers {:options {:global ["--help"]}}}] + :case "global-help" + :skip-build true + :run-command (fn [{:keys [cmd] :as opts}] + (reset! command-opts opts) + {:cmd cmd + :exit 0 + :out "ok" + :err ""}) + :run-case (fn [case {:keys [run-command]}] + (let [result (run-command {:cmd (first (:cmds case)) + :phase :main + :step-index 1 + :step-total 1})] + {:id (:id case) + :status :ok + :cmd (:cmd result)}) )}))] + (is (string/includes? output "==> Detailed case logging enabled (--case global-help)")) + (is (string/includes? output " [main 1/1] $ node static/logseq-cli.js --help")) + (is (true? (:stream-output? @command-opts))))) + +(deftest cleanup-help-prints-usage + (let [output (with-out-str (main/cleanup! {:help true}))] + (is (string/includes? output "Usage: bb -f cli-e2e/bb.edn cleanup")) + (is (string/includes? output "Terminate cli-e2e db-worker-node processes")) + (is (string/includes? output "Terminate db-sync server listeners on port 18080")) + (is (string/includes? output "Remove cli-e2e temp roots")) + (is (string/includes? output "--dry-run")))) + +(deftest cleanup-prints-summary-and-returns-status + (with-redefs [cleanup/cleanup-db-worker-processes! (fn [_] {:found-pids [101 202] + :killed-pids [101] + :failed-pids [202]}) + cleanup/cleanup-db-sync-port-processes! (fn [_] {:found-pids [303] + :killed-pids [303] + :failed-pids []}) + cleanup/cleanup-temp-roots! (fn [_] {:found-dirs ["/tmp/logseq-cli-e2e-a" + "/tmp/logseq-cli-e2e-b"] + :removed-dirs ["/tmp/logseq-cli-e2e-a"] + :failed-dirs ["/tmp/logseq-cli-e2e-b"]})] + (let [result (atom nil) + output (with-out-str + (reset! result (main/cleanup! {})))] + (is (= :ok (:status @result))) + (is (= [101] (get-in @result [:processes :killed-pids]))) + (is (= [303] (get-in @result [:db-sync-port-processes :killed-pids]))) + (is (= ["/tmp/logseq-cli-e2e-a"] (get-in @result [:temp-roots :removed-dirs]))) + (is (string/includes? output "db-worker-node processes: found 2, killed 1, failed 1")) + (is (string/includes? output "db-sync server processes (port 18080): found 1, killed 1, failed 0")) + (is (string/includes? output "temp roots: found 2, removed 1, failed 1"))))) + +(deftest cleanup-dry-run-prints-summary-and-passes-option + (let [process-opts (atom nil) + db-sync-opts (atom nil) + dir-opts (atom nil)] + (with-redefs [cleanup/cleanup-db-worker-processes! (fn [opts] + (reset! process-opts opts) + {:dry-run? true + :found-pids [101 202] + :would-kill-pids [101 202] + :killed-pids [] + :failed-pids []}) + cleanup/cleanup-db-sync-port-processes! (fn [opts] + (reset! db-sync-opts opts) + {:dry-run? true + :found-pids [303] + :would-kill-pids [303] + :killed-pids [] + :failed-pids []}) + cleanup/cleanup-temp-roots! (fn [opts] + (reset! dir-opts opts) + {:dry-run? true + :found-dirs ["/tmp/logseq-cli-e2e-a"] + :would-remove-dirs ["/tmp/logseq-cli-e2e-a"] + :removed-dirs [] + :failed-dirs []})] + (let [result (atom nil) + output (with-out-str + (reset! result (main/cleanup! {:dry-run true})))] + (is (= {:dry-run true} @process-opts)) + (is (= {:dry-run true} @db-sync-opts)) + (is (= {:dry-run true} @dir-opts)) + (is (= :ok (:status @result))) + (is (true? (get-in @result [:processes :dry-run?]))) + (is (true? (get-in @result [:db-sync-port-processes :dry-run?]))) + (is (true? (get-in @result [:temp-roots :dry-run?]))) + (is (string/includes? output "[dry-run] db-worker-node processes: found 2, would kill 2")) + (is (string/includes? output "[dry-run] db-sync server processes (port 18080): found 1, would kill 1")) + (is (string/includes? output "[dry-run] temp roots: found 1, would remove 1")))))) diff --git a/cli-e2e/test/logseq/cli/e2e/manifests_test.clj b/cli-e2e/test/logseq/cli/e2e/manifests_test.clj new file mode 100644 index 0000000000..70f535230b --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/manifests_test.clj @@ -0,0 +1,225 @@ +(ns logseq.cli.e2e.manifests-test + (:require [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.manifests :as manifests])) + +(deftest load-cases-requires-new-map-format + (with-redefs [manifests/read-edn-file (fn [_] + [{:id "legacy-a"} + {:id "legacy-b"}])] + (let [error (try + (manifests/load-cases :non-sync) + nil + (catch clojure.lang.ExceptionInfo error + error))] + (is (some? error)) + (is (re-find #"Invalid cli-e2e manifest format" (.getMessage error))) + (is (= "{:templates {...} :cases [...]}" + (:expected (ex-data error))))))) + +(deftest load-cases-merges-templates-and-cases-by-rules + (with-redefs [manifests/read-edn-file + (fn [_] + {:templates + {:base {:setup ["setup-a"] + :cmds ["cmd-a"] + :cleanup ["cleanup-a"] + :tags [:base] + :vars {:nested {:left 1} + :only-base true} + :covers {:commands ["base-command"] + :options {:global ["--base"]}} + :expect {:stdout-json-paths {[:status] "ok" + [:data :base] 1}} + :graph "base-graph"} + :addon {:setup ["setup-b"] + :cmds ["cmd-b"] + :cleanup ["cleanup-b"] + :tags [:addon] + :vars {:nested {:right 2}} + :covers {:options {:graph ["--addon"]}} + :expect {:stdout-json-paths {[:data :addon] 2}} + :graph "addon-graph"} + :shared {:setup ["setup-shared"] + :cmds ["cmd-shared"] + :cleanup ["cleanup-shared"] + :tags [:shared] + :vars {:nested {:shared 9}} + :graph "shared-graph"}} + :cases + [{:id "single-parent" + :extends :base + :cmds ["cmd-case"] + :expect {:stdout-json-paths {[:data :case] 3}} + :graph "single-parent-graph"} + {:id "multi-parent" + :extends [:base :addon] + :setup ["setup-case"] + :cmds ["cmd-case"] + :cleanup ["cleanup-case"] + :tags [:case] + :vars {:nested {:leaf 3} + :only-case true} + :covers {:commands ["case-command"] + :options {:graph ["--case"]}} + :expect {:stdout-json-paths {[:data :case] 3}} + :graph "case-graph"} + {:id "negative-case" + :extends :shared + :expect {:exit 1 + :stdout-json-paths {[:error :code] "boom"}} + :graph "negative-graph"}]})] + (let [[single-parent multi-parent negative-case] (manifests/load-cases :sync)] + (testing "supports :extends keyword" + (is (= "single-parent" (:id single-parent))) + (is (= ["cmd-a" "cmd-case"] (:cmds single-parent))) + (is (= "single-parent-graph" (:graph single-parent)))) + (testing "append merge keys" + (is (= ["setup-a" "setup-b" "setup-case"] (:setup multi-parent))) + (is (= ["cmd-a" "cmd-b" "cmd-case"] (:cmds multi-parent))) + (is (= ["cleanup-a" "cleanup-b" "cleanup-case"] (:cleanup multi-parent))) + (is (= [:base :addon :case] (:tags multi-parent)))) + (testing "deep merge keys" + (is (= {:nested {:left 1 :right 2 :leaf 3} + :only-base true + :only-case true} + (:vars multi-parent))) + (is (= {:commands ["case-command"] + :options {:global ["--base"] + :graph ["--case"]}} + (:covers multi-parent))) + (is (= {[:status] "ok" + [:data :base] 1 + [:data :addon] 2 + [:data :case] 3} + (get-in multi-parent [:expect :stdout-json-paths])))) + (testing "child expect overrides inherited maps when parent omits them" + (is (= {:exit 1 + :stdout-json-paths {[:error :code] "boom"}} + (:expect negative-case)))) + (testing "base templates can omit happy-path defaults" + (is (= ["setup-shared"] (:setup negative-case))) + (is (= ["cmd-shared"] (:cmds negative-case))) + (is (= ["cleanup-shared"] (:cleanup negative-case)))) + (testing "scalar keys are overridden by child" + (is (= "case-graph" (:graph multi-parent))))))) + +(deftest load-cases-detects-circular-template-inheritance-with-cycle-path + (with-redefs [manifests/read-edn-file (fn [_] + {:templates + {:a {:extends :b + :setup ["a"]} + :b {:extends :a + :setup ["b"]}} + :cases [{:id "cycle" :extends :a}]})] + (let [error (try + (manifests/load-cases :sync) + nil + (catch clojure.lang.ExceptionInfo error + error))] + (is (some? error)) + (is (re-find #"Circular template inheritance" (.getMessage error))) + (is (= [:a :b :a] (:cycle (ex-data error))))))) + +(deftest load-cases-validates-extends-entries + (with-redefs [manifests/read-edn-file (fn [_] + {:templates {:base {:cmds ["base"]}} + :cases [{:id "invalid-extends" + :extends [:base "bad"] + :cmds ["case"]}]})] + (let [error (try + (manifests/load-cases :non-sync) + nil + (catch clojure.lang.ExceptionInfo error + error))] + (is (some? error)) + (is (re-find #"Invalid :extends entries" (.getMessage error))) + (is (= ["bad"] (:invalid-entries (ex-data error))))))) + +(deftest load-cases-lint-detects-invalid-extends-references + (with-redefs [manifests/read-edn-file (fn [_] + {:templates {:base {:cmds ["base"]} + :unused {:extends :missing-template + :cmds ["unused"]}} + :cases [{:id "valid" :extends :base :cmds ["case"]} + {:id "invalid" :extends :also-missing :cmds ["case"]}]})] + (let [error (try + (manifests/load-cases :sync) + nil + (catch clojure.lang.ExceptionInfo error + error)) + issues (:issues (ex-data error))] + (is (some? error)) + (is (re-find #"manifest lint failed" (.getMessage error))) + (is (= #{:also-missing :missing-template} + (set (map :target (filter #(= :invalid-extends (:type %)) issues)))))))) + +(deftest load-cases-lint-detects-duplicate-case-ids + (with-redefs [manifests/read-edn-file (fn [_] + {:templates {:base {:cmds ["base"]}} + :cases [{:id "dup" :extends :base :cmds ["case-a"]} + {:id "dup" :extends :base :cmds ["case-b"]}]})] + (let [error (try + (manifests/load-cases :non-sync) + nil + (catch clojure.lang.ExceptionInfo error + error)) + duplicate-issues (filter #(= :duplicate-case-id (:type %)) + (:issues (ex-data error)))] + (is (some? error)) + (is (= [{:type :duplicate-case-id :id "dup" :count 2}] + (vec duplicate-issues)))))) + +(deftest load-cases-lint-detects-unused-templates + (with-redefs [manifests/read-edn-file (fn [_] + {:templates {:base {:cmds ["base"]} + :unused {:cmds ["unused"]}} + :cases [{:id "only" :extends :base :cmds ["case"]}]})] + (let [error (try + (manifests/load-cases :sync) + nil + (catch clojure.lang.ExceptionInfo error + error)) + unused-issues (filter #(= :unused-template (:type %)) + (:issues (ex-data error)))] + (is (some? error)) + (is (= [{:type :unused-template :template :unused}] + (vec unused-issues)))))) + +(deftest sync-multi-batch-operations-uses-state-driven-waits-instead-of-fixed-sleeps + (let [cases (manifests/load-cases :sync) + multi-batch (some #(when (= "sync-multi-batch-operations" (:id %)) %) cases) + commands (:cmds multi-batch) + upload-commands (filter #(re-find #"sync upload --graph" %) commands) + wait-commands (filter #(re-find #"wait_sync_status\.py" %) commands)] + (is (some? multi-batch)) + (is (not-any? #(= "sleep 1" %) commands)) + (is (= 1 (count upload-commands))) + (is (= 5 (count wait-commands))) + (is (= 3 (count (filter #(re-find #"--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) wait-commands)))))) + +(deftest sync-manifest-includes-duplicate-upload-negative-case + (let [cases (manifests/load-cases :sync) + duplicate-upload (some #(when (= "sync-upload-rejects-duplicate-remote-graph" (:id %)) %) cases) + json-paths (get-in duplicate-upload [:expect :stdout-json-paths]) + setup-commands (:setup duplicate-upload) + main-commands (:cmds duplicate-upload)] + (is (some? duplicate-upload)) + (is (= 1 (get-in duplicate-upload [:expect :exit]))) + (is (= "graph-already-exists" + (get-in duplicate-upload [:expect :stdout-json-paths [:error :code]]))) + (is (= ["delete it before uploading again"] + (get-in duplicate-upload [:expect :stdout-contains]))) + (is (= 1 (count (filter #(re-find #"sync upload --graph" %) setup-commands)))) + (is (= 1 (count (filter #(re-find #"sync upload --graph" %) main-commands)))) + (is (false? (contains? json-paths [:data :pending-local]))) + (is (false? (contains? json-paths [:data :pending-asset]))) + (is (false? (contains? json-paths [:data :pending-server]))) + (is (false? (contains? json-paths [:data :last-error]))))) + +(deftest sync-status-steady-state-does-not-repeat-identical-b-side-steady-state-waits + (let [cases (manifests/load-cases :sync) + steady-state (some #(when (= "sync-status-steady-state" (:id %)) %) cases) + steady-waits (filter #(re-find #"wait_sync_status\.py.+--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) (:cmds steady-state))] + (is (some? steady-state)) + (is (= 1 (count steady-waits))) + (is (= 1 (count (filter #(re-find #"sync status --graph" %) (:cmds steady-state))))))) diff --git a/cli-e2e/test/logseq/cli/e2e/paths_test.clj b/cli-e2e/test/logseq/cli/e2e/paths_test.clj new file mode 100644 index 0000000000..0e07287a74 --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/paths_test.clj @@ -0,0 +1,9 @@ +(ns logseq.cli.e2e.paths-test + (:require [clojure.test :refer [deftest is]] + [logseq.cli.e2e.paths :as paths])) + +(deftest repo-root-does-not-depend-on-runtime-file-binding + (let [expected (paths/repo-root)] + (binding [*file* nil] + (is (= expected + (paths/repo-root)))))) diff --git a/cli-e2e/test/logseq/cli/e2e/preflight_test.clj b/cli-e2e/test/logseq/cli/e2e/preflight_test.clj new file mode 100644 index 0000000000..aa647b0d29 --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/preflight_test.clj @@ -0,0 +1,95 @@ +(ns logseq.cli.e2e.preflight-test + (:require [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.preflight :as preflight])) + +(def required-artifacts + ["/repo/static/logseq-cli.js" + "/repo/static/db-worker-node.js" + "/repo/dist/db-worker-node.js" + "/repo/dist/db-worker-node-assets.json" + "/repo/deps/db-sync/worker/dist/node-adapter.js"]) + +(defn- with-required-artifacts + [f] + (with-redefs [logseq.cli.e2e.paths/repo-root (constantly "/repo") + logseq.cli.e2e.paths/required-artifacts (fn [] required-artifacts)] + (f))) + +(deftest build-plan-matches-required-commands + (is (= ["clojure -M:cljs compile logseq-cli" + "pnpm db-worker-node:compile:bundle" + "pnpm --dir deps/db-sync build:node-adapter"] + (mapv :cmd preflight/build-plan)))) + +(deftest missing-artifacts-returns-unreadable-paths + (let [artifacts ["/repo/static/logseq-cli.js" + "/repo/static/db-worker-node.js" + "/repo/dist/db-worker-node.js"] + present? #{"/repo/static/logseq-cli.js"}] + (is (= ["/repo/static/db-worker-node.js" + "/repo/dist/db-worker-node.js"] + (preflight/missing-artifacts artifacts present?))))) + +(deftest skip-build-avoids-running-shell-commands + (let [called? (atom false) + result (preflight/run! {:skip-build true + :run-command (fn [_] + (reset! called? true)) + :file-exists? (constantly false)})] + (is (= :skipped (:status result))) + (is (false? @called?)))) + +(deftest build-runs-commands-even-when-artifacts-are-ready + (let [calls (atom [])] + (with-required-artifacts + #(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}] + (swap! calls conj cmd) + {:cmd cmd + :exit 0 + :out "" + :err ""}) + :file-exists? (set required-artifacts)})] + (is (= :ok (:status result))) + (is (= ["clojure -M:cljs compile logseq-cli" + "pnpm db-worker-node:compile:bundle" + "pnpm --dir deps/db-sync build:node-adapter"] + @calls)))))) + +(deftest build-runs-commands-when-artifacts-are-partially-present + (let [calls (atom []) + existing (atom (disj (set required-artifacts) + "/repo/dist/db-worker-node-assets.json"))] + (with-required-artifacts + #(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}] + (swap! calls conj cmd) + (reset! existing (set required-artifacts)) + {:cmd cmd + :exit 0 + :out "" + :err ""}) + :file-exists? (fn [path] + (contains? @existing path))})] + (is (= :ok (:status result))) + (is (= ["clojure -M:cljs compile logseq-cli" + "pnpm db-worker-node:compile:bundle" + "pnpm --dir deps/db-sync build:node-adapter"] + @calls)))))) + +(deftest build-runs-commands-when-artifacts-are-absent + (let [calls (atom []) + existing (atom #{})] + (with-required-artifacts + #(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}] + (swap! calls conj cmd) + (reset! existing (set required-artifacts)) + {:cmd cmd + :exit 0 + :out "" + :err ""}) + :file-exists? (fn [path] + (contains? @existing path))})] + (is (= :ok (:status result))) + (is (= ["clojure -M:cljs compile logseq-cli" + "pnpm db-worker-node:compile:bundle" + "pnpm --dir deps/db-sync build:node-adapter"] + @calls)))))) diff --git a/cli-e2e/test/logseq/cli/e2e/random_bidirectional_block_ops_test.py b/cli-e2e/test/logseq/cli/e2e/random_bidirectional_block_ops_test.py new file mode 100644 index 0000000000..ec457c6948 --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/random_bidirectional_block_ops_test.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +import importlib.util +from pathlib import Path +import sys + + +MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "random_bidirectional_block_ops.py" +spec = importlib.util.spec_from_file_location("random_bidirectional_block_ops", MODULE_PATH) +random_bidirectional_block_ops = importlib.util.module_from_spec(spec) +assert spec.loader is not None +sys.modules[spec.name] = random_bidirectional_block_ops +spec.loader.exec_module(random_bidirectional_block_ops) + + +def test_default_profile_is_faster_than_high_stress(monkeypatch) -> None: + monkeypatch.setattr( + random_bidirectional_block_ops.sys, + "argv", + [ + "random_bidirectional_block_ops.py", + "--cli", + "/tmp/logseq-cli.js", + "--graph", + "demo", + "--config-a", + "/tmp/a.edn", + "--root-dir-a", + "/tmp/a", + "--config-b", + "/tmp/b.edn", + "--root-dir-b", + "/tmp/b", + "--page", + "Home", + ], + ) + default_args = random_bidirectional_block_ops.parse_args() + + monkeypatch.setattr( + random_bidirectional_block_ops.sys, + "argv", + [ + "random_bidirectional_block_ops.py", + "--cli", + "/tmp/logseq-cli.js", + "--graph", + "demo", + "--config-a", + "/tmp/a.edn", + "--root-dir-a", + "/tmp/a", + "--config-b", + "/tmp/b.edn", + "--root-dir-b", + "/tmp/b", + "--page", + "Home", + "--profile", + "high-stress", + ], + ) + high_stress_args = random_bidirectional_block_ops.parse_args() + + assert default_args.root_dir_a == "/tmp/a" + assert default_args.root_dir_b == "/tmp/b" + assert default_args.profile == "default" + assert high_stress_args.profile == "high-stress" + assert default_args.rounds_per_client < high_stress_args.rounds_per_client + + +def test_explicit_rounds_override_profile_default(monkeypatch) -> None: + monkeypatch.setattr( + random_bidirectional_block_ops.sys, + "argv", + [ + "random_bidirectional_block_ops.py", + "--cli", + "/tmp/logseq-cli.js", + "--graph", + "demo", + "--config-a", + "/tmp/a.edn", + "--root-dir-a", + "/tmp/a", + "--config-b", + "/tmp/b.edn", + "--root-dir-b", + "/tmp/b", + "--page", + "Home", + "--profile", + "high-stress", + "--rounds-per-client", + "12", + ], + ) + + args = random_bidirectional_block_ops.parse_args() + + assert args.profile == "high-stress" + assert args.rounds_per_client == 12 diff --git a/cli-e2e/test/logseq/cli/e2e/runner_test.clj b/cli-e2e/test/logseq/cli/e2e/runner_test.clj new file mode 100644 index 0000000000..a2a1ae1570 --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/runner_test.clj @@ -0,0 +1,152 @@ +(ns logseq.cli.e2e.runner-test + (:require [clojure.test :refer [deftest is]] + [logseq.cli.e2e.runner :as runner])) + +(deftest render-case-expands-template-values-recursively + (let [rendered (runner/render-case + {:id "graph-create" + :setup ["{{cli}} --graph {{graph-arg}}"] + :cmds ["{{cli}} graph info --graph {{graph-arg}}"] + :expect {:stdout-json-paths {[:data :graph] "{{graph}}" + [:status] "ok"}}} + {:cli "node /tmp/logseq-cli.js" + :graph "demo" + :graph-arg "'demo'"})] + (is (= ["node /tmp/logseq-cli.js --graph 'demo'"] (:setup rendered))) + (is (= ["node /tmp/logseq-cli.js graph info --graph 'demo'"] (:cmds rendered))) + (is (= "demo" (get-in rendered [:expect :stdout-json-paths [:data :graph]]))))) + +(deftest run-case-executes-setup-before-main-command + (let [calls (atom []) + result (runner/run-case! + {:id "graph-info" + :setup ["setup one" "setup two"] + :cmds ["main command one" "main command two"] + :expect {:exit 0}} + {:context {} + :run-command (fn [{:keys [cmd]}] + (swap! calls conj cmd) + {:cmd cmd + :exit 0 + :out "" + :err ""})})] + (is (= ["setup one" "setup two" "main command one" "main command two"] @calls)) + (is (= "graph-info" (:id result))) + (is (= "main command two" (get-in result [:result :cmd]))))) + +(deftest run-case-includes-command-phase-metadata-when-detailed + (let [calls (atom [])] + (runner/run-case! + {:id "graph-info" + :setup ["setup one" "setup two"] + :cmds ["main command one" "main command two"] + :cleanup ["cleanup one"] + :expect {:exit 0}} + {:context {} + :detailed-log? true + :run-command (fn [{:keys [cmd phase step-index step-total case-id throw?] :as opts}] + (swap! calls conj (select-keys opts [:cmd :phase :step-index :step-total :case-id :throw?])) + {:cmd cmd + :exit 0 + :out "" + :err ""})}) + (is (= [{:cmd "setup one" :phase :setup :step-index 1 :step-total 2 :case-id "graph-info" :throw? true} + {:cmd "setup two" :phase :setup :step-index 2 :step-total 2 :case-id "graph-info" :throw? true} + {:cmd "main command one" :phase :main :step-index 1 :step-total 2 :case-id "graph-info" :throw? true} + {:cmd "main command two" :phase :main :step-index 2 :step-total 2 :case-id "graph-info" :throw? false} + {:cmd "cleanup one" :phase :cleanup :step-index 1 :step-total 1 :case-id "graph-info" :throw? false}] + @calls)))) + +(deftest run-case-collects-step-timings-when-enabled + (let [result (runner/run-case! + {:id "graph-info" + :setup ["setup one"] + :cmds ["main command"] + :cleanup ["cleanup one"] + :expect {:exit 0}} + {:context {} + :timings? true + :run-command (fn [{:keys [cmd]}] + {:cmd cmd + :exit 0 + :out "" + :err ""})})] + (is (= 3 (count (:timings result)))) + (is (= [:setup :main :cleanup] + (mapv :phase (:timings result)))))) + +(deftest run-case-attaches-timings-to-error-when-enabled + (let [error (try + (runner/run-case! + {:id "graph-info" + :setup ["setup one"] + :cmds ["main command"] + :cleanup ["cleanup one"] + :expect {:exit 0}} + {:context {} + :timings? true + :run-command (fn [{:keys [cmd]}] + (when (= cmd "main command") + (throw (ex-info "boom" {:cmd cmd}))) + {:cmd cmd + :exit 0 + :out "" + :err ""})}) + nil + (catch clojure.lang.ExceptionInfo error + error)) + timings (:timings (ex-data error))] + (is (= "graph-info" (:case-id (ex-data error)))) + (is (= [:setup :main :cleanup] + (mapv :phase timings))) + (is (= :failed (:status (second timings)))))) + +(deftest run-case-validates-json-paths-and-nonzero-exit + (let [result (runner/run-case! + {:id "invalid-shell" + :cmds ["node static/logseq-cli.js completion fish"] + :expect {:exit 1 + :stdout-json-paths {[:status] "error" + [:error :code] "invalid-options"}}} + {:context {} + :run-command (fn [{:keys [cmd]}] + {:cmd cmd + :exit 1 + :out "{\"status\":\"error\",\"error\":{\"code\":\"invalid-options\"}}" + :err ""})})] + (is (= 1 (get-in result [:result :exit]))))) + +(deftest assert-result-validates-edn-paths + (is (nil? (runner/assert-result! + {:id "graph-list-edn" + :expect {:exit 0 + :stdout-edn-paths {[:status] :ok + [:data :graphs] ["demo"]}}} + {:cmd "node cli.js graph list" + :exit 0 + :out "{:status :ok, :data {:graphs [\"demo\"]}}" + :err ""})))) + +(deftest assert-result-validates-stdout-not-contains + (is (thrown-with-msg? + clojure.lang.ExceptionInfo + #"stdout contained forbidden text" + (runner/assert-result! + {:id "remove-block" + :expect {:exit 0 + :stdout-not-contains ["Alpha block"]}} + {:cmd "node cli.js show --page Home" + :exit 0 + :out "Home\n- Alpha block" + :err ""})))) + +(deftest assert-result-normalizes-json-escaped-windows-paths-for-contains + (is (nil? (runner/assert-result! + {:id "doctor-dev-script-json" + :expect {:exit 0 + :stdout-contains ["static/db-worker-node.js" + "C:\\Users\\demo\\tmp\\graph-export.edn"]}} + {:cmd "node cli.js doctor --dev-script" + :exit 0 + :out "{\"status\":\"ok\",\"data\":{\"path\":\"C:\\\\Users\\\\demo\\\\tmp\\\\graph-export.edn\",\"message\":\"Found readable file: C:\\\\Users\\\\demo\\\\project\\\\static\\\\db-worker-node.js\"}}" + :err ""})))) diff --git a/cli-e2e/test/logseq/cli/e2e/shell_test.clj b/cli-e2e/test/logseq/cli/e2e/shell_test.clj new file mode 100644 index 0000000000..9d696d1dc6 --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/shell_test.clj @@ -0,0 +1,54 @@ +(ns logseq.cli.e2e.shell-test + (:require [babashka.process :as process] + [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.shell :as shell])) + +(deftest preserves-the-exact-command-string + (let [captured (atom nil) + command "node static/logseq-cli.js --graph 'Graph With Space' --help" + result (shell/run! {:cmd command + :dir "/repo" + :executor (fn [cmd opts] + (reset! captured {:cmd cmd + :opts opts}) + {:exit 0 + :out "ok" + :err ""})})] + (is (= command (:cmd result))) + (is (= command (:cmd @captured))) + (is (= "/repo" (get-in @captured [:opts :dir]))))) + +(deftest shell-failures-include-command-context + (let [command "node static/logseq-cli.js completion fish"] + (try + (shell/run! {:cmd command + :executor (fn [_ _] + {:exit 2 + :out "" + :err "unsupported shell"})}) + (is false "Expected shell/run! to throw") + (catch clojure.lang.ExceptionInfo ex + (is (= command (:cmd (ex-data ex)))) + (is (= 2 (:exit (ex-data ex)))) + (is (= "unsupported shell" (:err (ex-data ex)))))))) + +(deftest allow-failure-returns-result-without-throwing + (let [command "node static/logseq-cli.js graph info --graph missing" + result (shell/run! {:cmd command + :throw? false + :executor (fn [_ _] + {:exit 1 + :out "failed" + :err "missing"})})] + (is (= command (:cmd result))) + (is (= 1 (:exit result))) + (is (= "failed" (:out result))) + (is (= "missing" (:err result))))) + +(deftest default-executor-uses-resolvable-shell-command + (let [captured (atom nil)] + (with-redefs [process/process (fn [_opts & args] + (reset! captured args) + (future {:exit 0}))] + (shell/run! {:cmd "echo ok"})) + (is (= ["bash" "-c" "echo ok"] @captured)))) diff --git a/cli-e2e/test/logseq/cli/e2e/sync_fixture_test.clj b/cli-e2e/test/logseq/cli/e2e/sync_fixture_test.clj new file mode 100644 index 0000000000..dd7a84ab6e --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/sync_fixture_test.clj @@ -0,0 +1,114 @@ +(ns logseq.cli.e2e.sync-fixture-test + (:require [clojure.string :as string] + [clojure.test :refer [deftest is testing]] + [logseq.cli.e2e.sync-fixture :as sync-fixture])) + +(deftest prepare-case-injects-case-local-sync-resources + (let [suite-context {:sync-port "18080" + :sync-http-base "http://127.0.0.1:18080" + :sync-ws-url "ws://127.0.0.1:18080/sync/%s"} + input-case {:id "sync-case" + :vars {:existing true} + :setup ["mkdir -p '{{tmp-dir}}/graphs-b'" + "mkdir -p '{{tmp-dir}}/home/logseq'" + "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + "python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' start --port 18080" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"] + :cleanup ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}" + "python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' stop --pid-file '{{tmp-dir}}/db-sync-server.pid'"]} + prepared (sync-fixture/prepare-case input-case suite-context)] + (is (= "sync-case" (:id prepared))) + (is (= ["mkdir -p '{{tmp-dir}}/graphs-b'" + "mkdir -p '{{tmp-dir}}/home/logseq'" + "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"] + (:setup prepared))) + (is (= ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"] + (:cleanup prepared))) + (is (= "http://127.0.0.1:18080" (get-in prepared [:vars :sync-http-base]))) + (is (= "ws://127.0.0.1:18080/sync/%s" (get-in prepared [:vars :sync-ws-url]))) + (is (= "18080" (get-in prepared [:vars :sync-port]))) + (is (= "11111" (get-in prepared [:vars :e2ee-password]))) + (is (= "11111" (get-in prepared [:vars :e2ee-password-arg]))) + (is (not (contains? (:vars prepared) :suite-auth-path))) + (is (not (contains? (:vars prepared) :suite-config-path))) + (is (not-any? #(string/includes? % "suite-auth-path") (:setup prepared))) + (is (not-any? #(string/includes? % "suite-config-path") (:setup prepared))) + (is (not-any? #(string/includes? % "db_sync_server.py' start") (:setup prepared))) + (is (not-any? #(string/includes? % "db_sync_server.py' stop") (:cleanup prepared))))) + +(deftest prepare-case-places-case-local-auth-before-cli-sync-commands + (let [suite-context {:suite-tmp-dir "/tmp/sync-suite" + :sync-port "18080" + :sync-http-base "http://127.0.0.1:18080" + :sync-ws-url "ws://127.0.0.1:18080/sync/%s"} + prepared (sync-fixture/prepare-case + {:id "sync-case" + :setup ["mkdir -p '{{tmp-dir}}/graphs-b'" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"] + :cleanup []} + suite-context) + setup (:setup prepared) + bootstrap-cmd (nth setup 5)] + (is (= "mkdir -p '{{tmp-dir}}/graphs-b'" (first setup))) + (is (= "mkdir -p '{{tmp-dir}}/home/logseq'" (second setup))) + (is (= "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" (nth setup 2))) + (is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + (nth setup 3))) + (is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'" + (nth setup 4))) + (is (string/includes? bootstrap-cmd "sync ensure-keys")) + (is (string/includes? bootstrap-cmd "--upload-keys")) + (is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.lock")) + (is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.ready")) + (is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null" + (nth setup 6))) + (is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}" + (nth setup 7))) + (is (not-any? #(and (string/includes? % "sync ensure-keys") + (not= % bootstrap-cmd) + (string/includes? % "--upload-keys")) + setup)))) + +(deftest prepare-case-accepts-custom-e2ee-password + (let [suite-context {:sync-port "18080" + :sync-http-base "http://127.0.0.1:18080" + :sync-ws-url "ws://127.0.0.1:18080/sync/%s" + :e2ee-password "pass word"} + prepared (sync-fixture/prepare-case {:id "sync-case" + :setup [] + :cleanup []} + suite-context)] + (is (= "pass word" (get-in prepared [:vars :e2ee-password]))) + (is (= "'pass word'" (get-in prepared [:vars :e2ee-password-arg]))))) + +(deftest before-and-after-suite-only-manage-shared-sync-server-lifecycle + (let [calls (atom []) + run-command (fn [opts] + (swap! calls conj opts) + {:exit 0 + :out "" + :err ""}) + suite-context (sync-fixture/before-suite! {:run-command run-command})] + (testing "before-suite only starts the shared server" + (is (= 1 (count @calls))) + (is (string/includes? (:cmd (first @calls)) "db_sync_server.py")) + (is (string/includes? (:cmd (first @calls)) " start ")) + (is (string/includes? (:cmd (first @calls)) "--data-dir")) + (is (not (string/includes? (:cmd (first @calls)) "--root-dir"))) + (is (string/includes? (:cmd (first @calls)) "--port 18080")) + (is (string/includes? (:cmd (first @calls)) "--startup-timeout-s 60")) + (is (not (string/includes? (:cmd (first @calls)) "prepare_sync_config.py"))) + (is (not (contains? suite-context :suite-auth-path))) + (is (not (contains? suite-context :suite-config-path)))) + (sync-fixture/after-suite! suite-context {:run-command run-command}) + (testing "after-suite only stops the shared server once" + (is (= 2 (count @calls))) + (is (string/includes? (:cmd (last @calls)) "db_sync_server.py")) + (is (string/includes? (:cmd (last @calls)) " stop ")) + (is (false? (:throw? (last @calls))))))) diff --git a/cli-e2e/test/logseq/cli/e2e/wait_sync_status_test.py b/cli-e2e/test/logseq/cli/e2e/wait_sync_status_test.py new file mode 100644 index 0000000000..ba9df4b78d --- /dev/null +++ b/cli-e2e/test/logseq/cli/e2e/wait_sync_status_test.py @@ -0,0 +1,55 @@ +from __future__ import annotations + +import importlib.util +from pathlib import Path +from types import SimpleNamespace + + +MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "wait_sync_status.py" +spec = importlib.util.spec_from_file_location("wait_sync_status", MODULE_PATH) +wait_sync_status = importlib.util.module_from_spec(spec) +assert spec.loader is not None +spec.loader.exec_module(wait_sync_status) + + +def test_resolved_status_command_is_built_once_and_reused() -> None: + args = SimpleNamespace( + cli="~/repo/static/logseq-cli.js", + root_dir="~/tmp/root", + config="~/tmp/cli.edn", + graph="demo", + ) + + command = wait_sync_status.status_command(args) + + assert command[0] == "node" + assert command[2] == "--root-dir" + assert command[-2:] == ["--graph", "demo"] + assert Path(command[1]).is_absolute() + assert Path(command[3]).is_absolute() + assert Path(command[5]).is_absolute() + assert wait_sync_status.status_command(args) is command + + +def test_run_status_uses_precomputed_command() -> None: + command = ["node", "/abs/cli.js", "--graph", "demo"] + args = SimpleNamespace(status_command=command) + calls = [] + + def fake_run(cmd, capture_output, text): + calls.append(cmd) + return SimpleNamespace( + returncode=0, + stdout='{"status":"ok","data":{"pending-local":0,"pending-asset":0,"pending-server":0}}', + stderr="", + ) + + original_run = wait_sync_status.subprocess.run + wait_sync_status.subprocess.run = fake_run + try: + payload = wait_sync_status.run_status(args) + finally: + wait_sync_status.subprocess.run = original_run + + assert payload["status"] == "ok" + assert calls == [command] diff --git a/deps.edn b/deps.edn index a75ca1655f..25544fddde 100644 --- a/deps.edn +++ b/deps.edn @@ -24,6 +24,8 @@ cljs-http/cljs-http {:mvn/version "0.1.49"} org.babashka/sci {:mvn/version "0.12.51"} org.clj-commons/hickory {:mvn/version "0.7.7"} + org.clj-commons/humanize {:mvn/version "1.2"} + org.babashka/cli {:mvn/version "0.8.67"} hiccups/hiccups {:mvn/version "0.3.0"} tongue/tongue {:mvn/version "0.4.4"} org.clojure/core.async {:mvn/version "1.8.741"} diff --git a/deps/cli/src/logseq/cli.cljs b/deps/cli/src/logseq/cli.cljs index cfcd4557fc..d53bc0bc0a 100644 --- a/deps/cli/src/logseq/cli.cljs +++ b/deps/cli/src/logseq/cli.cljs @@ -2,6 +2,7 @@ "Main ns for Logseq CLI" (:require ["fs" :as fs] ["path" :as node-path] + ["child_process" :as child-process] [babashka.cli :as cli] [clojure.string :as string] [logseq.cli.common.graph :as cli-common-graph] @@ -58,6 +59,29 @@ (print-command-help "help" cmd-map) (println "Command" (pr-str command) "does not exist")))) +(defn- extract-graph-names + "Extract graph names from parsed opts for unlock-graph." + [{:keys [graph graphs]}] + (cond-> [] + (string? graph) (conj graph) + (sequential? graphs) (into graphs))) + +(defn- unlock-graphs! + "Stop the db-worker-node server(s) for graph-names so this CLI can open the + SQLite database directly" + [graph-names] + (doseq [graph graph-names] + (let [result (.spawnSync child-process + "logseq" + #js ["server" "stop" "-g" graph] + #js {:timeout 5000 + :stdio "pipe"})] + (if (= 0 (.-status result)) + (println "Unlocked graph:" graph) + ;; Ignore server not started + (when-not (string/includes? (str (.-stdout result)) "server-not-found") + (println "Failed to unlock graph:" (pr-str result))))))) + (defn- lazy-load-fn "Lazy load fn to speed up start time. After nbb requires ~30 namespaces, start time gets close to 1s. Also handles --help on all commands" @@ -65,13 +89,17 @@ (fn [& args] (if (get-in (first args) [:opts :help]) (help-command {:opts {:command (-> args first :dispatch first)}}) - (-> (p/let [_ (require (symbol (namespace fn-sym)))] - (apply (resolve fn-sym) args)) - (p/catch (fn [err] - (if (= :sci/error (:type (ex-data err))) - (nbb.error/print-error-report err) - (js/console.error "Error:" err)) - (js/process.exit 1))))))) + (do + (when (get-in (first args) [:opts :unlock-graph]) + (let [graphs (extract-graph-names (get (first args) :opts))] + (unlock-graphs! graphs))) + (-> (p/let [_ (require (symbol (namespace fn-sym)))] + (apply (resolve fn-sym) args)) + (p/catch (fn [err] + (if (= :sci/error (:type (ex-data err))) + (nbb.error/print-error-report err) + (js/console.error "Error:" err)) + (js/process.exit 1)))))))) (def ^:private table* [{:cmds ["list"] :desc "List local graphs" @@ -122,10 +150,14 @@ :spec default-spec :fn default-command}]) + ;; Spec shared with all commands (def ^:private shared-spec {:help {:alias :h - :desc "Print help"}}) + :desc "Print help"} + :unlock-graph {:alias :u + :coerce :boolean + :desc "Unlock graph(s) before running command"}}) (def ^:private table (mapv (fn [m] (update m :spec #(merge % shared-spec))) table*)) diff --git a/deps/cli/src/logseq/cli/commands/graph.cljs b/deps/cli/src/logseq/cli/commands/graph.cljs index 73e5a0b00c..eb764d5525 100644 --- a/deps/cli/src/logseq/cli/commands/graph.cljs +++ b/deps/cli/src/logseq/cli/commands/graph.cljs @@ -38,6 +38,6 @@ (defn list-graphs [] (let [db-graphs (->> (cli-common-graph/get-db-based-graphs) - (map #(string/replace-first % common-config/db-version-prefix "")) + (map common-config/strip-leading-db-version-prefix) sort)] - (println (string/join "\n" db-graphs)))) \ No newline at end of file + (println (string/join "\n" db-graphs)))) diff --git a/deps/cli/src/logseq/cli/common/graph.cljs b/deps/cli/src/logseq/cli/common/graph.cljs index 87644110f2..d6c15ec8a2 100644 --- a/deps/cli/src/logseq/cli/common/graph.cljs +++ b/deps/cli/src/logseq/cli/common/graph.cljs @@ -1,23 +1,19 @@ (ns ^:node-only logseq.cli.common.graph "Graph related fns shared between CLI and electron" (:require ["fs-extra" :as fs-extra] - ["os" :as os] - ["path" :as node-path] [clojure.string :as string] [logseq.common.config :as common-config] - [logseq.common.graph :as common-graph])) + [logseq.common.graph :as common-graph] + [logseq.common.graph-dir :as graph-dir])) (defn ^:api graph-name->path [graph-name] - (when graph-name - (-> graph-name - (string/replace "+3A+" ":") - (string/replace "++" "/")))) + (graph-dir/decode-graph-dir-name graph-name)) (defn get-db-graphs-dir "Directory where DB graphs are stored" [] - (node-path/join (os/homedir) "logseq" "graphs")) + (common-graph/expand-home (common-graph/get-default-graphs-dir))) (defn get-db-based-graphs [] @@ -27,5 +23,7 @@ (remove (fn [s] (= s common-config/unlinked-graphs-dir))) (map graph-name->path) (keep (fn [s] - (when-not (string/starts-with? s common-config/file-version-prefix) - (str common-config/db-version-prefix s))))))) + (when (and (string? s) + (not (string/starts-with? s common-config/file-version-prefix))) + (common-config/canonicalize-db-version-repo s)))) + distinct))) diff --git a/deps/cli/src/logseq/cli/common/mcp/tools.cljs b/deps/cli/src/logseq/cli/common/mcp/tools.cljs index eba43a1da0..2856177335 100644 --- a/deps/cli/src/logseq/cli/common/mcp/tools.cljs +++ b/deps/cli/src/logseq/cli/common/mcp/tools.cljs @@ -26,7 +26,7 @@ (if expand (cond-> (into {} e) true - (dissoc e :block/tags :block/order :block/refs :block/name :db/index + (dissoc :block/tags :block/order :block/refs :block/name :db/index :logseq.property/default-value) true (update :block/uuid str) @@ -46,7 +46,7 @@ (if expand (cond-> (into {} e) true - (dissoc e :block/tags :block/order :block/refs :block/name) + (dissoc :block/tags :block/order :block/refs :block/name) true (update :block/uuid str) (:logseq.property.class/extends e) diff --git a/deps/cli/src/logseq/cli/util.cljs b/deps/cli/src/logseq/cli/util.cljs index 42bc50fa19..9bd9e0fc90 100644 --- a/deps/cli/src/logseq/cli/util.cljs +++ b/deps/cli/src/logseq/cli/util.cljs @@ -4,7 +4,7 @@ ["path" :as node-path] [clojure.string :as string] [logseq.cli.common.graph :as cli-common-graph] - [logseq.db.common.sqlite :as common-sqlite] + [logseq.common.graph-dir :as graph-dir] [nbb.error] [promesa.core :as p])) @@ -17,7 +17,7 @@ (string/includes? graph "/") ((juxt node-path/dirname node-path/basename) graph) :else - [(cli-common-graph/get-db-graphs-dir) (common-sqlite/sanitize-db-name graph)])) + [(cli-common-graph/get-db-graphs-dir) (graph-dir/repo->encoded-graph-dir-name graph)])) (defn get-graph-path "If graph is a file, return its path. Otherwise returns the graph's dir" diff --git a/deps/common/.carve/config.edn b/deps/common/.carve/config.edn index 954cbd2144..bdd5a7cab0 100644 --- a/deps/common/.carve/config.edn +++ b/deps/common/.carve/config.edn @@ -8,6 +8,8 @@ logseq.common.util.date-time logseq.common.date logseq.common.util.macro + logseq.common.cognito-config logseq.common.config - logseq.common.defkeywords] + logseq.common.defkeywords + logseq.common.graph-dir] :report {:format :ignore}} diff --git a/deps/common/.carve/ignore b/deps/common/.carve/ignore index e70a067333..2d0fac603b 100644 --- a/deps/common/.carve/ignore +++ b/deps/common/.carve/ignore @@ -3,6 +3,10 @@ logseq.common.graph/get-files ;; API fn logseq.common.graph/read-directories ;; API fn +logseq.common.graph/get-default-graphs-dir +;; API fn +logseq.common.graph/expand-home +;; API fn logseq.common.authorization/verify-jwt ;; Profile utils diff --git a/deps/common/src/logseq/common/cognito_config.cljs b/deps/common/src/logseq/common/cognito_config.cljs new file mode 100644 index 0000000000..d4f7d19450 --- /dev/null +++ b/deps/common/src/logseq/common/cognito_config.cljs @@ -0,0 +1,13 @@ +(ns logseq.common.cognito-config + "Shared Cognito configuration for frontend and CLI-safe consumers.") + +(def COGNITO-CLIENT-ID + "69cs1lgme7p8kbgld8n5kseii6") + +(def CLI-COGNITO-CLIENT-ID + "69cs1lgme7p8kbgld8n5kseii6") + +(def OAUTH-DOMAIN + "logseq-prod.auth.us-east-1.amazoncognito.com") + +(def OAUTH-SCOPE "email openid phone") diff --git a/deps/common/src/logseq/common/config.cljs b/deps/common/src/logseq/common/config.cljs index 09e690b84c..9f8f238e66 100644 --- a/deps/common/src/logseq/common/config.cljs +++ b/deps/common/src/logseq/common/config.cljs @@ -33,6 +33,26 @@ (defonce db-version-prefix "logseq_db_") (defonce file-version-prefix "logseq_local_") +(defn strip-leading-db-version-prefix + "Strip exactly one leading db prefix for user-facing display values." + [s] + (if (and (string? s) + (string/starts-with? s db-version-prefix)) + (subs s (count db-version-prefix)) + s)) + +(defn canonicalize-db-version-repo + "Normalize any repo/graph name to exactly one leading db prefix." + [s] + (when (seq s) + (let [s (str s) + stripped (loop [name' s] + (if (string/starts-with? name' db-version-prefix) + (recur (subs name' (count db-version-prefix))) + name'))] + (str db-version-prefix stripped)))) + +(defonce default-graphs-dir "~/logseq/graphs") (defonce local-assets-dir "assets") (defonce unlinked-graphs-dir "Unlinked graphs") diff --git a/deps/common/src/logseq/common/graph.cljs b/deps/common/src/logseq/common/graph.cljs index b255ac3b59..1cd38ae736 100644 --- a/deps/common/src/logseq/common/graph.cljs +++ b/deps/common/src/logseq/common/graph.cljs @@ -1,8 +1,10 @@ (ns ^:node-only logseq.common.graph "This ns provides common fns for a graph directory and only runs in a node environment" (:require ["fs" :as fs] + ["os" :as os] ["path" :as node-path] [clojure.string :as string] + [logseq.common.config :as common-config] [logseq.common.path :as path])) (def ^:private win32? @@ -97,3 +99,15 @@ Rules: (->> (readdir graph-dir) (remove (partial ignored-path? graph-dir)) (filter #(contains? allowed-formats (get-ext %))))) + +(defn get-default-graphs-dir + "Get default dir for storing graphs by first looking in env var." + [] + (or js/process.env.LOGSEQ_GRAPHS_DIR common-config/default-graphs-dir)) + +(defn expand-home + "Expands path if it starts with '~'" + [path] + (if (and (seq path) (string/starts-with? path "~")) + (node-path/join (os/homedir) (subs path 1)) + path)) diff --git a/deps/common/src/logseq/common/graph_dir.cljs b/deps/common/src/logseq/common/graph_dir.cljs new file mode 100644 index 0000000000..d57b68f37f --- /dev/null +++ b/deps/common/src/logseq/common/graph_dir.cljs @@ -0,0 +1,57 @@ +(ns logseq.common.graph-dir + "Platform-agnostic graph directory naming helpers." + (:require [clojure.string :as string] + [logseq.common.config :as common-config])) + +(defn encode-graph-dir-name + [graph-name] + (let [encoded (js/encodeURIComponent (or graph-name ""))] + (-> encoded + (string/replace "%20" " ") + (string/replace "~" "%7E") + (string/replace "%" "~")))) + +(defn decode-graph-dir-name + [dir-name] + (when-not (and (string? dir-name) + (or (string/includes? dir-name "++") + (string/includes? dir-name "+3A+"))) + (when (some? dir-name) + (try + (js/decodeURIComponent (string/replace dir-name "~" "%")) + (catch :default _ + nil))))) + +(def ^:private legacy-dir-pattern #"(?:\+\+|\+3A\+|%)") + +(defn decode-legacy-graph-dir-name + [dir-name] + (when (and (string? dir-name) + (re-find legacy-dir-pattern dir-name)) + (let [compat-name (-> dir-name + (string/replace "+3A+" ":") + (string/replace "++" "/"))] + (try + (let [decoded (js/decodeURIComponent compat-name)] + (when (seq decoded) + decoded)) + (catch :default _ + nil))))) + +(defn repo->graph-dir-key + [repo] + (when (seq repo) + (if (string/starts-with? repo common-config/db-version-prefix) + (subs repo (count common-config/db-version-prefix)) + repo))) + +(defn graph-dir-key->encoded-dir-name + [graph-dir-key] + (when (some? graph-dir-key) + (encode-graph-dir-name graph-dir-key))) + +(defn repo->encoded-graph-dir-name + [repo] + (some-> repo + repo->graph-dir-key + graph-dir-key->encoded-dir-name)) diff --git a/deps/db-sync/.carve/ignore b/deps/db-sync/.carve/ignore index 7979afb245..f234f1b130 100644 --- a/deps/db-sync/.carve/ignore +++ b/deps/db-sync/.carve/ignore @@ -26,6 +26,8 @@ logseq.db-sync.worker/worker ;; debugging logseq.db-sync.worker.timing/summary ;; API +logseq.db-sync.snapshot/finalize-datoms-jsonl-buffer +;; API logseq.db-sync.checksum/recompute-checksum-diagnostics ;; API (test runner entrypoint) logseq.db-sync.test-runner/main diff --git a/deps/db-sync/src/logseq/db_sync/malli_schema.cljs b/deps/db-sync/src/logseq/db_sync/malli_schema.cljs index b3ed276242..d9140a94b6 100644 --- a/deps/db-sync/src/logseq/db_sync/malli_schema.cljs +++ b/deps/db-sync/src/logseq/db_sync/malli_schema.cljs @@ -148,7 +148,7 @@ (def graphs-list-response-schema [:map [:graphs [:sequential graph-info-schema]] - [:user-rsa-keys-exists? :boolean]]) + [:user-rsa-keys-exists? {:optional true} :boolean]]) (def graph-create-request-schema [:map diff --git a/deps/db-sync/src/logseq/db_sync/node/graph.cljs b/deps/db-sync/src/logseq/db_sync/node/graph.cljs index 5e171b9d70..8cb7707ed5 100644 --- a/deps/db-sync/src/logseq/db_sync/node/graph.cljs +++ b/deps/db-sync/src/logseq/db_sync/node/graph.cljs @@ -8,14 +8,18 @@ :removeWebSocket (fn [ws] (swap! sockets disj ws))})) (defn- env-object [cfg index-db assets-bucket] - (doto (js-obj) - (aset "DB" index-db) - (aset "LOGSEQ_SYNC_ASSETS" assets-bucket) - ;; Keep node-adapter snapshot stream uncompressed. - (aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false") - (aset "COGNITO_ISSUER" (:cognito-issuer cfg)) - (aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg)) - (aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))) + (let [allow-unverified-jwt-claims (some-> js/process .-env (aget "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS")) + env (doto (js-obj) + (aset "DB" index-db) + (aset "LOGSEQ_SYNC_ASSETS" assets-bucket) + ;; Keep node-adapter snapshot stream uncompressed. + (aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false") + (aset "COGNITO_ISSUER" (:cognito-issuer cfg)) + (aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg)) + (aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))] + (when (some? allow-unverified-jwt-claims) + (aset env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" allow-unverified-jwt-claims)) + env)) (defn graph-context [{:keys [config index-db assets-bucket]} graph-id] diff --git a/deps/db-sync/src/logseq/db_sync/node/server.cljs b/deps/db-sync/src/logseq/db_sync/node/server.cljs index 887f222a88..46d6340ba0 100644 --- a/deps/db-sync/src/logseq/db_sync/node/server.cljs +++ b/deps/db-sync/src/logseq/db_sync/node/server.cljs @@ -24,15 +24,19 @@ (logging/install!) (defn- make-env [cfg index-db assets-bucket] - (doto (js-obj) - (aset "DB" index-db) - (aset "LOGSEQ_SYNC_ASSETS" assets-bucket) - ;; Node adapter serves snapshot transit stream without gzip to avoid - ;; browser/adapter content-encoding mismatches during graph download. - (aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false") - (aset "COGNITO_ISSUER" (:cognito-issuer cfg)) - (aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg)) - (aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))) + (let [allow-unverified-jwt-claims (some-> js/process .-env (aget "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS")) + env (doto (js-obj) + (aset "DB" index-db) + (aset "LOGSEQ_SYNC_ASSETS" assets-bucket) + ;; Node adapter serves snapshot transit stream without gzip to avoid + ;; browser/adapter content-encoding mismatches during graph download. + (aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false") + (aset "COGNITO_ISSUER" (:cognito-issuer cfg)) + (aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg)) + (aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))] + (when (some? allow-unverified-jwt-claims) + (aset env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" allow-unverified-jwt-claims)) + env)) (defn- access-allowed? [env graph-id request] diff --git a/deps/db-sync/src/logseq/db_sync/worker/auth.cljs b/deps/db-sync/src/logseq/db_sync/worker/auth.cljs index 66bff29f1c..f9f62aa950 100644 --- a/deps/db-sync/src/logseq/db_sync/worker/auth.cljs +++ b/deps/db-sync/src/logseq/db_sync/worker/auth.cljs @@ -34,12 +34,28 @@ (def ^:private recoverable-auth-errors #{"invalid" "iss not found" "aud not found" "exp" "kid"}) +(def ^:private truthy-env-values + #{"1" "true" "yes" "on"}) + (defn- recoverable-auth-error? [error] (when error (let [message (or (ex-message error) (some-> error .-message))] (contains? recoverable-auth-errors message)))) +(defn- env-flag-enabled? + [env k] + (let [v (some-> env (aget k))] + (cond + (true? v) true + (false? v) false + (string? v) (contains? truthy-env-values (string/lower-case v)) + :else false))) + +(defn- allow-unverified-jwt-claims? + [env] + (env-flag-enabled? env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS")) + (defn- expired-token? [token] (when-let [claims (unsafe-jwt-claims token)] @@ -55,7 +71,13 @@ (p/resolved nil) (-> (authorization/verify-jwt token env) (p/catch (fn [error] - (if (recoverable-auth-error? error) + (cond + (recoverable-auth-error? error) nil + + (allow-unverified-jwt-claims? env) + (unsafe-jwt-claims token) + + :else (p/rejected error)))))) (p/resolved nil)))) diff --git a/deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs b/deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs index 6a17a1fd06..eacf6ad040 100644 --- a/deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs +++ b/deps/db-sync/src/logseq/db_sync/worker/handler/sync.cljs @@ -274,9 +274,9 @@ (let [value (.-value chunk) {:keys [rows buffer]} (snapshot/parse-framed-chunk buffer value) rows-count (count rows) - reset? (and @reset-pending? (seq rows))] + reset? (boolean (and @reset-pending? (seq rows)))] (when (seq rows) - (import-snapshot! self rows (true? reset?)) + (import-snapshot! self rows reset?) (vreset! reset-pending? false)) (vswap! total-count + rows-count) (p/recur buffer))))) diff --git a/deps/db-sync/src/logseq/db_sync/worker/ws.cljs b/deps/db-sync/src/logseq/db_sync/worker/ws.cljs index 5c4846530b..5eed65c3d1 100644 --- a/deps/db-sync/src/logseq/db_sync/worker/ws.cljs +++ b/deps/db-sync/src/logseq/db_sync/worker/ws.cljs @@ -28,7 +28,9 @@ (.send ws (protocol/encode-message {:type "error" :message "server error"})))))) (defn broadcast! [^js self sender msg] - (let [clients (.getWebSockets (.-state self))] - (doseq [ws clients] - (when (and (not= ws sender) (ws-open? ws)) - (send! ws msg))))) + (when-let [state (some-> self .-state)] + (when (fn? (.-getWebSockets state)) + (let [clients (.getWebSockets state)] + (doseq [ws clients] + (when (and (not= ws sender) (ws-open? ws)) + (send! ws msg))))))) diff --git a/deps/db-sync/test/logseq/db_sync/worker_auth_test.cljs b/deps/db-sync/test/logseq/db_sync/worker_auth_test.cljs index da187c325e..ba57d2b032 100644 --- a/deps/db-sync/test/logseq/db_sync/worker_auth_test.cljs +++ b/deps/db-sync/test/logseq/db_sync/worker_auth_test.cljs @@ -57,6 +57,22 @@ (is (= "jwks" (ex-message error))) (done))))))) +(deftest auth-claims-jwks-error-falls-back-to-unsafe-claims-when-enabled-test + (async done + (let [token "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJ1MSJ9.signature" + request (js/Request. "http://localhost/graphs" + #js {:headers #js {"authorization" (str "Bearer " token)}}) + env #js {"DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" "true"}] + (-> (p/with-redefs [authorization/verify-jwt + (fn [_token _env] + (p/rejected (ex-info "jwks" {})))] + (p/let [claims (auth/auth-claims request env)] + (is (= "u1" (aget claims "sub"))))) + (p/then (fn [] (done))) + (p/catch (fn [error] + (is false (str error)) + (done))))))) + (deftest auth-claims-expired-jwt-short-circuits-verification-test (async done (let [expired-token "eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjEsInN1YiI6InUxIn0.signature" diff --git a/deps/db-sync/test/logseq/db_sync/worker_handler_sync_test.cljs b/deps/db-sync/test/logseq/db_sync/worker_handler_sync_test.cljs index d867321bc6..f9e8667fcb 100644 --- a/deps/db-sync/test/logseq/db_sync/worker_handler_sync_test.cljs +++ b/deps/db-sync/test/logseq/db_sync/worker_handler_sync_test.cljs @@ -561,6 +561,56 @@ (is (= "pull/ok" (:type pull-response))) (is (empty? (:txs pull-response))))))) +(deftest tx-batch-keeps-created-by-ref-lookup-payload-test + (testing "created-by lookup payload is preserved for save-block tx" + (let [sql (test-sql/make-sql) + conn (storage/open-conn sql) + self #js {:sql sql + :conn conn + :schema-ready true} + page-uuid (random-uuid) + missing-user-uuid (random-uuid) + missing-user-ref [:block/uuid missing-user-uuid] + tx-entry {:tx (protocol/tx->transit [[:db/add -1 :block/uuid page-uuid] + [:db/add -1 :block/name "created-by-sanitize-page"] + [:db/add -1 :block/title "created-by-sanitize-page"] + [:db/add -1 :logseq.property/created-by-ref missing-user-ref]]) + :outliner-op :save-block} + response (with-redefs [ws/broadcast! (fn [& _] nil)] + (sync-handler/handle-tx-batch! self nil [tx-entry] 0)) + page (d/entity @conn [:block/uuid page-uuid])] + (is (= "tx/batch/ok" (:type response))) + (is (= 1 (:t response))) + (is (some? page)) + (is (= "created-by-sanitize-page" (:block/title page))) + (is (= missing-user-ref (:logseq.property/created-by-ref page)))))) + +(deftest tx-batch-rejects-missing-page-ref-lookups-test + (testing "missing page refs/tags lookup refs reject create-page tx" + (let [sql (test-sql/make-sql) + conn (storage/open-conn sql) + self #js {:sql sql + :conn conn + :schema-ready true} + page-uuid (random-uuid) + missing-ref-uuid (random-uuid) + missing-tag-uuid (random-uuid) + missing-user-uuid (random-uuid) + tx-entry {:tx (protocol/tx->transit [[:db/add -1 :block/uuid page-uuid] + [:db/add -1 :block/name "optional-ref-sanitize-page"] + [:db/add -1 :block/title "optional-ref-sanitize-page"] + [:db/add -1 :block/refs [:block/uuid missing-ref-uuid]] + [:db/add -1 :block/tags [:block/uuid missing-tag-uuid]] + [:db/add -1 :logseq.property/created-by-ref [:block/uuid missing-user-uuid]]]) + :outliner-op :create-page} + response (with-redefs [ws/broadcast! (fn [& _] nil)] + (sync-handler/handle-tx-batch! self nil [tx-entry] 0)) + page (d/entity @conn [:block/uuid page-uuid])] + (is (= "tx/reject" (:type response))) + (is (= "db transact failed" (:reason response))) + (is (= 0 (:t response))) + (is (nil? page))))) + (deftest tx-batch-rejects-while-snapshot-upload-is-in-progress-test (let [sql (test-sql/make-sql) conn (d/create-conn db-schema/schema) @@ -640,6 +690,33 @@ (is false (str error)) (done))))))) +(deftest import-snapshot-stream-first-non-empty-chunk-applies-reset-test + (async done + (let [rows [[42 "payload" nil]] + frame (#'sync-handler/frame-bytes (snapshot/encode-rows rows)) + stream (js/ReadableStream. + #js {:start (fn [controller] + (.enqueue controller frame) + (.close controller))}) + applied (atom []) + self #js {:sql (test-sql/make-sql) + :conn (d/create-conn db-schema/schema) + :schema-ready true}] + (-> (p/with-redefs [sync-handler/import-snapshot! + (fn [_self rows* reset?] + (swap! applied conj {:rows rows* + :reset? reset?}))] + (p/let [count (#'sync-handler/import-snapshot-stream! self stream true)] + (is (= 1 count)) + (is (= [{:rows rows + :reset? true}] + @applied)))) + (p/then (fn [] + (done))) + (p/catch (fn [error] + (is false (str error)) + (done))))))) + (deftest tx-batch-rejects-when-a-tx-entry-fails-test (testing "db transact failure rejects the batch" (let [sql (test-sql/make-sql) diff --git a/deps/db/.carve/config.edn b/deps/db/.carve/config.edn index df4528d2d1..8068ab7858 100644 --- a/deps/db/.carve/config.edn +++ b/deps/db/.carve/config.edn @@ -9,6 +9,8 @@ logseq.db.sqlite.create-graph logseq.db.frontend.malli-schema logseq.db.frontend.asset + ;; Used outside deps/db by frontend worker and electron modules. + logseq.db.sqlite.backup ;; Some fns are used by frontend but not worth moving over yet logseq.db.frontend.schema logseq.db.frontend.validate diff --git a/deps/db/src/logseq/db/common/sqlite.cljs b/deps/db/src/logseq/db/common/sqlite.cljs index 9bfae51898..3c658ce3a9 100644 --- a/deps/db/src/logseq/db/common/sqlite.cljs +++ b/deps/db/src/logseq/db/common/sqlite.cljs @@ -3,6 +3,8 @@ (:require ["path" :as node-path] [clojure.string :as string] [datascript.core :as d] + [logseq.common.graph-dir :as graph-dir] + [logseq.common.path :as path] [logseq.db.sqlite.util :as sqlite-util])) (defn create-kvs-table! @@ -26,11 +28,11 @@ (defn get-db-full-path [graphs-dir db-name] - (let [db-name' (sanitize-db-name db-name) - graph-dir (node-path/join graphs-dir db-name')] - [db-name' (node-path/join graph-dir "db.sqlite")])) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name) + graph-dir (node-path/join graphs-dir graph-dir-name)] + [graph-dir-name (path/path-join graph-dir "db.sqlite")])) (defn get-db-backups-path [graphs-dir db-name] - (let [db-name' (sanitize-db-name db-name)] - (node-path/join graphs-dir db-name' "backups"))) + (let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name)] + (path/path-join graphs-dir graph-dir-name "backups"))) diff --git a/deps/db/src/logseq/db/common/sqlite_cli.cljs b/deps/db/src/logseq/db/common/sqlite_cli.cljs index 778e6a970b..13cd4231bb 100644 --- a/deps/db/src/logseq/db/common/sqlite_cli.cljs +++ b/deps/db/src/logseq/db/common/sqlite_cli.cljs @@ -1,11 +1,11 @@ (ns ^:node-only logseq.db.common.sqlite-cli "Primary ns to interact with DB files with node.js based CLIs" (:require ["better-sqlite3" :as sqlite3] - ["os" :as os] ["path" :as node-path] [cljs-bean.core :as bean] [clojure.string :as string] [datascript.storage :refer [IStorage]] + [logseq.common.graph :as common-graph] [logseq.db.common.sqlite :as common-sqlite] [logseq.db.frontend.schema :as db-schema] [logseq.db.sqlite.util :as sqlite-util])) @@ -22,6 +22,7 @@ (defn- upsert-addr-content! "Upsert addr+data-seq. Should be functionally equivalent to db-worker/upsert-addr-content!" [db data] + (assert db ::upsert-addr-content!) (let [insert (.prepare db "INSERT INTO kvs (addr, content, addresses) values ($addr, $content, $addresses) on conflict(addr) do update set content = $content, addresses = $addresses") insert-many (.transaction ^object db (fn [data] @@ -99,5 +100,4 @@ ;; $ORIGINAL_PWD used by bb tasks to correct current dir (node-path/join (or js/process.env.ORIGINAL_PWD ".") %))] ((juxt node-path/dirname node-path/basename) (resolve-path' graph-dir-or-path))) - ;; TODO: Reuse with get-db-graphs-dir when there is a db ns that is usable by electron i.e. no better-sqlite3 - [(node-path/join (os/homedir) "logseq" "graphs") graph-dir-or-path]))) + [(common-graph/expand-home (common-graph/get-default-graphs-dir)) graph-dir-or-path]))) diff --git a/deps/db/src/logseq/db/frontend/db.cljs b/deps/db/src/logseq/db/frontend/db.cljs index 4f875b5e4f..b67461ea05 100644 --- a/deps/db/src/logseq/db/frontend/db.cljs +++ b/deps/db/src/logseq/db/frontend/db.cljs @@ -21,7 +21,7 @@ (:db/ident property-entity)))) (defn private-built-in-page? - "Private built-in pages should not be navigable or searchable by users. Later it + "Private built-in pages should not be navigable, searchable or editable by users. Later it could be useful to use this for the All Pages view" [page] (cond (entity-util/property? page) diff --git a/deps/db/src/logseq/db/frontend/db_ident.cljc b/deps/db/src/logseq/db/frontend/db_ident.cljc index b681d15b2e..03e33e9a99 100644 --- a/deps/db/src/logseq/db/frontend/db_ident.cljc +++ b/deps/db/src/logseq/db/frontend/db_ident.cljc @@ -75,9 +75,9 @@ (assert (not (re-find #"^(logseq|block)(\.|$)" (name user-namespace))) "New ident is not allowed to use an internal namespace") (if #?(:org.babashka/nbb true - :cljs (exists? js/process) + ;; Use $LOGSEQ_STABLE_IDENTS when we want stable idents e.g. tests + :cljs (and (exists? js/process) js/process.env.LOGSEQ_STABLE_IDENTS) :default false) - ;; Used for contexts where we want repeatable idents e.g. tests and CLIs (keyword user-namespace (normalize-ident-name-part name-string)) (let [plugin? (string/starts-with? user-namespace "plugin.class.") suffix (str "-" diff --git a/deps/db/src/logseq/db/frontend/property.cljs b/deps/db/src/logseq/db/frontend/property.cljs index fb349d263b..3d7eb3de8d 100644 --- a/deps/db/src/logseq/db/frontend/property.cljs +++ b/deps/db/src/logseq/db/frontend/property.cljs @@ -26,7 +26,7 @@ * :cardinality - property cardinality. Default to one/single cardinality if not set * :hide? - Boolean which hides property when set on a block or exported e.g. slides * :public? - Boolean which allows property to be used by user: add and remove property to blocks/pages - and queryable via property and has-property rules + and queryable via property and has-property rules. When it's not set, it's the same as false * :view-context - Keyword to indicate which view contexts a property can be seen in when :public? is set. Valid values are :page, :block and :never. Property can be viewed in any context if not set @@ -635,6 +635,11 @@ :public? false :hide? true}}))) +(def public-built-in-properties + (->> built-in-properties + (keep (fn [[k v]] (when (get-in v [:schema :public?]) k))) + set)) + (def db-attribute-properties "Internal properties that are also db schema attributes" #{:block/alias :block/tags :block/parent @@ -741,6 +746,11 @@ ;; Disallow tags or page refs as they would create unreferenceable page names (not (re-find #"^(#|\[\[)" s))) +(defn built-in-closed-values + "Gets :closed-values for given built-in property ident" + [ident] + (get-in built-in-properties [ident :closed-values])) + (defn get-closed-property-values [db property-id] (when db diff --git a/deps/db/src/logseq/db/frontend/property/type.cljs b/deps/db/src/logseq/db/frontend/property/type.cljs index e04eb033d3..16c44eca4d 100644 --- a/deps/db/src/logseq/db/frontend/property/type.cljs +++ b/deps/db/src/logseq/db/frontend/property/type.cljs @@ -23,7 +23,7 @@ (def user-allowed-internal-property-types "Internal property types that users are allowed to store. These aren't available in the UI so these would normally be created via EDN or the API." - #{:map}) + #{:map :json :string}) (assert (set/subset? user-allowed-internal-property-types internal-built-in-property-types)) diff --git a/deps/db/src/logseq/db/sqlite/backup.cljs b/deps/db/src/logseq/db/sqlite/backup.cljs new file mode 100644 index 0000000000..1bcee037f9 --- /dev/null +++ b/deps/db/src/logseq/db/sqlite/backup.cljs @@ -0,0 +1,46 @@ +(ns logseq.db.sqlite.backup + "Shared SQLite backup utilities for Node runtimes." + (:require ["node:sqlite" :as node-sqlite] + [clojure.string :as string] + [goog.object :as gobj] + [promesa.core :as p])) + +(defn- resolve-database-sync-ctor + [] + (or (gobj/get node-sqlite "DatabaseSync") + (some-> (gobj/get node-sqlite "default") + (gobj/get "DatabaseSync")) + (let [default-export (gobj/get node-sqlite "default")] + (when (fn? default-export) + default-export)) + (throw (ex-info "node:sqlite DatabaseSync constructor missing" + {:module-keys (js->clj (js/Object.keys node-sqlite))})))) + +(def ^:private DatabaseSync + (resolve-database-sync-ctor)) + +(defn- resolve-sqlite-backup-fn + [] + (or (gobj/get node-sqlite "backup") + (some-> (gobj/get node-sqlite "default") + (gobj/get "backup")) + (throw (ex-info "node:sqlite backup function missing" + {:module-keys (js->clj (js/Object.keys node-sqlite))})))) + +(def ^:private sqlite-backup-fn + (resolve-sqlite-backup-fn)) + +(defn backup-connection! + [^js db path] + (sqlite-backup-fn db path)) + +(defn backup-db-file! + [src-path dst-path] + (let [db (new DatabaseSync src-path)] + (-> (backup-connection! db dst-path) + (p/finally (fn [] + (try + (.close db) + (catch :default error + (when-not (string/includes? (str error) "database is not open") + (throw error))))))))) diff --git a/deps/graph-parser/README.md b/deps/graph-parser/README.md index 7de5848680..f67565c795 100644 --- a/deps/graph-parser/README.md +++ b/deps/graph-parser/README.md @@ -36,10 +36,8 @@ This step is not needed if you're just running the frontend application. ### Testing -Since this library is compatible with cljs and nbb-logseq, tests are run against both languages. - -Nbb tests use [nbb-test-runner](https://github.com/nextjournal/nbb-test-runner). -Some basic usage: +Testing is done with nbb-logseq and +[nbb-test-runner](https://github.com/nextjournal/nbb-test-runner). Some basic usage: ``` # Run all tests @@ -50,13 +48,6 @@ $ pnpm test -H $ pnpm test -i focus ``` -ClojureScript tests use https://github.com/Olical/cljs-test-runner. To run tests: -``` -clojure -M:test -``` - -To see available options that can run specific tests or namespaces: `clojure -M:test --help` - ### Managing dependencies The package.json dependencies are just for testing and should be updated if there is diff --git a/deps/outliner/src/logseq/outliner/core.cljs b/deps/outliner/src/logseq/outliner/core.cljs index 5b84b3c118..3bd06b0d40 100644 --- a/deps/outliner/src/logseq/outliner/core.cljs +++ b/deps/outliner/src/logseq/outliner/core.cljs @@ -461,6 +461,11 @@ (throw (ex-info "Block eid doesn't exist" {:block block}))) (when-let [entity (d/entity db eid)] + (when (outliner-validate/built-in-entity? entity) + (throw (ex-info "Built-in nodes can't be modified" + {:type :notification + :payload {:message "Built-in nodes can't be modified" + :type :error}}))) (let [*txs-state (atom []) block' (if (de/entity? block) block @@ -838,15 +843,15 @@ (let [top-level-blocks (filter-top-level-blocks db blocks) non-consecutive? (and (> (count top-level-blocks) 1) (seq (ldb/get-non-consecutive-blocks db top-level-blocks))) top-level-blocks* (get-top-level-blocks top-level-blocks non-consecutive?) - top-level-blocks (remove :logseq.property/built-in? top-level-blocks*) + top-level-blocks (remove outliner-validate/built-in-entity? top-level-blocks*) txs-state (ds/new-outliner-txs-state) block-ids (map (fn [b] [:block/uuid (:block/uuid b)]) top-level-blocks) start-block (first top-level-blocks) end-block (last top-level-blocks) delete-one-block? (or (= 1 (count top-level-blocks)) (= start-block end-block))] - ;; Validate before `when` since top-level-blocks will be empty when deleting one built-in block - (when (seq (filter :logseq.property/built-in? top-level-blocks*)) + ;; Validate before `when` since top-level-blocks will be empty when deleting one built-in/internal block + (when (seq (filter outliner-validate/built-in-entity? top-level-blocks*)) (throw (ex-info "Built-in nodes can't be deleted" {:type :notification :payload {:message "Built-in nodes can't be deleted." @@ -934,6 +939,13 @@ :as opts}] {:pre [(seq blocks)]} (when (m/validate block-map-or-entity target-block) + (doseq [b blocks] + (let [entity (d/entity @conn (:db/id b))] + (when (outliner-validate/built-in-entity? entity) + (throw (ex-info "Built-in nodes can't be modified" + {:type :notification + :payload {:message "Built-in nodes can't be modified" + :type :error}}))))) (let [db @conn top-level-blocks (filter-top-level-blocks db blocks) [target-block sibling?] (get-target-block db top-level-blocks target-block opts) diff --git a/deps/outliner/src/logseq/outliner/page.cljs b/deps/outliner/src/logseq/outliner/page.cljs index eb8f0fa169..0df4d7baae 100644 --- a/deps/outliner/src/logseq/outliner/page.cljs +++ b/deps/outliner/src/logseq/outliner/page.cljs @@ -195,11 +195,9 @@ ;; TODO: Revisit title cleanup as this was copied from file implementation (defn ^:api sanitize-title [title] - (let [title (-> (string/trim title) - (text/page-ref-un-brackets!) - ;; remove `#` from tags - (string/replace #"^#+" "")) - title (common-util/remove-boundary-slashes title)] + (let [title (-> (string/trim title) + (text/page-ref-un-brackets!)) + title (common-util/remove-boundary-slashes title)] title)) (defn- get-page-by-parent-name @@ -310,6 +308,7 @@ class? (or class? (some (fn [t] (= :logseq.class/Tag (:db/ident t))) tags)) class-ident-namespace? (and class? class-ident-namespace (string? class-ident-namespace)) title (sanitize-title title*) + _ (outliner-validate/validate-page-title-no-hashtag title {:node {:block/title title}}) types (cond class? #{:logseq.class/Tag} today-journal? diff --git a/deps/outliner/src/logseq/outliner/property.cljs b/deps/outliner/src/logseq/outliner/property.cljs index 767cc6fc4a..91098a4959 100644 --- a/deps/outliner/src/logseq/outliner/property.cljs +++ b/deps/outliner/src/logseq/outliner/property.cljs @@ -94,6 +94,7 @@ [entities property-ident] (throw-error-if-deleting-protected-property (map :db/ident entities) property-ident) (when (= :block/tags property-ident) (throw-error-if-removing-private-tag entities)) + (outliner-validate/disallow-editing-private-built-in-nodes entities) (throw-error-if-deleting-required-property property-ident)) (defn- build-property-value-tx-data @@ -184,14 +185,18 @@ (and new-type old-ref-type? (not ref-type?)) (conj [:db/retract (:db/id property) :db/valueType])))) -(defn- update-property - [conn db-ident property schema {:keys [property-name properties]}] +(defn- validate-property-name-update + [conn property property-name] (when (and (some? property-name) (not= property-name (:block/title property))) (outliner-validate/validate-page-title property-name {:node property}) (outliner-validate/validate-page-title-characters property-name {:node property}) (outliner-validate/validate-block-title @conn property-name property) - (outliner-validate/validate-property-title property-name)) + (outliner-validate/validate-property-title property-name))) +(defn- update-property + [conn db-ident property schema {:keys [property-name properties]}] + (validate-property-name-update conn property property-name) + (outliner-validate/validate-editing-built-in-property property schema) (let [changed-property-attrs ;; Only update property if something has changed as we are updating a timestamp (cond-> (->> (dissoc schema :db/cardinality) @@ -219,13 +224,21 @@ (mapcat (fn [[property-id v]] (build-property-value-tx-data conn property property-id v)) properties))) - many->one? (and (db-property/many? property) (= :one (:db/cardinality schema)))] + many->one? (and (db-property/many? property) + ;; For UI calls, :db/cardinality can have :one and :many values + (contains? #{:one :db.cardinality/one} (:db/cardinality schema)))] (when (and many->one? (seq (d/datoms @conn :avet db-ident))) (throw (ex-info "Disallowed many to one conversion" {:type :notification :payload {:message "This property can't change from multiple values to one value because it has existing data." :i18n-key :property.validation/many-to-one :type :warning}}))) + (when (and (contains? changed-property-attrs :logseq.property/type) + (seq (d/datoms @conn :avet db-ident))) + (throw (ex-info "Disallowed type change with existing data" + {:type :notification + :payload {:message "This property's type can't be changed because it has existing data." + :type :error}}))) (when (seq tx-data) (ldb/transact! conn tx-data {:outliner-op :update-property :property-id (:db/id property)})) @@ -407,14 +420,37 @@ v)] (find-or-create-property-value conn property-id v'))))) +(defn- convert-ref-property-values + [conn property-id value property-type {:keys [many?]}] + (if-not (and many? (or (sequential? value) (set? value))) + (convert-ref-property-value conn property-id value property-type) + (try + (mapv #(convert-ref-property-value conn property-id % property-type) value) + (catch :default e + (throw (ex-info "Failed to convert many property values" + (merge + {:property-id property-id + :property-type property-type + :value value + :many? many? + :value-shape (cond + (vector? value) :vector + (set? value) :set + (sequential? value) :seq + :else :single)} + (ex-data e)) + e)))))) + (defn- throw-error-if-self-value [block value ref?] - (when (and ref? (= value (:db/id block))) - (throw (ex-info "Can't set this block itself as own property value" - {:type :notification - :payload {:message "Can't set this block itself as own property value." - :i18n-key :property.validation/cant-set-self-value - :type :error}})))) + (let [values (if (or (sequential? value) (set? value)) value [value])] + (when (and ref? + (some #(= % (:db/id block)) values)) + (throw (ex-info "Can't set this block itself as own property value" + {:type :notification + :payload {:message "Can't set this block itself as own property value." + :i18n-key :property.validation/cant-set-self-value + :type :error}}))))) (defn batch-remove-property! [conn block-ids property-id] @@ -467,9 +503,21 @@ (ldb/transact! conn txs {:outliner-op :batch-remove-property}))))))) +(defn- validate-batch-set-property + [conn block-eids property-id v] + (outliner-validate/disallow-editing-private-built-in-nodes (mapv #(d/entity @conn %) block-eids)) + (when (= property-id :block/tags) + (outliner-validate/validate-tags-property @conn block-eids v)) + (when (= property-id :logseq.property.class/extends) + (outliner-validate/validate-extends-property + @conn + (if (number? v) (d/entity @conn v) v) + (map #(d/entity @conn %) block-eids)))) + (defn batch-set-property! "Sets properties for multiple blocks. Automatically handles property value refs. - Does no validation of property values." + Does no validation of property values. For :many properties, passing a collection + replaces existing values in one call, while passing a scalar preserves add-single-value behavior." ([conn block-ids property-id v] (batch-set-property! conn block-ids property-id v {})) ([conn block-ids property-id v options] @@ -478,24 +526,18 @@ (if (nil? v) (batch-remove-property! conn block-ids property-id) (let [block-eids (map ->eid block-ids) - _ (when (= property-id :block/tags) - (outliner-validate/validate-tags-property @conn block-eids v)) + _ (validate-batch-set-property conn block-eids property-id v) property (d/entity @conn property-id) - blocks (keep #(d/entity @conn %) block-eids) - _ (when (= (:db/ident property) :logseq.property.class/extends) - (outliner-validate/validate-extends-property - @conn - (if (number? v) (d/entity @conn v) v) - blocks)) _ (when (nil? property) (throw (ex-info (str "Property " property-id " doesn't exist yet") {:property-id property-id}))) property-type (get property :logseq.property/type :default) + many? (= :db.cardinality/many (:db/cardinality property)) entity-id? (and (:entity-id? options) (number? v)) ref? (contains? db-property-type/all-ref-property-types property-type) default-url-not-closed? (and (contains? #{:default :url} property-type) (not (seq (entity-plus/lookup-kv-then-entity property :property/closed-values)))) v' (if (and ref? (not entity-id?)) - (convert-ref-property-value conn property-id v property-type) + (convert-ref-property-values conn property-id v property-type {:many? many?}) v) _ (when (nil? v') (throw (ex-info "Property value must be not nil" {:v v}))) @@ -505,11 +547,17 @@ (if-let [block (d/entity @conn eid)] (let [v' (if (and default-url-not-closed? (not (and (keyword? v) entity-id?))) - (do - (when (number? v') - (throw-error-if-invalid-property-value @conn property v')) - (let [v (if (number? v') (:block/title (d/entity @conn v')) v')] - (convert-ref-property-value conn property-id v property-type))) + (let [normalize-default-url-value + (fn [value] + (if (number? value) + (do + (throw-error-if-invalid-property-value @conn property value) + (:block/title (d/entity @conn value))) + value)) + value' (if (and many? (or (sequential? v') (set? v'))) + (mapv normalize-default-url-value v') + (normalize-default-url-value v'))] + (convert-ref-property-values conn property-id value' property-type {:many? many?})) v')] (throw-error-if-self-value block v' ref?) (throw-error-if-invalid-property-value @conn property v') @@ -665,6 +713,7 @@ (prn "property-id: " property-id ", property-name: " property-name)) (outliner-validate/validate-page-title k-name {:node {:db/ident db-ident'}}) (outliner-validate/validate-page-title-characters k-name {:node {:db/ident db-ident'}}) + (outliner-validate/validate-property-title k-name {:node {:db/ident db-ident'}}) (let [db-id (:db/id properties) opts' (cond-> {:title k-name :properties properties} @@ -672,6 +721,7 @@ (assoc :block-uuid (:block/uuid (d/entity db db-id)))) tx-data (concat [(sqlite-util/build-new-property db-ident' schema opts')] + ;; Convert page to property (when db-id [[:db/retract db-id :block/tags :logseq.class/Page]]))] (ldb/transact! conn tx-data diff --git a/deps/outliner/src/logseq/outliner/validate.cljs b/deps/outliner/src/logseq/outliner/validate.cljs index b744ea1409..b537027d0a 100644 --- a/deps/outliner/src/logseq/outliner/validate.cljs +++ b/deps/outliner/src/logseq/outliner/validate.cljs @@ -9,10 +9,11 @@ [logseq.db :as ldb] [logseq.db.frontend.class :as db-class] [logseq.db.frontend.entity-util :as entity-util] + [logseq.db.frontend.malli-schema :as db-malli-schema] [logseq.db.frontend.property :as db-property])) -(defn ^:api validate-page-title-characters - "Validates characters that must not be in a page title" +(defn ^:api validate-page-title-no-hashtag + "Validates a page title doesn't include hashtag character" [page-title meta-m] (when (string/includes? page-title "#") (throw (ex-info "Page name can't include \"#\"." @@ -20,7 +21,12 @@ {:type :notification :payload {:message "Page name can't include \"#\"." :i18n-key :page.validation/name-no-hash - :type :warning}})))) + :type :warning}}))))) + +(defn ^:api validate-page-title-characters + "Validates characters that must not be in a page title" + [page-title meta-m] + (validate-page-title-no-hashtag page-title meta-m) (when (and (string/includes? page-title ns-util/parent-char) (not (common-date/normalize-date page-title nil))) (throw (ex-info "Page name can't include \"/\"." @@ -154,6 +160,22 @@ :i18n-key :property.validation/invalid-name :type :error}})))))) +(defn validate-editing-built-in-property + "Validates if built-in property entity is editable for the given attributes to be updated" + [entity attribute-map-to-update] + ;; Update allowed as needed. Keep this as an allowed list to default to safe editing for built-in entities + (let [allowed-attributes #{:logseq.property/hide-empty-value :logseq.property/description}] + (when-let [disallowed (and (:logseq.property/built-in? entity) + (not-empty (set/difference (set (keys attribute-map-to-update)) + allowed-attributes)))] + (throw (ex-info "Given built-in property's attributes are not editable" + (merge + {:type :notification + :payload {:message "Can't change the given attributes for a built-in property" + :type :error}} + {:property (:db/ident entity) + :disallowed-attributes disallowed})))))) + (defn- validate-extends-property-have-correct-type "Validates whether given parent and children are classes" [parent-ent child-ents] @@ -231,9 +253,21 @@ :property-id :block/tags :property-value v}))))) +(defn built-in-entity? + "Returns true when the entity is a built-in. Ideally checking + :logseq.property/built-in? would be enough but not all built-in nodes have + that property. Covers: + - entities marked with :logseq.property/built-in? (built-in pages, classes, properties) + - file entities (logseq/config.edn, custom.css, etc.) + - entities whose :db/ident belongs to an internal namespace (KV entries, empty-placeholder)" + [ent] + (or (:logseq.property/built-in? ent) + (:file/path ent) + (some-> (:db/ident ent) db-malli-schema/internal-ident?))) + (defn- disallow-tagging-a-built-in-entity [db block-eids & {:keys [delete?]}] - (when-let [built-in-ent (some #(when (:logseq.property/built-in? %) %) + (when-let [built-in-ent (some #(when (built-in-entity? %) %) (map #(d/entity db %) block-eids))] (throw (ex-info (str (if delete? "Can't remove tag" "Can't add tag") " on built-in " (pr-str (:block/title built-in-ent))) @@ -328,3 +362,17 @@ (disallow-tagging-a-built-in-entity db block-eids {:delete? true}) (disallow-node-cant-tag-with-private-tags db block-eids v {:delete? true}) (disallow-removing-page-tag db block-eids v)) + +(defn disallow-editing-private-built-in-nodes + "Disallow editing private :built-in nodes. This explicit validation is needed for contexts + like CLI and API which allow users to edit any built-in entity whereas the app guards this + by not allowing users to navigate to private built-in nodes" + [entities] + (doseq [entity entities] + (when (and (built-in-entity? entity) + ;; This also checks private status of non-page ents like ents with :kv/value + (ldb/private-built-in-page? entity)) + (throw (ex-info "Built-in private nodes can't be modified" + {:type :notification + :payload {:message "Built-in private nodes can't be modified" + :type :error}}))))) diff --git a/deps/outliner/test/logseq/outliner/core_test.cljs b/deps/outliner/test/logseq/outliner/core_test.cljs index 276f144829..30ed0fa6b6 100644 --- a/deps/outliner/test/logseq/outliner/core_test.cljs +++ b/deps/outliner/test/logseq/outliner/core_test.cljs @@ -3,7 +3,8 @@ [datascript.core :as d] [logseq.db :as ldb] [logseq.db.test.helper :as db-test] - [logseq.outliner.core :as outliner-core])) + [logseq.outliner.core :as outliner-core] + [logseq.common.config :as common-config])) (deftest test-delete-block-with-default-property (testing "Delete block with default property hard retracts the block subtree" @@ -50,3 +51,49 @@ child' (db-test/find-block-by-content @conn "child")] (is (nil? parent')) (is (nil? child'))))) + +(deftest delete-blocks-rejects-built-in-entities + (let [conn (db-test/create-conn)] + (testing "built-in page is rejected" + (let [recycle-page (ldb/get-page @conn common-config/recycle-page-name)] + (is (true? (:logseq.property/built-in? recycle-page))) + (is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted" + (db-test/silence-stderr (outliner-core/delete-blocks! conn [recycle-page] {})))))) + + (testing "built-in idents that are not a class or property like empty-placeholder are rejected" + (let [placeholder (d/entity @conn :logseq.property/empty-placeholder)] + (is (some? (:block/uuid placeholder))) + (is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted" + (db-test/silence-stderr (outliner-core/delete-blocks! conn [placeholder] {})))))) + + (testing "file entity is rejected" + (let [file (->> (d/datoms @conn :avet :file/path) + first + :e + (d/entity @conn))] + (is (some? (:file/path file))) + (is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted" + (db-test/silence-stderr (outliner-core/delete-blocks! conn [file] {})))))) + + (testing "KV entity is rejected" + (let [kv (d/entity @conn :logseq.kv/db-type)] + (is (some? (:db/id kv))) + (is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted" + (db-test/silence-stderr (outliner-core/delete-blocks! conn [kv] {})))))))) + +(deftest save-block-rejects-built-in-entity + (let [conn (db-test/create-conn) + placeholder (d/entity @conn :logseq.property/description)] + (is (thrown-with-msg? js/Error #"Built-in.*can't be modified" + (db-test/silence-stderr + (outliner-core/save-block! conn {:db/id (:db/id placeholder) :block/title "hacked"})))))) + +(deftest move-blocks-rejects-built-in-entity + (let [conn (db-test/create-conn-with-blocks + [{:page {:block/title "page1"} + :blocks [{:block/title "target"}]}]) + placeholder (d/entity @conn :logseq.property/description) + target (db-test/find-block-by-content @conn "target")] + (is (thrown-with-msg? js/Error #"Built-in.*can't be modified" + (db-test/silence-stderr + (outliner-core/move-blocks! conn [placeholder] target {:sibling? true})))))) \ No newline at end of file diff --git a/deps/outliner/test/logseq/outliner/page_test.cljs b/deps/outliner/test/logseq/outliner/page_test.cljs index 6b977d9413..d5f679dabf 100644 --- a/deps/outliner/test/logseq/outliner/page_test.cljs +++ b/deps/outliner/test/logseq/outliner/page_test.cljs @@ -97,7 +97,19 @@ js/Error #"can't include \"/" (outliner-page/create! conn "foo/bar" {})) - "Page can't have '/'n title"))) + "Page can't have '/'n title") + + (is (thrown-with-msg? + js/Error + #"can't include \"#\"" + (outliner-page/create! conn "foo#bar" {})) + "Page can't have '#' in title") + + (is (thrown-with-msg? + js/Error + #"can't include \"#\"" + (outliner-page/create! conn "#tagstyle" {})) + "Page can't have leading '#' in title"))) (deftest delete-page (let [conn (db-test/create-conn-with-blocks diff --git a/deps/outliner/test/logseq/outliner/property_test.cljs b/deps/outliner/test/logseq/outliner/property_test.cljs index 336d682961..c0548df578 100644 --- a/deps/outliner/test/logseq/outliner/property_test.cljs +++ b/deps/outliner/test/logseq/outliner/property_test.cljs @@ -45,6 +45,20 @@ (:block/title (d/entity @conn :user.property/p1-2))) "3rd property gets unique ident")))) +(deftest upsert-property-rejects-type-change-with-existing-data + (testing "Changing type is rejected when property has values" + (let [conn (db-test/create-conn-with-blocks + [{:page {:block/title "page1"} + :blocks [{:block/title "b1" :build/properties {:status "active"}}]}])] + (is (thrown-with-msg? + js/Error #"Disallowed type change with existing data" + (outliner-property/upsert-property! conn :user.property/status {:logseq.property/type :number} {}))))) + + (testing "Changing type is allowed when property has no values" + (let [conn (db-test/create-conn-with-blocks {:properties {:empty-prop {:logseq.property/type :default}}})] + (outliner-property/upsert-property! conn :user.property/empty-prop {:logseq.property/type :number} {}) + (is (= :number (:logseq.property/type (d/entity @conn :user.property/empty-prop))))))) + (deftest convert-property-input-string (testing "Convert property input string according to its schema type" (let [test-uuid (random-uuid)] @@ -185,17 +199,68 @@ (is (nil? (:user.property/default updated-block)) "Block property is deleted"))) (deftest batch-set-property! - (let [conn (db-test/create-conn-with-blocks - [{:page {:block/title "page1"} - :blocks [{:block/title "item 1"} - {:block/title "item 2"}]}]) - block-ids (map #(-> (db-test/find-block-by-content @conn %) :block/uuid) ["item 1" "item 2"]) - _ (outliner-property/batch-set-property! conn block-ids :logseq.property/order-list-type "number") - updated-blocks (map #(db-test/find-block-by-content @conn %) ["item 1" "item 2"])] - (is (= ["number" "number"] - (map #(db-property/property-value-content (:logseq.property/order-list-type %)) - updated-blocks)) - "Property values are batch set"))) + (testing "Set built-in property values for multiple blocks" + (let [conn (db-test/create-conn-with-blocks + [{:page {:block/title "page1"} + :blocks [{:block/title "item 1"} + {:block/title "item 2"}]}]) + block-ids (map #(-> (db-test/find-block-by-content @conn %) :block/uuid) ["item 1" "item 2"]) + _ (outliner-property/batch-set-property! conn block-ids :logseq.property/order-list-type "number") + updated-blocks (map #(db-test/find-block-by-content @conn %) ["item 1" "item 2"])] + (is (= ["number" "number"] + (map #(db-property/property-value-content (:logseq.property/order-list-type %)) + updated-blocks)) + "Property values are batch set"))) + + (testing "Set custom default-many property values from string vector" + (let [conn (db-test/create-conn-with-blocks + {:properties {:user.property/reproducible-steps {:logseq.property/type :default + :db/cardinality :db.cardinality/many + :logseq.property/public? true}} + :pages-and-blocks [{:page {:block/title "page1"} + :blocks [{:block/title "target"}]}]}) + block-id (:block/uuid (db-test/find-block-by-content @conn "target"))] + (outliner-property/batch-set-property! conn [block-id] :user.property/reproducible-steps ["Step 1" "Step 2" "Step 3"]) + (is (= #{"Step 1" "Step 2" "Step 3"} + (->> (:user.property/reproducible-steps (d/entity @conn [:block/uuid block-id])) + (map db-property/property-value-content) + set)) + "String vector values are persisted as many property values"))) + + (testing "Set custom default-many property values from id vector" + (let [conn (db-test/create-conn-with-blocks + {:properties {:user.property/reproducible-steps {:logseq.property/type :default + :db/cardinality :db.cardinality/many + :logseq.property/public? true}} + :pages-and-blocks [{:page {:block/title "page1"} + :blocks [{:block/title "source" + :build/properties {:user.property/reproducible-steps #{"Step 1" "Step 2" "Step 3"}}} + {:block/title "target"}]}]}) + source-values (->> (:user.property/reproducible-steps (db-test/find-block-by-content @conn "source")) + (map :db/id) + vec) + target-id (:block/uuid (db-test/find-block-by-content @conn "target"))] + (outliner-property/batch-set-property! conn [target-id] :user.property/reproducible-steps source-values) + (is (= #{"Step 1" "Step 2" "Step 3"} + (->> (:user.property/reproducible-steps (d/entity @conn [:block/uuid target-id])) + (map db-property/property-value-content) + set)) + "Id vector values are persisted as many property values"))) + + (testing "Invalid many values throw and don't partially persist" + (let [conn (db-test/create-conn-with-blocks + {:properties {:user.property/reproducible-steps {:logseq.property/type :default + :db/cardinality :db.cardinality/many + :logseq.property/public? true}} + :pages-and-blocks [{:page {:block/title "page1"} + :blocks [{:block/title "target"}]}]}) + block-id (:block/uuid (db-test/find-block-by-content @conn "target"))] + (is (thrown-with-msg? + js/Error + #"Schema validation failed" + (outliner-property/batch-set-property! conn [block-id] :user.property/reproducible-steps [999999]))) + (is (nil? (:user.property/reproducible-steps (d/entity @conn [:block/uuid block-id]))) + "No partial values are persisted on failure")))) (deftest status-property-setting-classes (let [conn (db-test/create-conn-with-blocks @@ -223,6 +288,14 @@ (mapv :db/ident (:block/tags (d/entity @conn [:block/uuid project])))) "Doesn't add Task to block when it is already tagged"))) +(deftest batch-set-property-rejects-private-built-in-entity + (let [conn (db-test/create-conn) + placeholder (d/entity @conn :logseq.property/empty-placeholder) + prop (d/entity @conn :logseq.property/description)] + (is (thrown-with-msg? js/Error #"Built-in.*can't be modified" + (db-test/silence-stderr + (outliner-property/batch-set-property! conn [(:db/id placeholder)] (:db/ident prop) "hacked")))))) + (deftest batch-remove-property! (let [conn (db-test/create-conn-with-blocks {:classes {:C1 {}} @@ -247,6 +320,14 @@ #"Can't remove required" (outliner-property/batch-remove-property! conn [(:db/id (d/entity @conn :user.class/C1))] :logseq.property.class/extends))))) +(deftest batch-remove-property-rejects-private-built-in-entity + (let [conn (db-test/create-conn) + placeholder (d/entity @conn :logseq.property/empty-placeholder) + prop (d/entity @conn :logseq.property/description)] + (is (thrown-with-msg? js/Error #"Built-in.*can't be modified" + (db-test/silence-stderr + (outliner-property/batch-remove-property! conn [(:db/id placeholder)] (:db/ident prop))))))) + (deftest add-existing-values-to-closed-values! (let [conn (db-test/create-conn-with-blocks [{:page {:block/title "page1"} @@ -379,4 +460,4 @@ (:db/id (d/entity @conn :user.class/C1))) (is (= [:logseq.class/Root] (:logseq.property.class/extends (db-test/readable-properties (d/entity @conn :user.class/C3)))) - "Extends property is restored back to Root"))) + "Extends property is restored back to Root"))) \ No newline at end of file diff --git a/deps/outliner/test/logseq/outliner/validate_test.cljs b/deps/outliner/test/logseq/outliner/validate_test.cljs index 1334c5799b..6d0590a41c 100644 --- a/deps/outliner/test/logseq/outliner/validate_test.cljs +++ b/deps/outliner/test/logseq/outliner/validate_test.cljs @@ -248,6 +248,27 @@ (outliner-validate/validate-tags-property-deletion @conn [(:db/id page)] :logseq.class/Page)) "Page without parent can't remove #Page"))) +(deftest validate-editing-built-in-property + (let [conn (db-test/create-conn-with-blocks + {:properties {:myprop {:logseq.property/type :default}}}) + user-prop (d/entity @conn :user.property/myprop) + built-in-prop (d/entity @conn :block/tags)] + + (testing "user property can edit any attribute" + (is (nil? (outliner-validate/validate-editing-built-in-property + user-prop {:db/cardinality :db.cardinality/many})))) + + (testing "built-in property cannot edit disallowed attribute" + (is (thrown-with-msg? + js/Error + #"not editable" + (outliner-validate/validate-editing-built-in-property + built-in-prop {:block/title "renamed"})))) + + (testing "built-in property can edit allowed attribute" + (is (nil? (outliner-validate/validate-editing-built-in-property + built-in-prop {:logseq.property/hide-empty-value true})))))) + ;; Try as many of the validations against a new graph to confirm ;; that validations make sense and are valid for a new graph (deftest new-graph-should-be-valid diff --git a/dist/logseq.js b/dist/logseq.js new file mode 100755 index 0000000000..28b93e441c --- /dev/null +++ b/dist/logseq.js @@ -0,0 +1,6 @@ +#!/usr/bin/env node +"use strict"; + +const path = require("path"); + +require(path.resolve(__dirname, "../static/logseq-cli.js")); diff --git a/docs/agent-guide/001-logseq-cli.md b/docs/agent-guide/001-logseq-cli.md new file mode 100644 index 0000000000..0be698f64c --- /dev/null +++ b/docs/agent-guide/001-logseq-cli.md @@ -0,0 +1,166 @@ +# Logseq CLI Implementation Plan + +Goal: Build a new Logseq CLI in ClojureScript that runs on Node.js and connects to the db-worker-node server. + +Architecture: The CLI is a Node-targeted ClojureScript program built via shadow-cljs and packaged with a small JavaScript launcher. +The CLI speaks a simple request and response protocol to the existing db-worker-node HTTP or WebSocket API and exposes high-level subcommands for users. + +Tech Stack: ClojureScript, shadow-cljs :node-script target, Node.js runtime, existing db-worker-node server. + +Related: Relates to docs/agent-guide/task--basic-logseq-cli.md and docs/agent-guide/task--db-worker-nodejs-compatible.md. + +## Problem statement + +We need a new Logseq CLI that is independent of any existing CLI code in the repo. +The CLI must run in Node.js, be written in ClojureScript, and connect to the db-worker-node server started from dist/db-worker-node.js. +The CLI should provide a stable interface for scripting and troubleshooting, and it should be easy to extend with new commands. + +## Testing Plan + +I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple graph/content request. +I will add unit tests for command parsing, configuration precedence, and error formatting. +I will add unit tests for the client transport layer to ensure timeouts behave correctly. +I will add unit tests for new graph/content commands (parsing, validation, and request mapping). +I will add integration tests for graph lifecycle commands and content commands against a real db-worker-node. +I will follow @test-driven-development for all behavior changes. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +The CLI is a Node program that parses flags, loads config, and sends requests to db-worker-node. +The db-worker-node server is already built from the :db-worker-node shadow-cljs target and listens on a random localhost TCP port recorded in the lock file. + +ASCII diagram: + ++--------------+ HTTP or WS +---------------------+ +| logseq-cli | -----------------------> | db-worker-node | +| node script | <----------------------- | server on random port | ++--------------+ +---------------------+ + +## Assumptions + +The db-worker-node server exposes a stable API for a small set of requests needed by the CLI. +The CLI always uses localhost and discovers the server port from the lock file. +The CLI will use JSON for request and response bodies for ease of scripting. + +## Implementation plan + +1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps. +2. Read @test-driven-development guidelines and confirm the red phase will include all CLI tests first. +3. Identify existing db-worker-node request handlers and document their request and response shapes. +4. Define the initial CLI command surface as a table that includes command, input, output, and errors. +5. Decide on transport protocol based on db-worker-node capabilities and document the selection. +6. Add a new shadow-cljs build target named :logseq-cli with :target :node-script and a dedicated output file in static/. +7. Create a new namespace for the CLI entrypoint in src/main/cli/main.cljs and wire it as the :main for the build. +8. Create src/main/cli/config.cljs with config resolution order of CLI flags, env vars, then config file. +9. Create src/main/cli/transport.cljs with a small client that can send requests and parse responses. +10. Create src/main/cli/commands.cljs with pure functions that map parsed args to transport requests. +11. Create src/main/cli/format.cljs that formats success and error output for human and machine usage. +12. Add unit tests in src/test/logseq/cli for config precedence, command parsing, and error formatting behavior. +13. Add integration tests in src/test/logseq/cli that start db-worker-node and invoke the CLI entrypoint. +14. Run tests in red phase with bb dev:test -v and confirm failures are behavior-related. +15. Implement the minimal code to make the tests pass and re-run in green phase. +16. Refactor for naming and reuse while keeping tests green. +17. Document how to build and run the CLI in a short section in README.md. + +## Current status (2026-01-14) + +Implemented: +- CLI build target, entrypoint, config resolution, transport, formatting, and command wiring. +- Graph commands: list/create/switch/remove/validate/info. +- Content commands: add/remove/search/tree. +- Unit tests for config/commands/format/transport and integration tests for graph/content commands. +- CLI docs moved to `docs/cli/logseq-cli.md` and linked from README. + +Not fully aligned with plan: +- Red-first TDD sequence was not strictly followed (some tests added after initial implementation). +- README section was replaced by a link to the dedicated doc. +- `search` currently queries `:block/title` only (no page name/content search). + +Open follow-ups (optional): +- Expand `search` to include page name/content and update tests. +- Add any additional graph metadata to `graph-info` beyond `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version`. + +## Command surface definition + +| Command | Input | Output | Errors | +| --- | --- | --- | --- | +| graph-list | none | list of graphs | server unavailable, timeout | +| graph-create | graph name | created graph + set current graph | invalid name, server unavailable | +| graph-switch | graph name | switched graph + set current graph | missing graph, server unavailable | +| graph-remove | graph name | removal confirmation | missing graph, server unavailable | +| graph-validate | graph name or current graph | validation result | missing graph, server unavailable | +| graph-info | graph name or current graph | graph metadata/info | missing graph, server unavailable | +| add | block/page payload | created block IDs | invalid input, server unavailable | +| remove | block/page id or name | removal confirmation | invalid input, server unavailable | +| search | text query | matched blocks/pages | invalid input, server unavailable | +| tree | block/page id or name | hierarchical tree output | invalid input, server unavailable | + +## Edge cases + +The db-worker-node server is not running or the lock file points to a stale server. +The response payload is invalid JSON or missing fields. +The request times out or the server closes the connection early. +The user passes incompatible flags or unknown commands. +The CLI is run on Windows where path and quoting rules differ. +Graph commands are invoked without a current graph configured. +Content commands are invoked without specifying a graph and no current graph is set. +Content commands refer to missing pages/blocks. +Graph removal is attempted while a graph is open. + +## Testing commands and expected output + +Run a single unit test in red phase. + +```bash +bb dev:test -v 'logseq.cli.config-test/test-config-precedence' +``` + +Expected output includes a failing assertion and ends with a non-zero exit code. + +Run the full unit test suite in green phase. + +```bash +bb dev:test -v 'logseq.cli.commands-test' +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and unit tests when all work is complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +I will add behavior-driven tests that verify the CLI connects to a real db-worker-node process and that each command returns the expected output for valid input. +I will keep unit tests focused on pure functions like parsing, formatting, and config resolution, and avoid mocking internal implementation details. + +## Implementation Details + +- Add a new shadow-cljs build target for the CLI with a node-script output in static/. +- Create a dedicated CLI entrypoint namespace that handles args, logging, and exit codes. +- Implement config resolution for flags, env vars, and optional config file. +- Implement a transport client with timeouts and explicit error mapping. +- Define a small command map with functions that return request objects and output renderers. +- Add structured JSON output mode for scripting alongside human-readable output. +- Ensure the CLI exits with non-zero status codes on errors. +- Document build and run steps, including starting db-worker-node first. +- Add graph management commands that map to db-worker thread-apis. +- Add graph content commands (add/remove/search/tree) with clear input formats and output. +- Persist/resolve a “current graph” for commands that default to current context. + +## Question + +Which exact db-worker-node endpoints and request schemas should the CLI use for graph/content commands. +- Answer: all thread-apis are available in http endpoint, check @src/main/frontend/worker/db_worker_node.cljs + +Do we want WebSocket or HTTP as the default transport for the CLI. +- HTTP + +Can I consult the clojure-expert and research-agent agents for architecture and reference implementations as required by the planning guidelines. +- yes +--- diff --git a/docs/agent-guide/002-logseq-cli-subcommands.md b/docs/agent-guide/002-logseq-cli-subcommands.md new file mode 100644 index 0000000000..8e14f38fd0 --- /dev/null +++ b/docs/agent-guide/002-logseq-cli-subcommands.md @@ -0,0 +1,165 @@ +# Logseq CLI Subcommands Implementation Plan + +Goal: Replace the CLI argument parser with babashka/cli and expose every command as a subcommand with consistent help and output formats. + +Architecture: The CLI remains a Node-targeted ClojureScript program built via shadow-cljs, but command parsing moves to babashka/cli with an explicit subcommand map. The CLI entrypoint will delegate to per-subcommand parsers and handlers that return a consistent result envelope that is rendered to human, JSON, or EDN output. + +Tech Stack: ClojureScript, babashka/cli, shadow-cljs, Node.js runtime, db-worker-node HTTP API. + +Related: Builds on docs/agent-guide/001-logseq-cli.md. + +## Problem statement + +The current CLI uses clojure.tools.cli with a flat flag set and manual command detection. +This limits help text, makes subcommand-specific options awkward, and complicates output formatting consistency. +We need to migrate to babashka/cli so that each command is a first-class subcommand with its own help, and so output formats are consistent across all commands. + +## Testing Plan + +I will add unit tests that validate babashka/cli subcommand parsing for every command and its flags. +I will add unit tests that assert each subcommand renders help and that top-level help includes all subcommands. +I will add unit tests that verify output formatting for human, JSON, and EDN across success and error paths for each subcommand. +I will add integration tests that invoke the Node CLI with subcommands and verify consistent output formats for graph and content commands. +NOTE: I will write all tests before I add any implementation behavior. + +## Architecture sketch + ++--------------+ HTTP +---------------------+ +| logseq-cli | -----------------> | db-worker-node | +| node script | <----------------- | server on random port | ++--------------+ +---------------------+ + +## Command and output surface + +The CLI will expose these subcommands and shared output controls. + +| Subcommand | Purpose | Output formats | Notes | +| --- | --- | --- | --- | +| graph list | List graphs | human, json, edn | Replaces graph-list | +| graph create | Create graph | human, json, edn | Replaces graph-create | +| graph switch | Switch current graph | human, json, edn | Replaces graph-switch | +| graph remove | Remove graph | human, json, edn | Replaces graph-remove | +| graph validate | Validate graph | human, json, edn | Replaces graph-validate | +| graph info | Graph metadata | human, json, edn | Replaces graph-info | +| block add | Add blocks | human, json, edn | Replaces add | +| block remove | Remove block or page | human, json, edn | Replaces remove | +| search | Search graph | human, json, edn | Replaces search | +| block tree | Show tree | human, json, edn | Replaces tree | + +The plan assumes a single global output flag that defaults to human, and each subcommand may also accept it. + +## Subcommand map design + +Global options apply to all subcommands and are parsed before subcommand options. + +| Option | Purpose | Notes | +| --- | --- | --- | +| --help | Show help | Available at top level and per subcommand. | +| --version | Show version | Prints build time and revision. | +| --config PATH | Config file path | Defaults to ~/logseq/cli.edn. | +| --graph GRAPH | Graph name | Used as current graph. | +| --timeout-ms MS | Request timeout | Integer milliseconds. | +| --output FORMAT | Output format | One of human, json, edn. | + +Each subcommand uses a nested path and its own options. + +| Subcommand path | Required args | Options | Notes | +| --- | --- | --- | --- | +| graph list | none | --output | Lists all graphs. | +| graph create | none | --graph GRAPH, --output | Creates and switches graph. | +| graph switch | none | --graph GRAPH, --output | Switches current graph. | +| graph remove | none | --graph GRAPH, --output | Removes graph. | +| graph validate | none | --graph GRAPH, --output | Validates graph. | +| graph info | none | --graph GRAPH, --output | Shows metadata, defaults to config repo if omitted. | +| block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. | +| block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. | +| search | QUERY | --type page|block|tag|property|all, --tag NAME, --case-sensitive, --sort updated-at|created-at, --order asc|desc, --output | Search text is positional and required. Human output columns: ID (db/id), TITLE. Block reference UUIDs in text are resolved recursively up to 10 levels. | +| block tree | none | --block UUID, --page PAGE, --format FORMAT, --output | One of block or page is required, and format controls tree rendering. | + +## Plan + +1. Consult the clojure-expert agent about babashka/cli idioms for nested subcommands and help generation. +2. Consult the research-agent for a reference implementation of babashka/cli subcommand parsing in ClojureScript, including Node usage. +3. Review current CLI documentation in docs/cli/logseq-cli.md and list all existing flags and examples that must be preserved. +4. Review the current parser and action mapping in src/main/logseq/cli/commands.cljs and list which options are command-specific versus global. +5. Create a babashka/cli command map design and capture it in this document as a table of subcommands, arguments, and defaults. +6. Write new unit tests for top-level help output in src/test/logseq/cli/commands_test.cljs that assert subcommand listing and usage text. +7. Write new unit tests for each subcommand parse path in src/test/logseq/cli/commands_test.cljs covering required args, missing args, and unknown flags. +8. Write new unit tests in src/test/logseq/cli/format_test.cljs that assert human, json, and edn output for success and error results. +9. Write new unit tests in src/test/logseq/cli/config_test.cljs for output format precedence between flags, env, and config file. +10. Write new integration tests in src/test/logseq/cli/integration_test.cljs that invoke the built CLI with subcommands and verify outputs for at least one graph and one block command in each format. +11. Run the new tests to confirm they fail for the current parser and output handling. +12. Replace the parser in src/main/logseq/cli/commands.cljs with babashka/cli, using a subcommand map and per-command option specs. +13. Update src/main/logseq/cli/main.cljs to route to babashka/cli and return subcommand-specific help when requested. +14. Update src/main/logseq/cli/config.cljs to add a unified output format option and ensure json and edn are both supported. +15. Update src/main/logseq/cli/format.cljs so that all commands emit consistent human, json, or edn output using a single option path. +16. Update docs/cli/logseq-cli.md to document subcommands, shared output flags, and per-subcommand help examples. +17. Run the unit test suite with bb dev:test -v 'logseq.cli.commands-test' and confirm 0 failures and 0 errors. +18. Run lint and tests with bb dev:lint-and-test and confirm a zero exit code. +19. Refactor for naming clarity, shared helpers, and reduced duplication while keeping tests green. + +## Status + +- Completed: Plan tasks 1-10, 12-19. +- Skipped: Plan task 11 (red-phase confirmation no longer applicable after parser swap). + +## Edge cases + +Missing subcommand should show top-level help with a non-zero exit code. +Unknown subcommands should show a helpful error that includes the available subcommands. +Subcommand-specific help should not require a working db-worker-node server. +Output format flags should be accepted both at the top level and at subcommand level without conflict. +Existing config keys such as :output-format and the legacy --json flag should either be preserved or mapped with a clear deprecation path. +Windows quoting should be covered for block add subcommand with multi-word content arguments. + +## Testing commands and expected output + +Run a single failing unit test in red phase. + +```bash +bb dev:test -v 'logseq.cli.commands-test/test-help-output' +``` + +Expected output includes a failing assertion about subcommand help text and ends with a non-zero exit code. + +Run the full unit test suite in green phase. + +```bash +bb dev:test -r logseq.cli.* +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and unit tests when all work is complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +The unit tests will exercise parsing and output formatting behavior without mocking internal parser details. +The integration tests will start db-worker-node on a random port and invoke the CLI entrypoint with subcommands to verify end-to-end behavior. + +## Implementation Details + +- Replace clojure.tools.cli usage with babashka/cli and define a nested subcommand map for graph and block groups. +- Keep global options for server connection and output format and merge them with per-subcommand options. +- Normalize output format selection to :human, :json, or :edn and route it through a single formatting function. +- Preserve config precedence across flags, env vars, and config file while adding the output format option. +- Ensure each subcommand has a help string and usage text generated by babashka/cli. +- Keep error envelopes consistent with current :status and :error keys to avoid breaking existing scripts. +- Update CLI docs to show subcommand usage and output format examples. +- Add a transition note for legacy command names if backward compatibility is required. + +## Question + +Should we keep backwards compatibility for legacy command names like graph-list and add, or require the new subcommand forms only. +- Answer: No need to keep backwards compatibility +Should we retain the --json flag as an alias for --output json or remove it after a deprecation period. +- Answer: remove --json, only keep --output +Do we want --output edn and --output json to be accepted at both the top level and per-subcommand level. +- Answer: yes, accept at both levels +--- diff --git a/docs/agent-guide/003-db-worker-node-cli-orchestration.md b/docs/agent-guide/003-db-worker-node-cli-orchestration.md new file mode 100644 index 0000000000..53fe6268d9 --- /dev/null +++ b/docs/agent-guide/003-db-worker-node-cli-orchestration.md @@ -0,0 +1,115 @@ +# db-worker-node and logseq-cli Orchestration Plan + +Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, refactor db-worker-node to be repo-bound with locking, make logseq-cli fully manage db-worker-node lifecycle, and add server subcommands for management. + +## Background and Current State (from existing code) + +- `db-worker-node` currently accepts `--repo` but it is optional; it can open/switch graphs via `thread-api/create-or-open-db` at runtime. Entrypoint: `src/main/frontend/worker/db_worker_node.cljs`. +- `logseq-cli` connects to an existing db-worker-node on localhost using the port recorded in the lock file; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`. +- Tests explicitly start db-worker-node (`src/test/logseq/cli/integration_test.cljs`, `src/test/frontend/worker/db_worker_node_test.cljs`). + +## Requirements + +1. Refactor `db-worker-node`: startup must require `--repo`; on startup it must open or create that graph; it must not switch graphs at runtime; it must create a lock file so a graph can be served by only one db-worker-node instance; it only needs to bind to localhost. +2. In `logseq-cli`, all commands requiring `--graph` or any graph operations must connect to or create the corresponding db-worker-node server. +3. db-worker-node server must not be started manually; logseq-cli is fully responsible. +4. Add `server` subcommand(s) to logseq-cli for managing db-worker-node servers. + +## Scope / Non-goals + +- In scope: db-worker-node startup and lifecycle, repo binding enforcement, lock files, CLI server orchestration, management commands, tests/docs. +- Out of scope: changing db-worker core query/write protocol; changing db-worker-node HTTP API semantics beyond repo binding constraints. + +## Proposed Design + +- **Repo-bound server**: db-worker-node opens a single repo at startup and refuses repo changes for the lifetime of the process. It only listens on localhost. +- **Lock file**: each repo directory has a lock file to ensure one server per graph. Lock contains metadata for status/cleanup; db-worker-node handles it by default, and logseq-cli handles cases db-worker-node cannot. +- **CLI orchestration**: logseq-cli discovers/starts/reuses db-worker-node servers per repo. It is the only entrypoint for starting servers. +- **Server subcommands**: add `server list|status|start|stop|restart` (or similar) to manage servers explicitly. + +## Detailed Design + +### 1) db-worker-node: required repo, repo binding, lock file + +Files: +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/platform/node.cljs` (for data-dir / repo dir resolution) +- Optional new helper: `src/main/frontend/worker/db_worker_node_lock.cljs` + +Key changes: +- **Argument parsing**: `--repo` becomes required. If missing, print help and exit non-zero. Host binding is restricted to localhost (e.g., `127.0.0.1`) regardless of input. +- **Startup flow**: replace `/db-worker.lock`). + - Content: JSON `{repo, pid, host, port, startedAt}`. + - Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”. + - Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM. + - Stale lock: if lock exists but pid is dead, allow replacement (db-worker-node first; CLI can repair when server cannot). + +### 2) logseq-cli: auto start/reuse db-worker-node per repo + +Files: +- `src/main/logseq/cli/commands.cljs` +- `src/main/logseq/cli/main.cljs` +- `src/main/logseq/cli/config.cljs` +- New: `src/main/logseq/cli/server.cljs` (process management + lock handling) + +Key changes: +- **Repo resolution**: for all graph/content commands, require `--graph` or resolved repo from config; otherwise error. +- **Ensure server** (new helper `ensure-server!`): + 1. Derive data-dir, repo dir, and lock file path from repo. + 2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`. + 3. If healthy, reuse existing server; build the connection URL from localhost and the lock file port. + 4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries. + 5. Spawn via `child_process.spawn`: `./dist/db-worker-node.js --repo --data-dir <...>`. + 6. Resolve actual port from the lock file written by db-worker-node. +- **Connection URL**: derived from the repo lock file; host is always localhost and the port is always discovered from the lock file. + +### 3) CLI `server` subcommands + +Suggested command group: +- `server list`: list servers from lock files (repo, pid, port, status). +- `server start --graph `: start server for repo. +- `server stop --graph `: stop server (SIGTERM or `/v1/shutdown`). +- `server restart --graph `: stop + start. + +Implementation notes: +- `start|stop|restart` require `--graph`. +- `list` scans data-dir for repo directories, reads lock files, and verifies status. +- Consider adding `/v1/shutdown` in db-worker-node for graceful stop. + +## Compatibility / Migration + +- No need to preserve compatibility for existing env vars, config keys, or flags related to db-worker-node or CLI server connectivity; remove them if they are no longer needed. + +## Test Plan + +- **Unit tests**: + - CLI: repo resolution, server orchestration logic, lock parsing, error codes (`src/test/logseq/cli/*`). + - db-worker-node: repo required, repo mismatch rejection, lock file create/cleanup (`src/test/frontend/worker/db_worker_node_test.cljs`). +- **Integration tests**: + - CLI runs graph/content commands without manual server startup (`src/test/logseq/cli/integration_test.cljs`). + - Concurrent startup of same repo must fail due to lock. + +## Milestones + +1. **db-worker-node binding & lock file**: repo required + repo enforcement + lock creation/cleanup. +2. **CLI server module**: `ensure-server!` with lock/health checks and spawning. +3. **CLI command updates**: graph/content commands require repo and auto-start server; add `server` subcommands. +4. **Tests + docs**: update/extend tests and adjust CLI docs (`docs/cli/logseq-cli.md`). + +## Open Questions + +1. Should `graph list` require `--graph`? If not, define a “global” server or out-of-band access to data-dir. + - Answer: No --graph needed, using 'out-of-band access to data-dir' way +2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions). + - lockfile name:`db-worker.lock`, + - Location: inside repo dir (e.g. `~/logseq/cli-graphs//db-worker.lock`). + - only consider linux/macos for now +3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle. +4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI? +5. db-worker-node servers should keep running unless `logseq-cli server stop` is invoked or the process exits unexpectedly; in the latter case, CLI should handle lockfile cleanup on restart. diff --git a/docs/agent-guide/004-logseq-cli-verb-subcommands.md b/docs/agent-guide/004-logseq-cli-verb-subcommands.md new file mode 100644 index 0000000000..44fad77d27 --- /dev/null +++ b/docs/agent-guide/004-logseq-cli-verb-subcommands.md @@ -0,0 +1,218 @@ +# Logseq CLI Verb-First Subcommands Implementation Plan + +Goal: Refactor logseq-cli to remove the block subcommand group and replace it with verb-first subcommands for list, add, remove, search, and show. + +Architecture: Keep the babashka/cli dispatch table but reorganize it into verb-first subcommands, and route each verb to typed actions for pages, blocks, tags, and properties through existing db-worker-node thread-apis. + +Tech Stack: ClojureScript, babashka/cli, db-worker-node thread-apis, Datascript queries. + +Related: Builds on docs/agent-guide/003-db-worker-node-cli-orchestration.md and docs/agent-guide/002-logseq-cli-subcommands.md. + +## Problem statement + +The current CLI uses a block subcommand group, which makes the interface noun-first and inconsistent with graph and server commands. + +We need to make the CLI verb-first, so that content operations are consistent with other tooling and easier to extend for new resource types. + +The refactor must preserve existing behaviors for add, remove, search, and tree while adding list commands for pages, tags, and properties, and renaming tree to show. + +## Testing Plan + +I will add unit tests that cover verb-first parsing, group help output, and option validation for list, add, remove, search, and tree. + +I will add unit tests that assert list option parsing for each list subtype and that invalid flag combinations are rejected. + +I will add integration tests that run list, add, remove, search, and tree against a real db-worker-node with a test graph and assert output shapes. + +I will follow @test-driven-development for all behavior changes. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Command surface + +The block subcommand group is removed and replaced with verb-first subcommands. + +Help output groups commands into two sections, with Graph Inspect and Edit first, and Graph Management last. + +Group names and order: + +| Group | Commands | Order | +| --- | --- | --- | +| Graph Inspect and Edit | list, add, remove, search, show | First | +| Graph Management | graph, server | Last | + +| Command | Subcommand | Purpose | +| --- | --- | --- | +| list | page | List pages | +| list | tag | List tags | +| list | property | List properties | +| add | block | Add blocks | +| add | page | Create page | +| remove | block | Remove block | +| remove | page | Remove page | +| search | none | Search across resources | +| show | none | Show block tree | + +Global options remain unchanged and are shared across all commands. + +## List options detail + +The list command should expose a consistent shape across resource types. + +Common list options: + +| Option | Applies to | Purpose | Notes | +| --- | --- | --- | --- | +| --expand | page, tag, property | Include expanded metadata | Maps to existing api-list-* expand behavior. | +| --limit N | page, tag, property | Limit results | Implemented in CLI after fetch unless server supports it. | +| --offset N | page, tag, property | Offset results | Implemented in CLI after fetch unless server supports it. | +| --sort FIELD | page, tag, property | Sort results | Field whitelist per type. | +| --order asc|desc | page, tag, property | Sort direction | Defaults to desc. | +| --output FORMAT | all | Output format | Existing output handling. | + +List page options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-journal | Include journal pages | Default is include all. | +| --journal-only | Only journal pages | Requires journal detection in api-list-pages. | +| --include-hidden | Include hidden pages | Requires a flag to bypass entity-util/hidden? filtering. | +| --updated-after ISO8601 | Filter by updated-at | Compare to :block/updated-at. | +| --created-after ISO8601 | Filter by created-at | Compare to :block/created-at. | +| --fields FIELD,FIELD | Select output fields | | + +List tag options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-built-in | Include built-in classes | Built-in tags are currently included by default, clarify behavior. | +| --with-properties | Include class properties | Uses :logseq.property.class/properties when expanded. | +| --with-extends | Include class extends | Uses :logseq.property.class/extends when expanded. | +| --fields FIELD,FIELD | Select output fields | | + +List property options: + +| Option | Purpose | Notes | +| --- | --- | --- | +| --include-built-in | Include built-in properties | Built-in properties are currently included by default, clarify behavior. | +| --with-classes | Include property classes | Uses :logseq.property/classes when expanded. | +| --with-type | Include property type | Uses :logseq.property/type. | +| --fields FIELD,FIELD | Select output fields | | + +List block is removed to avoid overlap with search. + +## Search options detail + +Search has no subcommands and searches across pages, blocks, tags, and properties by default. Human output columns are `ID` (db/id) and `TITLE`. +Search and show outputs resolve block reference UUIDs in text. Nested references are resolved recursively up to 10 levels (e.g., `[[]]` → `[[some text [[]]]]`, then `` is also replaced). + +| Option | Purpose | Notes | +| --- | --- | --- | +| QUERY | Search text | Required positional argument. | +| --type page|block|tag|property|all | Restrict types | Default is all. | +| --tag NAME | Restrict to a specific tag | Tag is a class page, e.g. Page, Asset, Task. | +| --case-sensitive | Case sensitive search | Default is case-insensitive. | +| --sort updated-at|created-at | Sort results | Default is relevance or stable order. | +| --order asc|desc | Sort direction | Defaults to desc for time sorts. | + +## Tree options detail + +Show has no subcommands and returns the block tree for a page or block. + +| Option | Purpose | Notes | +| --- | --- | --- | +| --id ID | Tree root by :db/id | Mutually exclusive with other identifiers. | +| --uuid UUID | Tree root by :block/uuid | Mutually exclusive with other identifiers. | +| --page-name NAME | Tree root by :block/title for a #Page block | Must be a page. | +| --level N | Limit tree depth | N >= 1, default 10. | +| --format text|json|edn | Output format | Existing behavior. | + +## Plan + +1. Review current CLI command parsing and action routing in src/main/logseq/cli/commands.cljs to map block group behavior to verb-first commands. +2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and show. +3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, and list property. +4. Add failing unit tests that assert search defaults to all types and respects --type and positional text. +5. Add failing unit tests that assert show accepts --page-name, --uuid, or --id and rejects missing targets. +6. Run bb dev:test -v 'logseq.cli.commands-test/test-parse-args' and confirm failures are about the new verbs and options. +7. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group. +8. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last. +9. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface. +10. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags. +11. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties. +12. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs. +13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --order. +14. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level. +15. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show. +16. Run bb dev:test -v 'logseq.cli.integration-test/test-cli-list-and-search' and confirm failures before implementation. +17. Implement behavior for list, add, remove, search, and show until all tests pass. +18. Update docs/cli/logseq-cli.md with new verb-first commands and examples. +19. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors. +20. Run bb dev:lint-and-test and confirm a zero exit code. + +## Edge cases + +Missing subcommand should return group help for list, add, or remove, and still exit with a non-zero status. + +Unknown subcommands should return a helpful error message that lists the valid subcommands. + +Add block should still default to today’s journal page when no page is provided and no parent is provided. + +Search across all types should avoid duplicate hits when a tag or property is also a page with the same title. + +Show should return a deterministic order based on :block/order. + +## Testing commands and expected output + +Run a single unit test in red phase. + +```bash +bb dev:test -v 'logseq.cli.commands-test/test-parse-args' +``` + +Expected output includes failing assertions about the new verb-first commands and ends with a non-zero exit code. + +Run the integration tests in red phase. + +```bash +bb dev:test -v 'logseq.cli.integration-test/test-cli-list-add-search-show-remove' +``` + +Expected output includes failing assertions about list and search output and ends with a non-zero exit code. + +Run the full suite in green phase. + +```bash +bb dev:test -r logseq.cli.* +``` + +Expected output includes 0 failures and 0 errors. + +Run lint and tests after all changes. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +The unit tests will validate parsing, help output, and option validation for each new verb-first command. + +The integration tests will create a temporary graph, add pages, tags, and properties, and verify list, search, and tree output against db-worker-node behavior. + +## Implementation Details + +- Replace block group entries with list, add, remove, search, and show in src/main/logseq/cli/commands.cljs. +- Add list subtype specs and validation, including common list options and per-type field filtering in src/main/logseq/cli/commands.cljs. +- Extend search to combine page, block, tag, and property queries and to enforce --type and --tag behavior in src/main/logseq/cli/commands.cljs. +- Preserve existing add block and remove block behavior while changing only the command paths and option names. +- Rename tree to show and add id, uuid, page-name, and level parsing in src/main/logseq/cli/commands.cljs. +- Update docs/cli/logseq-cli.md to show new usage and examples. + +## Question + +None. + +--- diff --git a/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md new file mode 100644 index 0000000000..c0d041d37c --- /dev/null +++ b/docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md @@ -0,0 +1,160 @@ +# Logseq CLI Output and db-worker-node Log Implementation Plan + +Goal: Improve all logseq-cli human output for clarity and usability, and write db-worker-node logs to //db-worker-node-YYYYMMDD.log with retention of the most recent 7 logs. + +Architecture: Add a dedicated human output formatter with per-command renderers and consistent error messaging in the CLI formatting layer. +Add a file-based glogi appender for db-worker-node that writes logs into the graph-specific data directory using dated filenames and retention while keeping console logging behavior explicit and configurable. + +Tech Stack: ClojureScript, babashka/cli, lambdaisland.glogi, Node.js fs/path. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Human Output Specification + +Target: plain text, no ANSI colors. Each command has a stable layout and ordering. + +| Command | OK output (human) | Empty output | Notes | +| --- | --- | --- | --- | +| graph list | Table with header `GRAPH` and rows of graph names, followed by `Count: N` | Header + `Count: 0` | Data from `{:graphs [...]}` | +| graph create | `Graph created: ` | n/a | Use graph name from action/options | +| graph switch | `Graph switched: ` | n/a | Use graph name from action/options | +| graph remove | `Graph removed: ` | n/a | Use graph name from action/options | +| graph validate | `Graph validated: ` | n/a | Use graph name from action/options | +| graph info | Lines: `Graph: `, `Created at: `, `Schema version: ` | n/a | Use `:logseq.kv/*` data; show `-` if missing; `Created at` should use the same human-friendly relative format as list outputs | +| server list | Table with header `REPO STATUS HOST PORT PID`, rows for servers, followed by `Count: N` | Header + `Count: 0` | Data from `{:servers [...]}` | +| server status/start/stop/restart | `Server : ` + details line `Host: Port: ` when available | n/a | Use `:status` keyword where present | +| list page/tag/property | Table with header (fields vary by command) and rows, followed by `Count: N` | Header + `Count: 0` | Defaults: page/tag/property `ID TITLE UPDATED-AT CREATED-AT` (ID uses `:db/id`); if `:db/ident` present, include `IDENT` column | +| add block | `Added blocks: (repo: )` | n/a | Count = number of blocks submitted | +| add page | `Added page: (repo: )` | n/a | | +| remove block | `Removed block: (repo: )` | n/a | Prefer UUID if available | +| remove page | `Removed page: (repo: )` | n/a | | +| search | Table with header `TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT`, rows in stable order, followed by `Count: N` | Header + `Count: 0` | For block rows use content snippet; for tag/property rows omit timestamps | +| show (text) | Raw tree text with `:db/id` as first column, trimmed | n/a | Tree lines should be prefixed by `:db/id` followed by the tree glyphs. Example:

`id1 block1`
`id2 ├── b2`
`id3 │ └── b3`
`id4 ├── b4`
`id5 │ ├── b5`
`id6 │ │ └── b6`
`id7 │ └── b7`
`id8 └── b8`

If a block title spans multiple lines, show the first line on the tree line and indent the remaining lines under the glyph column. Example:

`168 Jan 18th, 2026`
`169 ├── b1`
`173 ├── aaaxx`
`174 ├── block-line1`
` │ block-line2`
`175 └── cccc`

For `--format json|edn`, keep existing structured output | +| errors | `Error (): ` + optional `Hint: ` line | n/a | Ensure error codes are stable and consistent | + +## Problem statement + +The current logseq-cli human output is mostly raw pr-str output, which is hard to read and inconsistent across commands. +Users need clearer, command-specific summaries, stable table formats, and more helpful error messages for CLI usage. +The db-worker-node process currently logs only to console, but operational debugging requires a per-graph log file stored under the data directory with simple retention. + +## Testing Plan + +I will add unit tests for human output formatting functions to ensure stable, readable rendering for each command result shape. +I will add unit tests for error formatting to ensure consistent human output for common failure cases. +I will add an integration test that starts db-worker-node and verifies that a log file is created at //db-worker-node-YYYYMMDD.log. +I will add an integration test that exercises a log-producing db-worker-node action and asserts the log file contains the expected log entries. +I will follow @test-driven-development for all behavior changes. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +The CLI outputs structured results and the formatter converts them into human-friendly text based on the command and payload. +The db-worker-node daemon configures glogi to append to a per-graph log file under the repo-specific data directory. + +ASCII diagram: + ++-----------------+ format-result +------------------------+ +| logseq-cli | --------------------> | human output formatter | +| result payloads | <-------------------- | command renderers | ++-----------------+ +------------------------+ + ++------------------+ glogi appender +-------------------------------------+ +| db-worker-node | -----------------> | //db-worker-node-YYYYMMDD.log | ++------------------+ +-------------------------------------+ + +## Implementation plan + +1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps. +2. Read @test-driven-development guidelines and confirm the red phase will include all CLI output and log file tests first. +3. Review existing CLI output shapes in src/main/logseq/cli/commands.cljs to catalog the current :data payloads by command. +4. Review current formatting in src/main/logseq/cli/format.cljs and identify all human output paths that need command-specific rendering. +5. Define a human output specification table in docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md that maps each command to its target human output layout. +6. Add unit test scaffolding for CLI formatting in src/test/logseq/cli/format_test.cljs (or a new namespace) using representative :status/:data payloads. +7. Write a failing unit test for list commands to ensure human output renders a table with a header and row count. +8. Write a failing unit test for add/remove commands to ensure human output renders a succinct success line with key identifiers. +9. Write a failing unit test for graph management commands to ensure human output includes graph name and status text. +10. Write a failing unit test for server commands to ensure human output includes repo, status, host, and port when available. +11. Write a failing unit test for search and show commands to ensure human output includes result counts and stable ordering. +12. Write a failing unit test for error formatting to ensure error codes and helpful hints are included in human output. +13. Add a failing integration test in src/test/frontend/worker/db_worker_node_test.cljs (or a new namespace) that starts db-worker-node and asserts the log file exists at //db-worker-node-YYYYMMDD.log. +14. Add a failing integration test that performs a db-worker-node action and asserts at least one log line is appended to the log file. +15. Implement a command-aware human formatter in src/main/logseq/cli/format.cljs, using a dispatch on command or data shape. +16. Update src/main/logseq/cli/main.cljs to pass command context into the formatter so it can choose the correct renderer. +17. Normalize CLI result payloads in src/main/logseq/cli/commands.cljs to include explicit command identifiers where needed for formatting. +18. Ensure human output uses consistent spacing, headers, and ordering for list output, and avoids raw EDN dumps in normal cases. +19. Add a utility for table rendering with fixed column widths and truncation behavior in src/main/logseq/cli/format.cljs or a new helper namespace. +20. Implement db-worker-node log file setup in src/main/frontend/worker/db_worker_node.cljs using lambdaisland.glogi appenders. +21. Compute the log path using frontend.worker.db-worker-node-lock/repo-dir and ensure the directory exists before writing. +22. Configure glogi to append to //db-worker-node-YYYYMMDD.log and define whether console logging remains enabled. +23. Update help text in src/main/frontend/worker/db_worker_node.cljs to document the log file location and log-level flag behavior. +24. Update docs/cli/logseq-cli.md with the new human output expectations and any new formatting options. +25. Run unit tests in the red phase to confirm failures, then implement minimal changes to make them pass. +26. Run bb dev:test -v 'logseq.cli.commands-test' and bb dev:test -v 'frontend.worker.db-worker-node-test' in the green phase. +27. Run bb dev:lint-and-test after all changes to validate lint and unit tests. + +## Edge cases + +The repo name contains characters that change the pool directory name, so the log file path must use worker-util/get-pool-name consistently. +The data directory is on a filesystem without write permissions, which should surface a clear error message and non-zero exit code. +Multiple db-worker-node instances for different repos should not overwrite each other’s log files. +The log file should be created even if no requests are served yet and only startup logs are emitted. +Human output should remain stable when list results are empty or fields are missing. +The human formatter should avoid printing large nested maps by default for search or show results. + +## Testing commands and expected output + +Run a focused unit test during the red phase. + +```bash +bb dev:test -v 'logseq.cli.format-test/test-human-output-list' +``` + +Expected output includes a failing assertion and exits with a non-zero status code. + +Run the db-worker-node log integration test in the green phase. + +```bash +bb dev:test -v 'frontend.worker.db-worker-node-test/test-log-file-created' +``` + +Expected output includes 0 failures and 0 errors. + +Run the full lint and unit test suite when all changes are complete. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful linting and tests with exit code 0. + +## Testing Details + +I will validate human output formatting by asserting on complete rendered strings for representative payloads instead of inspecting internal formatting helpers. +I will validate db-worker-node logging by checking file existence, dated filename format, and that only the most recent 7 log files remain after multiple startups. +I will assert that a known log event is present after a startup or invoke action. + +## Implementation Details + +- Add a command-aware human output renderer that produces tables, summaries, and success lines based on command and result payloads. +- Standardize human error output to include error codes, messages, and actionable hints when possible. +- Ensure human output defaults to stable ordering and includes a count line for list and search commands. +- Add a table rendering helper with column width limits and truncation rules. +- Pass command context through CLI result objects so the formatter can select the correct renderer. +- Configure db-worker-node glogi to append logs to //db-worker-node-YYYYMMDD.log. +- Enforce log retention by keeping only the most recent 7 dated log files per graph directory. +- Ensure the log directory exists before log initialization and keep the log file path deterministic. +- Document log file location and new human output behavior in CLI documentation. +- Keep JSON and EDN outputs unchanged for scripting compatibility. +- Preserve existing exit codes and error handling semantics in the CLI. + +## Question + +Should the human output include color or ANSI styling, or should it remain plain text for maximal portability. +Answer: Remain plain text for maximal portability. +Should db-worker-node log to both console and file, or file-only to avoid duplicate logs in CLI output. +Answer: File-only to avoid duplicate logs in CLI output. +Is log rotation or size management required for db-worker-node.log, or is simple append-only acceptable. +Answer: Use dated log filenames and keep only the most recent 7 log files. + +--- diff --git a/docs/agent-guide/006-logseq-cli-import-export.md b/docs/agent-guide/006-logseq-cli-import-export.md new file mode 100644 index 0000000000..461edee5aa --- /dev/null +++ b/docs/agent-guide/006-logseq-cli-import-export.md @@ -0,0 +1,83 @@ +# Logseq CLI Import/Export Plan + +Goal: Add logseq-cli support for import/export with EDN and SQLite formats using the existing db-worker-node server. + +Architecture: Extend logseq-cli command parsing and execution to invoke db-worker-node thread APIs for export and import, with minimal new APIs to handle EDN import and SQLite binary payloads over HTTP. + +Tech Stack: ClojureScript, babashka/cli, db-worker-node HTTP /v1/invoke, datascript, sqlite-export helpers, Node fs/path. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Requirements + +- Import types: edn, sqlite. +- Export types: edn, sqlite. +- CLI must work against db-worker-node with repo binding and lock file behavior. +- Output files must be written to user-specified paths with clear success/error messages. + +## Proposed CLI UX + +Prefer graph-scoped subcommands to keep import/export with graph management: + +- `logseq graph export --type edn --file [--graph ]` +- `logseq graph export --type sqlite --file [--graph ]` +- `logseq graph import --type edn --input --graph ` +- `logseq graph import --type sqlite --input --graph ` + +Notes: +- `graph import` only supports importing into a new graph name; it must not overwrite an existing graph. +- `--graph` is required for import, and required unless the current graph is set in config for export. + +## Current Capabilities (Baseline) + +- db-worker-node supports `:thread-api/export-edn`, `:thread-api/export-db`, and `:thread-api/import-db`. +- logseq-cli can invoke db-worker-node via `src/main/logseq/cli/transport.cljs` and write files with `transport/write-output`. +- EDN import exists in the app layer (`frontend.handler.db-based.import`) but not in db-worker-node. +- SQLite import in db-worker-node writes the sqlite file, but CLI needs a full reload step to reflect new data. + +## Implementation Plan + +1. Review existing CLI command table and action pipeline in `src/main/logseq/cli/commands.cljs` and `src/main/logseq/cli/main.cljs` to locate insertion points for new graph import/export actions. +2. Add new CLI specs for import/export flags (type, input, output, export-type, mode) in `src/main/logseq/cli/commands.cljs`. +3. Extend the command table with `graph import` and `graph export` entries and ensure help output includes them. +4. Add action builders for import/export that validate repo presence, file paths, and allowed types (edn, sqlite). +5. Add CLI helpers for reading input files and writing output files in `src/main/logseq/cli/transport.cljs` (or a new helper namespace), keeping the existing `write-output` behavior for EDN and DB files. +6. Implement export execution: + - EDN: call `thread-api/export-edn` (graph-only) and write EDN file. + - SQLite: call a new db-worker-node export API that returns a base64 string or transit-safe binary; decode to Buffer and write `.sqlite` file. +7. Implement import execution: + - EDN: read EDN file, pass data to a new db-worker-node `thread-api/import-edn` (see below), and return a summary message. + - SQLite: read file as Buffer, pass to a new db-worker-node `thread-api/import-sqlite` (or reuse `import-db` with a wrapper that closes/reopens the repo). + - Always stop and restart the db-worker-node server from the CLI around import to ensure a clean reload. +8. Add db-worker-node thread APIs in `src/main/frontend/worker/db_core.cljs`: + - `:thread-api/import-edn` to convert export EDN into tx data via `logseq.db.sqlite.export/build-import` and transact with `:tx-meta` including `::sqlite-export/imported-data? true` so the pipeline rebuilds refs. + - `:thread-api/export-db-base64` (or similar) to return a base64 string for SQLite export over HTTP. + - `:thread-api/import-db-base64` (or similar) to accept base64 input, close existing sqlite connections, import db data, and reopen the repo (or invoke `:thread-api/create-or-open-db` with `:import-type :sqlite-db`). +9. Update db-worker-node server validation (repo binding) if the new thread APIs need special argument shapes. +10. Update CLI output formatting in `src/main/logseq/cli/format.cljs` to print concise success lines like `Exported to ` and `Imported from `. +11. Update documentation in `docs/cli/logseq-cli.md` with new commands, examples, and file format notes. + +## Testing Plan + +- Add CLI parsing tests for `graph import` and `graph export` options in `src/test/logseq/cli/commands_test.cljs` (or a new namespace). +- Add integration tests in `src/test/logseq/cli/integration_test.cljs` to: + - export EDN and SQLite from a test graph and assert output files exist and are non-empty. + - import EDN into a new graph and verify a known page/block exists via CLI `show` or `list`. + - import SQLite into a new graph and verify graph metadata or page count. +- Add db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` for the new import/export thread APIs (EDN build-import path and base64 DB export/import). +- Follow @test-driven-development: write failing tests before implementation. + +## Edge Cases + +- Large SQLite exports may exceed JSON limits if not base64/transit encoded; ensure streaming-safe or chunked base64 handling. +- Import should fail fast if the repo is missing and `--graph` is not provided, or if input file does not exist. +- SQLite import while the repo is open must close/reopen connections to avoid stale datascript state. +- EDN import should validate the export shape and surface readable errors when EDN is invalid or incompatible. +- Overwrite behavior should be explicit for SQLite imports to prevent accidental data loss. + +## Decisions + +1. `graph import` only imports into a new graph; it must not overwrite an existing graph. +2. No `--mode` flag; both EDN and SQLite imports are replace-style imports. +3. CLI always stops and restarts db-worker-node around imports. +4. `graph export --type edn` is graph-only for now (no page/view/blocks). diff --git a/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md new file mode 100644 index 0000000000..0b3a8d77db --- /dev/null +++ b/docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md @@ -0,0 +1,75 @@ +# Logseq CLI Thread API Keywords And Command Split Implementation Plan + +Goal: Replace thread-api string usage with keywords, standardize CLI repo/graph option to --graph, and split logseq.cli.commands into per-subcommand namespaces. + +Architecture: Update transport and db-worker-node boundaries to accept keyword methods while still serializing over HTTP. Refactor CLI command parsing into a shared dispatcher plus per-subcommand namespaces under a new command directory. Keep existing CLI behavior and output stable while updating option naming and error hints. + +Tech Stack: ClojureScript, babashka.cli, promesa, Logseq db-worker-node. + +Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md, docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md, docs/agent-guide/006-logseq-cli-import-export.md. + +## Problem statement + +The current CLI and db-worker-node codebase mixes thread-api method strings with keyword-based APIs, which makes it easy to introduce mismatches and reduces consistency with the thread-api macro design. The logseq.cli.commands namespace has grown large and mixes parsing, validation, and execution for multiple command groups, which makes maintenance and ownership difficult. + +## Testing Plan + +I will add unit tests to ensure all CLI thread-api method invocations use keywords and still serialize correctly through transport when invoking db-worker-node. I will add unit tests for command parsing and action building to cover the new per-subcommand namespaces and ensure summaries and help still match expected output. I will update db-worker-node tests to assert keyword method handling for repo validation and allowed non-repo methods. NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +1. Review @prompts/review.md to align plan with review expectations. + +2. Enumerate all thread-api string usages in CLI, db-worker-node, and tests using `rg -n -- "\"thread-api/" src/main src/test`. + +3. Enumerate all --graph references in CLI code and tests using `rg -n -- "--graph" src/main src/test` and confirm there are no docs references outside CLI. + +4. Define a small utility in `src/main/logseq/cli/transport.cljs` to normalize thread-api method arguments to keywords for callers while serializing over HTTP as string names. + +5. Add tests in `src/test/logseq/cli/transport_test.cljs` that pass a keyword method to transport invoke and assert the outgoing payload uses the string name and the response handling is unchanged. + +6. Update all CLI calls in `src/main/logseq/cli/commands.cljs` to pass keyword thread-api names to transport invoke and to any action maps that store method identifiers. + +7. Update CLI tests in `src/test/logseq/cli/commands_test.cljs` and `src/test/logseq/cli/integration_test.cljs` to expect keyword thread-api methods where they assert method calls. + +8. Update db-worker-node internal invocation and repo validation in `src/main/frontend/worker/db_worker_node.cljs` to coerce incoming method values to keywords for comparisons and logging, while still accepting string input from JSON payloads. + +9. Update any db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` that assert method strings to use keyword expectations and verify non-repo method handling. + +10. ~~Replace any --graph option references in CLI formatting and tests by updating `src/main/logseq/cli/format.cljs` and `src/test/logseq/cli/format_test.cljs` to use --repo.~~ + +11. Search for any `:graph` option wiring in `src/main/logseq/cli/commands.cljs` and remove CLI option parsing for --graph, including any help or usage text, while preserving graph-specific subcommands like `graph create`. + +12. Introduce a new directory `src/main/logseq/cli/command/` and split `logseq.cli.commands` into per-subcommand namespaces such as `logseq.cli.command.graph`, `logseq.cli.command.server`, `logseq.cli.command.list`, `logseq.cli.command.add`, `logseq.cli.command.remove`, `logseq.cli.command.search`, and `logseq.cli.command.show`. + +13. Create a small shared namespace like `src/main/logseq/cli/command/core.cljs` for global option spec, parsing helpers, summary formatting, and shared validation utilities, ensuring no behavior changes in parsing or summaries. + +14. Update `src/main/logseq/cli/commands.cljs` to become a thin facade that assembles tables from per-subcommand modules, delegates parse-args and build-action to those modules, and exposes execute dispatching without changing public API. + +15. Update `src/main/logseq/cli/main.cljs` requires to match any namespace changes and confirm the CLI usage summary still renders the same command list. + +16. Update tests in `src/test/logseq/cli/commands_test.cljs` to import any moved namespaces or use the facade namespace, and ensure all help summary snapshots still pass. + +17. Run unit tests for CLI and db-worker-node with `bb dev:test -v 'logseq.cli.commands-test'`, `bb dev:test -v 'logseq.cli.transport-test'`, and `bb dev:test -v 'frontend.worker.db-worker-node-test'` and fix failures. + +18. Run the full lint and unit test suite with `bb dev:lint-and-test` after all changes are complete. + +## Testing Details + +The tests will focus on behavior by asserting that CLI invocations still produce the same actions and outputs while enforcing keyword-based thread-api methods and updated --repo hints. The db-worker-node tests will assert that repo validation and non-repo method bypass logic behave correctly when methods are provided as keywords or strings. The command split will be validated by reusing existing parse-args and summary tests to ensure no behavioral regression in user-facing help or dispatch. + +## Implementation Details + +- Normalize thread-api method values at transport and db-worker-node boundaries to accept keywords and serialize as strings over HTTP. +- Replace all explicit "thread-api/..." literals in CLI and db-worker-node call sites with :thread-api/... keywords. +- Preserve graph subcommands and graph naming semantics while standardizing on :graph options. +- Move per-subcommand parsing and execution helpers into `src/main/logseq/cli/command/` namespaces and keep `logseq.cli.commands` as a facade. +- Keep action map shapes stable to avoid downstream changes in format or execution. +- Update tests to match keyword method expectations and new module layout. +- Ensure public CLI output and behavior remain unchanged + +## Question + +~~Resolved: Remove --graph entirely and fail fast on any --graph usage.~~ + +--- diff --git a/docs/agent-guide/008-logseq-cli-move-command.md b/docs/agent-guide/008-logseq-cli-move-command.md new file mode 100644 index 0000000000..281c613800 --- /dev/null +++ b/docs/agent-guide/008-logseq-cli-move-command.md @@ -0,0 +1,92 @@ +# Logseq CLI Move Command Implementation Plan + +Goal: Add a move subcommand to logseq-cli that moves a non-page block and its children under a target block or page with positional control. +Architecture: Extend the existing CLI command table with a new move command that resolves source and target entities via db-worker-node and invokes :thread-api/apply-outliner-ops using :move-blocks. +Architecture: Use existing outliner move semantics for ordering while validating CLI inputs and translating --pos into outliner options. +Tech Stack: ClojureScript, babashka.cli, db-worker-node :thread-api/apply-outliner-ops, Logseq outliner ops. +Related: Builds on docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md. + +## Problem statement + +Users need a CLI way to move a block and its subtree to a new parent or sibling position in a graph without opening the UI. +The current CLI supports add and remove but lacks a move operation even though db-worker-node and outliner already support :move-blocks. +The move command must align with existing CLI patterns, validate non-page sources, and support positioning under a block or page target. + +## Testing Plan + +I will add unit tests for command parsing and validation of move options, including invalid combinations and default --pos behavior. +I will add unit tests for human output formatting of move success messages. +I will add an integration test that creates blocks, moves a block to a target page and block, and verifies the resulting tree via show. +NOTE: I will write all tests before I add any implementation behavior. + +## Command behavior and options + +The move command will be a new inspect and edit verb alongside add and remove. +The command will accept exactly one source selector and exactly one target selector. +The command will move a single block per invocation. +The command will allow a page target via --page-name and a block target via a block selector. +The move position will default to first-child unless --pos is provided. +Source selectors will be --id or --uuid, and target selectors will be --target-id or --target-uuid. + +| Flag | Meaning | Required | Notes | +| --- | --- | --- | --- | +| --id | Source block db/id | Yes | Mutually exclusive with --uuid. | +| --uuid | Source block UUID | Yes | Mutually exclusive with --id. | +| --target-id | Target block db/id | Yes | Mutually exclusive with --target-uuid and --page-name. | +| --target-uuid | Target block UUID | Yes | Mutually exclusive with --target-id and --page-name. | +| --page-name | Target page name | Yes | Only valid when target is a page. | +| --pos | Position relative to target | No | Allowed values: first-child, last-child, sibling. | + +## Implementation Plan + +1. Read @prompts/review.md and capture any relevant review checklist items for CLI commands and db-worker-node usage. +2. Add a new command namespace at src/main/logseq/cli/command/move.cljs with a command entry, spec, and action builder. +3. Add option validation in src/main/logseq/cli/command/move.cljs for allowed --pos values and mutually exclusive selectors. +4. Add source resolution logic in src/main/logseq/cli/command/move.cljs that fetches the source block by id or uuid and rejects page entities. +5. Add target resolution logic in src/main/logseq/cli/command/move.cljs that fetches the target block or page and returns a target db/id. +6. Map --pos to outliner options for :move-blocks and document the mapping in comments for future maintenance. +7. Implement execute-move in src/main/logseq/cli/command/move.cljs that calls :thread-api/apply-outliner-ops with :move-blocks. +8. Wire the new command into src/main/logseq/cli/commands.cljs for parsing, validation, action building, and execution. +9. Update src/main/logseq/cli/command/core.cljs to include move in the Graph Inspect and Edit group in top-level summaries. +10. Update src/main/logseq/cli/main.cljs to include move in the usage command list string. +11. Add human output formatting for move in src/main/logseq/cli/format.cljs and include the relevant context keys. +12. Update src/test/logseq/cli/commands_test.cljs with parse and help coverage for move and validation error cases. +13. Update src/test/logseq/cli/format_test.cljs with a move success formatting test. +14. Update src/test/logseq/cli/integration_test.cljs with a move workflow that asserts the moved block appears under the new target. +15. Update docs/cli/logseq-cli.md to document the new move command, its flags, and examples. +16. Run bb dev:lint-and-test and fix any failures. + +## Edge cases to cover + +Moving a page block should fail with a clear error message and code. +Providing --pos sibling should return a validation error when the target is a page. +Moving a block to itself or into its descendants should be rejected by outliner and surfaced as an error. +Supplying both id and uuid selectors should return a validation error. +Supplying no target selector should return a validation error. + +## Notes on position mapping + +first-child will use :sibling? false and no :bottom? so that the moved block becomes the first child of the target. +last-child will use :bottom? true so outliner places the block after the last child of the target. +sibling will use :sibling? true so outliner places the block immediately after the target block. + +## Testing Details + +I will add command parsing tests that assert move is present in help output and that invalid flag combinations are rejected. +I will add a format test that ensures the human output for move references the source block and target. +I will add an integration test that creates a page, adds blocks, moves a block under a target, and verifies the show tree includes it in the expected position. + +## Implementation Details + +- Add a new command entry in src/main/logseq/cli/command/move.cljs with a spec that includes source and target selectors plus --pos. +- Resolve source and target entities via :thread-api/pull and reject page sources by checking page attributes. +- Translate --pos into outliner options for :move-blocks and pass them through :thread-api/apply-outliner-ops. +- Extend command parsing and execution switches in src/main/logseq/cli/commands.cljs to include :move-block. +- Extend human formatting in src/main/logseq/cli/format.cljs with a concise move success line. +- Update docs/cli/logseq-cli.md to list the new move command in the inspect and edit section and help output. +- Follow @skills/test-driven-development for all tests and implementation work. + +## Question + + +--- diff --git a/docs/agent-guide/009-cli-add-pos-show-tree-align.md b/docs/agent-guide/009-cli-add-pos-show-tree-align.md new file mode 100644 index 0000000000..e822d4b36d --- /dev/null +++ b/docs/agent-guide/009-cli-add-pos-show-tree-align.md @@ -0,0 +1,110 @@ +# CLI Add Pos And Show Tree Alignment Implementation Plan + +Goal: Add --pos support to logseq-cli add block, rename move --page-name to --target-page-name, and fix show tree alignment when db/id widths differ. + +Architecture: Extend the logseq-cli command layer to parse and validate add block target options and --pos, map it to outliner insert options sent over db-worker-node, and update tree rendering to use a fixed-width id column computed from the tree. + +Tech Stack: ClojureScript, babashka.cli, promesa, db-worker-node thread-api. + +Related: Builds on docs/agent-guide/008-logseq-cli-move-command.md. + +## Problem statement + +The logseq-cli add block command always inserts at the bottom, and it cannot express first-child or sibling insertion positions, and it relies on --page/--parent targets instead of explicit target ids. + +The move command uses --page-name for page targets, which is inconsistent with the target naming used by other move flags. + +The show command renders a text tree with id prefixes, but the glyph column shifts when db/id digit widths differ, making the tree hard to read. + +We need to add a --pos option to add block that mirrors existing move semantics and fix show tree alignment for variable id widths. + +## Testing Plan + +I will add a unit test that parses add block with --target-id/--target-uuid/--target-page-name and --pos and validates invalid pos values, ensuring it is rejected with a clear error. + +I will add a unit test that parses move with --target-page-name and rejects --page-name as an unknown option. + +I will add a unit test for show tree rendering that uses mixed-width db/id values and verifies glyph alignment is consistent. + +I will add an integration test that inserts two blocks with different --pos values using the new target flags and verifies the resulting order via show output or a query in db-worker-node. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation Plan + +1. Read the existing add block command spec and execution path in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs. + +2. Read the existing move --pos implementation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to mirror the allowed values and option mapping, and to scope the rename from --page-name to --target-page-name. + +3. Write a failing unit test that parses add block with --target-id/--target-uuid/--target-page-name and --pos in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. + +4. Write a failing unit test for show tree alignment with mixed-width ids in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. + +5. Run the CLI unit tests to confirm both tests fail for the correct reasons. + +6. Replace add block target options in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs with --target-id, --target-uuid, --target-page-name, and add :pos to the spec so help text includes the option. + +7. Add an invalid-options? helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to validate allowed positions and to reject sibling positioning when the target is a page or when no target is supplied. + +8. Wire add-block invalid option checks into command validation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs. + +9. Update build-add-block-action in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to normalize :pos, resolve --target-* options into a single target selector, and include it in the action payload while keeping the default behavior as last-child. + +10. Update execute-add-block in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs to resolve the target selector and translate :pos into insert-blocks options, using the same mapping as move and keeping compatibility with db-worker-node. + +11. Rename move --page-name to --target-page-name in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs and update parsing in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs. + +12. Update any move-related tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to use --target-page-name and to assert --page-name is rejected. + +13. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to compute a fixed id column width from all nodes in the tree and pad id cells consistently for all rows and multiline continuations. + +14. Update the show tree unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to reflect the new alignment behavior. + +15. Write a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that uses db-worker-node to insert blocks with different --pos values and checks the resulting order. + +16. Run the specific unit tests and the new integration test to verify they pass. + +17. Run the full CLI test suite with bb dev:lint-and-test to ensure no regressions. + +## Edge cases + +Add --pos sibling with --target-page-name should be rejected because a page target has no sibling context. + +Add --pos values should be case-insensitive and trimmed, matching move semantics. + +Show tree rendering should keep alignment for nodes that have no db/id or use a placeholder. + +Multiline block titles should continue to render under the glyph column even with mixed-width ids. + +## Testing Details + +Unit tests cover add --pos parsing and show tree alignment with mixed-width ids and multiline titles to validate visible behavior rather than internal helpers. + +Integration tests cover db-worker-node insert ordering by creating a page and inserting blocks with first-child and last-child positions using the new target flags, then asserting order via show output or a query. + +Move command tests cover renaming --page-name to --target-page-name and ensure the legacy flag is rejected. + +## Implementation Details + +- Replace add block target flags with --target-id, --target-uuid, --target-page-name and add :pos to the spec in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs. +- Mirror move position values and mapping to {:sibling? false} and {:sibling? false :bottom? true} and {:sibling? true}. +- Keep default add behavior as last-child when --pos is omitted for backward compatibility. +- Reject --pos sibling when the target is a page or when no target is provided. +- Normalize and validate :pos and target selector in add command parsing to avoid leaking invalid values to db-worker-node. +- Rename move --page-name to --target-page-name in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs and update parsing and help output accordingly. +- Compute max id width in tree->text by traversing root and descendants before rendering lines. +- Build id padding from the max width and use it for both first rows and multiline continuation rows. +- Update tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to assert alignment with mixed-width ids. +- Ensure db-worker-node invocations remain unchanged aside from the extra insert options map. + +## Question + +Should --pos default to last-child to match current add behavior, or should it default to first-child for consistency with move. + +Answer: Default to last-child to preserve current add behavior. + +Should the old --page and --parent flags be removed immediately or supported as deprecated aliases for one release. + +Answer: Remove immediately. + +--- diff --git a/docs/agent-guide/010-logseq-cli-show-linked-references.md b/docs/agent-guide/010-logseq-cli-show-linked-references.md new file mode 100644 index 0000000000..931563c135 --- /dev/null +++ b/docs/agent-guide/010-logseq-cli-show-linked-references.md @@ -0,0 +1,82 @@ +# Logseq CLI Show Linked References Implementation Plan + +Goal: Add task status prefixes and block tag rendering to show output, replace inline `[[]]` references with `[[]]`, and include linked references for the shown block or page. + +Architecture: The CLI show command will fetch status data via `:logseq.property/status` with the tree and will build a display label that prefixes the status before the block content, replaces inline `[[]]` with `[[]]`, and then appends inline block tags (e.g. `#RTC #Task`) to the content display. +Architecture: The CLI show command will also call db-worker-node thread-api/get-block-refs to fetch linked references and append them to text output in a tree form (db/id in the first column), while returning structured data in JSON and EDN output. +Architecture: Data flow will remain CLI -> db-worker-node -> db-core with no new worker endpoints, reusing existing thread-api functions. + +Tech Stack: ClojureScript, promesa, logseq-cli transport, db-worker-node thread-api, Datascript. + +Related: Builds on docs/agent-guide/009-cli-add-pos-show-tree-align.md. + +## Testing Plan + +I will follow @test-driven-development and write failing tests before any production changes. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text prefixes :logseq.property/status before the block title for TODO and CANCELED blocks. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text appends :block/tags in `#Tag` format to the rendered block content. +I will add a unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that asserts tree->text keeps multiline alignment when the status prefix is present. +I will add an integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page and a referencing block, runs show with --format json, and asserts that linked references are present and include the referencing block uuid and page title. +I will run the new unit tests with bb dev:test -v 'logseq.cli.commands-test' and the new integration test namespace with bb dev:test -v 'logseq.cli.integration-test' to confirm failures, then again to confirm passing. +I will run bb dev:lint-and-test after implementation to ensure no regressions. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Problem statement + +The current show command prints only the block title or page name without the task status, which hides task status context in CLI usage. +The show command also does not include linked references for the shown block or page, forcing users to query references separately. +We need to enhance the show output to include task status prefixes and linked references while keeping existing formats and db-worker-node integration stable. + +## Implementation Plan + +1. Read /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to identify the tree-block selector, label construction, and output formatting paths. +2. Read /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs to confirm the behavior and return shape of :thread-api/get-block-refs. +3. Read the existing show tree tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to align new expectations with current formatting rules. +4. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that builds a tree with :logseq.property/status values and asserts the status prefix appears before the block title in tree->text output. +5. Add a failing unit test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs that covers multiline block titles with statuses and asserts continuation lines still align under the glyph column. +6. Add a failing integration test in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs that creates a page, adds a block referencing that page, runs show for the page in JSON, and asserts the response contains linked references with block uuid and page title. +7. Run the two new unit tests and the integration test to confirm failures for the expected reasons. +8. Update the tree-block selector in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to pull :logseq.property/status alongside :block/title and :block/name. +9. Add a small helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that builds the display label by prefixing :logseq.property/status followed by a space when a status is present, while falling back to :block/name or :block/uuid when :block/title is missing, and appending `#Tag` strings for :block/tags. +10. Update tree->text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use the new label helper for both root and child nodes so that status prefixes appear in all output lines. +11. Add a linked references fetch step in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs that calls transport/invoke with :thread-api/get-block-refs using the root db/id. +12. Normalize linked references by pulling a minimal selector for each ref block, including :db/id, :block/uuid, :block/title, :logseq.property/status, and {:block/page [:db/id :block/name :block/title :block/uuid]}, so CLI output is predictable and lightweight. +13. Extend the show tree data structure to include :linked-references with a list of normalized blocks and a :count, and ensure this structure is returned for JSON and EDN output paths. +14. For text output, append a Linked References section after the tree that renders each referencing block in tree form with db/id in the first column (aligned to the glyph column), include the status-prefixed label, and show a count line when references exist. +15. Update the unit test expectations in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs to match the status-prefixed text output. +16. Update the new integration test assertions to match the final JSON structure for linked references. +17. Run the targeted tests again, then bb dev:lint-and-test, to verify all changes pass. + +## Edge cases + +Blocks without :logseq.property/status should render exactly as before with no extra spacing. +Blocks with :block/tags should render tag names as `#Tag` appended after the content. +Blocks containing inline `[[]]` references should render those tokens replaced with `[[]]`. +Blocks with :block/title nil should still render using :block/name or :block/uuid with the status prefix applied only when a title exists. +Linked references can be empty, in which case the Linked References section should be omitted from text output and :linked-references should contain a zero count in JSON and EDN. +Linked reference blocks that are missing a page or title should still render using their uuid fallback. +Linked references should render using the same tree layout rules as the main show tree, including db/id in the first column. +Show by block id and show by page name should both resolve linked references using the root db/id. + +## Testing Details + +The unit tests exercise tree->text output formatting to ensure status prefixes appear on the first line and multiline alignment is preserved, which validates CLI-visible behavior. +The integration test uses db-worker-node to create actual referencing blocks and verifies that show output includes linked references in the JSON response without inspecting internal worker details. + +## Implementation Details + +- Pull :logseq.property/status in the tree selector so task status is available for label rendering. +- Build a label helper that prefixes status values without changing existing fallback logic for titles and names, replaces inline `[[]]` tokens with `[[]]`, and appends block tags (e.g. `#RTC #Task`) after the content. +- Append a Linked References section in text output with a header, count, and tree-formatted block labels (db/id in the first column). +- Use :thread-api/get-block-refs for reference discovery and re-pull a minimal selector for stable CLI output. +- Return linked references in JSON and EDN outputs as {:linked-references {:count n :blocks [...]}}. +- Keep all changes inside the CLI show command and avoid new db-worker-node endpoints. + +## Question + +Should the status prefix use the stored uppercase value (for example CANCELED) or should it be title-cased to match the example with Canceled. +Answer: Use the stored uppercase status value (for example CANCELED). +Should linked references be grouped by page in text output, or listed as a flat list with page labels. +Answer: Group linked references by page in text output. + +--- diff --git a/docs/agent-guide/011-logseq-cli-search-optimization.md b/docs/agent-guide/011-logseq-cli-search-optimization.md new file mode 100644 index 0000000000..81b44f36d4 --- /dev/null +++ b/docs/agent-guide/011-logseq-cli-search-optimization.md @@ -0,0 +1,111 @@ +# Logseq CLI Search Optimization Implementation Plan + +Goal: Simplify the search command arguments and ensure search covers blocks, pages, tags, and properties by default. + +Architecture: The CLI parses args with babashka.cli, builds a search action, and queries db-worker-node through thread-api endpoints for pages, blocks, tags, and properties. +Architecture: The change stays in the CLI layer and relies on existing thread-api methods in db-worker-node for data access. + +Tech Stack: ClojureScript, babashka.cli, Datascript queries, db-worker-node thread-api. + +Related: Relates to docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md. + +## Problem statement + +The current search command requires the --text option or positional args and joins all positional args into the search string. + +The current search documentation and tests are still written around the --text flag and do not describe default type behavior. + +We need to remove the --text option, use the first positional string as the search text, and ensure searches cover block titles, page names, tag names, and property names with --type defaulting to all. + +We also need to remove the --include-content option and any :block/content usage from CLI search because :block/content is no longer present in db-graph. + +We also need to remove the --limit option because it currently only trims output and does not reduce query work. + +## Testing Plan + +I will add unit tests for CLI parsing that assert the search text is taken from the first positional argument and that --type defaults to all when omitted. + +I will add integration tests for the CLI search command that use the positional search text and verify results include at least one matching item from each type when the graph contains data. + +I will add a formatting test that validates the missing search text hint no longer references --text. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope + +This plan updates CLI search option parsing and action building for search types. + +This plan removes --include-content from CLI search options and eliminates :block/content usage from CLI search code and tests. + +This plan updates user-facing docs that describe the search command usage. + +## Non-goals + +This plan does not change db-worker-node thread-api method signatures or introduce new server endpoints. + +This plan does not alter vector search or inference-worker behavior. + +## Expected CLI behavior + +| Scenario | Input | Behavior | +| --- | --- | --- | +| Basic search | logseq-cli search "hello" | Uses "hello" as search text and searches pages, blocks, tags, and properties. | +| Type filter | logseq-cli search "hello" --type page | Searches only pages. | +| Missing text | logseq-cli search | Returns missing-search-text with a hint that positional text is required. | +| Block titles | logseq-cli search "todo" | Matches block titles only, not :block/content. | + +## Implementation Plan + +1. Read @test-driven-development and follow TDD for every behavior change in this plan. +2. Update CLI parsing tests in `src/test/logseq/cli/commands_test.cljs` to use positional search text and verify default types are all when --type is omitted. +3. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm the new tests fail for the expected reasons. +4. Update integration tests in `src/test/logseq/cli/integration_test.cljs` to call search without --text and to assert results still return ok with data. +5. Run `bb dev:test -v 'logseq.cli.integration-test'` and confirm the new tests fail for the expected reasons. +6. Update formatting expectations in `src/test/logseq/cli/format_test.cljs` if missing-search-text hints change. +7. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm the new tests fail for the expected reasons. +8. Remove the :text, :include-content, and :limit options from search spec in `src/main/logseq/cli/command/search.cljs` and update help text accordingly. +9. Update `src/main/logseq/cli/commands.cljs` to require positional search text and to stop referencing :text in missing-search-text logic. +10. Update `src/main/logseq/cli/command/search.cljs` build-action to use the first positional argument as text and ignore any additional positional args for search text. +11. Remove :block/content references from `src/main/logseq/cli/command/search.cljs` so block searches use block title fields only. +12. Update `src/main/logseq/cli/format.cljs` to remove the hint that references --text. +13. Update docs in `docs/cli/logseq-cli.md`, `docs/agent-guide/004-logseq-cli-verb-subcommands.md`, and `docs/agent-guide/002-logseq-cli-subcommands.md` to remove --text, --include-content, and --limit and document the new positional argument behavior and default --type behavior. +14. Update CLI tests in `src/test/logseq/cli/commands_test.cljs`, `src/test/logseq/cli/integration_test.cljs`, and `src/test/logseq/cli/format_test.cljs` to remove --include-content, --limit, and any :block/content expectations. +15. Run `bb dev:lint-and-test` and confirm all linters and unit tests pass. + +## Edge cases and validation + +Multiple positional arguments should not be concatenated into a single search string unless explicitly required by design. + +Search text containing spaces should still work when the shell passes it as a single quoted argument. + +When --type is provided, only the requested type set should be searched and defaults should not override the filter. + +Search with --tag should continue to filter block searches and should not filter page, tag, or property results unless explicitly required. + +The CLI must not attempt to query :block/content because the attribute is absent in db-graph. + +## Testing Details + +The CLI command tests will assert that a positional search term maps to the action :text and that missing text errors are raised when no positional arguments exist. + +The integration tests will execute the CLI against a sample graph, verify the command exits with ok, and confirm search results are a vector for each type requested. + +The formatting tests will assert the error hint no longer suggests the deprecated --text option. + +The CLI tests will assert that no --include-content or --limit option exists and that searches do not rely on :block/content. + +## Implementation Details + +- Remove :text, :include-content, and :limit from the search option spec and adjust help text generation to avoid advertising those options. +- Update missing-search-text validation to rely only on positional args. +- Use only the first positional argument as the search text to match the new spec. +- Confirm default search types flow through normalize-search-types when --type is absent. +- Remove :block/content usage from query-blocks and any related logic. +- Update docs and examples to show quoted positional search text. +- Ensure error hints reference positional text rather than options. + +## Question + +No open questions. + +--- diff --git a/docs/agent-guide/012-logseq-cli-graph-storage.md b/docs/agent-guide/012-logseq-cli-graph-storage.md new file mode 100644 index 0000000000..c044f8bc84 --- /dev/null +++ b/docs/agent-guide/012-logseq-cli-graph-storage.md @@ -0,0 +1,77 @@ +# Logseq CLI Graph Storage Plan + +## Context +logseq-cli and db-worker-node currently store CLI-managed graphs under `~/.logseq/db-worker/` and use per-graph directories named like `.logseq-pool-/` (with partial encoding). This plan captures the non-functional updates requested: + +1) Rename `~/.logseq/db-worker/` to `~/logseq/cli-graphs/` for CLI-managed graphs. +2) Rename per-graph directory format from `.logseq-pool-/` to `/`. +3) Ensure graph names that are not valid directory names are encoded, and decoding is symmetric when reading/listing. + +## Goals +- Move CLI graph storage to `~/logseq/cli-graphs` by default. +- Use a clean per-graph directory name equal to the (encoded) graph name, without `.` prefix or `logseq-pool-` prefix. +- Provide a reversible encode/decode for graph names so list/read operations reconstruct the original graph name. +- CLI commands and outputs should hide the internal `logseq_db_` prefix; user-facing graph names strip that db prefix. +- Maintain db-worker-node functionality (locks/logs/kv-store) with the new paths. + +## Non-goals +- Changing Electron or browser-based db graph storage (`~/logseq/graphs`) or OPFS behavior. +- Changing db schema or sqlite storage format. +- Changing db-worker-node API semantics. + +## Current Behavior (Key References) +- CLI default data dir is `~/.logseq/db-worker`: `src/main/logseq/cli/config.cljs`. +- db-worker-node default data dir is `~/.logseq/db-worker`: `src/main/frontend/worker/db_worker_node.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. +- Per-graph directory currently `.logseq-pool-`: + - `frontend.worker-common.util/get-pool-name` returns `logseq-pool-`: `src/main/frontend/worker_common/util.cljc`. + - `repo-dir` uses `"." + pool-name` in CLI server, db-worker-node lock, and node platform: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. +- Current graph decoding in list operations reverses only `+3A+` and `++` (file-based graphs); other characters are not reversible: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/platform/node.cljs`. + +## Proposed Approach +### 1) New default data dir +- Change default data dir for CLI and db-worker-node from `~/.logseq/db-worker` to `~/logseq/cli-graphs`. +- Update help text and any user-facing docs mentioning the old default. + +### 2) New per-graph directory naming +- Replace `.logseq-pool-/` with `/`. +- Remove the leading dot and `logseq-pool-` prefix entirely for CLI-managed graphs. + +### 3) Reversible graph name encoding +- Introduce a shared encode/decode pair for graph directory names that is bijective for all graph names. +- The encoding must avoid path separators and other invalid characters (esp. `/`, `\`, `:` on Windows). +- Suggested approach (reversible and simple): + - Encode: apply `encodeURIComponent` to the graph name, then replace `%` with a safe delimiter (e.g. `~`) to keep filenames readable and avoid `%` edge cases. + - Decode: reverse the delimiter replacement, then `decodeURIComponent`. +- Provide helper functions in a shared place (e.g. `frontend.worker-common.util` or a new shared CLI/worker helper) so CLI server, db-worker-node lock, and node platform list all use the same encode/decode logic. + +## Implementation Steps +1) Add encode/decode helpers + - Add new helpers for reversible graph name <-> directory name. + - Update `get-pool-name` or replace its usage for CLI/db-worker-node paths. + - Files: `src/main/frontend/worker_common/util.cljc`, potentially `deps/cli/src/logseq/cli/common/graph.cljs`. + +2) Update data dir defaults + - Change defaults to `~/logseq/cli-graphs` in: + - `src/main/logseq/cli/config.cljs` + - `src/main/logseq/cli/server.cljs` + - `src/main/frontend/worker/db_worker_node_lock.cljs` + - `src/main/frontend/worker/db_worker_node.cljs` (help text) + - `src/main/frontend/worker/platform/node.cljs` + - Update any CLI docs/tests that reference `db-worker` as default. + +3) Update repo-dir/path derivation + - Replace `"." + pool-name` usage with new `` directory naming. + - Update list-graphs and list-servers to decode from new directory names. + - Files: `src/main/logseq/cli/server.cljs`, `src/main/frontend/worker/db_worker_node_lock.cljs`, `src/main/frontend/worker/platform/node.cljs`. + +4) Tests & verification + - Update CLI integration tests that construct temp dirs named `db-worker*` to match new defaults or explicitly pass `--data-dir`. + - Update db-worker-node tests to use new naming and to validate encode/decode. + - Ensure `bb dev:lint-and-test` passes. + +## Open Questions +- The new encoding is CLI and db-worker-node only (no Electron changes). + +## Rollout Notes +- This is a filesystem layout change. Include release notes and ensure users can override via `--data-dir`. +- Provide a one-time warning if old layout is detected and not migrated. diff --git a/docs/agent-guide/013-logseq-cli-datascript-query.md b/docs/agent-guide/013-logseq-cli-datascript-query.md new file mode 100644 index 0000000000..9fb2903dc3 --- /dev/null +++ b/docs/agent-guide/013-logseq-cli-datascript-query.md @@ -0,0 +1,157 @@ +# Logseq CLI Datascript Query Implementation Plan + +Goal: Add a logseq-cli query subcommand that runs a Datascript query via db-worker-node and returns the raw datascript-query result shape. +Architecture: The CLI will parse a query form from arguments, call db-worker-node using the existing /v1/invoke transport with :thread-api/q, and return whatever datascript-query returns without normalization. +Architecture: No new db-worker-node HTTP endpoints are required because :thread-api/q already exists in the worker thread API. +Tech Stack: ClojureScript, babashka.cli, Datascript, db-worker-node HTTP transport. +Related: Relates to docs/agent-guide/012-logseq-cli-graph-storage.md. + +## Problem statement + +The current logseq-cli does not expose a query subcommand for running Datascript queries against a graph. +Users need a CLI command that executes a Datascript query and returns the same result shape as datascript-query for scripting and downstream tooling. +The solution should follow the existing logseq-cli and db-worker-node invocation patterns so it works with the current daemon and transport. + +## Testing Plan + +I will add an integration test that creates a graph, inserts blocks, runs the new query subcommand, and asserts that the returned IDs match the expected block IDs. +I will add a unit test that validates query argument parsing, including invalid EDN, missing query text, and optional inputs parsing. +I will add a unit test that verifies the query command returns the same shape as datascript-query without transformation. +I will follow @test-driven-development and write the failing tests before implementing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +Create a new command module at src/main/logseq/cli/command/query.cljs that defines the query command spec and action builder. +Use --query for the Datascript query form and --inputs for the optional EDN inputs vector, and parse them with cljs.reader/read-string or logseq.common.util/safe-read-string with error handling. +Return a structured action map that includes :type :query, :repo, :query, and :inputs to keep execution isolated from parsing. +Register the new command in src/main/logseq/cli/commands.cljs and include it in the command table so help output includes query. +Update src/main/logseq/cli/main.cljs usage text to include query in the command list. +Implement execution in src/main/logseq/cli/command/query.cljs using logseq.cli.transport/invoke with :thread-api/q and args [repo [query & inputs]]. +Return the raw Datascript result as-is, matching datascript-query output across human, json, and edn formats. +Add formatting in src/main/logseq/cli/format.cljs for :query to render the raw result in human output and pass-through for json or edn output. +Add command-level validation in src/main/logseq/cli/commands.cljs to return a missing-query error when no query is supplied. +Update src/test/logseq/cli/commands_test.cljs to expect query in help output and to validate parse errors for missing query or invalid input. +Add a CLI integration test in src/test/logseq/cli/integration_test.cljs that uses run-cli to execute query and verifies IDs in JSON output. +Confirm that no db-worker-node changes are required by verifying that :thread-api/q continues to accept the same argument shape in src/main/frontend/worker/db_core.cljs. + +## Edge cases + +A query string that cannot be read as EDN should return a clear invalid-options error and a non-zero exit code. +A query that returns no results should return an empty result with a successful status. +Queries with :in parameters should work when --inputs supplies the matching values in order. + +## Testing Details + +The integration test will create a graph, add known blocks, run a query that finds those blocks, and verify that the output matches the datascript-query result shape. +The unit tests will assert that parsing rejects invalid EDN for --inputs, that a missing query produces a :missing-query error, and that query execution returns raw results unchanged. + +## Implementation Details + +- Add a new command module at src/main/logseq/cli/command/query.cljs. +- Add command entries in src/main/logseq/cli/commands.cljs. +- Add output formatting in src/main/logseq/cli/format.cljs. +- Update usage text in src/main/logseq/cli/main.cljs. +- Use transport/invoke with :thread-api/q and [repo [query & inputs]]. +- Return datascript-query results without transformation. +- Keep all argument parsing and validation inside query command module using --query and --inputs. +- Keep db-worker-node changes to zero unless a new worker API is required. +- Add `custom-queries` to cli.edn for storing pre-defined Datascript queries that the CLI can list and run by name. +- Add built-in queries that appear in the query list alongside custom queries. +- Optional inputs should support default values in cli.edn, and built-in queries should ship with reasonable defaults for their optional inputs (required inputs can omit defaults). + - `block-search` (search-title) + ``` + [:find [?e ...] + :in $ ?search-title + :where + [?e :block/title ?title] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title)]] + ``` + - `task-search` (search-status, ?search-title, ?recent-days) + ``` + ;; Modify this query so search-title and recent-days are optional parameters. + ;; ?now-ms is injected by the CLI so users don't need to pass it (and should be hidden in query list output). + ;; Example: logseq query --name task-search --inputs '["doing"]' + [:find [?e ...] + :in $ ?search-status ?search-title ?recent-days ?now-ms + :where + [?e :block/title ?title] + [?e :logseq.property/status ?s] + [?s :db/ident ?status-ident] + [(= ?status-ident ?search-status)] + [(clojure.string/lower-case ?title) ?title-lower-case] + (or-join [?search-title ?title-lower-case] + [(nil? ?search-title)] + [(clojure.string/blank? ?search-title)] + (and [(clojure.string/lower-case ?search-title) ?search-title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title-lower-case)])) + [(get-else $ ?e :block/updated-at 0) ?updated-at] + (or + [(nil? ?recent-days)] + (and [(number? ?recent-days)] + [(<= ?recent-days 0)]) + (and [(number? ?recent-days)] + [(>= ?updated-at (- ?now-ms (* ?recent-days 86400000)))]) )] + ``` + +## cli.edn query shape + +Represent queries as a map keyed by query name. Keep the query form as EDN data (not a string) so it can be read directly. Optional fields like `:doc` and `:inputs` are included to help with listing and UX. `:inputs` should allow optional inputs to declare default values that are used when the CLI caller omits them. Internal inputs like `?now-ms` should be hidden from `query list` output. + +Suggested `:inputs` shapes: +- `["search-status" "?search-title" "?recent-days"]` (legacy string-only form) +- `[{:name "search-status"} {:name "?search-title" :default nil} {:name "?recent-days" :default nil}]` (explicit defaults) + +Example: +``` +{:custom-queries + {"block-search" + {:doc "Find blocks by title substring (case-insensitive)." + :inputs ["search-title"] + :query [:find [?e ...] + :in $ ?search-title + :where + [?e :block/title ?title] + [(clojure.string/lower-case ?title) ?title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title)]]} + + "task-search" + {:doc "Find tasks by status, optional title substring, optional recent-days." + :inputs [{:name "search-status"} + {:name "?search-title" :default nil} + {:name "?recent-days" :default nil} + ;; ?now-ms is internal; CLI fills it with current ms and query list should hide it. + {:name "?now-ms" :default :now-ms}] + :query [:find [?e ...] + :in $ ?search-status ?search-title ?recent-days ?now-ms + :where + [?e :block/title ?title] + [?e :logseq.property/status ?s] + [?s :db/ident ?status-ident] + [(= ?status-ident ?search-status)] + [(clojure.string/lower-case ?title) ?title-lower-case] + (or-join [?search-title ?title-lower-case] + [(nil? ?search-title)] + [(clojure.string/blank? ?search-title)] + (and [(clojure.string/lower-case ?search-title) ?search-title-lower-case] + [(clojure.string/include? ?title-lower-case ?search-title-lower-case)])) + [(get-else $ ?e :block/updated-at 0) ?updated-at] + (or + [(nil? ?recent-days)] + (and [(number? ?recent-days)] + [(<= ?recent-days 0)]) + (and [(number? ?recent-days)] + [(>= ?updated-at (- ?now-ms (* ?recent-days 86400000)))]) )]}}} +``` + +Notes: +- Built-in queries live in code but should be merged into the same map shape when listing or resolving by name. +- `:inputs` is optional metadata for CLI help. It does not affect execution. +## Question + +Use --query and --inputs options for the query subcommand. +Output should mirror datascript-query for scripting stability. + +--- diff --git a/docs/agent-guide/014-logseq-cli-show-multi-id.md b/docs/agent-guide/014-logseq-cli-show-multi-id.md new file mode 100644 index 0000000000..7b0587a855 --- /dev/null +++ b/docs/agent-guide/014-logseq-cli-show-multi-id.md @@ -0,0 +1,71 @@ +# Logseq CLI Show Multi-ID Implementation Plan + +Goal: Extend the logseq-cli show command so `--id` accepts one or more block ids, and when passed as `[ ...]`, it displays all blocks separated by a clear delimiter. +Architecture: Keep existing CLI parsing and db-worker-node transport, but allow `--id` to accept an EDN vector of ids; execute one fetch per id or a single multi-id fetch if supported by the worker thread API. +Tech Stack: ClojureScript, babashka.cli, db-worker-node HTTP transport, Logseq block formatting. +Related: Builds on existing show command behavior and db-worker-node thread API usage. + +## Problem statement + +The current `logseq show --id ` only supports a single block id, which makes it cumbersome to inspect multiple blocks from scripts. +We need `--id` to accept `[ ...]` and print each corresponding block, separated by a reasonable visual delimiter. +This should align with existing logseq-cli and db-worker-node patterns and preserve existing single-id behavior. +This also enables a pipeline workflow such as: `logseq query --name task-search --inputs '["todo"]' | xargs logseq show -id`. + +## Note on removing `search` + +Remove the `search` subcommand. No migration or compatibility work is required. + +## Note on `logseq query` output + +`logseq query` output handling: +1. Validate it is valid EDN. +2. Replace all spaces with commas. + +## Note on `logseq query task-search` inputs + +The first `task-search` input `status` should be a string like `"todo"` or `"doing"`, not `:logseq.property/status.todo`. + +## Testing Plan + +I will add unit tests for show argument parsing to accept a vector of ids and to reject invalid EDN in `--id`. +I will add an integration test that runs `logseq show --id '["id1" "id2"]'` and asserts that both blocks are present in output with the delimiter between them. +I will keep existing single-id tests intact and ensure no regressions in JSON/EDN output modes. +I will follow @test-driven-development and write the failing tests before implementing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +Locate the show command definition and parsing in `src/main/logseq/cli/command/show.cljs` (or equivalent) and identify current `--id` parsing behavior. +Update argument parsing so `--id` accepts either a single id (string) or an EDN vector of ids; parse via `cljs.reader/read-string` or `logseq.common.util/safe-read-string` with clear errors on invalid EDN. +Normalize the parsed value into a vector of ids, preserving the current single-id behavior by wrapping the string in a vector. +Update the show execution path to iterate through ids and fetch each block via existing db-worker-node transport; if a batch API exists, prefer it but keep compatibility with current API. +Add output formatting logic to insert a delimiter between each block in human output (for example `\n-----\n` or `\n====\n`), and keep JSON/EDN output structured as a vector of blocks instead of concatenated text. +Update command help/usage in `src/main/logseq/cli/main.cljs` and `src/main/logseq/cli/commands.cljs` to document vector form, including an example. +Add unit tests in `src/test/logseq/cli/commands_test.cljs` or a show-specific test file to validate parsing and error cases. +Add an integration test in `src/test/logseq/cli/integration_test.cljs` to verify multi-id output with delimiter and correct block ordering. +Confirm db-worker-node thread API endpoints used by show (likely in `src/main/frontend/worker/`) do not need changes; if they do, add a minimal batch fetch method and corresponding tests. + +## Edge cases + +`--id` contains invalid EDN (e.g., `[` without closing bracket) should return a clear invalid-options error and non-zero exit. +Mixed types in the id vector (e.g., numbers, maps) should either coerce to strings or be rejected with a clear error; prefer rejection to avoid surprising behavior. +Missing blocks (id not found) should return a clear message per block while still printing other valid blocks. +Output delimiter should not appear before the first block or after the last block. + +## Testing Details + +Unit tests should cover parsing of a single id, a vector of ids, and invalid EDN. +Integration tests should create two blocks, fetch them by ids, and verify both are present in order with the delimiter separating them in human output. +JSON/EDN outputs should be a vector of block structures matching current single-id output shape. + +## Implementation Details + +- Update `--id` parsing to accept EDN vectors of ids while preserving single-id strings. +- Normalize `id` input to `ids` vector for downstream handling. +- Loop fetches through existing db-worker-node transport, or add a batch fetch endpoint only if necessary. +- Insert a delimiter between blocks in human output; keep machine-readable outputs as structured vectors. +- Update help text and tests accordingly. + +--- diff --git a/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md b/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md new file mode 100644 index 0000000000..8efa52b043 --- /dev/null +++ b/docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md @@ -0,0 +1,64 @@ +# Logseq CLI and db-worker-node Housekeeping Implementation Plan + +Goal: Remove the --retries and --auth-token options from logseq-cli and db-worker-node, and add a --version option that prints build time and commit. + +Architecture: Keep option parsing centralized in logseq.cli.command.core and frontend.worker.db-worker-node, and move build metadata into a dedicated ClojureScript namespace injected via shadow-cljs closure defines. +Architecture: Ensure logseq-cli prints version info without needing a running db-worker-node, and db-worker-node no longer gates endpoints on auth tokens. + +Tech Stack: ClojureScript, babashka.cli, shadow-cljs, Node.js. + +Related: Relates to docs/agent-guide/001-logseq-cli.md. +Related: Relates to docs/agent-guide/002-logseq-cli-subcommands.md. +Related: Relates to docs/agent-guide/003-db-worker-node-cli-orchestration.md. + +## Problem statement + +The current logseq-cli and db-worker-node expose --retries and --auth-token options that are no longer desired, and the CLI lacks a version command that prints build time and commit. +The cleanup should remove these options without compatibility shims and introduce a clear version output backed by build metadata. + +## Testing Plan + +I will add a unit test for logseq-cli parsing that asserts --version short-circuits command execution and prints build metadata fields. +I will update the config and transport tests to remove retries and auth token expectations while still validating timeout behavior. +I will add a db-worker-node CLI test that verifies the help output no longer mentions --auth-token and that args parsing ignores the removed flag. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Plan + +1. Review current option definitions and call sites for retries and auth-token in src/main/logseq/cli/command/core.cljs, src/main/logseq/cli/config.cljs, src/main/logseq/cli/transport.cljs, and src/main/frontend/worker/db_worker_node.cljs. +2. Update src/main/logseq/cli/command/core.cljs to remove :auth-token and :retries from the global spec and option summary output. +3. Update src/main/logseq/cli/config.cljs to remove env parsing and defaults for auth-token and retries, and to stop persisting those keys in config files. +4. Update src/main/logseq/cli/transport.cljs to drop auth header handling and retry loops, and adjust invoke to pass only method, url, body, and timeout values. +5. Update any logseq-cli action builders or server helpers that still read :auth-token or :retries from config, and delete those plumbing paths if present. +6. Update src/main/frontend/worker/db_worker_node.cljs to remove --auth-token parsing, remove authorization checks on endpoints, and delete auth-token from daemon options and help output. +7. Add a new namespace such as src/main/logseq/cli/version.cljs that defines BUILD_TIME and REVISION via goog-define with safe defaults, and exposes a formatter for the CLI. +8. Add a global --version flag in logseq-cli by extending the global spec and by adding a short-circuit path in src/main/logseq/cli/main.cljs or src/main/logseq/cli/commands.cljs that prints build time and commit without requiring a command. +9. Add closure defines for the CLI build in shadow-cljs.edn under :logseq-cli and for the node test build under :test so tests can assert deterministic build metadata values. +10. Update build entry points that compile logseq-cli, such as package.json scripts or any CI workflow that calls clojure -M:cljs compile logseq-cli, to export LOGSEQ_BUILD_TIME and LOGSEQ_REVISION for the defines. +11. Update docs/cli/logseq-cli.md to remove auth-token and retries from the configuration list and to document --version output format with build time and commit. +12. Scan for remaining user-facing mentions of --auth-token or --retries in docs and README files, and update or remove them where appropriate. +13. Run unit tests for CLI and db-worker-node using bb dev:test for the modified namespaces, and run bb dev:lint-and-test if time allows. +14. Follow @prompts/review.md and @skills/test-driven-development throughout implementation and verification. + +## Testing Details + +The CLI tests will assert that --version returns a non-empty output containing build time and commit keys and that it exits successfully without requiring a subcommand. +The transport tests will still cover timeout behavior but will no longer assert retries behavior or auth header inclusion. +The db-worker-node tests will validate updated help output and ensure that argument parsing still recognizes required flags after removing --auth-token. + +## Implementation Details + +- Remove :auth-token and :retries from global CLI option specs and summaries in src/main/logseq/cli/command/core.cljs. +- Remove env parsing and defaults for auth-token and retries in src/main/logseq/cli/config.cljs. +- Simplify HTTP request and invoke logic in src/main/logseq/cli/transport.cljs to remove retries and auth headers. +- Remove auth-token CLI parsing and authorization gating in src/main/frontend/worker/db_worker_node.cljs. +- Add build metadata defines in a new CLI version namespace and wire --version output through logseq-cli entrypoints. +- Add closure defines for LOGSEQ_BUILD_TIME and LOGSEQ_REVISION in shadow-cljs.edn for :logseq-cli and :test builds. +- Update scripts or CI to populate LOGSEQ_BUILD_TIME and LOGSEQ_REVISION at compile time. +- Update docs/cli/logseq-cli.md and any other user-facing documentation to reflect the new option set. + +## Question + +Answer: remove all auth support entirely, including env vars and header checks. + +--- diff --git a/docs/agent-guide/016-recent-updated-query.md b/docs/agent-guide/016-recent-updated-query.md new file mode 100644 index 0000000000..427625ae51 --- /dev/null +++ b/docs/agent-guide/016-recent-updated-query.md @@ -0,0 +1,89 @@ +# Recent Updated Query Implementation Plan + +Goal: Add a built-in CLI query named recent-updated that filters entities updated within a configurable recent-days window. + +Architecture: The logseq CLI will expose a new built-in query spec in the query command module and pass inputs to db-worker-node via thread-api/q with no db-worker-node changes. +The query will rely on :block/updated-at and a now-ms input to compute a rolling cutoff in milliseconds. + +Tech Stack: ClojureScript, Datascript queries, logseq CLI, db-worker-node thread-api. + +Related: Builds on 015-logseq-cli-db-worker-node-housekeeping.md. + +## Problem statement + +Users need a built-in CLI query to list recently updated content without crafting ad hoc datalog each time. +The new recent-updated query should accept a recent-days parameter and integrate with existing logseq CLI query listing and execution paths. +The implementation must align with db-worker-node query execution and hide internal inputs from query list output. + +## Testing Plan + +I will add an integration test that creates a temporary graph, adds blocks, and runs the recent-updated query with a small recent-days window, asserting only recently updated entities are returned. +I will add a second integration test case in the same test to cover recent-days values of nil and non-positive numbers, confirming the CLI returns a clear invalid input error. +I will add a CLI query list assertion that recent-updated appears with the correct input spec and defaults, excluding the internal now-ms input from list output. +I will add a CLI show -id test to ensure duplicate trees are filtered when ids include parent/child relationships. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation Plan + +1. Write the failing integration test for recent-updated in `src/test/logseq/cli/integration_test.cljs` that uses `run-cli` to create a graph, insert blocks, and query by name, asserting the result is a vector of db/id values. +2. Write the failing assertion that `query list` returns a recent-updated entry with an inputs vector that only shows recent-days. +3. Run the single integration test to confirm failures are due to missing built-in query behavior. +4. Add a new built-in query spec entry in `src/main/logseq/cli/command/query.cljs` with name "recent-updated" and inputs [{:name "?recent-days" :default 0} {:name "?now-ms" :default :now-ms}]. +5. Define the query to find entities that have :block/updated-at and apply an or-join to bypass filtering when recent-days is nil or <= 0. +6. Ensure the query result shape matches other built-ins and returns a vector of entity ids. +7. Run the integration test again and confirm it now passes. +8. Refactor the built-in query spec and test setup for clarity if necessary while keeping behavior unchanged. +9. Run the CLI integration test suite or the focused test case to ensure no regressions. +10. Update the show -id output path to drop contained trees when multiple ids overlap. + +## Edge Cases and Considerations + +- recent-days must be greater than 0, and nil or non-positive values should be rejected with a clear CLI error. +- Blocks without :block/updated-at should be excluded consistently and not cause query errors. +- The query should return both pages and blocks. +- The query should exclude built-in entities by default. + +## Testing Details + +The tests will exercise the CLI path end to end by invoking db-worker-node and asserting the query results contain only the expected entities based on updated-at timestamps. +The tests will validate both the query list metadata and the data returned from the query invocation. + +## Implementation Details + +- Add a new entry to built-in-query-specs in `src/main/logseq/cli/command/query.cljs`. +- Use :block/updated-at and a computed cutoff (now-ms minus recent-days in milliseconds) inside the datalog query, and enforce recent-days > 0 in CLI input validation. +- Keep the inputs list consistent with existing built-in queries and rely on hide-internal-inputs to remove ?now-ms from list output. +- Reuse the same optional input handling logic as task-search for recent-days defaults. +- Ensure the query name is normalized and discoverable via `logseq query list`. +- Return a vector of db/id values with no ordering guarantees, matching task-search behavior. +- Keep db-worker-node unchanged since it already supports arbitrary datascript queries via thread-api/q. + +## Question + +recent-updated must return both pages and blocks, and exclude built-in entities by default. +recent-days must be greater than 0, and nil or non-positive values should be treated as invalid input. +Results must be returned as an unordered vector of db/id values, and users should use logseq show to view content and apply sorting. + +## Show -id duplicate filtering note + +When `logseq show -id` is given multiple ids, the output can include duplicate trees if some ids are children of others. +Filter out smaller (contained) trees from the result, so only the largest parent tree is kept. + +Example: + +```text +logseq show -id [7830,7831,7832,7833] +7830 Jan 23rd, 2026 #Journal +7831 ├── asdfasdfasdf +7832 │ └── yyyy +7833 └── [[xxxx]] yyy +================================================================ +7831 asdfasdfasdf +7832 └── yyyy +================================================================ +7832 yyyy +================================================================ +7833 [[xxxx]] yyy +``` + +--- diff --git a/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md new file mode 100644 index 0000000000..0763fe5b8b --- /dev/null +++ b/docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md @@ -0,0 +1,97 @@ +# Logseq CLI Housekeeping 2 Implementation Plan + +Goal: Simplify CLI options for show, move, and remove while keeping db-worker-node behavior unchanged. + +Architecture: The changes are limited to CLI option parsing, action building, and output formatting in the CLI layer. +The db-worker-node API calls remain the same, but we will verify expected input shapes for delete and move operations. +We will centralize shared id parsing so show and remove stay consistent. + +Tech Stack: ClojureScript, babashka.cli, promesa, Logseq CLI, db-worker-node thread-api. + +Related: Builds on docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md and relates to docs/agent-guide/014-logseq-cli-show-multi-id.md. + +## Testing Plan + +I will follow @test-driven-development by writing failing tests for each new CLI behavior before changing implementation. +I will add unit tests in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs for parsing and validation of the new remove options and renamed flags. +I will add unit tests for show option parsing to accept --page and reject --page-name and --format, while preserving global --output handling. +I will add unit tests for move option parsing to accept --target-page and reject --target-page-name. +I will add unit tests for remove parsing to accept --id, --uuid, and --page, and to reject multiple selectors or missing target. +I will add unit tests for remove parsing to accept multi-id vectors and to reject invalid or empty vectors. +I will run the CLI test namespace and confirm the new tests fail before any implementation changes. +I will rerun the CLI tests after each behavioral change to confirm they pass. + +Command to run tests is shown below. + +```bash +bb dev:test -v 'logseq.cli.commands-test' +``` + +Expected test output is described below. + +The output should include zero failures and zero errors for logseq.cli.commands-test. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Problem statement + +The CLI currently exposes overlapping option names and per-command format flags that conflict with the global output option. +The remove command is split into remove block and remove page, which makes scripting awkward and inconsistent with show and move selectors. +The move and show commands use page-name flags that should be renamed for clarity and consistency across commands. +We need a small, coordinated change that updates CLI parsing, action building, and documentation without changing db-worker-node APIs. + +## Plan + +1. Review current CLI option specs and validation in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs, and /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to confirm existing behavior and data shapes. +2. Review db-worker-node call sites for delete and move operations by searching for :delete-blocks and :delete-page usages to confirm expected argument shapes in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs and related call sites. +3. Add failing parsing and validation tests for the unified remove command and renamed flags in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. +4. Add failing tests that assert legacy flags and subcommands are rejected in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs. +5. Run the CLI test namespace and record the failing cases using the test command in the testing plan. +6. Extract the show id parsing logic into a shared helper in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/id.cljs or a similar shared namespace, and update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to use it. +7. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs to define a single remove spec with --id, --uuid, and --page, and to build actions based on which selector is present. +8. Update remove execution to support single and multiple id deletion while preserving page deletion behavior, and ensure returned data matches existing format expectations. +9. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs to rename --page-name to --page and remove the --format option and related validation. +10. Update show execution to use the resolved output format from config instead of a command-specific flag, while preserving human-readable output for the default format. +11. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/move.cljs to rename --target-page-name to --target-page and adjust validation and target resolution accordingly. +12. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs to reflect the new remove command, show selector, and move target flag in validation, action building, and help routing. +13. Update /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs if the new remove action type changes the command name used for human formatting. +14. Update CLI usage text in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/main.cljs to remove the remove block and remove page references. +15. Update CLI documentation and examples in /Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md to use --page, --target-page, and the unified remove command without --format. +16. Rerun the CLI test namespace and confirm all tests pass. +17. Run bb dev:lint-and-test if time permits and confirm there are no regressions in other CLI tests. +18. Review the changes against @prompts/review.md and ensure the updated flags are reflected everywhere in docs and tests. + +## Edge cases + +Removing blocks with a vector of ids that contains non-integers should produce a clear invalid options error. +Removing blocks with an empty id vector should produce a clear invalid options error. +Removing with both --id and --uuid should fail validation with a single-selector error. +Removing with both --page and a block selector should fail validation with a single-selector error. +Showing with --page should still reject invalid level values and missing targets. +Showing with --output json or edn should return structured data rather than human text for both single and multi-id cases. +Moving with --target-page should still reject --pos sibling. +Legacy flags like --page-name, --target-page-name, and show --format should be rejected by the parser with invalid-options errors. + +## Testing Details + +I will focus tests on CLI behavior by asserting parse-args results, invalid option errors, and build-action normalization for ids and selectors. +I will avoid mock-only tests and instead assert actual validation behavior and action shapes that drive CLI execution. + +## Implementation Details + +- Consolidate remove command parsing around a single spec and selector validation. +- Share id parsing between show and remove to keep behavior identical. +- Keep db-worker-node API calls unchanged and only adjust CLI argument shapes. +- Use config output-format in show execution to decide between human text and structured data. +- Remove show-specific format validation and option from the CLI help output. +- Rename show and move page flags and update all associated validation logic. +- Update CLI documentation and examples to match the new flags and remove subcommands. +- Update CLI help routing so logseq remove behaves like a command, not a group. + +## Decisions + +- Remove `--page-name` and `--target-page-name` entirely (no aliases or warnings). +- For remove with multiple ids, continue with best-effort deletion (do not fail on the first missing id). +- For remove ids, allow only the show-style vector and single value formats (no repeated `--id` flags). + +--- diff --git a/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md new file mode 100644 index 0000000000..f9158c0b4b --- /dev/null +++ b/docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md @@ -0,0 +1,199 @@ +# Logseq CLI Add Tag And Built-in Property Support Implementation Plan + +Goal: Extend logseq-cli add block and add page to accept tags and built-in properties with correct type handling. + +Architecture: Parse tag and property options in the CLI, resolve them via db-worker-node, and apply them using existing outliner ops. +Architecture: Use built-in property definitions and property type rules to coerce values before invoking :batch-set-property or :create-page. + +Tech Stack: ClojureScript, babashka.cli, Datascript, db-worker-node, outliner ops. + +Related: Relates to docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md. + +## Problem statement + +Logseq CLI currently supports add block and add page with content and basic status support. +Users need to set tags and built-in properties at creation time from CLI, but built-in properties have multiple value types and validation rules. +The implementation must align with the built-in property definitions and property type system so that values are stored and validated correctly in db-worker-node. + +## Testing Plan + +I will add unit tests that parse new CLI options for tags and properties and validate error cases for invalid property names and invalid type values. +I will add integration tests that run add block and add page with tags and built-in properties and assert the resulting data in the graph. +I will add tests that cover ref-type properties like :logseq.property/deadline and :block/tags to ensure resolution behavior is correct. +I will add tests that cover scalar properties like :logseq.property/publishing-public? and :logseq.property/heading with proper type coercion. +I will verify that invalid types cause CLI errors before any outliner ops are sent. +I will add tests that missing tags in --add-tag fail with a clear error and do not create tags. +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior summary + +Add block and add page live in src/main/logseq/cli/command/add.cljs. +Add block supports content, blocks, blocks-file, target selection, position, and status, and it sets status via :batch-set-property. +Add page uses :create-page with an empty options map and does not apply tags or properties. +Built-in properties and their schema are defined in deps/db/src/logseq/db/frontend/property.cljs and type rules in deps/db/src/logseq/db/frontend/property/type.cljs. + +## Requirements + +Add block supports setting tags on all inserted blocks. +Add page supports setting tags on the created page. +Add block supports setting built-in properties with correct type coercion. +Add page supports setting built-in properties with correct type coercion. +Add block and add page accept tag and property references by id, :block/title, or :db/ident for --tags and --properties. +CLI rejects non built-in properties for these new options. +CLI rejects built-in properties that are not public and provides a clear error message. +CLI rejects adding non public tags to blocks. +CLI rejects combining --blocks or --blocks-file with --add-property, --remove-property, --add-tag, or --remove-tag. + +## Non goals + +Do not change db-worker-node HTTP APIs or add new endpoints. +Do not change outliner property validation logic. +Do not add support for user properties in this change. + +## Built-in property type considerations + +Built-in property configuration lives in deps/db/src/logseq/db/frontend/property.cljs. +Built-in property types and validation live in deps/db/src/logseq/db/frontend/property/type.cljs. +Property types include user types (:default, :number, :date, :datetime, :checkbox, :url, :node) and internal types (:string, :keyword, :map, :coll, :any, :entity, :class, :page, :property, :raw-number). +Some built-in properties are not public and must not be user-settable via CLI. + +### Value coercion table + +| Property type | Expected CLI input | Resolution behavior | +| --- | --- | --- | +| :checkbox | boolean or "true"/"false" | Coerce to boolean and send directly. | +| :number | number or numeric string | Coerce to number and send directly. | +| :raw-number | number | Send directly and reject non numeric. | +| :datetime | ISO string | Convert to epoch ms number before sending. | +| :date | date string or journal page name | Resolve to journal page entity id. | +| :default | string or EDN | If string, pass as text value and let outliner create property value block. | +| :url | string | Validate with db-property-type/url? or allow macro urls and send as string. | +| :string | string | Send directly. | +| :keyword | keyword or string | Coerce string to keyword if safe. | +| :map | EDN map | Send directly. | +| :coll | EDN vector or list | Send directly. | +| :entity | block uuid or db/id | Resolve to entity id with :thread-api/pull or reject. | +| :page | page name or uuid | Resolve to page entity id, create page if missing. | +| :class | tag name or uuid | Resolve to class entity id and fail if missing. | +| :property | property name or keyword | Resolve to property entity id using built-in properties map. | +| :node | block uuid or page name | Resolve to entity id or allow node text block if required. | +| :any | EDN value | Send directly. | + +## Data flow overview + +CLI input is parsed by babashka.cli and normalized in src/main/logseq/cli/command/add.cljs. +CLI resolves tags and property values via db-worker-node using :thread-api/pull and existing outliner ops. +CLI applies properties using :batch-set-property for blocks and :create-page options for pages. + +ASCII architecture sketch. + +CLI add command + -> parse options + -> resolve tags and properties + -> db-worker-node :thread-api/apply-outliner-ops + -> outliner ops set properties and tags + +## Design decisions + +Tags are applied through :block/tags because it is the built-in tags property and is validated by outliner validation. +Page creation uses :create-page with :tags and :properties options to avoid separate post-create transactions. +Block creation uses existing insert blocks then batch-set-property for tags and each built-in property to keep behavior consistent with status handling. +Tags are always written to :block/tags and never to :logseq.property/page-tags. +Add block rejects any combination of --blocks or --blocks-file with --add-property, --remove-property, --add-tag, or --remove-tag. +Properties provided via CLI apply to all inserted blocks unless the input blocks already include explicit values in their block maps. +Properties provided via CLI override existing values on the newly created blocks to avoid ambiguity. + +## Implementation plan + +### 1. CLI option design and parsing + +Add new options to content-add-spec in src/main/logseq/cli/command/add.cljs for tags and properties. +Add new options to add-page-spec in src/main/logseq/cli/command/add.cljs for tags and properties. +Use a single EDN map option like --properties for multiple properties and a repeated option like --property for key value pairs if needed. +Use a single EDN vector option like --tags and a repeated option like --tag for convenience. +For --tags, allow each entry to be either a tag id, :db/ident, or the tag page's :block/title. +For --properties, allow each property key to be a property id, :db/ident, or the property's :block/title. +Update command summary and help output in src/main/logseq/cli/command/core.cljs if needed to reflect new options. + +### 2. Tag resolution helpers + +Add a helper in src/main/logseq/cli/command/add.cljs to normalize tag inputs into a vector of tag names or uuids. +Add a helper that validates each tag exists and fails fast when a tag is missing. +Use :thread-api/pull with lookup refs to resolve existing tag pages by name or uuid. +Return a vector of tag entity ids or uuids suitable for outliner ops. + +### 3. Built-in property resolution helpers + +Add a helper in src/main/logseq/cli/command/add.cljs to parse --properties EDN and validate keys against logseq.db.frontend.property/built-in-properties. +Reject keys not present in built-in-properties or not public based on :schema :public?. +Add a helper to get property type from built-in-properties and then coerce values using rules in deps/db/src/logseq/db/frontend/property/type.cljs. +Add a helper to resolve ref values into entity ids via :thread-api/pull for :page, :class, :property, :entity, and :node. +Add a helper to resolve :date to a journal page, creating it when missing if that is consistent with UI behavior. +Do not create tags implicitly when missing for --add-tag. + +### 4. Add page execution changes + +Extend build-add-page-action to carry tags and properties in the action context. +Modify execute-add-page in src/main/logseq/cli/command/add.cljs to pass :tags and :properties into the :create-page op options map. +Ensure property values are coerced before being sent so outliner validation passes. + +### 5. Add block execution changes + +Extend build-add-block-action to carry tags and properties in the action context. +After insert blocks and status application, apply tags via :batch-set-property with :block/tags and the resolved tag ids. +Apply each built-in property via :batch-set-property for the newly created block uuids. +Keep :keep-uuid? behavior for status so tags and properties can reference inserted block ids. + +### 6. CLI formatting and errors + +Update error messages in src/main/logseq/cli/commands.cljs to include new invalid option errors. +Add error formatting in src/main/logseq/cli/format.cljs if needed to show applied tags or properties in human output. +Ensure JSON output includes any new context if the CLI returns it. + +### 7. Tests and fixtures + +Add unit tests in src/test/logseq/cli/commands_test.cljs for option parsing and error handling. +Add integration tests in src/test/logseq/cli/integration_test.cljs that create blocks and pages with tags and built-in properties. +Add tests for at least one ref property (e.g. :logseq.property/deadline) and one scalar property (e.g. :logseq.property/publishing-public?). +Add tests for tag creation when tag pages do not exist. + +## Edge cases + +Tags that collide with private or built-in non-tag classes should be rejected by validation and surfaced to the CLI user. +Missing tags in --add-tag should produce a clear missing tag error without creating new tag pages. +Properties with closed values like :logseq.property/status should accept keyword idents as well as string labels where supported. +Date properties must resolve to journal pages or fail with a clear error if parsing is invalid. +Properties with cardinality many should accept vectors and sets and maintain ordering when required. +Inline tags or page namespaces should not be created implicitly without validation of allowed characters. + +## Resolved decisions + +CLI must not allow setting non public built-in properties, even with a force option. +Tags are applied via :block/tags and not :logseq.property/page-tags. +Add block must reject --blocks or --blocks-file when combined with --add-property, --remove-property, --add-tag, or --remove-tag. +Datetime values are provided as ISO strings. + +## Testing Details + +I will add CLI tests that run add page and add block end to end and assert the actual persisted properties and tags using list or show commands. +I will add tests that confirm invalid inputs fail fast and do not produce partial writes. +I will add tests that assert correct ref resolution for tag pages and journal pages. +I will ensure tests cover behavior rather than internal data structures and follow @test-driven-development. + +## Implementation Details + +- Update src/main/logseq/cli/command/add.cljs with new option parsing and action fields. +- Add tag and property normalization helpers in src/main/logseq/cli/command/add.cljs. +- Use deps/db/src/logseq/db/frontend/property.cljs to validate built-in property keys. +- Use deps/db/src/logseq/db/frontend/property/type.cljs to coerce values by type. +- Use :thread-api/pull in src/main/logseq/cli/command/add.cljs to resolve pages, tags, properties, and blocks. +- Pass tags and properties into :create-page ops in src/main/logseq/cli/command/add.cljs. +- Apply :batch-set-property for :block/tags and built-in properties in src/main/logseq/cli/command/add.cljs. +- Update src/test/logseq/cli/commands_test.cljs with parsing validation tests. +- Update src/test/logseq/cli/integration_test.cljs with behavior tests for tags and built-in properties. + +## Question + +No open questions. + +--- diff --git a/docs/agent-guide/019-logseq-cli-data-dir-permissions.md b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md new file mode 100644 index 0000000000..9819e53edf --- /dev/null +++ b/docs/agent-guide/019-logseq-cli-data-dir-permissions.md @@ -0,0 +1,82 @@ +# Logseq CLI Data-dir Permission Checks Plan + +Goal: Make logseq-cli validate read/write access for `data-dir` before it tries to start or communicate with db-worker-node, and surface a clear error when permissions are missing. + +Architecture: The CLI resolves `data-dir` in `logseq.cli.config` and uses it via `logseq.cli.server` to spawn and manage `db-worker-node`. `db-worker-node` also depends on `data-dir` for logs, locks, and SQLite storage via `frontend.worker.platform.node`. + +Tech Stack: ClojureScript, Node.js fs APIs, promesa, logseq-cli, db-worker-node. + +## Problem statement + +`data-dir` is used for locks, logs, and the local SQLite DB. Today, permission issues show up as late runtime exceptions (e.g., during lock creation or log file writes) with unclear error output. The CLI should proactively check that `data-dir` is readable and writable and return a clear error before attempting db-worker-node actions. + +## Current behavior summary + +- `logseq.cli.config/resolve-config` defaults `:data-dir` to `~/logseq/cli-graphs`. +- `logseq.cli.server` resolves and uses `data-dir` for locks and server discovery. +- `frontend.worker.db-worker-node` writes logs and lock files under `data-dir` and delegates storage to `frontend.worker.platform.node`, which creates directories as needed. +- No explicit read/write permission checks exist; errors bubble up from fs operations. + +## Requirements + +- CLI validates that `data-dir` is a directory with read and write permission. +- If `data-dir` does not exist, CLI attempts to create it (recursive). If creation or access fails, CLI returns an error. +- CLI surfaces a clear error code and message that includes the failing path. +- The check must run before any db-worker-node lifecycle or graph access that relies on `data-dir`. + +## Non-goals + +- Do not change db-worker-node storage layout or lock format. +- Do not add new CLI options for data-dir. +- Do not change API server behavior. + +## Design decisions + +- Treat `data-dir` as required to be read/write for all local-graph CLI commands. +- Convert permission failures into a consistent CLI error code (e.g., `:data-dir-permission`) and message. +- Reuse the same permission check in db-worker-node entrypoint to guard direct invocation. + +## Implementation plan + +### 1) Add a data-dir permission helper + +- Create a helper namespace (e.g., `src/main/logseq/cli/data_dir.cljs`) that: + - Expands `~` and normalizes the path. + - If missing, attempts `fs.mkdirSync` with `{:recursive true}`. + - Verifies the path is a directory (`fs.statSync`). + - Verifies read/write access with `fs.accessSync` (R_OK | W_OK). + - Throws `ex-info` with `{:code :data-dir-permission :path :cause }` on failure. + +### 2) Wire the check into CLI flow + +- In `src/main/logseq/cli/main.cljs`, after `config/resolve-config`, call the permission helper before `commands/build-action`/`commands/execute`. +- If the CLI supports API-token-only commands that do not touch local graphs, gate the check to only run for actions that require local graph access or server management. +- Map thrown permission errors into CLI error output with a clear message (e.g., "data-dir is not readable/writable: "). + +### 3) Add a safety check in db-worker-node + +- In `src/main/frontend/worker/db_worker_node.cljs`, run the same permission helper (or a small local equivalent) before `install-file-logger!` and before `platform-node/node-platform`. +- When this check fails, print a concise error to stderr and exit with code 1 to avoid partial startup. + +### 4) Update CLI error formatting + +- In `src/main/logseq/cli/format.cljs`, add an error hint for `:data-dir-permission` (e.g., "Check filesystem permissions or set LOGSEQ_CLI_DATA_DIR"). +- Ensure error output contains the path and permission type (read/write). + +### 5) Tests + +- Add unit tests in `src/test/logseq/cli` for the permission helper: + - Non-existent path that can be created succeeds. + - Path that is a file (not directory) fails. + - Read-only directory fails (use chmod to remove write permission in tmp dir). +- Add an integration test that runs CLI with `--data-dir` pointing to a non-writable directory and asserts the CLI returns error code `:data-dir-permission`. +- Add a graph-create case where `graph-dir` cannot be created (no mkdir permission) and assert a clear error is returned. +- Add a db-worker-node test (if there is a suitable harness) or extend existing CLI integration tests to assert db-worker-node start fails fast with the new error. + +## Open questions + +Resolved: +- Always check `data-dir` permissions, even when an API-server token is provided. +- Only create `data-dir` when a command needs it (local graph or server operations), not eagerly for all commands. + +--- diff --git a/docs/agent-guide/020-logseq-cli-default-paths-move.md b/docs/agent-guide/020-logseq-cli-default-paths-move.md new file mode 100644 index 0000000000..fea9f1bcad --- /dev/null +++ b/docs/agent-guide/020-logseq-cli-default-paths-move.md @@ -0,0 +1,77 @@ +# Logseq CLI Default Paths Move Plan + +Goal: Move the default `--data-dir` location to `~/logseq/cli-graphs` and the default `cli.edn` location to `~/logseq/cli.edn`, keeping logseq-cli and db-worker-node consistent. + +Architecture: logseq-cli resolves defaults in `logseq.cli.config` and `logseq.cli.data-dir`, then hands `data-dir` into `logseq.cli.server` which spawns and manages db-worker-node. db-worker-node itself also resolves `data-dir` for logs, locks, and SQLite storage via `frontend.worker.platform.node` and `frontend.worker.db-worker-node-lock`. + +Tech Stack: ClojureScript, Node.js fs/path, logseq-cli, db-worker-node. + +## Problem statement + +Defaults currently live under `~/.logseq/`, but CLI data is not the same as desktop app data and should live under `~/logseq/` for better discoverability and separation. We need to update the defaults in both logseq-cli and db-worker-node, and update docs/help text to match. + +## Current behavior summary + +- logseq-cli uses `~/logseq/cli-graphs` as the default data dir. +- db-worker-node help text and internal resolution also default to `~/logseq/cli-graphs`. +- logseq-cli defaults config path to `~/logseq/cli.edn`. +- Docs reference `~/logseq/cli.edn` and `~/logseq/cli-graphs`. + +## Requirements + +- Default `data-dir` becomes `~/logseq/cli-graphs` everywhere it is derived. +- Default config path becomes `~/logseq/cli.edn`. +- `--data-dir` and `--config` flags continue to override defaults. +- `LOGSEQ_CLI_DATA_DIR` and `LOGSEQ_CLI_CONFIG` (if present) continue to override defaults. +- Help text and docs must match the new defaults. + +## Non-goals + +- Do not migrate existing data automatically. +- Do not change CLI flags, env var names, or db-worker-node storage layout. +- Do not change runtime behavior beyond the default locations. + +## Design decisions + +- Keep default paths defined in a single place per subsystem (CLI vs db-worker-node), but ensure they resolve to the same new location. +- Do not auto-detect the old location as a fallback to avoid surprises; users can pass `--data-dir` / `--config` explicitly if needed. +- Document the change and provide a brief migration note in CLI docs. + +## Implementation plan + +### 1) Update default `data-dir` constants and resolution + +- `src/main/logseq/cli/data_dir.cljs` + - Change `default-data-dir` from `~/logseq/cli-graphs` to `~/logseq/cli-graphs`. +- `src/main/logseq/cli/server.cljs` + - Update `resolve-data-dir` default to `~/logseq/cli-graphs` (keeps server defaults aligned when config is absent). +- `src/main/frontend/worker/db_worker_node_lock.cljs` + - Update `resolve-data-dir` default to `~/logseq/cli-graphs`. +- `src/main/frontend/worker/platform/node.cljs` + - Update `node-platform` default for `data-dir` to `~/logseq/cli-graphs`. +- `src/main/frontend/worker/db_worker_node.cljs` + - Update the `--data-dir` help text default to `~/logseq/cli-graphs`. + +### 2) Update default config path for CLI + +- `src/main/logseq/cli/config.cljs` + - Change `config-path` default from `~/logseq/cli.edn` to `~/logseq/cli.edn`. + - Update any inline default map (`resolve-config` default options) to match if present. + +### 3) Update docs and internal references + +- `docs/cli/logseq-cli.md` + - Replace references to `~/logseq/cli.edn` and `~/logseq/cli-graphs` with the new paths. + - Add a short migration note: existing data/config can be used by passing `--data-dir` / `--config`. +- `docs/agent-guide/*.md` + - Update any references to the old defaults (notably `docs/agent-guide/002-logseq-cli-subcommands.md`, `docs/agent-guide/003-db-worker-node-cli-orchestration.md`, `docs/agent-guide/012-logseq-cli-graph-storage.md`, `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`, and `docs/agent-guide/task--db-worker-nodejs-compatible.md`). + +### 4) Tests + +- Unit tests likely unaffected, but adjust any tests or snapshots that assert default path strings (search for `~/logseq/cli-graphs` or `~/logseq/cli.edn` in tests). +- If tests assert CLI help output or default config path, update expected strings accordingly. + +## Notes + +- Do not add a one-time warning for the old `~/logseq/cli-graphs` location. If a config is needed, prefer `cli.edn` under the selected `data-dir`. +- Do not add any fallback or compatibility for `~/logseq/cli.edn`. The old location is ignored. diff --git a/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md b/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md new file mode 100644 index 0000000000..e5c723a4bc --- /dev/null +++ b/docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md @@ -0,0 +1,87 @@ +# Logseq CLI Add Reference UUID Rewrite Plan + +Goal: For logseq-cli add block/page content, replace every `[[]]` with `[[block-uuid]]` before calling db-worker-node thread-api, creating missing pages when needed. + +Architecture: logseq-cli (`src/main/logseq/cli/command/add.cljs`) sends `:thread-api/apply-outliner-ops` calls through `logseq.cli.transport`. db-worker-node executes `outliner-op/apply-ops!` without normalizing refs. Frontend already normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI does not. + +Tech Stack: ClojureScript, logseq-cli, db-worker-node, Datascript. + +## Problem statement + +Today logseq-cli passes block content to db-worker-node as-is. If content includes `[[Page Name]]`, db-worker-node does not automatically resolve or create that page for outliner ops. We need to normalize content refs to uuid references up front so db-worker-node receives canonical refs and missing pages are created deterministically. + +## Current behavior summary + +- `logseq-cli add block` builds blocks and calls `:thread-api/apply-outliner-ops` with `:insert-blocks`. +- `logseq-cli add page` calls `:thread-api/apply-outliner-ops` with `:create-page` only (no content normalization). +- db-worker-node `:thread-api/apply-outliner-ops` applies ops verbatim and does not resolve page refs in block content. +- Frontend editor normalizes refs using `logseq.db.frontend.content/title-ref->id-ref`, but CLI paths do not use it. + +## Requirements + +- For add block and add page content, replace every `[[]]` with `[[block-uuid]]` before invoking db-worker-node thread-api. +- Page reference: + - If `` is not a UUID, treat it as a page title. + - If the page does not exist, create it first. + - Replace `[[Page Title]]` with `[[page-uuid]]` (case-insensitive). +- Block reference: + - If `` is a UUID, treat it as a block ref. + - The block must exist; otherwise return a CLI error. +- Do not change other syntax (e.g. `((uuid))` block refs, tags, or macros) unless they are inside `[[...]]`. + +## Non-goals + +- Do not alter how `((uuid))` block refs are parsed or stored. +- Do not introduce automatic block creation for missing block UUIDs. +- Do not change CLI command flags or output format. + +## Design decisions + +- Reuse `logseq.db.frontend.content/title-ref->id-ref` so CLI behavior matches frontend normalization rules. +- Extract `[[...]]` refs using the existing page-ref regex from `logseq.common.util.page-ref` to avoid implementing a new parser. +- Resolve refs once per CLI action, cache page-name -> uuid, and then update all affected blocks before the first `:thread-api/apply-outliner-ops` call. +- Handle page creation with the existing `ensure-page!` helper in `src/main/logseq/cli/command/add.cljs` for consistent behavior. + +## Implementation plan + +### 1) Add reference extraction and resolution helpers + +- `src/main/logseq/cli/command/add.cljs` + - Add a helper to extract `[[...]]` tokens from a block title using `logseq.common.util.page-ref/page-ref-re`. + - Add a helper that partitions refs into: + - `uuid-refs`: `[[]]` values + - `page-refs`: `[[]]` values + - Add a resolver that: + - For `page-refs`, calls `ensure-page!` (once per unique title) and returns `{:block/uuid uuid :block/title title}`. + - For `uuid-refs`, pulls the entity by `[:block/uuid uuid]` and errors if missing. + +### 2) Normalize add block content before outliner ops + +- `src/main/logseq/cli/command/add.cljs` + - In `execute-add-block`, before building ops: + - Walk the `:blocks` tree (top-level and any nested `:block/children`) and collect all `[[...]]` refs. + - Resolve refs once per action using the resolver from step 1. + - For each block with `:block/title`, rewrite it with `db-content/title-ref->id-ref` using the resolved refs and `{:replace-tag? false}`. + - Use the rewritten blocks in the `:insert-blocks` op. + +### 3) Normalize add page content (when present) + +- `src/main/logseq/cli/command/add.cljs` + - If add page grows to accept initial blocks/content (or if `:create-page` options start supporting content), reuse the same ref normalization flow from step 2 before invoking `:create-page` or `:insert-blocks`. + - If no content is provided, no change is needed in the current implementation. + +### 4) Tests + +- `test/logseq/cli/integration_test.cljs` + - Add an integration test for `add block` with content `"See [[New Page]]"`: + - Assert the page is created. + - Pull the inserted block and verify its title contains `[[]]` instead of `[[New Page]]`. + - Add an integration test for `add block` with content `"See [[]]"`: + - Assert the UUID is preserved and no new page is created. + - Add an integration test for `add block` with content `"See [[]]"`: + - Assert CLI returns an error with a clear message. + +## Notes + +- If we later decide to normalize tags (`#tag`) or macro-based refs, we can extend the resolver to call `db-content/title-ref->id-ref` with `:replace-tag? true` and add tests accordingly. +- Keep all normalization in CLI to avoid changing db-worker-node semantics for other callers. diff --git a/docs/agent-guide/022-logseq-cli-help-show-output.md b/docs/agent-guide/022-logseq-cli-help-show-output.md new file mode 100644 index 0000000000..00d72911bb --- /dev/null +++ b/docs/agent-guide/022-logseq-cli-help-show-output.md @@ -0,0 +1,122 @@ +# Logseq CLI Help and Show Output Cleanup Implementation Plan + +Goal: Improve logseq-cli help readability and ensure show JSON and EDN outputs use :db/id without :block/uuid. + +Architecture: The CLI help text is assembled in logseq.cli.command.core and babashka.cli spec descriptions, while show output is built in logseq.cli.command.show and formatted in logseq.cli.format before returning JSON or EDN. + +Tech Stack: ClojureScript, logseq-cli, babashka.cli, db-worker-node. + +Related: Relates to 018-logseq-cli-add-tags-builtin-properties.md. + +## Problem statement + +The current help output lists [options] for nearly every command, which clutters the help display and reduces readability. + +The tags and properties option help text does not clearly state that identifiers can be id, :db/ident, or :block/title across all relevant help contexts. + +The show command includes :block/uuid in JSON and EDN output, even though :db/id is already present and preferred for programmatic consumers. + +``` +logseq-cli + | help text built from babashka.cli specs + v +logseq.cli.command.core + | parse args and build command payloads + v +logseq.cli.command.show + | fetch tree data from db-worker-node + v +logseq.cli.format + | render human, json, edn output + v +stdout +``` + +## Testing Plan + +I will add or update unit tests that assert help summaries do not include repeated [options] in the command list, and that tags and properties descriptions include the identifier guidance. + +I will add or update unit tests that verify show JSON and EDN outputs do not include :block/uuid and still include :db/id for root and child nodes. + +I will add or update integration tests for show JSON output to assert :db/id is present and :block/uuid is absent in root, tags, and linked references where applicable. + +NOTE: I will write all tests before I add any implementation behavior. + +## Requirements + +The top level help output and group summaries must avoid repeating [options] for each command listing while still documenting that options exist in the usage line. + +The help description for --tags and --properties must explicitly state that identifiers can be id, :db/ident, or :block/title. + +The show command must omit :block/uuid from JSON and EDN outputs while preserving :db/id for the same entities. + +The show command human output must be unchanged. + +## Non-goals + +Do not change CLI command behavior or supported flags beyond help text updates. + +Do not change db-worker-node behavior or its API surface. + +Do not change the structure of human output for show, list, add, or query commands. + +## Design decisions + +Limit the help output adjustment to formatting in logseq.cli.command.core so command behavior and parsing remain unchanged. + +Apply the identifier clarification to all --tags and --properties options in logseq.cli.command.add and any other specs that expose those flags. + +Strip :block/uuid only for show outputs in JSON and EDN formats by post-processing tree data just before returning payloads. + +## Implementation plan + +1. Follow @test-driven-development for every change in this plan. + +2. Add a unit test in src/test/logseq/cli/commands_test.cljs that asserts command list rows in top level and group help do not contain [options] after the command name. + +3. Add a unit test in src/test/logseq/cli/commands_test.cljs that asserts the --tags and --properties option descriptions include the text supporting id, :db/ident, and :block/title identifiers. + +4. Add a unit test in src/test/logseq/cli/format_test.cljs or src/test/logseq/cli/commands_test.cljs that asserts show JSON and EDN outputs strip :block/uuid while retaining :db/id in root and child nodes. + +5. Update integration tests in src/test/logseq/cli/integration_test.cljs that currently assert :uuid or :block/uuid in show JSON output to instead assert :db/id and absence of :block/uuid. + +6. Adjust logseq.cli.command.core command listing formatting so only the usage line includes [options], and the command listing uses the bare command path without the suffix. + +7. Update the --tags and --properties option descriptions in src/main/logseq/cli/command/add.cljs to include the identifier guidance sentence in a consistent phrasing. + +8. Add a helper in src/main/logseq/cli/command/show.cljs or src/main/logseq/cli/format.cljs that removes :block/uuid keys from show JSON and EDN payloads, and apply it in execute-show when output-format is :json or :edn. + +9. Run the updated unit tests and integration tests from the Testing Plan and confirm all pass. + +## Edge cases + +Command help should still show [options] in the usage line for commands that accept options, but not in the command list table. + +Multi-id show results should strip :block/uuid from each tree entry without changing the error map shape. + +Linked references and tag entities should keep :db/id in the output even when :block/uuid is removed. + +## Testing Details + +I will add tests that verify help summaries and option descriptions at the command summary level and not by matching the raw babashka.cli output. + +I will add tests that parse show JSON and EDN output and assert :block/uuid is missing while :db/id remains on block nodes. + +I will update integration tests that read show JSON output to match the new key expectations without changing the test setup logic. + +## Implementation Details + +- Update logseq.cli.command.core formatting to render command rows without [options]. +- Keep usage lines intact so users still see options availability in usage sections. +- Align help text for --tags and --properties to a single wording that mentions id, :db/ident, and :block/title. +- Add a show-specific sanitization step for json and edn output only. +- Keep the show tree data used for human output unchanged to avoid regressions. +- Ensure strip logic is recursive so :block/uuid is removed from nested children and linked references. +- Prefer clojure.walk/postwalk for key removal to minimize custom traversal code. +- Document the new behavior in tests rather than adding new user-facing docs. + +## Question + +This is resolved. Only add supports --tags and --properties today, so we will update help text there only. + +--- diff --git a/docs/agent-guide/023-logseq-cli-help-show-styling.md b/docs/agent-guide/023-logseq-cli-help-show-styling.md new file mode 100644 index 0000000000..c0c25da8e3 --- /dev/null +++ b/docs/agent-guide/023-logseq-cli-help-show-styling.md @@ -0,0 +1,108 @@ +# Logseq CLI Help And Show Styling Implementation Plan + +Goal: Add picocolors-based styling for help output and show human output in logseq-cli and db-worker-node. + +Architecture: Introduce a small shared styling helper that wraps picocolors and is used by CLI help renderers and show tree formatting. +Help output headings and error strings will apply bold to keywords while show output will color status labels and bold tag suffixes without changing data payloads. + +Tech Stack: ClojureScript, Node.js, picocolors, babashka.cli. + +Related: Relates to docs/agent-guide/022-logseq-cli-help-show-output.md. + +## Problem statement + +The current help output in logseq-cli and db-worker-node is plain text and does not emphasize command names or option names, which makes scanning harder. +The show command human output also does not visually differentiate status values or tags, which makes scanning large trees harder. +The tree glyphs in show output currently have the same visual weight as content, which makes the structure harder to scan. + +## Testing Plan + +I will add unit tests for help summary formatting that assert bold styling is applied to command names, option names, and error messages for missing options. +I will add unit tests for show tree text rendering that verify status labels are colorized and tag suffixes are bolded while preserving the existing tree glyph alignment when ANSI codes are stripped. +I will add a unit test that verifies tree glyphs are rendered in a lighter style without altering alignment when ANSI is stripped. +I will add a unit test for db-worker-node help output that asserts bold styling on command names and option names and that the help text still omits auth-token. +I will add unit tests for the styling helper to ensure it can be disabled for tests by stripping ANSI when comparing output. +NOTE: I will write all tests before I add any implementation behavior. + +## Scope and constraints + +This plan targets logseq-cli in src/main/logseq/cli and db-worker-node help output in src/main/frontend/worker/db_worker_node.cljs. +This plan must use picocolors for color and bold styling and should not change JSON or EDN output formats. +This plan should not introduce new CLI options unless required to gate coloring for tests. +Styling must only be applied when color is supported, and dumb terminals must receive plaintext output. +The `logseq -h` help output should omit the commands list section. + +## Files and ownership + +| Area | Path | Notes | +| --- | --- | --- | +| npm dependency | package.json | Add picocolors dependency used by ClojureScript Node targets. | +| npm lockfile | pnpm-lock.yaml | Update to include picocolors. | +| CLI help summary | src/main/logseq/cli/command/core.cljs | Apply bold styling to command names and option names in help summaries and error text. | +| CLI show output | src/main/logseq/cli/command/show.cljs | Apply status color and tag bold styling in tree labels. | +| CLI show output | src/main/logseq/cli/command/show.cljs | Apply lighter styling to tree glyphs in human output. | +| CLI formatting helpers | src/main/logseq/cli/format.cljs | Avoid impacting non-human output, and ensure show uses styled message for human output only. | +| CLI legacy help | deps/cli/src/logseq/cli.cljs | Apply bold styling to command names and option names in help output for legacy cli entrypoint. | +| db-worker-node help | src/main/frontend/worker/db_worker_node.cljs | Apply bold styling to command names and option names in help output lines. | +| CLI tests | src/test/logseq/cli/commands_test.cljs | Update help summary and show tree tests to tolerate ANSI and assert styling intent. | +| CLI format tests | src/test/logseq/cli/format_test.cljs | Add or update tests to ensure human show output includes styled text but JSON and EDN do not. | +| db-worker-node tests | src/test/frontend/worker/db_worker_node_test.cljs | Extend help output test to validate bold styling. | + +## Implementation plan + +1. Add picocolors to package.json dependencies and update pnpm-lock.yaml with the new dependency using pnpm. +2. Create a small styling helper in a new namespace such as src/main/logseq/cli/style.cljs that wraps picocolors functions for bold and color and exposes a no-color flag for tests. +3. Add a companion helper in a shared location for db-worker-node, or reuse the same namespace if it is available in that build target, to avoid duplicated color logic. +4. In src/main/logseq/cli/style.cljs, add a color support check that disables styling when color is not supported or TERM is dumb. +5. In src/main/logseq/cli/command/core.cljs, wrap help summary command names and option names with the new bold helper. +6. In src/main/logseq/cli/command/core.cljs, update the invalid options error formatting so missing required option names are bolded in the error message. +7. In deps/cli/src/logseq/cli.cljs, apply the same bold styling to command names and option names in the help output for the legacy cli entrypoint. +8. In src/main/frontend/worker/db_worker_node.cljs, update show-help! output to bold command names and option names in the help text. +9. In src/main/logseq/cli/command/show.cljs, add a status style function that maps known status labels to distinct colors, and bolds the status text, using picocolors. +10. In src/main/logseq/cli/command/show.cljs, update the tag suffix rendering to wrap each #tag with bold styling and ensure tags remain separated by spaces. +11. In src/main/logseq/cli/command/show.cljs, style the tree glyphs with a dim or gray color using picocolors while leaving ids and labels unstyled. +12. In src/main/logseq/cli/command/show.cljs, ensure status formatting and glyph styling are only applied to the human output path and do not alter the underlying data used for JSON or EDN outputs. +13. Update src/test/logseq/cli/commands_test.cljs to compare help summaries using an ANSI-stripping helper so assertions remain stable, and to assert bold styling for command and option names. +14. Update src/test/logseq/cli/commands_test.cljs show tree text tests to assert that the status prefix and tag suffix are styled when ANSI is preserved, and to verify tree alignment and glyph lightening using stripped output. +15. Add or update tests in src/test/logseq/cli/format_test.cljs to verify that human show output includes styled prefixes while JSON and EDN outputs remain unchanged. +16. Update src/test/frontend/worker/db_worker_node_test.cljs to assert that the help output bolds command and option names and still omits auth-token. +17. Run bb dev:lint-and-test to ensure all lint and unit tests pass. + +## Edge cases + +The status value may be a keyword with namespaces such as :logseq.property.status/todo and should still map to the same color for TODO. +The status label may be missing or blank, and the show output should remain unchanged in that case. +Tag labels may include uppercase or punctuation and should still render as bolded tags without losing the leading #. +Help output should still be readable when ANSI colors are not supported, and tests should be resilient by stripping ANSI sequences. +Tree glyph styling should not break alignment when ANSI codes are stripped. +Styling should be fully disabled when color is not supported or TERM is dumb. + +## Open questions + +Should picocolors styling be applied only when stdout is a TTY, or should it always render for human output regardless of terminal support. +Which specific status to color mapping is preferred for the full set of Logseq statuses such as NOW, LATER, WAITING, CANCELLED, and TODO variants. + +## Testing Details + +The tests will verify visible behavior by asserting that help output includes bolded command names and option names and that show output includes styled status and tags when rendered to human text. +The tests will also assert that JSON and EDN outputs remain unchanged and that ANSI codes do not break alignment by validating stripped output. +The tests will continue to avoid asserting internal data structures and instead focus on rendered output behavior. + +## Implementation Details + +- Use a small helper that can apply bold and color via picocolors and also expose a strip-ansi helper for tests. +- Keep styling limited to human output paths and avoid touching transport or data payloads. +- Centralize the status to color mapping in one function to keep future changes easy. +- Apply bold to command names and option names in help output and error strings. +- Preserve existing spacing and alignment by applying styling after label construction rather than before width calculations. +- Apply a lighter style to tree glyphs only, not to ids or labels. +- Gate styling behind color support checks so dumb terminals get plaintext output. +- Ensure any new helper is available to both the CLI and db-worker-node build targets. +- Update tests to use ANSI stripping for alignment assertions and explicit style presence for keyword checks. +- Avoid adding new configuration flags unless tests cannot reliably assert output without them. + +## Question + +Styling is limited to help info and show human output for now. + +--- diff --git a/docs/agent-guide/024-logseq-cli-show-updates.md b/docs/agent-guide/024-logseq-cli-show-updates.md new file mode 100644 index 0000000000..4bc6d1d19e --- /dev/null +++ b/docs/agent-guide/024-logseq-cli-show-updates.md @@ -0,0 +1,99 @@ +# Logseq CLI Show Output & Linked References Options Plan + +Goal: Update the `logseq show` command to (1) render the ID column in human output with lighter styling (matching tree glyphs), (2) include `:db/ident` on block entities in JSON/EDN output when present, and (3) add an option to toggle Linked References (default enabled). +Architecture: Keep the existing logseq-cli → db-worker-node HTTP transport and thread-api calls; adjust show command selectors and output formatting only. +Tech Stack: ClojureScript, babashka.cli, db-worker-node transport, Logseq pull selectors, CLI styling helpers. +Related: Builds on `logseq.cli.command.show` tree rendering and linked references fetch; see `docs/agent-guide/010-logseq-cli-show-linked-references.md` and `docs/agent-guide/023-logseq-cli-help-show-styling.md`. + +## Problem Statement + +The `show` command currently renders the ID column with the same styling as regular text, does not expose `:db/ident` in JSON/EDN output for blocks, and always fetches/prints Linked References. We need to make the ID column visually lighter, surface `:db/ident` when it exists, and add a switch to disable linked references (defaulting to true). + +## Non-Goals + +- Changing db-worker-node APIs or transport protocols. +- Altering the show command’s existing tree structure or block ordering. +- Changing JSON/EDN output structure beyond adding `:db/ident` when present. + +## Current Behavior (Key Points) + +- Human output is rendered by `tree->text` in `src/main/logseq/cli/command/show.cljs`, with tree glyphs styled via `style/dim` but IDs unstyled. +- JSON/EDN output pulls specific selectors via `tree-block-selector` / `linked-ref-selector` and strips only `:block/uuid`. +- Linked References are always fetched and rendered via `fetch-linked-references` + `tree->text-with-linked-refs`. + +## Proposed Changes + +1) **Lighter ID column in human output** + - Style the padded ID column with the same dim styling used for tree glyphs. + - Apply the change in `tree->text` so both root and child rows render IDs dimmed. + +2) **Include `:db/ident` in JSON/EDN for blocks (when present)** + - Add `:db/ident` to block pull selectors: + - `tree-block-selector` (children) + - `linked-ref-selector` (linked references) + - root entity pulls in `fetch-tree` for `:id`, `:uuid`, and `:page` + - No post-processing is required because absent attributes will not appear in the pulled maps. + +3) **Optional Linked References (default true)** + - Add a boolean option `--linked-references` to `show-spec`, defaulting to true. + - Pass the option through `build-action` and into `build-tree-data`. + - When disabled: + - Skip `fetch-linked-references` entirely. + - Omit the `Linked References` section from human output. + - Omit `:linked-references` from JSON/EDN output. + +## Implementation Plan + +1) **Add show option wiring** + - Update `show-spec` in `src/main/logseq/cli/command/show.cljs` with a new boolean option (choose name + description, note default true). + - Extend `build-action` to include the option in `:action` (e.g., `:linked-references?`). + - Update `invalid-options?` if needed for any new validation. + +2) **Gate linked references fetching/rendering** + - Update `build-tree-data` to honor the new option: + - If enabled, keep current logic. + - If disabled, skip `fetch-linked-references`, avoid UUID label resolution based on linked refs, and avoid `tree->text-with-linked-refs`. + - Update `execute-show` to select `tree->text` vs `tree->text-with-linked-refs` based on the option. + - Ensure JSON/EDN output omits `:linked-references` when disabled. + +3) **Add `:db/ident` to selectors** + - Update `tree-block-selector`, `linked-ref-selector`, and root entity pull vectors in `fetch-tree` to include `:db/ident`. + - Ensure this does not add nil values (pull should omit missing attributes). + +4) **Dim ID column in human output** + - In `tree->text`, apply `style/dim` to the padded ID string (e.g., `pad-id` output) for both root and child rows. + - Keep existing branch/prefix dim styling unchanged. + +5) **Help output updates** + - Ensure `logseq show --help` lists the new linked references option (covered automatically by `show-spec`). + +## Testing Plan + +- Update or add unit tests in: + - `src/test/logseq/cli/format_test.cljs` to verify: + - ANSI dim styling is applied to the ID column in human output when color is enabled. + - No regressions in strip-ansi output. + - `src/test/logseq/cli/commands_test.cljs` to verify: + - `logseq show --help` includes the new linked references option. +- Add show-specific tests for JSON/EDN output: + - Ensure `:db/ident` appears when present (use a stubbed tree map or test fixtures if available). + - Ensure `:linked-references` is omitted when the option disables them. + +## Edge Cases + +- Blocks with no `:db/ident` should not gain a nil or empty key in output. +- When linked refs are disabled, UUID label resolution should not depend on linked refs (ensure `collect-uuid-refs` handles empty refs). +- Multi-id output should still honor the linked references option per target and not render linked refs when disabled. + +## Files to Touch + +- `src/main/logseq/cli/command/show.cljs` (options, selectors, tree output, linked refs gating) +- `src/test/logseq/cli/format_test.cljs` (ID dim styling test) +- `src/test/logseq/cli/commands_test.cljs` (help output) +- Possibly `src/test/logseq/cli/command_show_test.cljs` or similar if a dedicated show test file exists + +## Open Questions + +- None. + +--- diff --git a/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md b/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md new file mode 100644 index 0000000000..b185015d8b --- /dev/null +++ b/docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md @@ -0,0 +1,94 @@ +# Logseq CLI Built-in Status/Priority Queries Plan + +Goal: Add built-in `query` names `list-status` and `list-priority` that return the available Status/Priority options from a graph. +Architecture: Keep the existing logseq-cli → db-worker-node transport and thread-api usage; implement `list-status`/`list-priority` via `:thread-api/q` rather than adding a dedicated thread-api. +Tech Stack: ClojureScript, logseq-cli, db-worker-node, Datascript, logseq.db.frontend.property. +Related: `src/main/logseq/cli/command/query.cljs`, `src/main/frontend/worker/db_core.cljs`, `deps/db/src/logseq/db/frontend/property.cljs`. + +## Problem Statement + +Logseq CLI’s `query` built-ins are limited to Datascript queries. Users cannot easily discover valid Status or Priority options for a graph (e.g., TODO/DOING/DONE, Urgent/High/Low) without knowing the property internals or running custom queries. We need simple built-ins that list the closed values configured for Status and Priority. + +## Non-Goals + +- Changing property schemas or defaults. +- Replacing the existing Datascript query flow. +- Adding new UI commands outside `query` or changing CLI output formats. + +## Current Behavior (Key Points) + +- Built-in queries are defined in `built-in-query-specs` in `src/main/logseq/cli/command/query.cljs` and executed via `:thread-api/q`. +- Status and Priority options are stored as closed values on `:logseq.property/status` and `:logseq.property/priority` (see `logseq.db.frontend.property/get-closed-property-values`). +- There is no CLI helper to list those closed values. + +## Proposed Changes + +1) **Use `:thread-api/q` for closed values** + - Implement `list-status`/`list-priority` by issuing a Datascript query via `:thread-api/q`. + - Return a vector of maps with `:db/ident` and `:db/id`. + - Use `:find` with an ellipsis form to return a vector, e.g. `:find [(pull ?e [:db/ident :db/id]) ...]` (instead of `:find (pull ?e [:db/ident :db/id])`). + +2) **Add built-in query names `list-status` and `list-priority`** + - Add entries to `built-in-query-specs` that specify a non-Datascript execution path (e.g., `:method`/`:handler` metadata). + - Wire `build-action` to create a dedicated action type when these names are used. + - Update `execute-query` to call the new thread API and return results as `{:result ["TODO" ...]}` so existing output formatting works unchanged. + +3) **Expose built-ins in `query list`** + - Ensure `query list` includes the new entries (with `doc` and `inputs: []`). + - Keep existing “custom overrides built-in” semantics. + +## Implementation Plan + +1) **db-worker-node API** + - No new thread-api should be added; use `:thread-api/q` only. + +2) **CLI built-in wiring** + - Extend `built-in-query-specs` in `src/main/logseq/cli/command/query.cljs`: + - `list-status` → `:property-ident :logseq.property/status` + - `list-priority` → `:property-ident :logseq.property/priority` + - Include `:doc` and `:inputs []`. + - Update `normalize-query-entry` / `find-query` handling to preserve the extra metadata. + - Update `build-action`: + - When the built-in includes a `:property-ident` (or `:method`), create an action like + `{:type :query-closed-values :repo ... :property-ident ...}`. + - Update `execute-query`: + - Branch on the new action type to call `transport/invoke` with `:thread-api/q` and return `{:result values}` (vector of maps with `:db/ident` and `:db/id`). + +3) **Output and docs** + - No change to formatters; `format-query-results` already prints vectors. + - Update CLI docs if needed (e.g., `docs/cli/logseq-cli.md`) to mention the two built-ins. + +## Testing Plan + +- **Unit tests** (`src/test/logseq/cli/command/query_test.cljs`): + - `list-queries` includes `list-status` and `list-priority` with empty inputs. + - `build-action` for `--name list-status` returns `:query-closed-values` action with property ident. + - Custom query overrides built-in name still works. + +- **db-worker-node tests**: no new thread-api, so no additional db-worker-node test needed. + +- **Integration** (`src/test/logseq/cli/integration_test.cljs`): + - Start a graph, run `logseq query --name list-status` / `list-priority`, assert status `ok` and a non-empty vector. + - If stable defaults are known, assert a known value is present; otherwise, seed closed values in the test graph. + +## Edge Cases + +- Property has no closed values → return an empty vector (not an error). +- Closed values may be stored in either `:block/title` or `:logseq.property/value`; if `:db/ident` is absent, the map should still include the key with a nil value. +- Ensure ordering follows `:block/order` from `get-closed-property-values`. + +## Files to Touch + +- `src/main/frontend/worker/db_core.cljs` (no new thread-api) +- `src/main/logseq/cli/command/query.cljs` (built-in specs, build-action branching, execute-query) +- `src/test/logseq/cli/command/query_test.cljs` (unit coverage) +- `src/test/frontend/worker/db_worker_node_test.cljs` or a db-core test (not required for new thread-api) +- `src/test/logseq/cli/integration_test.cljs` (CLI end-to-end) +- `docs/cli/logseq-cli.md` (optional docs update) + +## Open Questions + +- Use `:thread-api/q` to return structured values: `{:db/ident .., :db/id ..}`. +- Expose `list-status`/`list-priority` via `query --name` only (no dedicated subcommands). + +--- diff --git a/docs/agent-guide/026-logseq-cli-query-output.md b/docs/agent-guide/026-logseq-cli-query-output.md new file mode 100644 index 0000000000..7c36123334 --- /dev/null +++ b/docs/agent-guide/026-logseq-cli-query-output.md @@ -0,0 +1,70 @@ +# Logseq CLI Query Output Piping Implementation Plan + +Goal: Remove the space-to-comma transformation in CLI query human output while preserving EDN output and ensuring `logseq show --id` accepts ids both via argument and stdin pipelines. + +Architecture: Keep human formatting for query results as EDN output (no line-oriented transformation) and update tests to validate pipeline usage with intact EDN. +Architecture: Extend logseq show --id to read ids from stdin when present, whether or not an id argument is provided, and update integration tests to validate both xargs and direct stdin pipelines. + +Tech Stack: ClojureScript, Logseq CLI, db-worker-node, Node-based integration tests. + +Related: Relates to docs/agent-guide/025-logseq-cli-builtin-status-priority-queries.md. + +## Testing Plan + +I will add an integration test that ensures a human-output query can be piped into xargs and then into logseq show --id for multiple ids, with the query output preserved as EDN. +I will add an integration test that ensures a human-output query can be piped directly into logseq show --id via stdin, with or without an explicit id argument. +I will update the existing integration test that currently pipes human query output directly into logseq show so it asserts EDN preservation and stdin ingestion. +I will add a focused unit test for format-query-results that covers EDN output for scalar collections, non-scalar collections, and nil results. +I will add a unit test for show id parsing that covers stdin provided, stdin blank, and explicit id values. + +NOTE: I will write all tests before I add any implementation behavior. + +## Problem statement + +The CLI currently replaces spaces with commas in format-query-results, which is a lossy transformation that makes output less readable. +We need to remove the space-to-comma logic while preserving EDN output (including spaces) for query results. +Pipelines should remain usable by allowing logseq show --id to accept ids from stdin and from explicit arguments interchangeably. + +## Plan + +1. Read the existing formatting logic in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs and document current behavior in a quick note. +2. Read the show command option parsing in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs and /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/id.cljs to understand current id validation. +3. Locate the current integration test that verifies human query output piping in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs and map the required changes. +4. Define the new output rule for format-query-results as a comment in the plan: always return EDN output unchanged (including spaces), and remove space-to-comma logic. +5. Define the new show --id stdin rule as a comment in the plan: when stdin is non-empty, parse it as the id or id vector string and use it regardless of whether an id argument is provided. +6. Write the failing unit tests for format-query-results in a new test namespace under /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs that covers nil, scalar-id sequences, and nested maps, all preserved as EDN strings. +7. Write failing unit tests for show id parsing in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/show_test.cljs that cover stdin provided (overriding or complementing args), stdin blank, and explicit id values. +8. Run the unit tests to confirm the failure before implementation using bb dev:test with the new namespaces. +9. Write a failing integration test that uses the exact pipeline command from the request by invoking a shell command from the test harness in /Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs for xargs usage. +10. Write a failing integration test that uses the exact pipeline command from the request for direct stdin piping into logseq show --id with or without an argument. +11. Run the integration tests to confirm the failure before implementation using bb dev:test with the specific test names. +12. Implement the new format-query-results behavior in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs by removing the space-to-comma logic and preserving EDN output. +13. Implement stdin ingestion for show --id in /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/show.cljs, using a helper to read from stdin when non-empty regardless of whether an id argument is provided. +14. Update the existing human-output pipeline integration test to align with the EDN-preserving output and assert both xargs and direct stdin usage. +16. Re-run the unit tests and the integration tests to validate the new behavior. +17. Update deps/cli/README.md examples if they reference comma-transformed output, and add a short note describing the new xargs-friendly and stdin-friendly behavior for id lists. + +## Testing Details + +The unit tests will exercise behavior by calling format-query-results with representative results and asserting the exact emitted EDN string for id lists and non-id data. +The integration tests will execute logseq query with a task-search or custom datalog query, pipe the EDN output through xargs into logseq show --id, and pipe directly into logseq show --id via stdin. +The show stdin behavior test will assert that missing --id input without stdin returns a clear error and that stdin is parsed the same as an explicit id argument. + +## Implementation Details + +- Preserve the existing safe-read-string validation to avoid changing behavior for invalid EDN strings. +- Keep non-scalar results as their original EDN string output. +- Maintain nil handling so that "nil" still prints as "nil". +- Remove the space-to-comma transformation while keeping EDN output intact, including spaces. +- Add stdin reading to show --id when stdin is non-empty, even if an id argument is provided. +- Parse stdin through the same id parsing function used for explicit --id values. +- Keep existing errors when --id is missing and stdin is empty or whitespace. +- Update integration tests to use the exact pipeline command from the requirement. +- Use @test-driven-development for all implementation steps. + +## Question + +Query results remain in EDN form (e.g., `[1 2 3]`) with no line-oriented transformation. +Logseq show --id accepts ids via explicit argument and via stdin pipelines (stdin takes precedence when provided). + +--- diff --git a/docs/agent-guide/027-logseq-cli-update-command.md b/docs/agent-guide/027-logseq-cli-update-command.md new file mode 100644 index 0000000000..1911937726 --- /dev/null +++ b/docs/agent-guide/027-logseq-cli-update-command.md @@ -0,0 +1,133 @@ +# Logseq CLI Update Command (Move Refactor) + +## Summary +Introduce a new `update` CLI command that subsumes the current `move` command and adds tag/property updates. The new command keeps all existing move capabilities/options, makes move targets optional (no target means no move), and adds update/remove semantics for tags and properties. The plan is grounded in current `logseq-cli` parsing/execution and db-worker-node outliner ops. + +## Goals +- Replace `move` command behavior with `update` while keeping all current move options and semantics. +- Allow `update` to move blocks *and/or* update tags/properties in one command. +- Support new options: + - `--update-tags`, `--update-properties` + - `--remove-tags`, `--remove-properties` +- Reuse tag/property parsing and resolution rules from `add block`: + - Identifiers accept `db/id`, `db/ident`, `block/title`. + - `--update-tags` and `--update-properties` accept the same EDN format as `add block` `--tags`/`--properties`. + - `--remove-tags` and `--remove-properties` accept EDN vectors of tag/property identifiers. + +## Non-goals +- Changing db-worker-node APIs or introducing new outliner ops. +- Expanding `update` beyond blocks (no page-level update scope in this iteration). +- Changing move semantics or target resolution. + +## Background: Current Move Command +- CLI move implementation lives in `src/main/logseq/cli/command/move.cljs`. +- It requires a source (`--id` or `--uuid`) and a target (`--target-id`, `--target-uuid`, or `--target-page`). +- Execution uses `:thread-api/apply-outliner-ops` with `[:move-blocks ...]`. +- Move output formatting in `src/main/logseq/cli/format.cljs` uses `format-move-block`. + +## Proposed Behavior (Update Command) +### Required/Optional Inputs +- **Source is required**: one of `--id` or `--uuid`. +- **Move targets are optional**: + - `--target-page`, `--target-uuid`, `--target-id`, `--pos` are optional. + - If no target is provided, no move occurs and the command only updates tags/properties. + - If a target is provided, move executes with the same semantics as today. +- **Update/remove options are optional**: + - `--update-tags` (EDN vector, same as `add block --tags`) + - `--update-properties` (EDN map, same as `add block --properties`) + - `--remove-tags` (EDN vector of tag identifiers) + - `--remove-properties` (EDN vector of property identifiers) + +### Validation Rules +- Only one source selector is allowed: `--id` or `--uuid`. +- Only one target selector is allowed: `--target-id`, `--target-uuid`, or `--target-page`. +- `--pos sibling` is only valid when the target is a block (not a page), same as `move`. +- At least one of the following must be provided: target (move) or update/remove options. +- `--update-tags` and `--update-properties` accept the same EDN grammar and validations as in `add`. +- `--remove-tags` and `--remove-properties` must be non-empty vectors (EDN), with identifiers validated similarly to add. + +### Execution Semantics +- Resolve source block to `db/id` using existing move helpers. +- If move target provided, resolve target entity using existing move helpers and compute `pos` opts. +- Tag/property updates use current outliner ops (no new db-worker changes): + - **Add/update tags**: `[:batch-set-property [block-ids :block/tags tag-id {}]]` for each tag. + - **Add/update properties**: `[:batch-set-property [block-ids property-id value {}]]`. + - **Remove tags**: `[:batch-delete-property-value [block-ids :block/tags tag-id]]`. + - **Remove properties**: `[:batch-remove-property [block-ids property-id]]`, with property resolution aligned to `add` (built-in/public property rules). +- Combine operations into a single `apply-outliner-ops` call when possible to keep updates atomic and reduce roundtrips. + +## Design and Implementation Plan +### 1) Create `update` command module +- New file: `src/main/logseq/cli/command/update.cljs`. +- Start from `move.cljs` and expand the spec to include update/remove tag/property options. +- Extract shared helpers from `move.cljs` (e.g., `resolve-source`, `resolve-target`, `pos->opts`, `invalid-options?`) into a small shared namespace or move into `update.cljs`. + +### 2) Reuse tag/property parsing and resolution from `add` +- Refactor `src/main/logseq/cli/command/add.cljs` to expose reusable helpers for: + - `parse-tags-option`, `parse-properties-option` (for update) + - `resolve-tags`, `resolve-properties` (for update) + - Tag/property identifier normalization functions if needed for remove vectors +- Keep the parsing behavior exactly consistent with `add block`. +- Add new helper in `add` (or a shared namespace) to parse **remove vectors**: + - `parse-tags-vector-option` (vector of tag identifiers) + - `parse-properties-vector-option` (vector of property identifiers) + +### 3) Update top-level command registry +- Add `update` to `src/main/logseq/cli/commands.cljs` table and summary groups. +- Remove `move` from the command registry and help output (no alias/compat). + +### 4) Update parse/validation logic +- In `finalize-command` (`src/main/logseq/cli/commands.cljs`): + - Add `update`-specific validation for sources, targets, and update/remove options. + - Reuse `update-command/invalid-options?`. + - Allow missing target when update/remove options are present. + - Keep error messaging aligned with existing CLI patterns. + +### 5) Implement update action building +- `build-action` returns a combined action with: + - `:type :update-block` + - `:id`/`:uuid` for source, optional target selectors, `:pos` default `first-child` when move is requested + - `:update-tags`, `:update-properties`, `:remove-tags`, `:remove-properties` +- `execute-update`: + - Resolve source; resolve target only when move is requested. + - Build a vector of outliner ops, in order: move (if present), then remove tags/properties, then update/add tags/properties (order can be adjusted if needed for predictable results). + - Use `:thread-api/apply-outliner-ops` once with all ops. + +### 6) Update output formatting +- `src/main/logseq/cli/format.cljs`: + - Add `format-update-block` to describe move + updates in a concise line. + - Update dispatcher to handle `:update-block`. + +### 7) Tests +- Unit tests in `src/test/logseq/cli/commands_test.cljs`: + - Parsing: `update` accepts `--id/--uuid`, optional target, and new options. + - Validation: missing source, invalid target selector combinations, invalid pos, invalid EDN options. + - Ensure that `update` without target but with update/remove options is accepted. +- Format tests in `src/test/logseq/cli/format_test.cljs` for `:update-block`. +- Integration tests in `src/test/logseq/cli/integration_test.cljs`: + - Move-only update should behave the same as current `move`. + - Update tags/properties on an existing block. + - Remove tags/properties and validate via `show` or query. + +## Open Questions +- If only `--pos` is provided without any target selector, return the error: `--pos is only valid when a target is provided`. + +## Risks / Edge Cases +- If tag/property parsing rules diverge from `add`, user experience becomes inconsistent. Refactor shared parsing to avoid drift. +- Combining move and property updates in one `apply-outliner-ops` call needs to preserve correct operation order; keep move first unless property updates depend on position. +- Ensure non-page validation for source and block target remains intact when refactoring. + +## Implementation Checklist (Concrete File Touches) +- Add: `src/main/logseq/cli/command/update.cljs`. +- Update: `src/main/logseq/cli/commands.cljs` (table, validation, action dispatch). +- Update: `src/main/logseq/cli/command/core.cljs` (top-level summary group list to include update). +- Update: `src/main/logseq/cli/format.cljs` (formatting for update). +- Update: `src/test/logseq/cli/commands_test.cljs`. +- Update: `src/test/logseq/cli/format_test.cljs`. +- Update: `src/test/logseq/cli/integration_test.cljs`. + +## Verification +- Run unit tests: `bb dev:test -v logseq.cli.commands-test`. +- Run format tests: `bb dev:test -v logseq.cli.format-test`. +- Run CLI integration tests (move/update subset): `bb dev:test -v logseq.cli.integration-test`. +- Optional full suite: `bb dev:lint-and-test`. diff --git a/docs/agent-guide/028-logseq-cli-verbose-debug.md b/docs/agent-guide/028-logseq-cli-verbose-debug.md new file mode 100644 index 0000000000..ff4b1b867f --- /dev/null +++ b/docs/agent-guide/028-logseq-cli-verbose-debug.md @@ -0,0 +1,74 @@ +# Logseq CLI Verbose Debug Logging Implementation Plan + +Goal: Add a global --verbose flag that enables structured debug logging for CLI options and all db-worker-node API calls without polluting normal command output. + +Architecture: The CLI will install a stderr log handler and enable debug level logging when --verbose is set. +Architecture: The CLI transport and db-worker-node lifecycle utilities will emit structured debug logs for each request and response, using a shared truncation helper to cap large payloads. +Architecture: The db-worker-node process log level remains unchanged, and --verbose only controls logseq-cli logging. + +Tech Stack: ClojureScript, lambdaisland.glogi, Node.js, db-worker-node HTTP API. + +Related: Relates to docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md. + +## Problem statement + +The CLI currently has no verbose debug mode for troubleshooting, so engineers cannot easily see which options were parsed or what db-worker-node API calls were made. +This makes it hard to diagnose failures, especially when db-worker-node responses are large and need safe, truncated logging. +The goal is to add a global --verbose option that emits debug logs for command options and db-worker-node API calls without breaking existing output contracts. + +## Testing Plan + +I will add a unit test that verifies the log truncation helper returns a preview with a stable maximum length and a flag indicating truncation. +I will add a unit test that verifies the log truncation helper handles strings, collections, and nil without throwing. +I will add a unit test that verifies the CLI debug logger emits records only when verbose is enabled by capturing glogi records via a temporary handler. +I will add an integration test that runs the CLI with --verbose and asserts that stderr contains a db-worker-node invoke debug line while stdout remains valid JSON for a simple command. +NOTE: I will write all tests before I add any implementation behavior. + +## Logging coverage + +| Area | File path | Log events | Notes | +| --- | --- | --- | --- | +| CLI option intake | src/main/logseq/cli/main.cljs | Parsed options and resolved config values | Use truncation for long option values like content or blocks. | +| db-worker-node invoke | src/main/logseq/cli/transport.cljs | Request and response debug logs with timing | Truncate response preview and include size metadata. | +| db-worker-node health checks | src/main/logseq/cli/server.cljs | /healthz, /readyz, /v1/shutdown debug logs | Keep quiet unless verbose is true. | +| db-worker-node server logs | src/main/frontend/worker/db_worker_node.cljs | Keep existing logging behavior unchanged. | --verbose does not affect db-worker-node. | + +## Implementation Plan + +1. Read @prompts/review.md and note any CLI and db-worker-node review checklist items that affect logging or output stability. +2. Add :verbose to the CLI global option spec in src/main/logseq/cli/command/core.cljs with a description and boolean coercion so it appears in help output. +3. Add a new logging helper namespace in src/main/logseq/cli/log.cljs that sets a stderr handler, toggles log levels, and exposes a truncate-preview helper that caps output length and records the original size. +4. Add unit tests for the new logging helper in src/test/logseq/cli/log_test.cljs to cover truncation behavior and the verbose gating behavior using a temporary glogi handler. +5. Initialize CLI logging in src/main/logseq/cli/main.cljs after resolving config so --verbose turns on debug level and installs the handler once per run. +6. Emit a debug log in src/main/logseq/cli/main.cljs that includes the command, args, and full options map from the parsed command, using the truncation helper for large values. +7. Add request and response debug logs in src/main/logseq/cli/transport.cljs around invoke, including method, directPass, args preview, response preview, response size, and elapsed time. +8. Add request and response debug logs in src/main/logseq/cli/server.cljs for db-worker-node HTTP calls, and ensure logs are emitted only when --verbose is set. +9. Confirm no changes are made to src/main/frontend/worker/db_worker_node.cljs behavior or log levels for this feature. +10. Add an integration test in src/test/logseq/cli/integration_test.cljs to ensure stdout remains valid JSON when verbose logs are emitted to stderr. +11. Update docs/cli/logseq-cli.md to document the new --verbose flag, that it only affects logseq-cli, and that debug logs go to stderr with large payloads truncated. + +## Edge Cases + +Large query or export results can exceed the preview limit, so logs must include a length field and a truncated preview instead of full payloads. +CLI commands that output JSON or EDN must keep stdout clean, so debug logs must go to stderr only. +Options that contain large text content or EDN blocks must be truncated in logs to avoid massive log lines. +db-worker-node can be started independently, so its debug logs should still be gated by its log-level flag even when the CLI is not involved. + +## Testing Details + +Tests will validate that debug logging is gated by --verbose, that truncation is applied consistently, and that stdout output remains parseable while stderr contains debug logs for a simple db-worker-node invocation. + +## Implementation Details + +- Use lambdaisland.glogi for CLI logging so log-level control is consistent with db-worker-node. +- Install a stderr log handler in the CLI to avoid polluting stdout output formats. +- Truncate previews by character count and include metadata such as original length and truncation flag. +- Log db-worker-node invoke timings so slow calls are visible in verbose mode. +- Keep db-worker-node API behavior and log levels unchanged. +- Ensure all verbose logs are emitted by logseq-cli only. + +## Question + +None. + +--- diff --git a/docs/agent-guide/029-logseq-cli-show-properties.md b/docs/agent-guide/029-logseq-cli-show-properties.md new file mode 100644 index 0000000000..d4da572d5b --- /dev/null +++ b/docs/agent-guide/029-logseq-cli-show-properties.md @@ -0,0 +1,150 @@ +# Logseq CLI Show Properties Implementation Plan + +Goal: Display block properties in the human-readable output of the logseq-cli show command. + +Architecture: Extend the show command pull selectors to include property data from db-worker-node, then enrich tree->text to append formatted property lines per block. +Architecture: Keep JSON and EDN outputs structurally the same, while only altering the human text renderer to include properties beneath each block label. + +Tech Stack: ClojureScript, Datascript pull selectors, logseq-cli, db-worker-node thread-api. + +Related: Builds on 028-logseq-cli-verbose-debug.md. + +## Problem statement + +The logseq-cli show command currently renders block trees without any property visibility. +Users need to see each block's properties directly under the block content in the human output so that show reflects the same metadata they rely on in the UI. +Property lines belong to the same block tree element, so they must render with the same tree glyphs and indentation rules used for multiline block content (no blank space-only prefix). +The display must show ": " and must handle single-value properties as a single line and multi-value properties as an indented list as shown in the example. + +## Testing Plan + +I will add a unit test that asserts tree->text renders single-value properties in one line after the block content. +I will add a unit test that asserts tree->text renders multi-value properties as a dash list aligned under the property name. +I will add a unit test that asserts user property order follows a stable key order. +I will add a unit test that asserts properties do not break multiline block alignment for both root and child blocks. +I will add a unit test that asserts property values only appear in property-kvs and never in block/children. +I will update selector coverage tests to assert property selectors are included in show pulls. +I will add an integration test that asserts show output includes property lines. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Context and integration points + +The show command data path is: CLI action -> transport/invoke -> db-worker-node -> pull/query -> tree data -> tree->text human rendering. +The pull selectors in logseq.cli.command.show define what properties are available to tree->text. +User properties are stored directly on block entities with keys in the :user.property/* namespace. + +ASCII diagram of the flow: + +CLI show + | + | transport/invoke :thread-api/pull / :thread-api/q + v +DB worker node + | + | returns tree data with block maps + v +logseq.cli.command.show/tree->text + | + | renders block label + property lines + v +stdout + +## Files to touch + +| File path | Purpose | +| --- | --- | +| src/main/logseq/cli/command/show.cljs | Add property selectors, build property lines, and render properties in tree->text. | +| src/test/logseq/cli/commands_test.cljs | Add tree->text unit tests for property rendering and selector coverage. | +| src/test/logseq/cli/integration_test.cljs | Add integration coverage for show property rendering output. | + +## Implementation plan (TDD-ordered) + +1. Read @prompts/review.md to align with review expectations. +2. Add failing unit tests for tree->text: + - single-value property rendering + - multi-value property rendering with dash list formatting + alignment + - user property ordering + - multiline alignment with properties +3. Add failing selector coverage test asserting show pull patterns include property selectors. +4. Add failing integration test asserting show output includes property lines. +5. Run each new test and confirm failures reference missing property behavior (not test errors). +6. Update show pull selectors in src/main/logseq/cli/command/show.cljs to include :user.property/* attributes so db-worker-node returns needed data. +7. Add helpers in src/main/logseq/cli/command/show.cljs to: + - collect and sort user property keys + - normalize user property values into a vector of display strings + - format property lines for single vs multi values +8. Extend tree->text in src/main/logseq/cli/command/show.cljs to append property lines after block label lines, preserving indentation rules and no tree glyphs. +9. Re-run unit tests and selector test; confirm they pass. +10. Re-run integration test; confirm it passes. +11. Run `bb dev:lint-and-test` and confirm all linters and tests pass. + +## Edge cases to handle + +User properties with empty values should be skipped to avoid blank lines in the output. +User properties with values that are sets, vectors, or lists should render each item as a separate dash line. +User properties with values that are entities or maps should render as :block/title, :block/name, or :logseq.property/value when present. +User properties must be detected only via the :user.property/* namespace and no other property sources should be displayed. +Blocks without properties should render exactly as they do today. +Linked references output should include properties for each block without breaking the existing tree formatting. +Property lines should include the same tree glyphs/branch markers used for multiline block content so they align with block text. + +## Implementation details for formatting + +Use a stable sort order for :user.property/* keys, such as ascending by keyword name. +Do not use :block/properties-text-values for db-graph output. +Format user property values from :user.property/* attributes and coerce values into strings using :block/title, :block/name, or :logseq.property/value. +Render single values as "Property: value" and multi values as "Property: - v1" with subsequent lines aligned under the dash list. +Align property lines with the block content column and reuse the same tree glyph indentation rules used for multiline block content. + +## Questions + +Properties should display using their :block/title for the property name, derived from the property entity when available, and never fall back to :db/ident. +Hidden or internal properties must be filtered out to avoid noise in CLI output. +Tags must render as #tags only and must not be listed as properties. +:block/properties-text-values is file-graph only and should be ignored for db-graph output. + +## Testing Details + +The new tests will directly exercise tree->text with synthetic tree data that includes :user.property/* attributes to ensure the human output matches the requested format. +The new tests will assert property values are only present in property-kvs and do not appear in block/children output. +The selector test will validate that the show pull patterns include property attributes so db-worker-node can provide the necessary data. +These tests verify output behavior, ordering, and indentation rather than internal helper logic. +The integration test will validate that show output includes property lines end-to-end. + +## Implementation Details + +- Update tree-block-selector and linked-ref-selector to pull :user.property/* attributes. +- Update the id and uuid fetch pull patterns to include the same property fields. +- Add a property normalization helper that returns ordered [key values] pairs. +- Add a value formatting helper that converts each property value into displayable strings. +- Extend tree->text to append property lines after block content lines for both root and child nodes. +- Keep JSON and EDN outputs intact aside from the additional property keys pulled from the db. +- Property values should only appear in property-kvs; do not surface property values in block/children. +- Add new unit tests for single-value, multi-value, ordering, and multiline alignment cases. +- Add a selector test that asserts property selectors are included. + +## Example output + +Current: +``` +5137 Done Add git sha when graph is created for improved debugging #Issue +5138 ├── Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 +5139 ├── When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha +5140 └── When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha +``` + +Target: +``` +5137 Done Add git sha when graph is created for improved debugging #Issue + │ Background: Motivated by wanting to ensure missing addresses bug isn't happening in new graphs - https://logseq.slack.com/archives/C04ENNDPDFB/p1748290483138269 + │ Acceptance Criteria: + │ - When a DB graph is created in app, store git SHA used to create it in entity :logseq.kv/graph-git-sha + │ - When a DB graph is created with a script, store git SHA used to create it in entity :logseq.kv/graph-git-sha +``` + +## Question + +JSON and EDN outputs must include property values alongside the existing data. + +--- diff --git a/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md new file mode 100644 index 0000000000..1384d26901 --- /dev/null +++ b/docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md @@ -0,0 +1,191 @@ +# Logseq CLI DB Graph Default Dir And Write Exclusion Implementation Plan + +Goal: Move the default CLI data directory to `~/logseq/graphs`, store graph directories without requiring the `logseq_db_` prefix, and enforce single-writer behavior for graph files while one db-worker-node instance owns a graph. + +Architecture: Keep `logseq_db_` as the internal repo identifier used by thread-api calls, but introduce a canonical graph directory key that strips the db prefix for filesystem paths. +Architecture: Centralize graph directory resolution and lock ownership checks so `logseq.cli.server`, `frontend.worker.db-worker-node-lock`, and `frontend.worker.platform.node` enforce the same rules. + +Tech Stack: ClojureScript, Node.js `fs` and `path`, promesa, logseq-cli command pipeline, db-worker-node daemon, existing lock file protocol. + +Related: Builds on `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`. +Related: Supersedes `docs/agent-guide/020-logseq-cli-default-paths-move.md`. +Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. + +## Problem statement + +The current CLI default data directory is `~/logseq/cli-graphs`, while this change requires `~/logseq/graphs`. + +The current filesystem directory naming for a graph is based on repo identifiers that frequently include `logseq_db_`, so users still observe prefixed names in the data directory. + +The current lock model prevents launching a second db-worker-node for one repo, but it does not define an explicit write-lease boundary that all graph file mutations must validate before writing. + +We need one coherent model where graph directories are user-facing names, internal repo identifiers stay db-prefixed for thread-api compatibility, and graph writes are denied for non-owners while a server is running. + +This plan does not include compatibility logic, migration, or special handling for old on-disk graph directories that start with `logseq_db_`. + +This plan treats old on-disk `logseq_db_` prefixed graph directories as ignored entries. + +## Testing Plan + +I will follow `@test-driven-development` and add all failing tests before implementation edits. + +I will add unit tests for default path resolution and help text defaults in CLI and db-worker-node code paths. + +I will add unit tests for graph directory canonicalization that prove `logseq_db_demo` resolves to the same on-disk directory as `demo` in the default data directory. + +I will not add migration tests for old prefixed directories because compatibility and migration are out of scope for this change. + +I will add db-worker-node tests that validate write-lease ownership checks fail for non-owner lock metadata and pass for the active owner. + +I will run targeted test namespaces first for fast red and green loops, then run the full lint and test suite. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Required change | +|---|---|---| +| Default data dir | `~/logseq/cli-graphs` in CLI and db-worker-node defaults. | `~/logseq/graphs` in all default derivation and help output locations. | +| Graph dir naming | Graph directory derivation commonly receives repo values that include `logseq_db_`. | Graph directory derivation must use canonical graph names without requiring `logseq_db_`. | +| Graph type assumption | DB graph identity is inferred from repo prefix in several paths. | In default data dir, treat every graph directory as a db graph and map to internal repo with prefix only at invocation boundaries. | +| Write exclusivity | `db-worker.lock` blocks duplicate daemon start, but write ownership is not verified by all mutation paths. | Introduce write-lease ownership checks for all graph file mutation operations executed by db-worker-node. | + +## Integration sketch + +```text +CLI --graph demo + -> command-core resolves internal repo: logseq_db_demo + -> graph-dir resolver maps repo to graph key: demo + -> fs paths use ~/logseq/graphs/demo + -> thread-api calls still use logseq_db_demo + +db-worker-node owner lease + -> creates db-worker.lock with pid + lock-id + -> mutation path checks lock-id + pid ownership + -> non-owner mutation attempt returns :repo-locked +``` + +## Implementation plan + +### Phase 1: Add failing tests for path defaults and naming. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs` that `resolve-config` defaults `:data-dir` to `~/logseq/graphs`. +2. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/data_dir_test.cljs` that `normalize-data-dir` resolves to `$HOME/logseq/graphs`. +3. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `show-help!` prints `default ~/logseq/graphs` for `--data-dir`. +4. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that global help includes `Path to db-worker data dir (default ~/logseq/graphs)`. +5. Run `bb dev:test -v 'logseq.cli.config-test'` and confirm the new default path assertion fails. +6. Run `bb dev:test -v 'logseq.cli.data-dir-test'` and confirm the new default path assertion fails. + +### Phase 2: Add failing tests for prefix-free graph directory semantics. + +7. Add a new test namespace `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that asserts repo `logseq_db_demo` resolves to graph directory key `demo` under default data dir. +8. Add a second test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that repo `demo` also resolves to graph directory key `demo`. +9. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that verifies no migration helper is invoked for legacy prefixed on-disk graph directories. +10. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_lock_test.cljs` that canonical directory resolution only targets `` naming in the default data dir. +11. Add a failing CLI server test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that `lock-path` for repo `logseq_db_demo` points to `/demo/db-worker.lock`. +12. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and confirm failures for unimplemented canonicalization and non-migration logic. +13. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm lock-path behavior fails before implementation. + +### Phase 3: Add failing tests for write-lease ownership. + +14. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that simulates lock metadata mismatch and expects write mutation to fail with `:repo-locked`. +15. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that verifies owner write mutation succeeds when lock metadata matches current owner. +16. Add a test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that stale lock ownership is rejected after lock replacement. +17. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm the write-lease tests fail before implementation. + +### Phase 4: Implement default directory switch to `~/logseq/graphs`. + +18. Update default path constants in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs` from `~/logseq/cli-graphs` to `~/logseq/graphs`. +19. Update CLI config defaults in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` so `:data-dir` defaults to `~/logseq/graphs`. +20. Update server fallback in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so `resolve-data-dir` defaults to `~/logseq/graphs`. +21. Update db-worker-node default path in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` so `resolve-data-dir` defaults to `~/logseq/graphs`. +22. Update node platform fallback in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` so `node-platform` defaults to `~/logseq/graphs`. +23. Update CLI help text in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` to show `~/logseq/graphs`. +24. Re-run `bb dev:test -v 'logseq.cli.config-test'` and `bb dev:test -v 'logseq.cli.data-dir-test'` and confirm green results. + +### Phase 5: Implement canonical graph directory resolution without required `logseq_db_` prefix. + +25. Add shared graph directory canonicalization helpers in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` to map internal repo names to canonical graph directory keys. +26. Ensure canonicalization strips `logseq_db_` only when the repo is a db repo and preserves encoded filename safety through existing `encode-graph-dir-name` helpers. +27. Explicitly avoid adding legacy directory migration logic from `logseq_db_` to `` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs`. +28. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` to use canonical graph directory resolution for `repo-dir`, `lock-path`, and graph enumeration. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` to use canonical graph directory keys in `storage-pool-name` and `db-exists?` path resolution for node runtime. +30. Re-run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and `bb dev:test -v 'logseq.cli.server-test'` and confirm canonical naming tests pass without migration behavior. + +### Phase 6: Implement write-lease ownership checks for graph file mutations. + +32. Extend lock payload in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` with a generated `lock-id` and keep ownership fields immutable after acquisition. +33. Add `assert-lock-owner!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node_lock.cljs` that validates lock existence, pid ownership, and `lock-id` match before mutation. +34. Pass a write-guard callback from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` into `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. +35. Invoke the write-guard callback before every graph file mutation path in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`, including sqlite import, text writes, and recursive delete paths. +36. Return consistent `:repo-locked` errors from ownership failures and ensure CLI formatting remains readable through `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +37. Re-run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm ownership tests pass. + +### Phase 7: Update docs and run full verification. + +38. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` for the new default data dir and for prefix-free on-disk graph directory naming. +39. Add a breaking-change note in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` stating old on-disk prefixed graph directories are not automatically migrated. +40. Re-run `bb dev:test -v 'logseq.cli.commands-test'` and `bb dev:test -v 'logseq.cli.main-test'` to verify help text and command behavior coverage. +41. Run `bb dev:lint-and-test` and confirm the suite passes with no regressions. +42. Review changed code against `@prompts/review.md` to catch Clojure and ClojureScript correctness pitfalls before merge. + +## Edge cases to cover + +| Scenario | Expected behavior | +|---|---| +| Graph name contains `/`, `:`, `%`, or unicode. | Directory naming remains reversible through encode and decode helpers. | +| Legacy on-disk directory `logseq_db_demo` exists. | No compatibility or migration is performed by this change, and discovery commands ignore this directory. | +| Lock file is stale with dead pid. | Startup removes stale lock and acquires a fresh lease. | +| Lock file exists with alive non-owner pid. | Startup and direct mutation fail fast with `:repo-locked`. | +| Default data dir has non-graph directories. | Enumeration ignores directories that are not valid graph directories after canonicalization checks. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.config-test' +bb dev:test -v 'logseq.cli.data-dir-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'frontend.worker.db-worker-node-lock-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +bb dev:test -v 'logseq.cli.commands-test' +bb dev:lint-and-test +``` + +Each command should finish with zero failures and zero errors. + +If a red phase command is run before implementation, it should fail specifically on the newly added assertions in that phase. + +## Testing Details + +The tests validate behavior through public CLI and daemon entrypoints instead of validating only implementation internals. + +The naming tests prove that internal db repo prefixes remain stable while on-disk names are canonicalized for user-facing graph directories. + +The ownership tests verify that only the active lock owner can execute graph file mutation paths in db-worker-node. + +The tests intentionally avoid migration coverage because compatibility with old prefixed on-disk graph directories is out of scope. + +## Implementation Details + +- Keep internal repo values db-prefixed for thread-api compatibility and db metadata reads. +- Use canonical graph directory keys for all on-disk path construction in node runtime. +- Do not add compatibility branches or migration logic for old `logseq_db_` prefixed on-disk graph directories. +- Ignore old `logseq_db_` prefixed on-disk graph directories during graph discovery and server listing. +- Add lock ownership token fields and verify ownership before each graph mutation path. +- Enforce ownership checks for all files under a graph directory, including sqlite files, search and client-op files, debug logs, backups, and text files. +- Keep error code semantics stable by reusing `:repo-locked` for ownership and lock conflicts. +- Keep lock-conflict error semantics unified across CLI and db-worker-node as `:repo-locked`. +- Update all default data-dir strings and help text to `~/logseq/graphs`. +- Keep CLI output graph names user-facing and prefix-free. +- Update docs with a breaking-change note for old prefixed on-disk graph directories. +- Follow `@test-driven-development` and `@prompts/review.md` through implementation and verification. + +## Question + +Resolved decisions: + +1. Ignore old on-disk `logseq_db_` prefixed graph directories. +2. Requirement three covers all files under the graph directory. +3. Lock conflicts and ownership failures use unified error code `:repo-locked`. + +--- diff --git a/docs/agent-guide/031-logseq-cli-doctor-command.md b/docs/agent-guide/031-logseq-cli-doctor-command.md new file mode 100644 index 0000000000..4a2872cd3b --- /dev/null +++ b/docs/agent-guide/031-logseq-cli-doctor-command.md @@ -0,0 +1,183 @@ +# Logseq CLI Doctor Command Implementation Plan + +Goal: Add a `doctor` command that verifies logseq-cli runtime availability before normal command execution, including `db-worker-node.js` existence and `data-dir` read and write readiness. + +Architecture: Add a dedicated `logseq.cli.command.doctor` namespace and wire it into the existing `parse-args` -> `build-action` -> `execute` pipeline in `logseq.cli.commands`. +Architecture: Reuse existing helpers in `logseq.cli.data-dir` and `logseq.cli.server` for permission checks and daemon liveness probes, then return one structured diagnostics report. + +Tech Stack: ClojureScript, babashka.cli command table, Node.js `fs` and `path`, Promesa, existing CLI formatter and test harness. + +Related: Builds on `docs/agent-guide/019-logseq-cli-data-dir-permissions.md`. +Related: Relates to `docs/agent-guide/015-logseq-cli-db-worker-node-housekeeping.md`. +Related: Relates to `docs/agent-guide/017-logseq-cli-db-worker-node-housekeeping-2.md`. +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. + +## Problem statement + +The current CLI fails only when a concrete command touches startup paths, so users discover environment problems late. + +We need a fast explicit health command that confirms whether logseq-cli can run reliably in the current machine context. + +The minimum required checks are the presence of `db-worker-node.js` and read and write access for `data-dir`. + +Note: `dist/db-worker-node.js` is a thin entry wrapper that loads `static/db-worker-node.js`. Doctor should validate the actual runtime target in `static/` rather than only the `dist/` wrapper. + +We should also surface practical runtime risks already modeled by current code, especially stale or unready db-worker instances discovered from lock files and health endpoints. + +This plan keeps scope to diagnostics and does not change daemon lifecycle semantics, lock protocol, or graph migration behavior. + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before adding implementation behavior. + +I will add parser and action-dispatch tests for `doctor` in `commands_test` so command discovery and help output are guarded. + +I will add dedicated `doctor` command tests that cover success, missing script file, and `data-dir` permission failure behavior. + +I will add `format` tests to ensure human and machine-readable output for `doctor` are stable and useful. + +I will run focused test namespaces first to validate RED and GREEN transitions, then run the full lint and test suite. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Required change | +|---|---|---| +| Runtime script path | `spawn-server!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` starts `../dist/db-worker-node.js`, which delegates to `../static/db-worker-node.js`, but no explicit diagnostic command validates that runtime target path readiness. | Add `doctor` check that validates the effective script file existence and readability before startup commands fail. | +| Data-dir readiness | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs` enforces directory creation and read or write access in `ensure-data-dir!`. | Reuse `ensure-data-dir!` inside `doctor` and report a dedicated failing check item. | +| Daemon liveness visibility | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` has `list-servers`, `server-status`, `ready?`, and `healthy?`, but no consolidated health summary command. | Add optional runtime checks in `doctor` that flag non-ready running servers discovered from lock files. | +| CLI discoverability | Top-level help and command table in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` do not include diagnostics entrypoint. | Add `doctor` to command entries and help summaries. | + +## Proposed doctor checks + +| Check id | Behavior | Existing helper to reuse | Failure signal | +|---|---|---|---| +| `db-worker-script` | Verify `../static/db-worker-node.js` exists and is readable as a file (and optionally verify `../dist/db-worker-node.js` wrapper exists). | New shared path helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` plus Node `fs` checks in doctor command. | `:doctor-script-missing` or `:doctor-script-unreadable`. | +| `data-dir` | Verify configured or default data dir can be created and is read and write accessible. | `logseq.cli.data-dir/ensure-data-dir!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/data_dir.cljs`. | Existing `:data-dir-permission` surfaced as doctor failure detail. | +| `running-servers` | Verify currently locked db-worker instances are reachable on readiness endpoint. | `logseq.cli.server/list-servers` status derivation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`. | `:doctor-server-not-ready` for any server reported as `:starting`. | + +## Integration sketch + +```text +logseq doctor + -> parse-args (commands table) + -> build-action {:type :doctor} + -> execute-doctor + 1) check effective db-worker-node.js runtime path (`static/db-worker-node.js`). + 2) check data-dir accessibility. + 3) inspect running server readiness. + -> format result for human/json/edn. +``` + +## Implementation plan + +### Phase 1: RED for command plumbing. + +1. Add failing assertions in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that top-level help includes `doctor` and bold-styled `doctor` command text. +2. Add a failing parse test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `commands/parse-args ["doctor"]` returning `:ok? true` with command `:doctor`. +3. Add a failing build-action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `{:command :doctor}` producing action type `:doctor`. +4. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm failures are specifically on new doctor assertions. + +### Phase 2: RED for doctor behavior. + +5. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` with namespace and fixtures consistent with existing command tests. +6. Add a failing test that marks script check as failed when `static/db-worker-node.js` path does not exist. +7. Add a failing test that marks data-dir check as failed when `ensure-data-dir!` throws `:data-dir-permission`. +8. Add a failing test that returns all checks passed when script and data-dir are both valid and no running server is unready. +9. Add a failing test that reports runtime warning or failure when `list-servers` includes entries with status `:starting`. +10. Run `bb dev:test -v 'logseq.cli.command.doctor-test'` and confirm all new tests fail for expected reasons. + +### Phase 3: GREEN for command integration. + +11. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` with `entries`, `build-action`, and `execute-doctor` returning structured check results. +12. Wire doctor namespace into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` requires and append `doctor-command/entries` into `table`. +13. Add `:doctor` branch in `build-action` inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +14. Add `:doctor` branch in `execute` inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +15. Update top-level command grouping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` to show `doctor` in help output. +16. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm doctor parse and help tests are green. + +### Phase 4: GREEN for doctor checks. + +17. Extract or add a shared db-worker script path helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so spawn and doctor share one source of truth. +18. Implement script existence and readability check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` using Node `fs` metadata checks. +19. Implement data-dir check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` by invoking `logseq.cli.data-dir/ensure-data-dir!`. +20. Implement running-server readiness check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` using `logseq.cli.server/list-servers`. +21. Return deterministic check ordering and include actionable message per check in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs`. +22. Re-run `bb dev:test -v 'logseq.cli.command.doctor-test'` and confirm all doctor behavior tests are green. + +### Phase 5: RED and GREEN for formatting and docs. + +23. Add failing output tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human summary rendering of doctor checks. +24. Add failing output tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for json and edn output preserving structured check payload. +25. Implement doctor-specific human formatter in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +26. Ensure `doctor` output includes overall status and per-check status in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +27. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` with `doctor` command description, examples, and expected failure hints. +28. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm new doctor formatting tests are green. + +### Phase 6: Verify RED to GREEN cycle completion, refactor, and full validation. + +29. Run `bb dev:test -v 'logseq.cli.commands-test'` and ensure no regressions in help parsing and action dispatch. +30. Run `bb dev:test -v 'logseq.cli.command.doctor-test'` and ensure all doctor checks are behavior-driven and stable. +31. Run `bb dev:test -v 'logseq.cli.main-test'` to confirm entrypoint behavior remains compatible. +32. Run `bb dev:test -v 'logseq.cli.server-test'` to verify shared script path changes do not break server startup assumptions. +33. Run `bb dev:test -v 'logseq.cli.format-test'` to validate output contracts. +34. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +35. Review changed code against `@prompts/review.md` before merge. + +## Edge cases to cover + +| Scenario | Expected behavior | +|---|---| +| `static/db-worker-node.js` path exists but points to a directory. | `doctor` reports script check failure with explicit path and reason. | +| `data-dir` path points to a file. | `doctor` fails with `:data-dir-permission` detail and does not continue to misleading pass status. | +| `data-dir` is readable but not writable. | `doctor` fails data-dir check and returns actionable permission hint. | +| Running server lock exists but `/readyz` is not healthy. | `doctor` reports runtime check as failed for that repo. | +| No running server exists. | Runtime server check passes with empty server list and does not force daemon startup. | +| `--output json` is used. | Doctor returns stable machine-readable check list for scripts and automation. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.commands-test' +bb dev:test -v 'logseq.cli.command.doctor-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.main-test' +bb dev:lint-and-test +``` + +Each command should finish with zero failures and zero errors in GREEN phase. + +Each RED phase run should fail on newly added doctor assertions and not on unrelated setup errors. + +## Testing Details + +The tests focus on command behavior and diagnostics outcomes through public parser and executor boundaries. + +The tests avoid implementation-detail assertions and instead validate user-observable results for success and failure cases. + +The formatter tests ensure the same doctor payload is usable for both human troubleshooting and automation output modes. + +## Implementation Details + +- Keep `doctor` as a first-class command in the existing CLI command table. +- Reuse `ensure-data-dir!` instead of reimplementing permission checks. +- Reuse server health status discovery through existing `list-servers` behavior. +- Keep check execution deterministic and output stable for CI parsing. +- Keep command scope read-only for diagnostics and avoid auto-remediation side effects. +- Return explicit error codes for script and runtime health failures. +- Preserve current graph and repo naming semantics and lock protocol behavior. +- Add targeted formatter support so human output is concise and actionable. +- Verify all changes via focused tests before full lint and test pass. +- Follow `@test-driven-development` and `@prompts/review.md` throughout implementation. + +## Question + +Resolved: `doctor` will fail fast on the first failed check. + +Resolved: `doctor` will treat `:starting` servers as warnings when script and data-dir checks pass. + +Resolved: `doctor` will support a future `--graph` scoped deep check that verifies per-graph lock path and repo directory access without starting the daemon. + +--- diff --git a/docs/agent-guide/032-logseq-cli-show-property-key-bold.md b/docs/agent-guide/032-logseq-cli-show-property-key-bold.md new file mode 100644 index 0000000000..703f9a84d5 --- /dev/null +++ b/docs/agent-guide/032-logseq-cli-show-property-key-bold.md @@ -0,0 +1,119 @@ +# Logseq CLI Show Property Key Bold Implementation Plan + +Goal: Make `` render in bold in `show` human output while keeping the existing `property-kvs` layout and values unchanged. + +Architecture: Keep db-worker-node data fetching and property title resolution unchanged, and only add styling at the final text rendering layer in `logseq.cli.command.show`. + +Tech Stack: ClojureScript, logseq-cli show renderer, db-worker-node transport/thread-api, cljs.test. + +Related: Relates to `docs/agent-guide/029-logseq-cli-show-properties.md` and `docs/agent-guide/023-logseq-cli-help-show-styling.md`. + +## Problem statement + +The current `show` human output prints property lines as plain text in the form `: `. + +`property-kvs` are already rendered in the correct position and indentation under each block, but `` does not have visual emphasis. + +The requested behavior is to bold only `` while preserving `:`, value text, multiline/list formatting, and tree alignment. + +Because logseq-cli `show` depends on db-worker-node data, this change must not alter db-worker-node API contracts, pull results, or non-human output formats. + +## Testing Plan + +I will add a unit test that verifies a single-value property line styles only the property key with ANSI bold when color is enabled. + +I will add a unit test that verifies multi-value property blocks style the property key heading in bold while list item rows remain non-bold. + +I will update existing property rendering tests to keep `strip-ansi` expectations unchanged so layout behavior is locked. + +I will add a format-level test to ensure human `show` output preserves the new bold key styling through `format/format-result`. + +I will run one integration test for show properties to confirm db-worker-node end-to-end behavior remains unchanged after ANSI stripping. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +In scope is styling `` in human output generated by `show`. + +Out of scope is changing property query logic, property sorting rules, property value normalization, and db-worker-node request/response schemas. + +Out of scope is any change to JSON or EDN output. + +## Files to touch + +| File path | Purpose | +| --- | --- | +| `src/main/logseq/cli/command/show.cljs` | Apply bold style to `` in property line formatter only. | +| `src/test/logseq/cli/commands_test.cljs` | Add renderer-focused tests for ANSI bold presence and unchanged stripped layout. | +| `src/test/logseq/cli/format_test.cljs` | Verify formatted human output keeps bold property-key styling. | +| `src/test/logseq/cli/integration_test.cljs` | Confirm end-to-end show property output remains stable after strip-ansi. | + +## Implementation plan + +1. Follow `@test-driven-development` and `@prompts/review.md` before code edits. +2. Add failing renderer unit tests in `src/test/logseq/cli/commands_test.cljs` for bold `` in single and multi-value property output. +3. Run targeted tests and confirm they fail for missing styling behavior and not for test setup errors. +4. Update `format-property-lines` in `src/main/logseq/cli/command/show.cljs` to wrap only the property title segment with `style/bold`. +5. Keep `indent`, colon placement, and value rendering unchanged so current tree alignment remains intact. +6. Add or update `src/test/logseq/cli/format_test.cljs` to ensure `format/format-result` does not strip the new bold style in human output. +7. Run targeted unit tests again and confirm all new assertions pass. +8. Run the existing integration test that covers show properties and confirm stripped output still matches the same textual content. +9. Run full lint and unit test suite for regression coverage. + +## Verification commands + +```bash +bb dev:test -v 'logseq.cli.commands-test/test-tree->text-renders-properties-single-value' +bb dev:test -v 'logseq.cli.commands-test/test-tree->text-renders-properties-multi-value' +bb dev:test -v 'logseq.cli.format-test/test-human-output-show' +bb dev:test -v 'logseq.cli.integration-test/test-cli-show-properties-human-output' +bb dev:lint-and-test +``` + +Expected outcome is that new property-key bold tests pass, pre-existing strip-ansi assertions remain unchanged, integration behavior remains stable, and the full lint/test command exits successfully. + +## Edge cases + +When color is disabled, output should remain plain text and still include `: ` with no ANSI codes. + +For multiline block titles, property-key bold styling must not shift spacing or tree glyph alignment. + +For multi-value properties, only the heading key line should be bold and bullet item values should remain unchanged. + +If a property title is missing or blank, existing skip behavior should remain unchanged. + +## Rollout and risk + +Risk is low because the change is limited to a string formatting helper used only by human output. + +The main regression risk is accidental styling leakage into values or alignment shifts caused by ANSI code placement. + +This risk is controlled by preserving existing strip-ansi golden assertions and adding explicit ANSI presence tests. + +## Testing Details + +The tests verify behavior at three levels, which are renderer output, formatter passthrough, and db-worker-node integration stability. + +Renderer tests assert exact ANSI bold placement around property keys and unchanged plain text after stripping ANSI. + +Formatter tests confirm human output still carries ANSI styling while JSON and EDN behavior is unchanged. + +Integration coverage confirms end-to-end property visibility still works with the same stripped output content. + +## Implementation Details +- Reuse `logseq.cli.style/bold` instead of introducing a new styling helper. +- Change only `format-property-lines` and avoid modifying property discovery helpers. +- Keep sorted property order logic exactly as-is. +- Keep `property-value->string` and `normalize-property-values` untouched. +- Preserve the existing list format for multi-value properties. +- Preserve root and child indentation prefixes from `tree->text`. +- Avoid any db-worker-node API or transport changes. +- Ensure ANSI checks in tests use existing helpers like `style/strip-ansi` or regex. +- Keep all JSON and EDN output paths unchanged. + +## Question + +Only the heading key should be considered the full `` target, so `Criteria` should be bold while `- One` and `- Two` should remain non-bold. + +--- diff --git a/docs/agent-guide/033-desktop-db-worker-node-backend.md b/docs/agent-guide/033-desktop-db-worker-node-backend.md new file mode 100644 index 0000000000..592d4ed0ab --- /dev/null +++ b/docs/agent-guide/033-desktop-db-worker-node-backend.md @@ -0,0 +1,224 @@ +# Desktop Db Worker Node Backend Implementation Plan + +Goal: Switch the Electron desktop app graph database backend from OPFS plus periodic disk export to db-worker-node with direct disk SQLite access, so desktop app and logseq-cli can safely use the same data-dir at the same time. + +Architecture: Reuse existing `logseq.cli.server` daemon orchestration and lock semantics, and add an Electron main process graph-scoped daemon manager. + +Architecture: Replace the Electron renderer `PersistentDB` implementation from `frontend.persist-db.browser` to an HTTP plus SSE remote client that talks to `/v1/invoke` and `/v1/events` on db-worker-node. + +Tech Stack: ClojureScript, Electron main plus renderer, Node child_process, db-worker-node HTTP plus SSE API, Electron IPC with transit-json payloads, SQLite files under data-dir, lock files. + +Related: Builds on `docs/agent-guide/003-db-worker-node-cli-orchestration.md`. + +Related: Relates to `docs/agent-guide/012-logseq-cli-graph-storage.md`. + +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. + +Related: Owner-aware lifecycle follow-up is documented in `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +The current desktop app uses an OPFS-backed SQLite worker in the renderer and periodically exports to disk through `persist-db/run-export-periodically!`. + +The current logseq-cli starts and connects to db-worker-node, and directly reads and writes SQLite files in data-dir. + +This split creates two write paths with eventual synchronization, and desktop plus CLI concurrent usage depends on export timing instead of one shared lock-governed write path. + +The goal is to make desktop and CLI share db-worker-node semantics and lock behavior so disk SQLite becomes the single source of truth. + +The desktop default DB graphs directory is `~/logseq/graphs`, defined in `deps/cli/src/logseq/cli/common/graph.cljs`, and used by Electron DB file operations in `src/electron/electron/db.cljs`. + +This plan focuses on backend access flow and lifecycle management, and does not change business-level thread-api semantics or SQLite schema. + +## Testing Plan + +I will follow `@test-driven-development` for every phase and write failing tests before implementation changes. + +I will prioritize pure-function and dependency-injected tests so core behavior can be validated without launching a full Electron GUI. + +I will add Electron main db-worker manager lifecycle tests for first graph open, multi-window reuse, last-window close, and app shutdown cleanup. + +I will add remote client transport tests for invoke success, invoke failure propagation, SSE disconnect and reconnect, and missing auth token handling. + +I will extend db-worker-node tests with desktop plus CLI coexistence cases, including lock contention and stale lock recovery. + +I will run targeted tests first and then run `bb dev:lint-and-test`, and I will apply the review checklist in `@prompts/review.md`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current implementation | Target implementation | +|---|---|---| +| Desktop graph DB runtime | Renderer uses `frontend.persist-db.browser` and an OPFS worker. | Renderer uses a remote `PersistentDB` client against db-worker-node. | +| Desktop persistence model | OPFS acts as source of truth and is periodically exported to disk through Electron IPC. | Disk SQLite in data-dir is the source of truth with no OPFS periodic export flow. | +| CLI persistence model | CLI starts db-worker-node and calls `/v1/invoke`. | Keep unchanged and align desktop with the same daemon semantics. | +| Lock and ownership | Desktop OPFS path bypasses db-worker-node lock semantics during in-memory writes. | Desktop and CLI both go through db-worker-node lock and single-writer enforcement. | +| Process lifecycle | Desktop has no graph-scoped daemon manager in main process. | Electron main manages per-graph daemon start, reuse, health checks, and stop. | + +## Integration sketch + +```text +Desktop Renderer + -> requests graph runtime from Electron Main via `electron.ipc/ipc` (transit-json) + -> receives {base-url, auth-token, repo} for db-worker-node + -> calls /v1/invoke and listens /v1/events through remote PersistentDB client + +Electron Main + -> on graph open: ensure db-worker-node started for graph in data-dir + -> on graph close or app quit: stop graph daemon when the last window exits + -> maintains graph -> daemon state cache + +Logseq CLI + -> uses existing logseq.cli.server ensure/start/stop + -> talks to the same graph data-dir and lock protocol + +db-worker-node + -> provides the single write path to disk SQLite files + -> enforces lock ownership and readiness checks +``` + +## Scope and non-goals + +In scope are Electron main daemon lifecycle management, renderer persistence client switch, OPFS export-path removal, and required tests plus docs. + +Out of scope are thread-api business behavior changes, SQLite schema changes, mobile behavior changes, and broad sync-system redesign. + +## Implementation plan + +### Phase 1: Add failing tests for the new desktop backend contract. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that verifies stable lock-conflict error reporting from daemon orchestration. +2. Add `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` and write a failing test for `ensure-started!` idempotency. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies one daemon is reused across multiple windows for the same graph. +4. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies stop on last-window close. +5. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that verifies stop-all behavior on app quit. +6. Add `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` and write failing tests for invoke success and invoke error propagation. +7. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` for SSE parsing and reconnect behavior. +8. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm new assertions fail first. +9. Run `bb dev:test -v 'frontend.persist-db.remote-test'` and confirm new assertions fail first. + +### Phase 2: Extract shared daemon orchestration for CLI and Electron. + +10. Extract CLI-output-independent daemon orchestration logic from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`. +11. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` and move `spawn`, `wait-ready`, `read-lock`, and `cleanup-stale-lock` core functions into it. +12. Keep command-facing API shape in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` stable and delegate internally to the new helper. +13. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` with regression tests for unchanged CLI behavior. +14. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm green. + +### Phase 3: Implement Electron main db-worker manager. + +15. Add `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs` with graph-to-daemon state cache and reference counting. +16. Implement `ensure-started!` in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs` using the shared daemon helper and return `base-url` plus `auth-token`. +17. Implement `ensure-stopped!` and `stop-all!` in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db_worker.cljs`. +18. Add `electron.ipc/ipc` handlers in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` for renderer requests of graph runtime configuration, encoded with transit-json. +19. Hook `stop-all!` into app lifecycle in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/core.cljs` at `before-quit` and `window-all-closed`. +20. Hook graph last-window stop logic into close flow in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/core.cljs`. +21. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm lifecycle tests pass. + +### Phase 4: Add Electron renderer remote PersistentDB client. + +22. Add `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` implementing `protocol/PersistentDB` with HTTP plus SSE transport. +23. Implement browser-safe invoke transport in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` and avoid Node-only `http` dependency in the renderer. +24. Implement event subscription and reconnect policy in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db/remote.cljs` compatible with current event handler signatures. +25. Extend runtime implementation selection in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db.cljs` to explicitly use remote client for Electron. +26. Add initialization flow in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/persist_db.cljs` that fetches runtime config from `electron.ipc/ipc` via transit-json before creating the remote client. +27. Update `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db/remote_test.cljs` to green after implementation. +28. Run `bb dev:test -v 'frontend.persist-db.remote-test'` and confirm green. + +### Phase 5: Replace OPFS startup path and remove periodic export workflow. + +29. Remove Electron-path dependency on `frontend.persist-db.browser/start-db-worker!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +30. Start remote `PersistentDB` initialization flow in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +31. Remove Electron-path invocation of `persist-db/run-export-periodically!` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler.cljs`. +32. Adjust `:graph/save-db-to-disk` behavior in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/events.cljs` to a no-op or user guidance path. +33. Adjust shortcut behavior for `:graph/db-save` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/modules/shortcut/config.cljs` so it no longer triggers legacy export flow. +34. Mark `:db-get` and `:db-export` `electron.ipc/ipc` endpoints in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` as compatibility-only or remove unused entry points. +35. Clean up OPFS-export-only logic in `/Users/rcmerci/gh-repos/logseq/src/electron/electron/db.cljs` while preserving needed utility paths. +36. Run `bb dev:test -v 'frontend.handler.route-test'` and `bb dev:test -v 'frontend.handler.common.config-edn-test'` for regression coverage. + +### Phase 6: Add desktop and CLI coexistence verification. + +37. Add concurrency tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` covering desktop plus CLI access to the same graph. +38. Add stale-lock recovery tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs`. +39. Add tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` for CLI reuse or conflict reporting when desktop already started the graph daemon. +40. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and confirm green for coexistence tests. +41. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm green for coexistence tests. + +### Phase 7: Docs and rollout safety. + +42. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document shared db-worker-node semantics between desktop and CLI. +43. Add `/Users/rcmerci/gh-repos/logseq/docs/developers/desktop-db-worker-node.md` describing Electron main lifecycle and renderer remote-client init ordering. +44. Add release notes describing that Electron no longer uses OPFS as the primary database storage path. +45. Add rollback notes for a temporary fallback switch if emergency recovery is required. +46. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +47. Run a final review against `@prompts/review.md` and fix findings before merge. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Desktop opens a graph before daemon readiness completes. | Renderer waits for main-process ready runtime or receives a retryable error, and does not silently fall back to OPFS. | +| Desktop and CLI both attempt first start for the same graph daemon. | Exactly one owner acquires lock and the other path reuses the existing daemon or retries after a `:repo-locked` status check. | +| Graph name contains special characters. | Existing graph-dir encoding resolves to the same on-disk directory for desktop and CLI. | +| SSE connection drops. | Remote client reconnects and keeps event handling consistent, while invoke calls remain independent. | +| App exits abnormally and leaves a stale lock. | Next startup cleans stale lock via existing lock housekeeping logic without manual lock deletion. | +| Version switch from OPFS-backed desktop behavior. | No one-time migration is required because desktop already writes to disk data-dir, and startup verification should check disk DB readability before enabling the new backend. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:test -v 'frontend.persist-db.remote-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +clojure -M:cljs compile db-worker-node +bb dev:lint-and-test +``` + +Each test command should finish with `0 failures, 0 errors`. + +`clojure -M:cljs compile db-worker-node` should produce a runnable `static/db-worker-node.js`. + +In manual smoke tests, desktop and CLI reads and writes for the same graph should be immediately visible to each other without periodic export dependency. + +## Rollout strategy + +Phase one ships behind a feature flag and enables by default in development builds for coexistence and recovery telemetry. + +Phase two enables by default in stable builds and keeps a short-lived rollback switch. + +Phase three removes legacy OPFS export-path code and removes the rollback switch. + +## Clarity required before implementation + +Confirm product-level messaging for desktop and CLI lock-contention conflicts on the same graph. + +Confirm whether `:graph/db-save` is removed or redefined as a checkpoint action in the new model. + +Confirm rollback-switch lifecycle and target removal release. + +## Testing Details + +Tests focus on behavior, not implementation details, by validating daemon lifecycle, invoke responses, and event-stream consistency. + +Core coexistence tests validate lock ownership, recovery, and cross-client visibility for the same graph instead of mock call counts. + +Regression tests ensure existing CLI behavior stays stable and Electron startup plus shutdown does not leave zombie processes or lock files. + +## Implementation Details + +- Reuse and extract daemon orchestration from `logseq.cli.server` to avoid duplicate process-management logic. +- Add a dedicated Electron main db-worker manager with graph-scoped daemon state cache. +- Use a renderer remote `PersistentDB` client against db-worker-node HTTP plus SSE endpoints. +- Use transit-json for frontend and Electron communication through `electron.ipc/ipc` (see also `ldb/write-transit-str`). +- Remove periodic OPFS export workflow in Electron so disk SQLite is the only source of truth. +- Keep thread-api and database schema unchanged to limit application-level regressions. +- Keep lock-file and `:repo-locked` semantics identical for desktop and CLI. +- Add reconnect and stale-lock recovery tests to cover availability risks. +- Roll out with feature-flag phases and a short rollback window. +- Follow `@test-driven-development` and `@prompts/review.md` through implementation and verification. + +## Question + +--- diff --git a/docs/agent-guide/034-db-worker-node-owner-process-management.md b/docs/agent-guide/034-db-worker-node-owner-process-management.md new file mode 100644 index 0000000000..6d9148b9f8 --- /dev/null +++ b/docs/agent-guide/034-db-worker-node-owner-process-management.md @@ -0,0 +1,205 @@ +# DB Worker Node Owner-Aware Process Management Implementation Plan + +Goal: Add owner-aware lock metadata and orphan-process recovery so CLI and Electron can safely share one graph daemon without cross-managing each other. + +Architecture: Keep one `db-worker.lock` per graph directory, but extend lock schema with `owner-source` so lifecycle actions can enforce owner boundaries. +Architecture: Keep read and write traffic reusable across clients, while restricting `stop` and `restart` to the side that originally started the daemon. +Architecture: Add orphan-process detection for lock-missing cases so `logseq server restart` does not hang on timeout when a legacy process is still alive. + +Tech Stack: ClojureScript, Node.js child process APIs, `promesa`, `logseq.cli.server`, `logseq.db-worker.daemon`, Electron main-process daemon manager, db-worker-node lock helpers. + +Related: Builds on `docs/agent-guide/033-desktop-db-worker-node-backend.md`. +Related: Relates to `docs/agent-guide/030-logseq-cli-db-graph-default-dir-locking.md`. +Related: Relates to `docs/agent-guide/003-db-worker-node-cli-orchestration.md`. + +## Problem statement + +Current lock payload only records repo, pid, host, and port, so ownership is implicit and lifecycle commands cannot distinguish CLI-started and Electron-started daemons. + +`stop-server!` and `restart-server!` can currently terminate any alive daemon if the lock exists, which violates the requirement that each client should manage only its own process. + +If a db-worker-node process remains alive but lock file is missing, `server restart` can wait until timeout because startup relies on lock appearance and has no orphan recovery path. + +When CLI starts a daemon first, Electron may treat the runtime as managed-by-self and attempt stop or restart logic, which can break graph open flow and produce user-facing errors. + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before each implementation change. + +I will add lock schema and owner compatibility tests in `src/test/frontend/worker/db_worker_node_lock_test.cljs`. + +I will add daemon-owner lifecycle and orphan-recovery tests in `src/test/logseq/cli/server_test.cljs` and `src/test/logseq/db_worker/daemon_test.cljs`. + +I will add Electron manager tests for external-runtime attachment and no-cross-stop behavior in `src/test/electron/db_worker_manager_test.cljs`. + +I will add db-worker-node argument and lock-write tests in `src/test/frontend/worker/db_worker_node_test.cljs`. + +I will run focused red-green loops first, then run `bb dev:lint-and-test`, and finish with a review pass against `@prompts/review.md`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current behavior map + +| Area | Current behavior | Target behavior | +|---|---|---| +| Lock metadata | No owner field in `db-worker.lock`. | Lock includes `owner-source` as `cli` or `electron`, plus versioned metadata. | +| Lifecycle authority | Any caller can stop or restart lock-owned daemon. | `stop` and `restart` are allowed only for matching owner-source. | +| Runtime reuse | Reuse happens, but manager cannot tell owned vs external runtime. | Reuse still happens, and runtime state tracks `owned?` to prevent cross-stop. | +| Lock missing + orphan process | Startup can timeout with no clear recovery path. | Orphan detection and cleanup path runs before or after failed startup wait. | +| Compatibility | Legacy lock without owner is ambiguous. | Legacy lock is treated as `owner-source: unknown` with explicit policy. | + +## Integration sketch + +```text +CLI or Electron request ensure-server(repo, requester-owner) + -> read lock + -> if lock exists and healthy: + return runtime + ownership(owned/external) + -> if lock missing: + scan orphan db-worker-node process for same repo/data-dir + if orphan found: + terminate orphan + spawn new daemon with --owner-source + -> db-worker-node writes lock {repo,pid,host,port,owner-source,lock-id,...} + +stop/restart(requester-owner) + -> read lock owner-source + -> if owner-source matches requester-owner: allow + -> else: deny with :server-owned-by-other +``` + +## Implementation plan + +### Phase 1: Add failing tests for owner-aware lock schema and policies. + +1. Add a failing test in `src/test/frontend/worker/db_worker_node_lock_test.cljs` that lock serialization includes `owner-source`. +2. Add a failing test in `src/test/frontend/worker/db_worker_node_lock_test.cljs` that missing owner metadata is normalized to `:unknown`. +3. Add a failing test in `src/test/frontend/worker/db_worker_node_test.cljs` that `--owner-source cli` is written into the lock. +4. Add a failing test in `src/test/frontend/worker/db_worker_node_test.cljs` that `--owner-source electron` is written into the lock. +5. Add a failing test in `src/test/logseq/cli/server_test.cljs` that `stop-server!` returns `:server-owned-by-other` on owner mismatch. +6. Add a failing test in `src/test/logseq/cli/server_test.cljs` that `restart-server!` does not SIGTERM external-owner daemon. +7. Add a failing test in `src/test/electron/db_worker_manager_test.cljs` that external runtime release does not call `stop-daemon!`. +8. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'` and confirm failures match new assertions. +9. Run `bb dev:test -v 'logseq.cli.server-test'` and confirm failures match new assertions. + +### Phase 2: Extend lock schema and daemon startup arguments. + +10. Update argument parsing in `src/main/frontend/worker/db_worker_node.cljs` to accept `--owner-source`. +11. Add owner-source validation in `src/main/frontend/worker/db_worker_node.cljs` with allowed values `cli`, `electron`, and fallback `unknown`. +12. Thread owner-source through `start-daemon!` in `src/main/frontend/worker/db_worker_node.cljs` into lock creation. +13. Update `create-lock!` in `src/main/frontend/worker/db_worker_node_lock.cljs` to persist `owner-source`. +14. Update `read-lock` normalization path in `src/main/frontend/worker/db_worker_node_lock.cljs` to inject `:owner-source :unknown` for legacy files. +15. Keep `update-lock!` in `src/main/frontend/worker/db_worker_node_lock.cljs` from mutating existing owner-source during port updates. +16. Add targeted tests for lock read-write roundtrip in `src/test/frontend/worker/db_worker_node_lock_test.cljs`. +17. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'` and make sure lock owner assertions pass. + +### Phase 3: Make CLI server orchestration owner-aware. + +18. Add requester owner config to `ensure-server!` in `src/main/logseq/cli/server.cljs`. +19. Pass owner-source to daemon spawn args from `spawn-server!` in `src/main/logseq/db_worker/daemon.cljs`. +20. Update `ensure-server-started!` in `src/main/logseq/cli/server.cljs` to return ownership metadata for caller state tracking. +21. Update `stop-server!` in `src/main/logseq/cli/server.cljs` to deny stop when lock owner-source differs from requester owner. +22. Update `restart-server!` in `src/main/logseq/cli/server.cljs` to preserve the same owner check semantics as `stop-server!`. +23. Update `server-status` and `list-servers` in `src/main/logseq/cli/server.cljs` to include owner-source in output payload. +24. Update server command response formatting so `logseq server list` human output includes an `OWNER` column mapped from `owner-source` (and preserves owner metadata in structured output). +25. Add regression tests in `src/test/logseq/cli/server_test.cljs` for owner-aware start, stop, and restart. +26. Run `bb dev:test -v 'logseq.cli.server-test'` and verify no timeout-based flaky failure remains. + +### Phase 4: Add orphan-process detection and recovery for lock-missing start. + +27. Add process listing helper in `src/main/logseq/db_worker/daemon.cljs` that discovers db-worker-node processes by repo and data-dir arguments. +28. Add parser helpers in `src/main/logseq/db_worker/daemon.cljs` to read `repo`, `data-dir`, and `owner-source` from command args. +29. Add `cleanup-orphan-process!` in `src/main/logseq/db_worker/daemon.cljs` to SIGTERM matched orphan pids before new spawn. +30. Call orphan cleanup path in `ensure-server-started!` in `src/main/logseq/cli/server.cljs` when lock is missing before spawn. +31. Add timeout fallback in `ensure-server-started!` in `src/main/logseq/cli/server.cljs` to emit `:server-start-timeout-orphan` with discovered pids. +32. Add unit tests in `src/test/logseq/db_worker/daemon_test.cljs` for process-arg parsing and orphan match logic. +33. Add CLI regression test in `src/test/logseq/cli/server_test.cljs` for lock-missing orphan scenario to avoid raw timeout. +34. Run `bb dev:test -v 'logseq.db-worker.daemon-test'` and `bb dev:test -v 'logseq.cli.server-test'`. + +### Phase 5: Make Electron manager attach external daemon without cross-management. + +35. Pass requester owner as `electron` from `start-managed-daemon!` in `src/electron/electron/db_worker.cljs`. +36. Save ownership flag in manager runtime state in `src/electron/electron/db_worker.cljs`. +37. Update stop flow in `src/electron/electron/db_worker.cljs` so `stop-daemon!` runs only when `owned?` is true. +38. Update unhealthy-runtime branch in `src/electron/electron/db_worker.cljs` to avoid stopping external owner daemon and re-resolve runtime instead. +39. Add tests in `src/test/electron/db_worker_manager_test.cljs` for external runtime reuse plus no-stop-on-release. +40. Run `bb dev:test -v 'electron.db-worker-manager-test'` and confirm lifecycle behavior. + +### Phase 6: Update docs and error surfaces. + +41. Update CLI docs in `docs/cli/logseq-cli.md` to document owner-aware `server stop` and `server restart` behavior. +42. Update desktop lifecycle docs in `docs/developers/desktop-db-worker-node.md` to explain external runtime attachment semantics. +43. Add explicit error messages for `:server-owned-by-other` and `:server-start-timeout-orphan` in `src/main/logseq/cli/format.cljs`. +44. Add one integration note in `docs/agent-guide/033-desktop-db-worker-node-backend.md` linking to owner-aware behavior. + +### Phase 7: Full verification and review gate. + +45. Run `bb dev:test -v 'frontend.worker.db-worker-node-test'`. +46. Run `bb dev:test -v 'frontend.worker.db-worker-node-lock-test'`. +47. Run `bb dev:test -v 'logseq.db-worker.daemon-test'`. +48. Run `bb dev:test -v 'logseq.cli.server-test'`. +49. Run `bb dev:test -v 'electron.db-worker-manager-test'`. +50. Run `bb dev:lint-and-test` and confirm zero failures and zero errors. +51. Perform final review checklist pass against `@prompts/review.md`. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Lock file missing but old CLI daemon still alive. | CLI restart detects orphan by repo and data-dir, cleans it, then starts cleanly. | +| Lock owner is `cli` and Electron calls ensure runtime. | Electron reuses runtime with `owned? false` and never stops it on window close. | +| Lock owner is `electron` and CLI calls `server stop`. | CLI returns `:server-owned-by-other` with owner metadata and no process kill. | +| Legacy lock file has no owner-source field. | System treats owner as `unknown` and allows CLI takeover with owner metadata rewrite. | +| Two owners race to start same graph. | First lock wins and second caller reuses healthy daemon without extra spawn. | +| Owner process crashes and lock remains stale. | Stale lock cleanup still works, and next owner can start daemon normally. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'frontend.worker.db-worker-node-lock-test' +bb dev:test -v 'frontend.worker.db-worker-node-test' +bb dev:test -v 'logseq.db-worker.daemon-test' +bb dev:test -v 'logseq.cli.server-test' +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +The owner-mismatch tests should return `:server-owned-by-other` instead of timeout or forced stop behavior. + +`logseq server list` human output should include an `OWNER` column. + +The orphan-recovery tests should return deterministic cleanup behavior instead of waiting until generic timeout. + +## Testing Details + +The tests validate behavior by asserting lifecycle authority boundaries, successful runtime reuse across clients, and orphan recovery outcomes. + +The tests avoid mock-only success criteria by asserting returned error codes and process-management side effects observable from public APIs. + +The critical regressions are lock-missing orphan restart and CLI-first then Electron-open graph flow, and both are explicitly covered. + +## Implementation Details + +- Extend lock payload with `owner-source` and preserve it across lock updates. +- Pass `--owner-source` when spawning db-worker-node from both CLI and Electron pathways. +- Return ownership metadata from server orchestration so callers can track `owned?` state. +- Enforce owner check for stop and restart while keeping read and write invoke reuse unchanged. +- Add orphan process discovery by command args for lock-missing recovery. +- Scope orphan process discovery in v1 to macOS and Linux, and use a Windows-safe no-op fallback. +- Keep stale-lock cleanup logic and layer orphan recovery without changing healthy lock reuse flow. +- Add explicit CLI error codes for owner mismatch and orphan timeout contexts. +- Prevent Electron manager from stopping external-owner runtime on release or health fallback. +- Document operator-visible behavior changes in CLI and desktop developer docs. +- Execute full suite and `@prompts/review.md` checks before merge. + +## Question + +Decision: CLI is allowed to take over `owner-source: unknown` and rewrite ownership metadata in v1. + +Decision: when lock file is missing, orphan cleanup terminates all matching repo and data-dir processes. + +Decision: v1 scopes orphan process-scan support to macOS and Linux only, with a Windows-safe no-op fallback. + +--- diff --git a/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md b/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md new file mode 100644 index 0000000000..0ad05a9d67 --- /dev/null +++ b/docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md @@ -0,0 +1,167 @@ +# Logseq CLI and db-worker-node deps/cli Decoupling Implementation Plan + +Goal: Make `logseq-cli` behavior independent from old `deps/cli` regressions while restoring list behavior that regressed after commit `0de3c337e` on February 12, 2026. + +Architecture: Apply a CLI-scoped decoupling only for the runtime path used by `logseq-cli` (`src/main/logseq/cli/*` plus db-worker API path it invokes), and defer non-CLI namespace migration to follow-up work. + +Tech Stack: ClojureScript, Datascript, `shadow-cljs` node-script builds (`:logseq-cli` and `:db-worker-node`), babashka test workflow. + +Related: Builds on `docs/agent-guide/003-db-worker-node-cli-orchestration.md`, `docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md`, and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +Developer note (CLI-scoped): The implementation for this plan only migrates the `logseq-cli -> db-worker-node -> thread-api` runtime path by adding `src/main/logseq/cli/common/mcp/tools.cljs`. It does not migrate or alter export/Electron/non-CLI namespaces still resolving from `deps/cli`. + +## Problem statement + +`deps/cli` is the old CLI codebase, but current runtime paths under `src/` still require namespaces from `deps/cli`, so old CLI commits can silently change new CLI behavior. + +Commit `0de3c337e` changed `deps/cli/src/logseq/cli/common/mcp/tools.cljs` and removed fields and filters that new CLI still depends on through db-worker thread-api calls. + +Current regressions are reproducible with existing tests. + +`bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` currently fails because list items no longer include numeric `:id`. + +`bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` currently fails because default tag/property lists no longer include built-in entities needed for ID-based add flows. + +The dependency path that causes this is shown below. + +```text +logseq-cli (src/main/logseq/cli/*) + -> HTTP invoke to db-worker-node + -> thread-api methods in frontend.worker.db-core + -> implementation helper logseq.cli.common.mcp.tools (currently loaded from deps/cli) +``` + +## Current dependency inventory + +| Namespace currently resolved from `deps/cli` | `src/` consumers | Build impact | Status in this plan | +| --- | --- | --- | --- | +| `logseq.cli.common.mcp.tools` | `src/main/frontend/worker/db_core.cljs`, `src/main/logseq/api/db_based/cli.cljs`, `src/main/logseq/sdk/utils.cljs` | `:db-worker-node`, app API, SDK | In scope now | +| `logseq.cli.common.file` | `src/main/frontend/worker/export.cljs` | `:db-worker-node` export path | Out of scope now | +| `logseq.cli.common.util` | `src/main/frontend/extensions/zip.cljs`, export handlers | app export and zip features | Out of scope now | +| `logseq.cli.common.export.common` | `src/main/frontend/handler/export/common.cljs`, `src/main/frontend/handler/export/html.cljs`, `src/main/frontend/handler/export/opml.cljs`, `src/main/frontend/handler/export/text.cljs` | app export features | Out of scope now | +| `logseq.cli.common.export.text` | `src/main/frontend/handler/export/text.cljs` | app markdown export | Out of scope now | +| `logseq.cli.common.graph` | `src/electron/electron/utils.cljs`, `src/electron/electron/db.cljs`, `src/electron/electron/handler.cljs` | Electron graph directory behavior | Out of scope now | +| `logseq.cli.common.mcp.server` | `src/electron/electron/server.cljs` | Electron MCP HTTP endpoint | Out of scope now | +| `logseq.cli.text-util` | `src/main/frontend/util/text.cljs` | frontend text helpers | Out of scope now | + +## Scope + +This plan only changes logseq-cli related runtime behavior. + +This plan migrates the `logseq-cli -> db-worker-node -> thread-api` path where the implementation currently resolves to `deps/cli`. + +In this PR, that means migrating `logseq.cli.common.mcp.tools` out of `deps/cli` because it is the thread-api-side implementation used by logseq-cli list/add flows. + +This plan does not modify any code under `deps/cli`. + +This plan does not migrate frontend export namespaces, Electron namespaces, or unrelated old CLI modules in `deps/cli`. + +This plan does not redesign CLI command UX or old CLI feature parity, and old CLI under `deps/cli` remains frozen unless a compatibility fix is required for release safety. + +Implementation must follow @test-driven-development, and any unexpected test failure while migrating must follow @clojure-debug before changing behavior. + +## Testing Plan + +I will first lock the current regressions by running the two failing integration tests as baseline checks and recording failure reasons in the PR notes. + +I will add focused tests for list data contract behavior so `:id`, built-in inclusion defaults, and page filtering options are guaranteed independent of old CLI code. + +I will run compile checks for both node builds to ensure the CLI path behaves correctly after the targeted namespace move. + +I will not include export/Electron migration tests in this PR because those modules are explicitly out of scope. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +1. Add a migration checklist section in the PR description that references this document and lists the two known failing integration tests. + +2. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` and confirm it fails before code changes. + +3. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` and confirm it fails before code changes. + +4. Add a focused test file at `src/test/logseq/cli/mcp_tools_contract_test.cljs` for `list-pages`, `list-tags`, and `list-properties` contract behavior used by new CLI. + +5. In that test file, add a case asserting non-expanded list output includes `:db/id`, `:block/title`, `:block/created-at`, and `:block/updated-at`. + +6. In that test file, add a case asserting `include-built-in` defaults to true and explicitly false excludes built-in tags and properties. + +7. In that test file, add a case asserting page list filtering honors `include-hidden`, `include-journal`, `journal-only`, `created-after`, and `updated-after`. + +8. Add `src/main/logseq/cli/common/mcp/tools.cljs` by porting the pre-`0de3c337e` behavior as baseline thread-api implementation and keeping API signatures used by `db-core` and SDK. + +9. Update `src/` callsites to resolve the new `src/main` implementation and keep `deps/cli` source files untouched. + +10. Re-run `bb dev:test -v 'logseq.cli.mcp-tools-contract-test'` and make it pass with no test skips. + +11. Re-run the two integration regressions and make both pass. + +12. Keep all files under `deps/cli` unchanged in this PR. + +13. Do not remove `logseq/cli` from `deps.edn` in this PR because unrelated frontend and Electron runtime modules still depend on it. + +14. Run `clojure -M:cljs compile logseq-cli` and verify compile success. + +15. Run `clojure -M:cljs compile db-worker-node` and verify compile success. + +16. Run `bb dev:test -v 'logseq.cli.commands-test/test-list-subcommand-parse'` to verify list option parsing remains stable. + +17. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-outputs-include-id'` to verify ID contract restoration end-to-end. + +18. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-add-tags-and-properties-by-id'` to verify built-in tag/property ID flows end-to-end. + +19. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-list-add-show-remove'` as a smoke test for normal create/list/show/remove flow. + +20. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-query-recent-updated'` as a smoke test for list timestamps and query interplay. + +21. Run `bb dev:lint-and-test` before merge to satisfy repository review checklist. + +## Edge cases to validate during implementation + +If `created-after` or `updated-after` is an invalid date string, filtering should not crash and should behave as no time filter. + +When `include-journal` is omitted, journals should remain included by default to preserve existing CLI expectations. + +When `include-built-in` is omitted, built-in tags and properties must remain included so ID resolution in add/update flows still works. + +Non-expanded list output must contain stable numeric IDs even if UUID and title are present. + +Expanded list output must keep UUID string conversion and keep relationship fields (`classes`, `extends`, `properties`) in expected shapes. + +Non-CLI frontend export and Electron behavior must remain untouched in this PR. + +## Open questions requiring clarity + +Should we do a second PR for non-CLI namespace migration (`export`, `electron`, `text-util`) immediately after this thread-api decoupling PR, or wait until after release. + +## Testing Details + +The key behavior tests are integration-first and validate user-observable outcomes, not internal helper implementation. + +`test-cli-list-outputs-include-id` verifies that real CLI JSON output includes usable IDs for list operations. + +`test-cli-add-tags-and-properties-by-id` verifies that list output can feed directly into add operations using IDs and complete a write/read roundtrip. + +`mcp-tools-contract-test` verifies option semantics and filtering behavior at the db-worker API boundary to prevent future regressions from unrelated old CLI edits. + +## Implementation Details + +- Keep migration atomic by changing only logseq-cli related runtime pieces in this PR. +- Preserve public function names and argument shapes to minimize callsite churn. +- Restore pre-`0de3c337e` list semantics for IDs and built-in filtering. +- Do not edit any file under `deps/cli`; all migration changes must happen in `src/` and `src/test/`. +- Do not change command parsing behavior in `src/main/logseq/cli/command/list.cljs` unless tests show incompatibility. +- Keep Electron/export/frontend non-CLI require lines unchanged in this PR. +- Treat `deps/cli` as frozen legacy code for non-CLI modules in this PR. +- Add a short developer note in `README.md` or `docs` explaining this PR is CLI-scoped only. +- Use smallest possible commits per namespace group to simplify rollback. +- Ensure all touched files remain ASCII and follow existing formatting conventions. +- Finish with `bb dev:lint-and-test` and include command outputs in the PR summary. + +## Decision + +This migration will keep `logseq.cli.common.*` namespace names stable and only move CLI-path runtime implementation needed for `logseq-cli`. + +Namespace renaming will be done in a follow-up PR after decoupling and regression fixes are complete. + +--- diff --git a/docs/agent-guide/036-db-worker-node-ncc-bundling.md b/docs/agent-guide/036-db-worker-node-ncc-bundling.md new file mode 100644 index 0000000000..dccc7f4929 --- /dev/null +++ b/docs/agent-guide/036-db-worker-node-ncc-bundling.md @@ -0,0 +1,199 @@ +# db-worker-node ncc Standalone Bundle Implementation Plan + +Goal: Build `db-worker-node.js` with `@vercel/ncc` so the runtime can run without `node_modules` present next to the executable. + +Architecture: Keep `shadow-cljs` as the source compiler for `:db-worker-node`, then run `ncc` on the generated entry and publish a single runtime artifact in `dist/` that is used by CLI daemon orchestration. +Architecture: Preserve local development ergonomics by keeping `static/db-worker-node.js` for fast dev loops, while production and package paths resolve to the ncc artifact first. + +Tech Stack: ClojureScript, `shadow-cljs` `:node-script`, `@vercel/ncc`, Node.js 22, `pnpm` scripts in `package.json`, existing CLI daemon and doctor checks. + +Related: Builds on `docs/agent-guide/031-logseq-cli-doctor-command.md`. +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. +Related: Relates to `docs/agent-guide/035-logseq-cli-db-worker-deps-cli-decoupling.md`. + +## Problem statement + +`/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js` is currently generated by `shadow-cljs`, and runtime behavior assumes dependencies are available from `node_modules`. + +The CLI server startup path in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` points to `../dist/db-worker-node.js`, which is currently a thin wrapper that forwards to `../static/db-worker-node.js`. + +This wrapper model keeps runtime coupled to workspace layout and to `node_modules`, so the daemon script is not independently portable. + +We need a deterministic packaging path that produces one runnable artifact plus copied native assets, and we need to verify that artifact works when `node_modules` is absent. + +Electron release packaging also needs to include the db-worker standalone bundle step, so `pnpm release-electron` always ships the same packaged runtime artifact. + +The solution must keep existing CLI and Electron daemon orchestration behavior unchanged, including lock-file semantics, owner-source semantics, and health endpoint behavior. + +## Current packaging map + +| Area | Current behavior | Limitation | +| --- | --- | --- | +| Build output | `pnpm db-worker-node:compile` writes `/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js`. | Output is not a standalone distribution artifact. | +| Dist entry | `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` only `require`s `../static/db-worker-node.js`. | Runtime still depends on static output and installed dependencies. | +| Daemon spawn | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` spawns `../dist/db-worker-node.js`. | Spawn path is stable, but executable is not standalone. | +| Doctor check | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` checks `../static/db-worker-node.js` by default. | Diagnostic target does not match the intended distributable runtime. | +| Package manifest | `/Users/rcmerci/gh-repos/logseq/package.json` includes `static/db-worker-node.js` in `files`. | Published package does not guarantee standalone daemon artifact contract. | +| Electron release | `pnpm release-electron` does not guarantee db-worker bundle refresh before packaging. | Desktop release artifact can drift from standalone db-worker bundle contract. | + +## Target packaging map + +| Area | Target behavior | Verification signal | +| --- | --- | --- | +| Bundle output | `ncc` emits a standalone `db-worker-node` runtime in `/Users/rcmerci/gh-repos/logseq/dist/` with required runtime assets copied adjacent to entrypoint. | Daemon starts and serves `/healthz` and `/readyz` without `node_modules`. | +| Spawn path | CLI server keeps spawning `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` as canonical runtime. | Existing `logseq.cli.server-test` assertions remain green with updated contract. | +| Doctor check | Doctor defaults to the same packaged runtime path used for spawn, and does not auto-fallback to static runtime. | Doctor check path matches runtime path in tests and manual runs. | +| Dev flow | Fast local dev command remains available using `static/db-worker-node.js` for watch and debug workflows. | `pnpm db-worker-node:compile` and `node ./static/db-worker-node.js` still work during development. | +| Publish flow | Package `files` include standalone runtime assets required by ncc output. | Installed package can execute daemon without extra dependency install. | +| Electron release | `pnpm release-electron` runs db-worker bundle build before Electron packaging steps. | Electron release artifact includes the same standalone db-worker runtime contract. | + +## Integration sketch + +```text +shadow-cljs (:db-worker-node) + -> /static/db-worker-node.js + -> ncc build step + -> /dist/db-worker-node.js + -> /dist/ + +logseq-cli runtime + -> logseq.cli.server/spawn-server! + -> /dist/db-worker-node.js + -> db-worker daemon HTTP + SSE API +``` + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before implementation changes in each phase. + +I will add behavior tests for runtime path resolution so spawn and doctor point to the same canonical bundle target. + +I will add a standalone smoke test that launches the bundled daemon from a temporary directory without `node_modules` and verifies `/healthz`, `/readyz`, and shutdown behavior. + +I will keep existing daemon lifecycle tests green to ensure no regression in lock cleanup, owner checks, and timeout error semantics. + +I will run focused tests first, then full validation with `pnpm cljs:lint && pnpm test`, and if any unexpected failures appear I will use `@clojure-debug` before changing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: RED for runtime artifact contract. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` asserting canonical db-worker runtime path resolves to `dist/db-worker-node.js` as the production target. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` asserting default doctor script check points to the same canonical runtime target as server spawn. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/doctor_test.cljs` asserting optional dev-mode check can still validate `static/db-worker-node.js` when explicitly requested. +4. Add a new failing bundle smoke test file at `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` that expects daemon startup success from a bundle-only temp directory. +5. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.server-test'` and confirm failures occur on the new path-contract assertions. +6. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm failures occur on the new default-path expectations. +7. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm standalone smoke test fails before implementation. + +### Phase 2: Add ncc build pipeline. + +8. Add `@vercel/ncc` to `/Users/rcmerci/gh-repos/logseq/package.json` as a dev dependency. +9. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to build `:db-worker-node` in release mode before bundling. +10. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to run `ncc` against `/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js`. +11. Add a dedicated script in `/Users/rcmerci/gh-repos/logseq/package.json` to normalize ncc output into `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` with adjacent assets preserved. +12. If script complexity is non-trivial, add `/Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs` to encapsulate output normalization and deterministic cleanup. +13. Add or update `pnpm` scripts in `/Users/rcmerci/gh-repos/logseq/package.json` for one-command bundle build and optional local run of the bundled artifact. +14. Update `pnpm release-electron` in `/Users/rcmerci/gh-repos/logseq/package.json` so it includes `db-worker-node:release:bundle` before Electron packaging. +15. Run `pnpm db-worker-node:release:bundle` and verify `dist/db-worker-node.js` is regenerated with executable permissions preserved. + +### Phase 3: Align runtime path and diagnostics. + +15. Refactor runtime path helpers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` so there is one canonical function for packaged runtime and one explicit dev fallback function. +16. Keep `spawn-server!` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` bound to the packaged runtime path to avoid ambiguity. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` to default-check the same packaged runtime path used by spawn. +18. Add an explicit action option in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/doctor.cljs` for fallback static-path diagnostics used only for development troubleshooting. +19. Ensure doctor failure codes remain stable as `:doctor-script-missing` and `:doctor-script-unreadable`. +21. Re-run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.server-test'` and `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.command.doctor-test'` to make the new path contract green. + +### Phase 4: Package manifest and docs alignment. + +22. Update `files` in `/Users/rcmerci/gh-repos/logseq/package.json` so packaged runtime includes `dist/db-worker-node.js` and ncc-emitted adjacent assets. +23. Keep `static/db-worker-node.js` inclusion only if required for development workflows, and document that distinction explicitly. +24. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build instructions to include the ncc bundle command and standalone runtime expectations. +25. Update daemon runtime notes in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` so `doctor` references packaged runtime as primary target. +26. Add troubleshooting notes for native module asset copy behavior from ncc output. + +### Phase 5: Standalone runtime behavior verification. + +27. Implement bundle smoke test setup in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to copy only bundle artifacts into a temp directory with no `node_modules`. +28. In that test, spawn `node ./db-worker-node.js --repo --data-dir ` from the bundle-only directory. +29. In that test, poll `/healthz` and `/readyz` and assert both return HTTP 200 after startup. +30. In that test, invoke `/v1/shutdown` and assert process exits and lock file is cleaned or becomes stale-removable. +31. In that test, assert failure output is actionable if native binary asset is missing, to guard accidental packaging regressions. +32. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and make it green. + +### Phase 6: Final validation and review checklist. + +33. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.server-test'` and confirm zero failures and zero errors. +34. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.command.doctor-test'` and confirm zero failures and zero errors. +35. Run `pnpm cljs:test && pnpm cljs:run-test -v 'logseq.db-worker.ncc-bundle-test'` and confirm zero failures and zero errors. +36. Run `pnpm db-worker-node:compile` and verify `static/db-worker-node.js` remains valid for local dev flow. +37. Run `pnpm db-worker-node:release:bundle` and verify `dist/db-worker-node.js` starts successfully with `node dist/db-worker-node.js --help`. +38. Run `pnpm release-electron` and verify the script execution includes `db-worker-node:release:bundle` before Electron packaging steps. +39. Run `pnpm cljs:lint && pnpm test` and confirm repository review checklist passes. +40. Validate changed code against `@prompts/review.md` before merge. + +## Edge cases to validate during implementation + +| Scenario | Expected behavior | +| --- | --- | +| `ncc` emits native `.node` assets for `better-sqlite3`. | Bundle output keeps those assets adjacent to entrypoint and runtime loads without `node_modules`. | +| Bundle is copied to another directory without static files. | Daemon still starts because packaged runtime no longer `require`s `../static/db-worker-node.js`. | +| Developer runs doctor in source workspace before bundle build. | Doctor reports missing packaged artifact by default, and only checks static runtime when explicitly requested. | +| `dist/db-worker-node.js` exists but is not readable or is a directory. | Doctor returns `:doctor-script-unreadable` with path detail. | +| Bundle build is run twice. | Build output remains deterministic and stale ncc artifacts are cleaned safely. | +| `pnpm release-electron` is run directly. | Release flow still builds db-worker standalone bundle before Electron packaging artifacts are produced. | +| CLI and Electron share lock for same repo under bundled runtime. | Existing ownership and lock semantics remain unchanged from current behavior. | + +## Verification commands and expected outputs + +```bash +pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.server-test' +pnpm cljs:test && pnpm cljs:run-test -v 'logseq.cli.command.doctor-test' +pnpm cljs:test && pnpm cljs:run-test -v 'logseq.db-worker.ncc-bundle-test' +pnpm db-worker-node:compile +pnpm db-worker-node:release:bundle +pnpm release-electron +node dist/db-worker-node.js --help +pnpm cljs:lint && pnpm test +``` + +All test commands should finish with `0 failures, 0 errors`. + +The bundle command should finish without `MODULE_NOT_FOUND` for runtime dependencies. + +`node dist/db-worker-node.js --help` should print daemon help text and exit with code `0`. + +## Testing Details + +Behavior-focused tests will validate that runtime path resolution, doctor diagnostics, and daemon startup behavior match user-visible expectations. + +The standalone smoke test will verify real process startup and HTTP readiness in a bundle-only filesystem layout, rather than asserting internal helper calls. + +Regression safety is provided by existing CLI server and doctor tests to ensure lock lifecycle and error-code contracts remain stable. + +## Implementation Details + +- Keep packaged runtime path as one canonical helper shared by spawn and doctor. +- Keep dev runtime path explicit and opt-in for local diagnostics only. +- Introduce ncc build scripts with deterministic output normalization into `dist/`. +- Preserve local `shadow-cljs` development flow and avoid slowing watch mode. +- Add a dedicated standalone bundle smoke test namespace for runtime validation. +- Keep error code contracts stable for doctor and server command callers. +- Ensure package `files` include all ncc runtime assets required at execution time. +- Document build and troubleshooting steps in CLI docs for contributors and release workflows. +- Use `@test-driven-development` for every behavior change and follow `@clojure-debug` for unexpected failures. +- Finish with full lint and test validation and checklist review from `@prompts/review.md`. + +## Question + +Resolved: choose option 1. + +`doctor` defaults to strict packaged-runtime validation only. + +`doctor` does not auto-fallback to static runtime without an explicit flag. + +--- diff --git a/docs/agent-guide/037-db-worker-node-node-sqlite.md b/docs/agent-guide/037-db-worker-node-node-sqlite.md new file mode 100644 index 0000000000..2701ba92b6 --- /dev/null +++ b/docs/agent-guide/037-db-worker-node-node-sqlite.md @@ -0,0 +1,166 @@ +# db-worker-node Node Built-in SQLite Migration Implementation Plan + +Goal: Replace `better-sqlite3` in `db-worker-node` with Node.js built-in `node:sqlite` while keeping `logseq-cli` behavior and db-worker thread-api contracts unchanged. + +Architecture: Keep the existing platform adapter boundary in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` and swap only the Node SQLite backend implementation to a compatibility wrapper around `DatabaseSync` and `StatementSync`. +Architecture: Preserve daemon lifecycle and lock ownership semantics, then update bundle/test/doc assumptions that currently depend on native `.node` assets from `better-sqlite3`. + +Tech Stack: ClojureScript, `shadow-cljs` `:node-script`, Node.js `>=22.20.0`, `node:sqlite`, `@vercel/ncc`, `logseq-cli` HTTP transport. + +Related: Builds on `docs/agent-guide/033-desktop-db-worker-node-backend.md` and `docs/agent-guide/036-db-worker-node-ncc-bundling.md`. +Related: Relates to `docs/agent-guide/task--db-worker-nodejs-compatible.md` and `docs/cli/logseq-cli.md`. + +## Problem statement + +`db-worker-node` currently requires `better-sqlite3` from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. + +`logseq-cli` and Electron desktop both depend on this daemon runtime through `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs`, so backend driver replacement must not change daemon API behavior. + +The current ncc bundle plan and tests assume native `.node` assets are emitted and copied next to `dist/db-worker-node.js`, which is specific to `better-sqlite3` and must be revised after migration. + +Node.js in this repository is already pinned to `>=22.20.0` in `/Users/rcmerci/gh-repos/logseq/package.json`, so `node:sqlite` is available, but it is still experimental and emits runtime warnings that we need to account for. + +## Current implementation map + +| Area | Current implementation | Migration impact | +| --- | --- | --- | +| Node sqlite adapter | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` wraps `better-sqlite3` with custom `exec` and `transaction` behavior. | Replace constructor and statement execution with `node:sqlite` API while preserving wrapper contract. | +| Daemon runtime contract | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` exposes `/healthz`, `/readyz`, `/v1/invoke`, `/v1/events`. | No protocol change allowed. | +| CLI runtime spawn | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` spawns `dist/db-worker-node.js`. | Behavior must remain unchanged. | +| Bundle smoke tests | `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` expects a missing native `.node` asset failure mode. | Replace asset assertions to match built-in sqlite runtime with zero native assets. | +| Dependency declaration | `/Users/rcmerci/gh-repos/logseq/package.json` includes `better-sqlite3`. | Remove dependency and refresh lockfile. | +| CLI documentation | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` references `dist/build/Release/better_sqlite3.node`. | Update build output description and troubleshooting text. | + +## Target architecture + +```text +logseq-cli / electron + -> db-worker-node HTTP API + -> frontend.worker.db-core + -> frontend.worker.platform.node sqlite wrapper + -> node:sqlite DatabaseSync + -> graph-dir/*.sqlite files +``` + +The wrapper contract consumed by db-core remains `open-db`, `exec`, `transaction`, and `close`. + +The wrapper implementation changes from `better-sqlite3` to `node:sqlite` internals only. + +## Testing Plan + +I will use `@test-driven-development` and add failing tests first for adapter behavior and bundle assumptions before modifying runtime code. + +I will add focused Node adapter tests for parameter binding, array-row reads, commit behavior, and rollback behavior so the compatibility wrapper is behavior-locked. + +I will keep existing daemon smoke tests and CLI sqlite import/export integration tests as end-to-end regression guards. + +I will update ncc bundle tests so they validate standalone runtime startup without relying on native `.node` artifacts. + +I will run `bb dev:test` for focused namespaces first, then run `bb dev:lint-and-test` for the repository checklist. + +I will use `@clojure-debug` if any ClojureScript test fails unexpectedly while porting the adapter. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Lock behavior with failing tests. + +1. Add a new test namespace at `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/platform_node_test.cljs` to cover Node sqlite wrapper behavior in isolation. +2. Add a failing test that `exec` with SQL string creates schema and writes data through the wrapper. +3. Add a failing test that `exec` with `{:sql ... :bind ... :rowMode "array"}` returns array rows in the same shape used by `restore-data-from-addr`. +4. Add a failing test that named bindings with `$name` and `:name` styles are both accepted by the wrapper. +5. Add a failing test that wrapper `transaction` commits writes on success. +6. Add a failing test that wrapper `transaction` rolls back writes when callback throws. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` asserting bundled runtime can start even when bundle manifest has zero assets. +8. Replace the current missing-native-asset expectation test with a failing test that checks bundle manifest format and actionable errors for missing manifest or missing entry script instead. +9. Run `bb dev:test -v 'frontend.worker.platform-node-test'` and confirm failures before implementation. +10. Run `bb dev:test -v 'logseq.db-worker.ncc-bundle-test'` and confirm failures before implementation. + +### Phase 2: Port Node sqlite adapter to node:sqlite. + +11. Edit `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs` to replace `"better-sqlite3"` require with `"node:sqlite"`. +12. Build a `DatabaseSync` constructor resolver compatible with Shadow-CLJS interop and avoid default-export assumptions. +13. Keep `open-sqlite-db` async shape unchanged and create `DatabaseSync` after ensuring parent directory exists. +14. Re-implement statement execution so `:rowMode "array"` maps to `StatementSync#setReturnArrays(true)`. +15. Re-implement positional and named parameter passing for array and object binds without changing db-core callsites. +16. Preserve existing bind key normalization behavior for `$name` and `:name` forms to avoid hidden regressions. +17. Re-implement wrapper `transaction` semantics using explicit SQL transaction control with rollback on exceptions. +18. Add nested-transaction safety via savepoint naming or equivalent deterministic strategy to avoid partial writes from nested calls. +19. Keep wrapper `close` idempotent and compatible with existing shutdown paths in `db-core` and daemon stop. +20. Keep all public platform map keys unchanged in `node-platform`. + +### Phase 3: Update dependency and bundle assumptions. + +21. Remove `better-sqlite3` from `/Users/rcmerci/gh-repos/logseq/package.json` dependencies. +22. Run `pnpm install` to refresh `/Users/rcmerci/gh-repos/logseq/pnpm-lock.yaml` and verify `better-sqlite3` is removed from runtime dependency graph. +23. Verify no remaining runtime require for `better-sqlite3` via `rg -n "better-sqlite3" /Users/rcmerci/gh-repos/logseq/src /Users/rcmerci/gh-repos/logseq/package.json /Users/rcmerci/gh-repos/logseq/pnpm-lock.yaml`. +24. Keep `/Users/rcmerci/gh-repos/logseq/scripts/package.json` unchanged in this task because scope is limited to `logseq-cli` and `db-worker-node` runtime paths. +25. Update `/Users/rcmerci/gh-repos/logseq/scripts/build-db-worker-node-bundle.mjs` only if manifest handling needs explicit support for empty asset arrays. + +### Phase 4: Refresh tests and docs around runtime packaging. + +26. Update `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/ncc_bundle_test.cljs` to stop asserting a required `.node` asset exists. +27. Keep startup smoke test that runs copied `db-worker-node.js` from a temporary runtime directory with no `node_modules`. +28. Add assertions that `/healthz`, `/readyz`, and `/v1/shutdown` still work under bundled runtime. +29. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` build section to remove `better_sqlite3.node` runtime-asset example. +30. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` troubleshooting text to describe expected bundle output when native assets are absent. +31. Update references in `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/task--db-worker-nodejs-compatible.md` to reflect that Node runtime now uses built-in sqlite. + +### Phase 5: Run regression and verification commands. + +32. Run `bb dev:test -v 'frontend.worker.platform-node-test'` and expect `0 failures, 0 errors`. +33. Run `bb dev:test -v 'frontend.worker.db-worker-node-test/db-worker-node-daemon-smoke-test'` and expect daemon startup and query path to pass. +34. Run `bb dev:test -v 'frontend.worker.db-worker-node-test/db-worker-node-import-db-base64'` and expect sqlite export/import behavior to pass. +35. Run `bb dev:test -v 'logseq.db-worker.ncc-bundle-test'` and expect standalone bundle smoke tests to pass. +36. Run `bb dev:test -v 'logseq.cli.integration-test/test-cli-graph-export-import-sqlite'` and expect end-to-end CLI sqlite flow to pass. +37. Run `clojure -M:cljs compile db-worker-node logseq-cli` and expect successful node-script builds. +38. Run `pnpm db-worker-node:release:bundle` and verify `dist/db-worker-node.js` still starts with `node ./dist/db-worker-node.js --help`. +39. Run `bb dev:lint-and-test` and expect full lint and test checks to pass. +40. Review changed files against `/Users/rcmerci/gh-repos/logseq/prompts/review.md` checklist before merge. + +## Edge cases to validate during implementation + +| Scenario | Expected behavior | +| --- | --- | +| Named parameter binding uses `$name` keys from current callsites. | Statement executes without bind-key mismatch errors. | +| Named parameter binding uses `:name` keys from normalized callsites. | Statement executes and returns same data as before. | +| `rowMode` is `"array"` for kv restore reads. | First row remains index-addressable for existing `first` and destructuring logic. | +| Transaction callback throws mid-write. | All writes in that transaction scope are rolled back. | +| Nested transaction callback occurs inside outer transaction. | Inner failure does not commit partial data and outer behavior is deterministic. | +| ncc bundle emits zero extra assets. | Bundle tests and CLI docs still treat runtime as valid standalone output. | +| Daemon starts under Node 22 and emits experimental sqlite warning. | Warning does not break health/readiness checks or CLI invoke flow. | +| Graph sqlite import/export uses large payloads. | Base64 transport and file writes still preserve binary integrity. | + +## Decisions confirmed + +1. Keep Node experimental `node:sqlite` warning output as-is for this migration phase; warning suppression and logging policy changes are out of scope. +2. Scope is limited to `logseq-cli` and `db-worker-node`; do not modify `/Users/rcmerci/gh-repos/logseq/scripts/package.json` in this task. +3. No temporary fallback flag to `better-sqlite3`; enforce one-way migration to built-in `node:sqlite`. +4. Treat Node.js `>=22.20.0` as a hard prerequisite for local and CI runtime to ensure `node:sqlite` availability. + +## Testing Details + +The adapter unit tests will validate observable behavior for SQL execution, parameter binding, row shape, and transaction semantics instead of testing internal helper structure. + +The daemon smoke tests will validate real process startup and thread-api calls so platform wiring and lock behavior stay stable. + +The CLI integration sqlite export/import test will verify user-visible behavior from command surface to db-worker storage backend. + +The bundle tests will validate standalone runtime packaging assumptions that changed because native `.node` assets are no longer required. + +## Implementation Details + +- Keep `db-worker-node` HTTP and SSE API contracts unchanged. +- Keep platform adapter keys unchanged to avoid db-core callsite churn. +- Implement a compatibility wrapper over `DatabaseSync` instead of refactoring db-core. +- Preserve bind normalization semantics for backward compatibility. +- Implement explicit rollback-safe transaction handling with nested safety. +- Remove `better-sqlite3` only from main runtime dependency declarations. +- Update bundle tests to assert behavior, not driver-specific artifact names. +- Update CLI docs and historical planning notes that mention native sqlite assets. +- Require Node.js `>=22.20.0` in local and CI verification environments. +- Run focused tests first, then repository-wide lint and tests. +- Follow `@test-driven-development` and `@clojure-debug` for implementation and debugging workflow. + +--- diff --git a/docs/agent-guide/038-electron-db-worker-switch-graph.md b/docs/agent-guide/038-electron-db-worker-switch-graph.md new file mode 100644 index 0000000000..eed3e7acb3 --- /dev/null +++ b/docs/agent-guide/038-electron-db-worker-switch-graph.md @@ -0,0 +1,187 @@ +# Electron Db Worker Switch Graph Implementation Plan + +Goal: Fix Electron graph switching with db-worker-node so switching to a new graph never leaves the renderer bound to a stopped runtime. + +Architecture: Keep db-worker runtime lifecycle in the `db-worker-runtime` path and window close path, and treat `setCurrentGraph` as window graph metadata synchronization only. + +Architecture: Remove or guard the duplicate release path in `setCurrentGraph` that can stop the newly started runtime after `persist-db/ persist-db/ ipc "db-worker-runtime" B. + -> main db-worker manager switches A -> B. + -> state/set-current-repo! B. + -> ipc "setCurrentGraph" B. + -> handler calls release-window! again. + -> B runtime may be stopped unexpectedly. + +Target sequence. +Renderer restore graph B. + -> persist-db/ ipc "db-worker-runtime" B. + -> main db-worker manager switches A -> B. + -> state/set-current-repo! B. + -> ipc "setCurrentGraph" B. + -> handler only updates window graph path. + -> B runtime stays available. +``` + +## Testing Plan + +I will follow `@test-driven-development` and add failing tests before each behavior change. + +I will add a failing regression test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that encodes the expected post-switch invariant that the new runtime remains active until explicit release on window close. + +I will add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db_test.cljs` for the graph switch flow to assert runtime rebinding happens once per target repo and is not reinitialized by graph metadata sync. + +I will add a focused unit test file `/Users/rcmerci/gh-repos/logseq/src/test/electron/graph_switch_flow_test.cljs` for extracted pure graph-switch decision logic so the release/no-release condition is testable without Electron GUI dependencies. + +I will run focused tests after each phase and finish with `bb dev:lint-and-test`. + +I will perform manual Electron smoke checks that open graph A, switch to graph B, execute thread-api reads and writes, and then switch back to graph A. + +I will review changes against `@prompts/review.md` before merge. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests that reproduce the switch-order regression. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/electron/db_worker_manager_test.cljs` that simulates A -> B switch and asserts no stop is triggered for B before explicit release. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/persist_db_test.cljs` for the sequence ` B -> A and invoking read and write actions after each switch. +26. Validate no immediate invoke failures after switch and confirm runtime stays available for the active graph. +27. Remove temporary logs that are not needed for long-term maintainability. + +### Phase 7: Final verification and review gate. + +28. Run `bb dev:test -v 'electron.graph-switch-flow-test'`. +29. Run `bb dev:test -v 'electron.db-worker-manager-test'`. +30. Run `bb dev:test -v 'frontend.persist-db-test'`. +31. Run `bb dev:lint-and-test`. +32. Confirm each command reports `0 failures, 0 errors`. +33. Run final review checklist pass against `@prompts/review.md`. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Switch A -> B in one window where both graphs are local db graphs. | Runtime for B remains active and calls succeed immediately after switch. | +| Switch A -> B -> A quickly before all async handlers settle. | Late `setCurrentGraph` sync does not stop the currently active runtime. | +| Two windows share graph A and one window switches to graph B. | Graph A runtime remains alive for the other window and graph B runtime starts for the switching window. | +| Re-select current graph B from UI without actual graph change. | No runtime restart and no runtime release occurs. | +| Window closes right after switch to B. | Runtime release happens exactly once via close flow and does not leave stale window mapping. | +| Runtime ownership is external (`:owned? false`). | Graph switch does not attempt to stop external runtime unexpectedly. | +| Restore fails after runtime bind but before UI route redirect. | Failure handling does not silently stop the newly bound runtime unless explicit cleanup path runs. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'electron.graph-switch-flow-test' +bb dev:test -v 'electron.db-worker-manager-test' +bb dev:test -v 'frontend.persist-db-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +Manual Electron switch checks should show successful read and write operations immediately after each switch. + +No `db-worker invoke failed` errors should appear during normal A -> B -> A switching. + +## Testing Details + +The new tests verify behavior around event ordering and runtime lifecycle boundaries instead of implementation details. + +The manager tests validate that stale or duplicate release operations cannot terminate the active repo runtime. + +The frontend tests validate that runtime rebinding is driven by repo changes and not by metadata synchronization calls. + +Manual smoke checks validate real Electron runtime behavior that unit tests cannot fully represent. + +## Implementation Details + +- Keep `setCurrentGraph` focused on graph-path synchronization and remove lifecycle side effects from that path. +- Keep runtime start and stop orchestration centralized in `electron.db-worker` manager APIs. +- Use a small pure helper for release decision logic so regression tests do not depend on Electron runtime modules. +- Preserve existing `db-worker-runtime` IPC contract from renderer to main process. +- Keep old graph cleanup tied to explicit switch lifecycle and window close lifecycle only. +- Validate switch behavior with both single-window and multi-window tests. +- Use `@test-driven-development` for red-green implementation order. +- Follow `@prompts/review.md` checks before merging. + +## Question + +Decision: On failed graph restore, keep the newly bound runtime alive for fast retry in the same window. + +Decision: Do not add a short-lived debug flag for switch sequencing logs in development builds. + +Decision: Add an E2E scenario in `/Users/rcmerci/gh-repos/logseq/clj-e2e` for switch graph with db-worker-node and treat it as a release gate. + +--- diff --git a/docs/agent-guide/039-worker-platform-abstraction-cleanup.md b/docs/agent-guide/039-worker-platform-abstraction-cleanup.md new file mode 100644 index 0000000000..2bc4e3762c --- /dev/null +++ b/docs/agent-guide/039-worker-platform-abstraction-cleanup.md @@ -0,0 +1,213 @@ +# Worker Platform Abstraction Cleanup Implementation Plan + +Goal: Route shared db-worker sync code through `frontend.worker.platform` wrappers so browser and node runtimes both work without runtime-specific branches in shared modules. + +Architecture: Keep runtime-specific APIs inside `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/browser.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform/node.cljs`. +Call platform capabilities from shared modules via `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` using `platform/current` at the call site. +Preserve existing key names and payload shapes to avoid data migration. + +Tech Stack: ClojureScript, promesa, cljs.test, clojure-lsp diagnostics, db-worker platform adapters. + +Related: Relates to `docs/agent-guide/038-electron-db-worker-switch-graph.md` and `docs/agent-guide/db-sync/db-sync-guide.md`. + +## Problem statement + +`frontend.worker.platform` currently exposes public wrappers that clojure-lsp reports as unused. + +The reported vars are `kv-get`, `kv-set!`, `read-text!`, `write-text!`, and `websocket-connect`. + +Shared worker modules still contain runtime-specific calls that bypass those wrappers. + +The bypasses are concentrated in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +This creates duplicated runtime assumptions and weakens node and browser parity. + +| Symptom | Current location | Impact | +|---|---|---| +| `clojure-lsp/unused-public-var` on `platform/kv-get` and `platform/kv-set!` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals shared code is not using the adapter path for kv persistence. | +| `clojure-lsp/unused-public-var` on `platform/read-text!` and `platform/write-text!` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals text file I/O in shared code can still hardcode a backend. | +| `clojure-lsp/unused-public-var` on `platform/websocket-connect` | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs` | Signals websocket creation in shared sync code may bypass node adapter (`ws`). | +| Direct `js/WebSocket.` in shared sync module | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` | Couples shared sync lifecycle to browser global API. | +| Direct `opfs/ sync.cljs -> js/WebSocket. +db_core -> sync/crypt.cljs -> opfs/idb-keyval. + +Target shared path. +db_core -> sync.cljs -> platform/websocket-connect. +db_core -> sync/crypt.cljs -> platform/read-text!/write-text!/kv-get/kv-set!. + +Runtime adapter ownership. +browser adapter -> OPFS + IndexedDB + browser WebSocket. +node adapter -> fs + JSON kv file + ws package. +``` + +## Testing Plan + +I will add a unit test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that asserts `#'db-sync/connect!` creates sockets through `platform/websocket-connect` and not direct globals. + +I will write the test by stubbing `platform/current`, `platform/websocket-connect`, and `#'db-sync/attach-ws-handlers!` to verify the adapter call receives the tokenized URL. + +I will add unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that assert non-native password read and write paths call `platform/read-text!` and `platform/write-text!`. + +I will add a unit test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that asserts native write fallback uses `platform/write-text!` when main-thread persistence fails. + +I will add unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that assert encrypted AES key cache I/O flows through `platform/kv-get`, `platform/kv-set!`, and `platform/current`. + +I will run each new test case first and confirm failure before implementation changes using `@test-driven-development`. + +I will then run targeted suites and expect all tests to pass. + +I will run `bb dev:test -v frontend.worker.db-sync-test` and expect `0 failures, 0 errors`. + +I will run `bb dev:test -v frontend.worker.sync.crypt-test` and expect `0 failures, 0 errors` for the selected non-`:fix-me` cases. + +I will run `bb dev:lint-and-test` and expect lint plus unit test completion without new warnings in touched namespaces. + +I will verify clojure-lsp diagnostics no longer report `clojure-lsp/unused-public-var` for the five wrapper vars in `frontend.worker.platform`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +This plan changes shared worker modules that should remain runtime-agnostic. + +This plan does not change browser or node adapter implementations except where signature alignment is required. + +This plan does not redesign db-sync protocol or e2ee crypto flow. + +This plan does not migrate persisted data keys. + +This plan is intentionally limited to the exact five wrapper warnings first. + +This plan includes migrating encrypted AES key cache access in `sync/crypt.cljs` to `platform/kv-get` and `platform/kv-set!` because it is in scope of those warnings. + +This plan does not include unrelated broader idb migration outside these touched shared worker paths. + +## Implementation steps + +1. Add a failing websocket adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for `#'db-sync/connect!`. + +2. Run `bb dev:test -v frontend.worker.db-sync-test` and confirm the new test fails because the code still uses direct `js/WebSocket.`. + +3. Add `frontend.worker.platform` require alias in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs`. + +4. Replace the socket constructor in `connect!` with `(platform/websocket-connect (platform/current) ...)`. + +5. Run `bb dev:test -v frontend.worker.db-sync-test` and confirm the websocket adapter test passes. + +6. Add a failing non-native read and write adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs`. + +7. Add a failing native fallback test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that forces native save failure and expects `platform/write-text!`. + +8. Add a failing kv adapter test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` for AES key cache persistence path. + +9. Run `bb dev:test -v frontend.worker.sync.crypt-test` and confirm new tests fail before implementation. + +10. Add `frontend.worker.platform` require alias in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +11. Replace direct password file I/O calls with `platform/read-text!` and `platform/write-text!` against `(platform/current)`. + +12. Replace direct encrypted AES cache idb calls with `platform/kv-get` and `platform/kv-set!` against `(platform/current)`. + +13. Keep key format exactly as `rtc-encrypted-aes-key###` to preserve existing browser data compatibility. + +14. Remove now-unused direct OPFS and idb-keyval requirements from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` if no longer referenced. + +15. Re-run `bb dev:test -v frontend.worker.sync.crypt-test` and confirm all new tests pass. + +16. Re-run `bb dev:test -v frontend.worker.db-sync-test` and confirm no regressions from sync namespace changes. + +17. Run `bb dev:lint-and-test` and confirm there are no new lint or test regressions. + +18. Verify editor or CI diagnostics no longer show the five `frontend.worker.platform` unused-public-var warnings. + +19. If any wrapper remains unused, decide whether to add a real call site or convert that wrapper to private in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/platform.cljs`. + +20. Document verification evidence and remaining caveats in the PR description. + +## Edge cases and risk controls + +Native worker password flow must keep current behavior that first attempts main-thread persistence and falls back on local storage write. + +Node runtime may not expose browser globals, so all shared websocket construction must pass through the adapter. + +Key-value persistence must keep existing key naming to avoid cache misses for already stored encrypted graph AES keys. + +The adapter functions may return promises, so call sites must preserve existing `p/let` sequencing and error paths. + +If `platform/current` is unset in tests, failures should be explicit and tests should set a minimal platform map. + +## Clarified decisions before coding + +Encrypted AES key cache in `sync/crypt.cljs` will migrate from direct idb store to platform kv now. + +Removal of the five `unused-public-var` warnings is mandatory acceptance criteria. + +Tests remain colocated in `db_sync_test.cljs` and `sync/crypt_test.cljs` instead of adding a dedicated platform test namespace. + +## Verification commands + +```bash +bb dev:test -v frontend.worker.db-sync-test +``` + +Expected output contains the new websocket adapter test name and ends with zero failures. + +```bash +bb dev:test -v frontend.worker.sync.crypt-test +``` + +Expected output contains new adapter usage tests and ends with zero failures for executed tests. + +```bash +bb dev:lint-and-test +``` + +Expected output finishes lint plus test pipeline without new errors in touched files. + +## Skills to apply during implementation + +Use `@test-driven-development` for all behavior changes. + +Use `@clojure-debug` immediately when any new test fails unexpectedly. + +Use `@clojure-paren-repair` if Clojure delimiter errors occur while editing touched namespaces. + +## Testing Details + +Tests focus on externally visible behavior of shared worker modules choosing runtime behavior through adapter calls. + +The websocket test validates the constructor path and tokenized URL input instead of testing internal locals. + +The crypt tests validate fallback and persistence behavior through adapter interaction and returned outcomes. + +The kv cache tests validate read and write behavior by key and value flow rather than implementation-specific helpers. + +## Implementation Details + +- Touch `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to route socket creation through `platform/websocket-connect`. +- Touch `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs` to route text and kv persistence through platform wrappers. +- Touch `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` to add websocket adapter behavior tests. +- Touch `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` to add adapter-backed storage and kv behavior tests. +- Preserve existing e2ee key naming and payload shapes. +- Keep adapter interface unchanged unless tests prove a missing capability. +- Prefer removing obsolete direct dependencies once wrappers are adopted. +- Keep all new logic promise-safe with existing `promesa` flow. +- Validate clojure-lsp warning cleanup for all five wrapper vars. +- Keep PR scoped to abstraction usage cleanup and tests only. + +## Question + +Confirmed scope: limit this effort to the exact five wrapper warnings first. + +Confirmed decision: migrate encrypted AES key cache usage in `sync/crypt.cljs` to platform kv in this pass. + +Confirmed quality gate: the five `unused-public-var` warnings must be cleared. + +Confirmed test placement: keep new tests in existing `db_sync_test.cljs` and `sync/crypt_test.cljs`. + +--- diff --git a/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md new file mode 100644 index 0000000000..e26f5a5c23 --- /dev/null +++ b/docs/agent-guide/040-hide-db-prefix-in-user-visible-graph-names.md @@ -0,0 +1,204 @@ +# Hide Db Prefix In User Visible Graph Names Implementation Plan + +Goal: Ensure user-visible graph names strip exactly one leading `logseq_db_` prefix while preventing new multi-prefix repos from being created. + +Architecture: Keep display normalization as single-pass prefix stripping for web, Electron, and CLI user-facing fields. + +Architecture: Add shared canonicalization at ingestion and graph-discovery boundaries so internal repo identifiers are normalized to exactly one leading prefix before persistence and routing. + +Architecture: Follow `@test-driven-development` with failing tests first across frontend, Electron boundary code, RTC ingestion, and CLI paths. + +Tech Stack: ClojureScript, Rum, Electron IPC, db-worker-node runtime, Logseq CLI formatting pipeline, Babashka tests. + +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +Related: Builds on `docs/agent-guide/038-electron-db-worker-switch-graph.md`. + +## Problem statement + +Graph names shown in the web graph list can expose `logseq_db_` when the stored repo value has multiple leading prefixes. + +Current rendering logic intentionally strips only one leading prefix, so malformed values like `logseq_db_logseq_db_demo` still render as `logseq_db_demo`. + +The product decision is to keep one-layer stripping behavior and fix the upstream causes that create multi-prefix repo identifiers. + +Multi-prefix data can enter through legacy disk graph discovery, Electron graph mapping, RTC remote metadata ingestion, and CLI graph-name conversion paths. + +The required outcome is stable single-prefix internal repo identifiers plus single-pass display normalization, so normal graphs render as `demo` and malformed legacy doubles render as `logseq_db_demo`. + +## Current and target normalization path + +```text +Current path with multi-prefix source data. +input repo: logseq_db_logseq_db_demo + -> display helper strips once + -> output: logseq_db_demo + -> user-visible prefix leak occurs. + +Target path with source canonicalization + single-pass display stripping. +ingress repo: logseq_db_logseq_db_demo + -> canonicalize to one internal prefix: logseq_db_demo + -> display helper strips once + -> output: demo + +Legacy persisted double-prefix fallback path (no migration). +input repo from old state: logseq_db_logseq_db_demo + -> display helper strips once + -> output: logseq_db_demo +``` + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before implementation. + +I will add frontend unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/util/text_test.cljs` to assert single-pass prefix stripping behavior. + +I will add frontend persist tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/db/persist_test.cljs` to assert merged graph sources are canonicalized to one internal prefix before UI state usage. + +I will add RTC tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handler/db_based/rtc_test.cljs` to assert remote graph payload ingestion cannot create local double-prefix repos. + +I will extend CLI formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to assert user-facing fields strip exactly one prefix and never introduce additional prefixes. + +I will extend CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `graph list` and server output paths with unprefixed and prefix-like graph names, and assert prefix-like `--graph` values are treated as graph-name content instead of invalid input. + +I will add legacy graph discovery tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` to verify old directory names do not produce multi-prefix repo ids. + +I will run focused tests after RED and GREEN phases, then run `bb dev:lint-and-test` before completion. + +I will review changes against `@prompts/review.md` before merge. + +NOTE: I will write all tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests for one-layer display stripping and multi-prefix prevention. + +1. Add failing unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/util/text_test.cljs` for `logseq_db_demo -> demo`, `logseq_db_logseq_db_demo -> logseq_db_demo`, and middle-substring preservation. +2. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/db/persist_test.cljs` to verify worker and Electron graph sources are canonicalized to one internal prefix. +3. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/handler/db_based/rtc_test.cljs` for remote graph mapping and download paths with prefixed and double-prefixed payload names. +4. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` with failing cases where `:repo` or `:graph` includes one or two prefixes and output uses one-layer stripping only. +5. Extend `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with failing cases for graph list and server output using unprefixed and prefix-like `--graph` values, and assert prefix-like values are not rejected by argument validation. +6. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/common/graph_test.cljs` for legacy directory names that already contain `logseq_db_`. +7. Run focused tests and confirm failures reflect behavior gaps rather than setup errors. + +### Phase 2: Implement shared helpers for display normalization and repo canonicalization. + +8. Add shared helpers in `/Users/rcmerci/gh-repos/logseq/deps/common/src/logseq/common/config.cljs` for single-pass display stripping and exact-one-prefix canonicalization. +9. Keep display helper semantics strict to remove only one leading prefix and preserve all middle substrings. +10. Keep canonicalization helper semantics strict to collapse any number of leading prefixes to exactly one. +11. Use function names that clearly separate display behavior from internal repo id normalization. +12. Keep helpers pure and dependency-light so they can be reused in frontend, Electron, and CLI namespaces. + +### Phase 3: Apply web app fixes with one-layer display semantics. + +13. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/util/text.cljs` so `get-graph-name-from-path` calls the shared single-pass display helper. +14. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` so `db-graph-name` remains one-layer stripping and does not over-strip. +15. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/db/conn.cljs` so `get-short-repo-name` uses shared one-layer display normalization. +16. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/components/repo.cljs` rendering paths only if needed to ensure all user-visible labels share one-layer stripping behavior. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/db/persist.cljs` so merged graph sources are canonicalized to one internal prefix before entering repo state. +18. Keep `data-testid` and internal routing keys unchanged unless canonicalization is required to prevent multi-prefix key creation. + +### Phase 4: Fix Electron and graph discovery paths that can create multi-prefix repos. + +19. Update `/Users/rcmerci/gh-repos/logseq/deps/cli/src/logseq/cli/common/graph.cljs` so `get-db-based-graphs` canonicalizes discovered graph repo names to one prefix. +20. Update `/Users/rcmerci/gh-repos/logseq/src/electron/electron/handler.cljs` and `/Users/rcmerci/gh-repos/logseq/src/electron/electron/utils.cljs` to canonicalize repo identifiers at IPC mapping boundaries. +21. Verify Electron IPC `getGraphs` keeps stable internal repo identifiers and no path emits new double-prefixed repos. + +### Phase 5: Fix RTC ingestion paths that can reintroduce multi-prefix repos. + +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/db_based/rtc.cljs` so remote graph payload mapping canonicalizes incoming graph names before repo/url construction. +23. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/rtc/full_upload_download_graph.cljs` so download naming canonicalizes to one internal prefix before local repo creation. +24. Ensure existing graphs with prefixed remote names remain accessible after normalization. +25. Ensure new uploads and downloads cannot create `logseq_db_logseq_db_*` local repo ids. + +### Phase 6: Apply CLI fixes for one-layer display and one-prefix internal ids. + +26. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` so `repo->graph` strips exactly one leading prefix for user-visible output. +27. Verify `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` uses one-layer display normalization consistently for human, JSON, and EDN user-facing graph fields. +28. Update `/Users/rcmerci/gh-repos/logseq/deps/cli/src/logseq/cli/commands/graph.cljs` to canonicalize internal repo identifiers before list rendering and server-target resolution. +29. Ensure CLI parsing and command execution treat `--graph` as a graph-name string, so leading `logseq_db_` is interpreted as part of graph name when present and is not rejected by input validation. + +### Phase 7: Verification and release gate. + +30. Run `bb dev:test -v 'frontend.util.text-test'` and confirm one-layer strip behavior and canonicalization tests pass. +31. Run `bb dev:test -v 'frontend.db.persist-test'` and confirm merged-source canonicalization behavior is stable. +32. Run `bb dev:test -v 'frontend.handler.db-based.rtc-test'` and confirm remote ingestion cannot produce multi-prefix repos. +33. Run `bb dev:test -v 'logseq.cli.format-test'` and confirm CLI display fields apply one-layer strip behavior. +34. Run `bb dev:test -v 'logseq.cli.commands-test'` and confirm graph command behavior remains stable with unprefixed and prefix-like `--graph` input. +35. Run `bb dev:test -v 'logseq.cli.common.graph-test'` and confirm legacy graph discovery does not emit double-prefixed repo names. +36. Run `bb dev:lint-and-test` and confirm `0 failures, 0 errors`. +37. Perform a manual graph-list smoke check on web and Electron to confirm normal graphs display without prefix and legacy doubles display with one remaining prefix. +38. Perform a manual CLI smoke check with `logseq graph list`, `logseq server status --graph demo`, and `logseq server status --graph logseq_db_demo` to confirm both inputs are accepted as graph names. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Internal repo is `logseq_db_demo`. | User-visible name is `demo`. | +| Internal repo is `logseq_db_logseq_db_demo`. | User-visible name is `logseq_db_demo`. | +| Graph name contains middle substring like `my_logseq_db_notes`. | Middle substring is preserved and only one leading prefix is removed when present. | +| Legacy disk directory is named `logseq_db_demo`. | Discovery canonicalization keeps exactly one prefix and does not create `logseq_db_logseq_db_demo`. | +| Legacy disk directory is named `logseq_db_logseq_db_demo`. | Discovery canonicalization collapses to internal repo `logseq_db_demo`. | +| Remote graph payload returns `graph-name` as `logseq_db_demo`. | RTC mapping keeps internal repo `logseq_db_demo` and user-visible name `demo`. | +| Remote graph payload returns `graph-name` as `logseq_db_logseq_db_demo`. | RTC mapping canonicalizes to internal repo `logseq_db_demo`, and user-visible name is `demo` after one-layer display strip. | +| CLI receives `--graph demo`. | Command works and output graph name is `demo`. | +| CLI receives `--graph logseq_db_demo`. | Command treats `logseq_db_` as part of graph name and does not fail argument validation. | +| CLI receives `--graph logseq_db_logseq_db_demo`. | Command treats the full value as graph name content and does not fail argument validation. | +| Non-user-visible fields like `data-testid` include repo id. | Existing selectors remain unchanged unless canonicalization is required to prevent duplicate graph entries. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'frontend.util.text-test' +bb dev:test -v 'frontend.db.persist-test' +bb dev:test -v 'frontend.handler.db-based.rtc-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.commands-test' +bb dev:test -v 'logseq.cli.common.graph-test' +bb dev:lint-and-test +``` + +Each command should finish with `0 failures, 0 errors`. + +Web and Electron manual checks should show no new multi-prefix repo entries. + +Web and Electron display should strip one prefix only at render time. + +CLI human output should match one-layer strip semantics, and CLI `--graph` should treat prefix-like values as normal graph names. + +## Testing Details + +The tests verify user-visible behavior and repo-id canonicalization at ingress and discovery boundaries. + +Frontend tests assert display output and merged-state behavior rather than helper internals alone. + +RTC and Electron tests assert that incoming prefixed names cannot generate additional prefixes in local repo ids. + +CLI tests assert command behavior and output formatting remain stable for unprefixed and prefix-like graph-name input. + +## Implementation Details + +- Keep display normalization as single-pass prefix stripping. +- Add one shared helper for exact-one-prefix repo canonicalization. +- Canonicalize ingress and discovery data before it reaches persistent repo state. +- Reuse shared helpers across frontend, Electron, and CLI. +- Preserve `data-testid` compatibility unless canonicalization makes key updates unavoidable. +- Avoid one-time metadata migration for existing persisted graphs. +- Treat CLI `--graph` input as raw graph name where `logseq_db_` may be part of the name. +- Keep internal thread-api contracts based on prefixed repo ids. +- Follow `@test-driven-development` for RED, GREEN, and REFACTOR order. +- Validate final patch with `@prompts/review.md` checklist. + +## Question + +Decision: Display normalization removes only one leading prefix, so `logseq_db_logseq_db_xxxx` displays as `logseq_db_xxxx` in app surfaces that read legacy uncanonicalized values. + +Decision: The implementation focus is to identify and fix all paths that can create multi-prefix repo identifiers, so newly produced data stays canonical. + +Decision: This change does not include a one-time metadata migration for existing persisted legacy values. + +Decision: CLI `--graph` option treats leading `logseq_db_` as graph-name content, not as forbidden prefix. + +Decision: Treat `data-testid` stability as a strict compatibility requirement for `clj-e2e`. + +--- diff --git a/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md b/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md new file mode 100644 index 0000000000..c3b0e28b86 --- /dev/null +++ b/docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md @@ -0,0 +1,171 @@ +# Logseq CLI Add Command Entity Id Output Implementation Plan + +Goal: Make `add page` and `add block` return newly created entity `db/id` in `--output human`, `--output json`, and `--output edn`. + +Architecture: Keep existing add command write paths and add a post-write identifier resolution step in the CLI command layer. +Architecture: Return a unified structured payload for both add commands where `:result` is a vector of created entity `db/id`. +Architecture: Update human formatter for add commands to include created entity ids while preserving existing success semantics. + +Tech Stack: ClojureScript, Logseq CLI command layer, db-worker-node thread API, Logseq CLI formatter. + +Related: Builds on docs/agent-guide/027-logseq-cli-update-command.md and docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md. + +Document naming follows @planning-documents using the next available sequence number `041`. + +## Problem statement + +Current behavior for `add block --output json` is `{"status":"ok","data":{"result":null}}` because `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` sets `:result nil` in `execute-add-block`. + +Current add command outputs are not consistent for machine and human flows when users need the new entity id immediately. + +Users now require `db/id` in all output formats for both `add page` and `add block`. + +Without this output, automation must run extra commands to locate newly created entities before `update` or `remove`. + +## Testing Plan + +I will follow @test-driven-development and complete all RED tests before implementation. + +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `add page --output json` and `add block --output json` asserting returned `db/id`. + +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `add page --output edn` and `add block --output edn` asserting returned `:db/id`. + +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting human output lines for add page and add block include new ids. + +I will add one integration chain test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that uses returned add ids directly in `update` and `remove` commands. + +I will add a focused unit test namespace at `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/add_test.cljs` for helper logic that builds the created-entity result payload. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Architecture sketch + +``` +add page/add block + -> /src/main/logseq/cli/command/add.cljs execute-add-* + -> thread-api write call + -> resolve created entity ids in CLI command layer + -> result payload {:result [id1 id2 ...]} + -> /src/main/logseq/cli/format.cljs human renderer includes id information +``` + +## Output contract + +JSON output for add page returns created page id vector in `data.result`. + +JSON output for add block returns created block ids in `data.result`. + +EDN output mirrors the same data shape using keyword keys. + +Human output includes created ids for both commands. + +Example human output for add page is: + +```text +Added page: +[123] +``` + +Example human output for add block is: + +```text +Added blocks: +[101 102] +``` + +## Plan + +1. Read `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and confirm current add page and add block result payload shapes. +2. Read `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` and confirm current human formatter paths for add commands. +3. Write RED integration test A in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add page JSON result containing created page `db/id`. +4. Write RED integration test B in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add block JSON result containing created block `db/id` list. +5. Write RED integration test C in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for add page and add block EDN output containing `:db/id`. +6. Write RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting human add output includes ids. +7. Write RED unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/add_test.cljs` for id vector normalization and deterministic ordering. +8. Run focused tests and verify failures are caused by missing behavior and not incorrect test setup. +9. Implement add block result enrichment in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` so it returns all created block ids including nested children. +10. Implement add page result enrichment in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` so it returns created page id as a one-element vector. +11. Keep result field naming stable with a shape `:data {:result [101 102]}`. +12. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` add page and add block human renderers to include id data from command result. +13. Ensure JSON and EDN formatter paths continue to serialize the enriched result without special-case regressions. +14. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` with add page and add block structured output examples that include `db/id`. +15. Run focused tests again and verify GREEN behavior for all newly added tests. +16. Refactor helper naming and duplicate mapping code in add command execution if needed without behavior change. +17. Re-run focused tests after refactor to verify tests stay green. +18. Run `bb dev:lint-and-test` for regression verification. + +## Edge cases + +Add block from `--content` must still return generated block id when UUID was not supplied in input. + +Add block from `--blocks` with multiple nested blocks must return all created ids and use deterministic ordering in returned payload. + +Add page should return the created page id even when tags and properties are added in the same command. + +Add block human output should remain readable when returning many ids including nested children. + +When id resolution fails unexpectedly after a successful write, command should return an error rather than a misleading success payload without ids. + +Error output format and exit codes must remain unchanged for invalid input and missing target cases. + +## Testing commands and expected output + +Run focused RED tests. + +```bash +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-json-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-block-json-output-returns-ids' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-block-edn-output-returns-id' +bb dev:test -v 'logseq.cli.format-test/test-human-output-add-remove' +``` + +Expected RED output includes assertion failures indicating missing id fields in add outputs. + +Run focused GREEN tests after implementation. + +```bash +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-json-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-block-json-output-returns-ids' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-page-block-edn-output-returns-id' +bb dev:test -v 'logseq.cli.integration-test/test-cli-add-identifiers-chain-update-remove' +bb dev:test -v 'logseq.cli.command.add-test' +bb dev:test -v 'logseq.cli.format-test/test-human-output-add-remove' +``` + +Expected GREEN output includes zero failures and zero errors for the focused namespaces. + +Run full verification. + +```bash +bb dev:lint-and-test +``` + +Expected output includes successful lint and tests with exit code `0`. + +## Testing Details + +Integration tests will assert actual CLI command output payloads and real persisted graph behavior for add page and add block. + +Formatter tests will assert exact human output strings for add page and add block including id fragments. + +Unit tests will validate helper behavior for id list assembly and ordering. + +## Implementation Details + +- Enrich `execute-add-block` result so `:result` is a vector of all created block ids including nested children. +- Enrich `execute-add-page` result so `:result` is a one-element vector containing created page id. +- Keep result payload stable across JSON and EDN output paths by using Clojure maps with keyword keys. +- Update human formatters for add page and add block to include id information: + - add page rendered as `Added page:` on one line and `[123]` on the next line. + - add block rendered as `Added blocks:` on one line and `[101 102]` on the next line. +- Preserve existing command success and error semantics besides the new id output fields. +- Add integration coverage for add outputs plus id-based `update` and `remove` chaining. +- Update CLI docs to show the new add page and add block result examples with ids. + +## Decisions + +Human output for add block must display all created ids including nested children. + +For add block and add page, `:result` must be an id vector such as `[101 102]`. + +--- diff --git a/docs/agent-guide/042-logseq-cli-add-tag-command.md b/docs/agent-guide/042-logseq-cli-add-tag-command.md new file mode 100644 index 0000000000..f5d3656573 --- /dev/null +++ b/docs/agent-guide/042-logseq-cli-add-tag-command.md @@ -0,0 +1,138 @@ +# Logseq CLI Add Tag Subcommand Implementation Plan + +Goal: Add `logseq add tag` so CLI users can create a tag entity before using that tag in `add block`, `add page`, and `update`. + +Architecture: Reuse the existing CLI to db-worker-node write path by sending `:create-page` with `{:class? true}` through `:thread-api/apply-outliner-ops`. +Architecture: Keep db-worker-node protocol and HTTP endpoints unchanged because the feature composes existing worker operations. +Architecture: Validate the result after write by pulling the page entity and asserting the page has `:logseq.class/Tag` semantics. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, db-worker-node, outliner ops. + +Related: Builds on docs/agent-guide/018-logseq-cli-add-tags-builtin-properties.md and docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md. + +## Problem statement + +Current CLI behavior can attach only existing tags because `resolve-tag-entity` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` fails with `:tag-not-found` when a tag is missing. + +Current CLI behavior has no `add tag` command entry in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, so users cannot create custom tags from CLI. + +Current db-worker-node flow already supports page and class creation through `:thread-api/apply-outliner-ops` and `:create-page`, so the missing capability is command orchestration in CLI instead of worker transport. + +`list tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` returns entities tagged with `:logseq.class/Tag`, so `add tag` must create that exact class shape instead of a plain page. + +## Testing Plan + +I will follow `@test-driven-development` and write parsing tests before changing command implementation code. + +I will add failing action-building tests that verify required options, normalized action payload, and explicit errors for invalid input. + +I will add failing execution tests that stub transport calls and verify emitted outliner ops include `:create-page` with `{:class? true}`. + +I will add failing format tests for human output so `:add-tag` has a stable success message contract. + +I will add one integration test that creates a new tag and then uses that tag in `add block` to prove end to end behavior through db-worker-node. + +I will add one integration test that confirms failure when the same title already exists as a non-tag page, so the command does not report false success. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and non-goals + +This plan adds only `add tag` under the existing `add` command group. + +This plan does not add new db-worker-node endpoints or thread-api methods. + +This plan does not add editing features such as setting class extends, class properties, or tag description during creation. + +This plan does not change existing `add block`, `add page`, or `update` option syntax. + +`add tag` accepts `--name` only, and does not support `--tag` alias. + +## Integration overview + +``` +logseq add tag --name "Quote" + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> db-worker-node /v1/invoke + -> :thread-api/apply-outliner-ops + -> :create-page [title {:class? true}] + -> Datascript entity tagged as :logseq.class/Tag +``` + +## Detailed implementation plan + +1. Add a failing parser help test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that expects `add` group help to include `add tag`. +2. Add a failing parser test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `["add" "tag" "--name" "Quote"]` to produce `:add-tag`. +3. Add a failing parser validation test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that missing `--name` returns `:missing-tag-name`. +4. Add a failing build-action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that ensures `:add-tag` action contains `:type`, `:repo`, `:graph`, and normalized `:name`. +5. Add a failing execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that stubs transport and verifies `:create-page` options include `:class? true`. +6. Add a failing execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that simulates existing non-tag page conflict and expects a deterministic CLI error code. +7. Add a failing format test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for `:add-tag` human output. +8. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that runs `add tag`, validates `list tag` contains the new tag, and confirms `add block --tags` with that tag succeeds. +9. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for duplicate title where a normal page exists and command must fail with clear error. +10. Run focused tests and confirm all new tests fail for behavior reasons, and use `@clojure-debug` only if failures are caused by test setup mistakes. +11. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` with a new `add-tag` command spec, entry, action builder, and executor. +12. In `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, implement execution via `:thread-api/apply-outliner-ops` and `[:create-page [name {:class? true}]]`. +13. In `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, add a post-create pull check that verifies the resulting entity is class-tagged and raise an explicit error if not. +14. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` parse validation with `:missing-tag-name` handling for `:add-tag`. +15. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` action dispatch and execute dispatch for `:add-tag`. +16. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` context propagation to include the new tag field used by formatter output. +17. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with `format-add-tag` and command routing for `:add-tag`. +18. Run focused unit and integration tests and confirm they pass without changing unrelated command behavior. +19. Run `bb dev:lint-and-test` and confirm the repository remains green after the feature. +20. Refactor only local helper naming and shared logic inside `add.cljs` while preserving behavior and keeping tests green. + +## Edge cases to cover + +Tag names with leading `#` should be normalized consistently, or rejected consistently, with one documented behavior. + +Tag names containing namespace separators like `A/B` should produce deterministic behavior aligned with existing page creation rules. + +Duplicate tag creation should be idempotent when the existing entity is already a tag class. + +If a page with the same name exists but is not a tag class, `add tag` should fail with a dedicated error instead of silently succeeding. + +Built-in tag names should remain valid and should return existing ids without creating duplicate entities. + +The command should reject blank names after trim. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` | New `add tag` appears in add group help assertions. | +| `bb dev:test -v logseq.cli.commands-test/test-verb-subcommand-parse-add` | New parse and validation tests for `add tag` pass. | +| `bb dev:test -v logseq.cli.commands-test/test-build-action-inspect-edit` | Build action includes `:add-tag` cases and passes. | +| `bb dev:test -v logseq.cli.commands-test/test-execute-add-tag-builds-create-page-op` | Outliner op assertions pass with `{:class? true}`. | +| `bb dev:test -v logseq.cli.format-test/test-human-output-add-remove` | Human output for `:add-tag` matches expected string. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-add-tag-create-and-use` | End to end creation and usage of a new tag passes. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-add-tag-rejects-existing-non-tag-page` | Conflict behavior test passes with explicit error. | +| `bb dev:lint-and-test` | Full lint and unit suite pass. | + +## Testing Details + +The tests validate user-visible behavior at parser, action, executor, formatter, and integration boundaries. + +The tests assert command success and failure contracts, and they verify persisted graph behavior with CLI reads such as `list tag` and `show`. + +The tests avoid asserting implementation details that are not externally observable. + +## Implementation Details + +- Add `add tag` spec and entry in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`. +- Add `build-add-tag-action` and `execute-add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`. +- Use existing server bootstrap and transport invoke path from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs`. +- Create tag through `:create-page` with `{:class? true}` and verify resulting entity semantics. +- Add parse validation and missing error mapping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Add build and execute routing for `:add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Add human formatter branch for `:add-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +- Add parser and executor unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +- Add integration coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. + +## Question + +No open questions. + +--- diff --git a/docs/agent-guide/043-logseq-cli-tag-property-management.md b/docs/agent-guide/043-logseq-cli-tag-property-management.md new file mode 100644 index 0000000000..906a892bcb --- /dev/null +++ b/docs/agent-guide/043-logseq-cli-tag-property-management.md @@ -0,0 +1,182 @@ +# Logseq CLI Tag and Property Management Implementation Plan + +Goal: Add first class CLI support for `upsert tag`, `upsert property`, `remove tag`, and `remove property`, and restructure existing remove behavior into `remove block` and `remove page`. + +Architecture: Replace current `add tag` command entry with `upsert tag` so tag creation and idempotent update semantics are unified under one verb. +Architecture: Expand `remove` into typed subcommands (`block`, `page`, `tag`, `property`) to make deletion intent explicit and prevent mixed selector ambiguity. +Architecture: Reuse `:thread-api/apply-outliner-ops` in db-worker-node so no new HTTP route or transport protocol is introduced. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, db-worker-node, outliner ops. + +Related: Builds on docs/agent-guide/042-logseq-cli-add-tag-command.md and docs/agent-guide/029-logseq-cli-show-properties.md. + +## Problem statement + +Current CLI supports tag creation through `add tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs`, but does not expose an `upsert` verb for tags and properties. + +Current `update` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` updates tag and property values on blocks, but does not upsert or remove tag/property entities. + +Current `remove` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` mixes block and page deletion under one command, and has no explicit tag/property deletion path. + +db-worker-node already supports required mutation primitives through `:thread-api/apply-outliner-ops` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and outliner ops in `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/op.cljs`. + +The implementation gap is CLI command surface design, parser wiring, validation, and output contract coverage. + +## Testing Plan + +I will follow `@test-driven-development` and write failing parser, action, executor, formatter, and integration tests before adding behavior. + +I will add parser tests for the new `upsert` verb and the new typed `remove` subcommands. + +I will add action builder tests that verify repo propagation, normalized names, schema coercion, and typed action payloads for each new command. + +I will add executor tests that stub transport and assert exact outliner ops emitted for each command path. + +I will add formatter tests for human and json output so command responses are stable for scripts. + +I will add integration tests against db-worker-node that verify graph state changes through `list`, `show`, and entity queries. + +I will use `@clojure-debug` only when failures are caused by test setup rather than behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Command contract + +The final command surface is top level verbs without a `manage` group. + +| Command | Required options | Optional options | Behavior | +| --- | --- | --- | --- | +| `logseq upsert tag` | `--name` | none in v1 | Creates tag when missing, returns existing tag when already present, and errors if same title exists as non-tag page. | +| `logseq upsert property` | `--name` | `--type`, `--cardinality`, `--hide`, `--public` | Creates property when missing, updates schema for existing property, and validates type or cardinality compatibility. | +| `logseq remove tag` | one of `--name`, `--id` | none | Deletes a tag entity after validating target type and removability, and fails with a candidate list when `--name` matches multiple tags. | +| `logseq remove property` | one of `--name`, `--id` | none | Deletes a property entity after validating target type and removability, and fails with a candidate list when `--name` matches multiple properties. | +| `logseq remove block` | one of `--id`, `--uuid` | none | Deletes one or more blocks with existing block remove behavior. | +| `logseq remove page` | `--name` | none | Deletes a page with existing page remove behavior. | + +`add tag` will be removed from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and replaced by `upsert tag`. + +Bare `remove` without a subcommand will be rejected with a clear parse error. + +No compatibility aliases or deprecation shims will be kept for removed commands. + +## Integration overview + +```text +logseq upsert property --name "owner" --type node --cardinality many + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs (/v1/invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs (:thread-api/apply-outliner-ops) + -> /Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/op.cljs (:upsert-property) +``` + +```text +logseq remove tag --name "Quote" + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs + -> remove tag resolver validates :logseq.class/Tag + -> :thread-api/apply-outliner-ops with [:delete-page [tag-uuid]] + -> outliner page delete flow applies class cleanup and persistence +``` + +## Detailed implementation plan + +1. Add failing parse help tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that expect top level `upsert` and `remove` subcommands (`block`, `page`, `tag`, `property`). +2. Add failing parse tests for `['upsert' 'tag' '--name' 'Quote']` and `['upsert' 'property' '--name' 'owner' '--type' 'node' '--cardinality' 'many']` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +3. Add failing parse tests for `['remove' 'tag' '--name' 'Quote']` and `['remove' 'property' '--id' '123']` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +4. Add failing parse tests for `['remove' 'block' '--id' '1']` and `['remove' 'page' '--name' 'Home']` to preserve old delete behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +5. Add failing parse validation tests that reject bare `remove` and old `add tag` command usage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +6. Add failing parse validation tests for invalid property type and cardinality values in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +7. Add failing build action tests for `upsert tag` name normalization and `#` prefix stripping in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +8. Add failing build action tests for `upsert property` schema coercion to `:logseq.property/type` and `:db/cardinality` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +9. Add failing executor tests that `upsert tag` emits `[:create-page [name {:class? true}]]` only when the tag is missing in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +10. Add failing executor tests that `upsert tag` is idempotent for existing tag entities and rejects non-tag title conflicts in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +11. Add failing executor tests that `upsert property` emits `[:upsert-property [property-id schema opts]]` and passes `{:property-name name}` when creating a new property in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +12. Add failing executor tests that `remove tag` and `remove property` resolve entities by `--name` or `--id` and emit `[:delete-page [uuid]]` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +13. Add failing executor tests that `remove tag --name` and `remove property --name` fail on multiple matches, return all matched candidates in output, and require rerun with `--id` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +14. Add failing executor tests that built in or hidden tag or property targets are rejected with explicit error codes in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +15. Add failing formatter tests for `upsert tag`, `upsert property`, `remove tag`, and `remove property` outputs in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +16. Add failing formatter tests that `remove block` and `remove page` outputs remain backward compatible in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +17. Add failing integration tests for `upsert tag` create and idempotent behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +18. Add failing integration tests for `upsert property` create and schema update behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +19. Add failing integration tests for `remove tag` and `remove property` ensuring entities disappear from `list tag` and `list property` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +20. Add failing integration tests for `remove tag --name` and `remove property --name` ambiguous matches to assert candidate list output and `--id` guidance in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +21. Add failing integration tests for `remove block` and `remove page` command migration behavior in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +22. Run focused tests and confirm all new tests fail for behavior reasons before implementation. +23. Create `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` with command specs, entries, validation helpers, action builders, and executors for tag and property upsert. +24. Refactor shared tag resolver helpers from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` into `upsert.cljs` or a shared helper namespace to avoid duplication. +25. Remove `add tag` command entry and related build or execute dispatch from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +26. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` to register `remove block`, `remove page`, `remove tag`, and `remove property` entries. +27. Keep existing block and page delete execution logic and map it behind `remove block` and `remove page` subcommands. +28. Implement new remove tag and remove property resolver and execution paths in `remove.cljs` using `:delete-page` outliner op after entity type validation and ambiguity failure behavior for `--name`. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` table, parse validation, action builder, context propagation, and execute dispatch for new upsert and remove contracts. +30. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to format results for `upsert tag`, `upsert property`, `remove tag`, and `remove property`, including ambiguous candidate lists. +31. Run focused unit and integration tests, then run `bb dev:lint-and-test`, and keep only behavior preserving refactors. +32. Update CLI help text snapshots and any docs references to remove `add tag` and bare `remove` usage. + +## Edge cases to cover + +Tag names with leading `#` should normalize consistently in `upsert tag`. + +Tag and property names that collide by title or case must fail resolution for `--name`, list all candidate matches, and require explicit `--id`. + +Remove selectors must reject ambiguous name matches with a clear error, include all candidate ids and names in output, and require `--id`. + +Built in tags and built in properties must not be removable. + +Property type and cardinality updates must reject invalid transitions already enforced by outliner validation. + +`remove block` multi id behavior must stay unchanged from current implementation. + +`remove page` must preserve current not found and built in page deletion behavior. + +Commands must reject blank names after trim. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` | Help output includes `upsert` and typed `remove` subcommands. | +| `bb dev:test -v logseq.cli.commands-test/test-verb-subcommand-parse-upsert-remove` | Parse and validation tests for new command contracts pass. | +| `bb dev:test -v logseq.cli.commands-test/test-build-action-upsert-remove` | Action payload tests pass for all new paths. | +| `bb dev:test -v logseq.cli.commands-test/test-execute-upsert-remove-tag-property` | Outliner op emission tests pass for upsert and entity removal flows. | +| `bb dev:test -v logseq.cli.format-test/test-human-output-upsert-remove` | Human output formatting for new commands is stable. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-and-remove-tag-property` | End to end behavior through db-worker-node passes for all four new commands. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-remove-block-page-subcommands` | Block and page deletion still work through new remove subcommands. | +| `bb dev:lint-and-test` | Full lint and unit suite pass. | + +## Migration and compatibility + +No db-worker-node route migration is required because this feature reuses `/v1/invoke` and existing thread-api methods. + +No schema migration is required because tag and property entities are created and deleted through existing outliner operations. + +This is a CLI breaking change because `add tag` is removed and bare `remove` is replaced by typed `remove block` and `remove page` commands. + +This breaking change will be applied directly with no compatibility alias period. + +## Testing Details + +Tests validate parser, action, execution, formatter, and integration behavior for `upsert tag`, `upsert property`, `remove tag`, and `remove property`. + +Tests also validate migration behavior for `remove block` and `remove page` so previous block and page deletion semantics are preserved. + +Tests focus on command contracts and graph outcomes instead of helper internals. + +## Implementation Details + +- Add new upsert command module at `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Register upsert entries and dispatch hooks in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Remove `add tag` command entry and related dispatch from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +- Refactor remove command entries in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/remove.cljs` to `remove block`, `remove page`, `remove tag`, and `remove property`. +- Reuse `:thread-api/apply-outliner-ops` with `:create-page`, `:upsert-property`, and `:delete-page` for all entity mutations. +- Add formatter branches in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` for new commands. +- Add parser and executor unit tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`. +- Add end to end coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +- Use `@clojure-debug` only when failures indicate fixture or async wiring issues. + +## Question + +No open questions. + +--- diff --git a/docs/agent-guide/044-logseq-cli-upsert-block-page.md b/docs/agent-guide/044-logseq-cli-upsert-block-page.md new file mode 100644 index 0000000000..43bf316684 --- /dev/null +++ b/docs/agent-guide/044-logseq-cli-upsert-block-page.md @@ -0,0 +1,191 @@ +# Logseq CLI Upsert Block and Upsert Page Implementation Plan + +Goal: Consolidate block and page write commands by replacing `add block`, `add page`, and `update` with `upsert block` and `upsert page` while preserving current db-worker-node write behavior. + +Architecture: Keep db-worker-node RPC and outliner operation contracts unchanged, and implement command consolidation in CLI parsing, action building, execution, and formatting layers. +Architecture: Reuse existing helper logic from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` first, then fold shared behavior into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +Architecture: Route all mutations through existing `:thread-api/apply-outliner-ops` and `:thread-api/pull` calls so `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` require no new thread APIs. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Logseq CLI transport, db-worker-node, and outliner ops. + +Related: Relates to `docs/agent-guide/027-logseq-cli-update-command.md` and builds on `docs/agent-guide/041-logseq-cli-add-block-json-identifiers.md`. +Document naming follows @planning-documents with sequence `044`. + +## Problem statement + +The current CLI splits block mutations across `add block` and `update`, while page writes are exposed as `add page`. +This creates an inconsistent user model and duplicates validation and formatting paths in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs`. +The current block update property options only support built-in properties, which prevents consistent upsert behavior for custom properties. +The current `add page` flow applies tags after create, but page property behavior for existing pages is not fully upsert-like because `create-page` may no-op when the page exists. +The db-worker-node layer already exposes stable generic APIs, so this feature should be implemented as a CLI surface refactor without protocol changes. + +## Testing Plan + +I will use @test-driven-development for all implementation batches. +I will write all RED tests for parser, action builder, formatter, and integration flows before changing implementation behavior. +I will add parser and builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert block` and `upsert page` command forms and for hard-removal behavior of `add` and `update`. +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `:upsert-block` and `:upsert-page`. +I will add integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that verify block creation, block update, block move, page creation, and page update through only `upsert` commands. +I will add one integration test that verifies `upsert page` updates properties on an existing page, which closes the current `add page` gap. +I will verify RED failures come from missing behavior and not from broken test setup. +I will run focused GREEN tests after minimal implementation, then refactor, then rerun focused tests and full lint-test. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Area | Current implementation | Change target | +| --- | --- | --- | +| Command entries | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` defines `["add" "block"]` and `["add" "page"]`, and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` defines `["update"]`. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` defines `["upsert" "block"]` and `["upsert" "page"]` together with existing upsert subcommands. | +| Validation and dispatch | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` validates and dispatches `:add-block`, `:add-page`, and `:update-block` separately. | Replace with `:upsert-block` and `:upsert-page` parse and dispatch paths. | +| Help and group summaries | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` top-level summary and group handling still expose `add` and `update`. | Expose only `upsert` for these write cases. | +| Formatter | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` has `format-add-block`, `format-add-page`, and `format-update-block`. | Replace with upsert-focused formatter routes while preserving existing output contract style. | +| Worker APIs | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` already provide `:thread-api/apply-outliner-ops`. | No new worker endpoint or transport shape. | +| Page create behavior | `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/page.cljs` `create!` can return existing page with no transaction. | `upsert page` applies properties and tags explicitly on existing page to ensure real upsert behavior. | + +## Interface contract proposal + +`upsert block` supports two modes with deterministic priority. +If `--id` or `--uuid` is provided, `upsert block` always runs update mode. +If neither `--id` nor `--uuid` is provided, `upsert block` runs create mode. +For `upsert block` update mode, add, update, and remove property options must support all existing properties, not only built-in properties. +`upsert page` requires `--page` and always resolves an existing or newly created page, then applies add, update, and remove semantics for tags and properties. +For `upsert page`, all add, update, and remove tag or property operations require the target tag and property to already exist, otherwise the command returns an error. + +| Command | Input signal | Behavior | Existing code path to reuse | +| --- | --- | --- | --- | +| `upsert block` create mode | `--id` and `--uuid` are absent and a content source is present. | Insert blocks under target with existing add semantics and support add, update, and remove tag or property options in post-insert ops. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` add helpers plus update option helpers. | +| `upsert block` update mode | `--id` or `--uuid` is present. | Move and or add, update, and remove tags or properties with existing update semantics, and support both built-in and custom properties in property options. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` update helpers. | +| `upsert page` | `--page` is present. | Create page if missing, then apply add, update, and remove tags and properties for both new and existing pages, with strict existing-tag and existing-property validation. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` plus explicit remove op wiring. | + +## Architecture sketch + +```text +CLI args + -> parse-args in commands.cljs + -> build-action returns :upsert-block or :upsert-page + -> execute routes to upsert command executor + -> transport/invoke :thread-api/apply-outliner-ops and :thread-api/pull + -> db-worker-node /v1/invoke passthrough + -> db_core thread-api handlers and outliner-op/apply-ops! +``` + +## Plan + +1. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert block` create mode with `--content` and for update mode with `--id` plus `--update-tags`. +2. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that confirm `--id` or `--uuid` forces update mode even when create inputs are also present. +3. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert page --page ` with tags and properties. +4. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to define expected unknown-command behavior for legacy `add block`, `add page`, and `update`. +5. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-block` action shape in create mode. +6. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-block` action shape in update mode, including custom property option inputs. +7. Add RED builder tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `:upsert-page` action shape including resolved options. +8. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-block` delegates to insert-style ops for create mode. +9. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-block` delegates to move and property ops for update mode across built-in and custom properties. +10. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-page` applies add, update, and remove property or tag ops on an already existing page. +11. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` to verify `:upsert-page` returns errors when any referenced tag or property does not exist. +12. Add RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output text of `:upsert-block` and `:upsert-page`. +13. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert block` create mode id outputs. +14. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert block` update mode move behavior and custom property updates. +15. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page` create and update-existing behaviors. +16. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page` erroring when referenced tags or properties do not exist. +17. Run focused RED commands and confirm failures are expectation failures rather than transport or fixture errors. +18. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` spec to include block and page options while keeping existing tag and property specs. +19. Implement `build-block-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to classify create mode versus update mode with `--id` or `--uuid` priority, and normalize property options for all property identifiers. +20. Implement `build-page-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` for `upsert page`. +21. Extract or reuse add helper functions from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` for reading blocks, parsing tags, parsing properties, and resolving ids. +22. Extract or reuse update helper functions from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/update.cljs` for source and target resolution and move option mapping. +23. Implement `execute-upsert-block` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` that branches by mode and calls reused logic without behavior drift, including custom property update support. +24. Implement `execute-upsert-page` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so add, update, and remove property or tag ops are applied after resolving the page entity in both create and existing-page paths. +25. Enforce strict `upsert page` validation and execution behavior where missing tags or properties fail fast instead of creating missing entities. +26. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` table entries and finalize-command validation for `:upsert-block` and `:upsert-page`. +27. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` build and execute case dispatch to remove `:add-block`, `:add-page`, and `:update-block` routing. +28. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` top-level summary and group-help triggers to reflect the new command family. +29. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with `format-upsert-block` and `format-upsert-page` and command dispatch keys. +30. Keep `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` unchanged unless test evidence proves a missing worker behavior. +31. Update CLI documentation in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to replace add and update examples with upsert equivalents. +32. Run focused GREEN tests and confirm parser, builder, formatter, and integration cases pass. +33. Refactor duplicated helper wiring between add, update, and upsert modules while preserving test behavior. +34. Rerun focused test set after refactor to confirm no regressions. +35. Run `bb dev:lint-and-test` as final regression verification. + +## Edge cases + +`upsert block` with `--id` or `--uuid` plus create inputs must deterministically run update mode because source selectors have priority. +`upsert block` update mode must keep current `--pos` validation where `sibling` is invalid for page targets and `--pos` requires a target. +`upsert block` update mode property options must support all existing properties, including non built-in properties. +`upsert block` create mode must preserve current default target fallback to today journal when no target selector is provided. +`upsert block` create mode with `--blocks` or `--blocks-file` must keep the existing restriction that tags and properties cannot be combined if that restriction is still required by current insert behavior. +`upsert block` and `upsert page` must support remove options for tags and properties in addition to add and update options. +`upsert page` must return stable `data.result` id vectors for JSON, EDN, and human output just like current add command id outputs. +`upsert page` on an existing page must apply property updates and removals explicitly so upsert semantics are true for both create and existing states. +`upsert page` must error when any tag or property referenced by add, update, or remove options does not already exist. +Legacy command behavior must be hard removal with standard `unknown-command` errors for `add block`, `add page`, and `update`. +Help output must not regress command grouping or ANSI formatting alignment in `commands_test`. + +## Verification commands and expected output + +Run parser and builder tests during RED. + +```bash +bb dev:test -v logseq.cli.commands-test +``` + +Expected RED behavior is failing assertions for missing `upsert block` and `upsert page` paths before implementation. + +Run formatter tests during RED. + +```bash +bb dev:test -v logseq.cli.format-test +``` + +Expected RED behavior is failing assertions for unknown command formatter branches for upsert block and upsert page. + +Run focused integration tests after implementation. + +```bash +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-block-create-json-output-returns-ids +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-block-update-move +bb dev:test -v logseq.cli.integration-test/test-cli-upsert-page-create-and-update-existing +``` + +Expected GREEN behavior is zero failures and zero errors for these tests. + +Run full verification. + +```bash +bb dev:lint-and-test +``` + +Expected GREEN behavior is full suite pass with exit code `0`. + +## Testing Details + +Behavior tests will assert command-level outcomes through real CLI execution and Datascript queries instead of only checking mock invocation counts. +Unit-level command tests will assert parse, validation, and action-shape behavior at module boundaries. +Integration tests will verify persisted graph state changes for both create and update paths of upsert block and upsert page. + +## Implementation Details + +- Keep db-worker-node API contracts unchanged and implement all command-surface changes in CLI modules only. +- Add `upsert block` and `upsert page` entries to `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Reuse add and update helper functions to minimize behavior drift and reduce migration risk. +- Ensure `upsert block` and `upsert page` support add, update, and remove options for tags and properties. +- Ensure `upsert block` property update options accept all existing properties, including custom properties, not only built-in properties. +- Ensure `upsert page` applies tags and properties after resolving page entity so existing pages are updated too. +- Ensure `upsert page` fails fast when referenced tags or properties do not exist, and never auto-creates them through upsert-page mutation options. +- Remove old command routes in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` for `add block`, `add page`, and `update`, returning standard `unknown-command`. +- Update formatter dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` for new command ids. +- Update command summaries in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` to keep help output accurate. +- Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` examples and command reference text. +- Keep implementation batches aligned to @test-driven-development RED, GREEN, and refactor phases. + +## Question + +No open questions. +Decided: remove `add block`, `add page`, and `update` immediately. +Decided: in `upsert block`, `--id` or `--uuid` means update mode and absence of both means create mode. +Decided: support add, update, and remove semantics for tags and properties. +Decided: in `upsert block` update mode, property mutation options support all existing properties, including custom properties. +Decided: for `upsert page`, add, update, and remove tag or property options require existing tags and properties, otherwise return error. + +--- diff --git a/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md b/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md new file mode 100644 index 0000000000..ce6d2caa8d --- /dev/null +++ b/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md @@ -0,0 +1,180 @@ +# Logseq CLI Property Type and Upsert Option Unification Implementation Plan + +Goal: Add a property type column to `list property`, add `--id` update-mode semantics to `upsert block/page/tag/property`, and remove duplicated `--tags` or `--properties` options from `upsert block/page` in favor of `--update-tags` or `--update-properties`. + +Architecture: Keep the existing `logseq-cli -> transport/invoke -> db-worker-node :thread-api/*` contract unchanged and implement behavior changes in CLI parsing, action building, execution, and formatting. +Architecture: Extend property list payload shaping so non-expanded property items include `:logseq.property/type`, then render a dedicated property table in human output with a `TYPE` column. +Architecture: Treat `--id` as an explicit update signal for all upsert entity commands, and keep create paths only when `--id` is absent. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript, logseq-cli command modules, db-worker-node thread APIs. + +Related: Builds on `docs/agent-guide/044-logseq-cli-upsert-block-page.md` and relates to `docs/agent-guide/043-logseq-cli-tag-property-management.md`. + +## Problem statement + +Current `list property` human output in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` uses the same formatter as `list tag`, so no property-type column is rendered. + +Current non-expanded property list items from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` are built by `minimal-list-item`, which does not include `:logseq.property/type`. + +Current `upsert block` already supports `--id` and treats it as update mode via `update-mode?` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, but `upsert page`, `upsert tag`, and `upsert property` do not accept `--id`. + +Current `upsert block/page` specs include both `--tags` or `--properties` and `--update-tags` or `--update-properties` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, which duplicates semantics and increases parser and action complexity. + +Current parser validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` requires `--page` for `upsert page` and requires `--name` for `upsert property`, so there is no update-by-id mode for those commands. + +## Testing Plan + +I will use `@test-driven-development` for all implementation batches. + +I will add parser and action RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for new `--id` contracts on `upsert page`, `upsert tag`, and `upsert property`. + +I will add formatter RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for the `list property` `TYPE` column and its value normalization. + +I will add contract RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/mcp_tools_contract_test.cljs` to ensure non-expanded property list items carry `:logseq.property/type`. + +I will add integration RED tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for update-by-id flows and for rejecting removed `--tags` or `--properties` flags on `upsert block/page`. + +I will use `@clojure-debug` only when failures indicate fixture or async harness issues rather than missing behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Requirement | Current behavior | Gap | +| --- | --- | --- | +| `list property` shows type column. | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` renders tag and property with the same columns (`ID`, `TITLE`, optional `IDENT`, timestamps). | No `TYPE` column in human output, and non-expanded payload currently omits type. | +| `upsert block/page/tag/property` supports `--id` and `--id` forces update mode. | `upsert block` supports this already, but `upsert page/tag/property` specs do not expose `--id` and still depend on page/name creation-first contracts. | Missing update-by-id mode for three upsert commands. | +| `upsert block/page` removes `--tags` or `--properties` and uses `--update-tags` or `--update-properties` only. | Both old and new options are accepted and merged in action building and execution. | Duplicate option surface and duplicate parsing paths remain. | + +## Target contract + +`upsert block` keeps current `--id` update-mode behavior and remove legacy create-only `--tags` or `--properties` options. + +`upsert page` accepts `--id` as update mode, and accepts `--page` only for create mode. + +`upsert tag` accepts `--id` as update mode, and keeps `--name` for create mode. + +`upsert property` accepts `--id` as update mode, and keeps `--name` for create mode. + +`upsert tag --id ` with no additional mutation options is a successful no-op after id lookup and tag-class validation. + +`upsert page --id --page ` is invalid and must fail as conflicting selectors. + +When `--id` is provided for any upsert command, create-specific resolution paths must be skipped and the command must fail if the target id does not exist or has the wrong entity class. + +`upsert block/page` should reject `--tags` and `--properties` as unknown options after spec cleanup, with guidance to use `--update-tags` and `--update-properties`. + +Update-by-id failures should use new id-mode specific error codes so scripts can distinguish id lookup and id class mismatch from create-mode validation failures. + +## Architecture sketch + +```text +list property + -> /src/main/logseq/cli/command/list.cljs execute-list-property + -> /src/main/frontend/worker/db_core.cljs :thread-api/cli-list-properties + -> /src/main/logseq/cli/common/mcp/tools.cljs list-properties + -> /src/main/logseq/cli/format.cljs format-list-property (new dedicated formatter) +``` + +```text +upsert page/tag/property --id + -> /src/main/logseq/cli/commands.cljs parse + finalize-command + -> /src/main/logseq/cli/command/upsert.cljs update-mode detection and action build + -> transport/invoke :thread-api/pull for id/entity validation + -> transport/invoke :thread-api/apply-outliner-ops for update ops only +``` + +## Detailed implementation plan + +1. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert page --id 10 --update-properties ...` parses as `:upsert-page` and no longer requires `--page`. +2. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert tag --id 10` parses and `upsert property --id 10 --type node` parses. +3. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert block --tags ...` and `upsert page --properties ...` fail with `:invalid-options` due to removed flags. +4. Add RED parser tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert page --id --page ` fails with a selector conflict error. +5. Add RED build-action tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for `upsert page/tag/property` actions that include `:mode :update` when `--id` is present. +6. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` verifying `upsert tag --id ` with no update fields returns `:ok` and no mutation ops. +7. Add RED execute tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` verifying update-by-id rejects missing ids and wrong entity classes with new id-mode specific error codes. +8. Add RED formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting `list property` human output includes `TYPE` header and per-row values. +9. Add RED contract tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/mcp_tools_contract_test.cljs` asserting non-expanded `list-properties` items include `:logseq.property/type`. +10. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `upsert page --id`, `upsert tag --id`, and `upsert property --id` update mode behavior, including tag no-op behavior. +11. Add RED integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` verifying `upsert page --id --page` fails with selector conflict and `upsert block/page` reject removed `--tags` and `--properties` options. +12. Run focused RED commands and verify failures are behavior assertions, not fixture failures. +13. Update specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to remove `:tags` and `:properties` from block/page specs and add `:id` to page/tag/property specs. +14. Update option validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` so `upsert page` and `upsert property` required-field checks are mode-aware instead of unconditional, and add explicit selector-conflict validation for `upsert page --id --page`. +15. Refactor `build-page-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to support create mode by `--page` and update mode by `--id`. +16. Refactor `build-tag-action` and `build-property-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to support update mode by `--id` with mode-specific required options. +17. Add shared helper(s) in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to pull entities by id and validate class/type constraints before updates. +18. Update `execute-upsert-page`, `execute-upsert-tag`, and `execute-upsert-property` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so update mode uses id lookup, skips creation paths, and applies only update semantics, with `upsert tag --id` no-op when no mutation fields are provided. +19. Introduce explicit id-mode error codes in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` for id-not-found and id-type-mismatch failures. +20. Remove all `:tags` and `:properties` action wiring from page/block upsert flows in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`, and keep only `:update-tags` and `:update-properties`. +21. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` to include `:logseq.property/type` in non-expanded property list items. +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to split property formatting from tag formatting and render a dedicated `TYPE` column for `:list-property`. +23. Update docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to remove `--tags` or `--properties` from `upsert block/page` docs, document update-by-id behavior across all upsert commands, and document selector conflict behavior. +24. Run focused GREEN tests for commands, format, and integration, then run `bb dev:lint-and-test`. +25. Refactor only after GREEN to reduce duplication in upsert mode branching, then rerun focused tests and full suite. + +## Edge cases + +`upsert page --id ` must fail when id points to a block that is not a page entity. + +`upsert tag --id ` must fail when id points to a page not tagged with `:logseq.class/Tag`. + +`upsert property --id ` must fail when id points to an entity without `:logseq.property/type`. + +`upsert tag --id ` with no mutation options must return success without issuing mutation ops. + +`upsert page --id --page ` must fail with a dedicated selector conflict error. + +Update-by-id missing target and class mismatch failures must return new id-mode specific error codes. + +`upsert block` create mode with `--blocks` or `--blocks-file` must preserve existing validation behavior after removing `--tags` and `--properties` options. + +Property type display should remain stable for built-in and custom properties, and missing type values should render as `-` instead of throwing. + +JSON and EDN list outputs should remain backward compatible except for the additive `type` field on property items. + +## Verification commands and expected output + +| Command | Expected output | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test` | Parser, action, and execute tests for mode switching and option removal pass. | +| `bb dev:test -v logseq.cli.format-test` | Human formatter tests pass with `TYPE` column coverage for `list property`. | +| `bb dev:test -v logseq.cli.mcp-tools-contract-test` | Contract tests pass with `:logseq.property/type` present in non-expanded property items. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-page-create-and-update-existing` | Existing page upsert flow still passes after mode refactor. | +| `bb dev:test -v logseq.cli.integration-test` | New `--id` update-mode and removed-option behavior pass end to end. | +| `bb dev:lint-and-test` | Full suite passes with exit code `0`. | + +## Testing Details + +Tests cover CLI behavior at parser, action, executor, formatter, and end-to-end levels, and they assert entity outcomes instead of internal helper wiring. + +Tests verify that update-by-id mode never creates entities and that legacy duplicated options are no longer accepted for block/page upsert. + +Tests verify that `list property` human and structured output both include property-type information in their respective contracts. + +## Implementation Details + +- Keep db-worker-node thread API names unchanged and avoid adding new transport methods. +- Add `--id` to `upsert page/tag/property` specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Remove `--tags` and `--properties` from `upsert block/page` specs in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Make finalize validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` mode-aware for page/property required options. +- Rework `build-page-action`, `build-tag-action`, and `build-property-action` to branch on create vs update mode. +- Add id-based entity validation helpers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Make `upsert tag --id` with no mutation fields a successful no-op after id and class validation. +- Reject `upsert page --id --page` as explicit selector conflict. +- Add new id-mode specific error codes for id-not-found and id-type-mismatch paths. +- Include `:logseq.property/type` in non-expanded property list payload from `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs`. +- Split property-specific table rendering in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to add `TYPE` column. +- Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` command reference and examples. +- Keep implementation and debugging workflow aligned with `@test-driven-development` and `@clojure-debug`. + +## Question + +No open questions. + +Decided: `upsert tag --id ` with no additional mutation options is a successful no-op. + +Decided: `upsert page --id --page ` is rejected as conflicting selectors. + +Decided: update-by-id failures use new id-mode specific error codes. + +--- diff --git a/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md new file mode 100644 index 0000000000..3c8824ad74 --- /dev/null +++ b/docs/agent-guide/046-logseq-cli-upsert-tag-rename-by-id.md @@ -0,0 +1,147 @@ +# Logseq CLI Upsert Tag Rename by ID Implementation Plan + +Goal: Allow `logseq upsert tag --id --name ` to rename an existing tag while preserving the current create-by-name and validate-by-id behavior. + +Architecture: Keep the current `logseq-cli -> transport/invoke -> :thread-api/apply-outliner-ops` integration and implement rename-by-id entirely in the CLI command layer. +Architecture: Reuse the existing db-worker-node `:rename-page` outliner op instead of introducing a new thread API. +Architecture: Keep `upsert tag --id ` with no `--name` as an id-validation no-op for backward compatibility. + +Tech Stack: ClojureScript, babashka.cli, Promesa, Datascript pull queries, db-worker-node outliner ops. + +Related: Builds on `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/042-logseq-cli-add-tag-command.md`, `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/043-logseq-cli-tag-property-management.md`, and `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md`. + +## Problem statement + +Current `upsert tag` validation in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` rejects `--id` and `--name` together with the error `only one of --id or --name is allowed`. + +Current update mode in `execute-upsert-tag` only calls `ensure-tag-by-id!` and returns success without mutation when `:mode` is `:update`. + +Current db-worker-node path already supports page rename through `:thread-api/apply-outliner-ops` with `[:rename-page [page-uuid new-title]]` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`, so rename behavior can be reused without adding new APIs. + +The user-visible gap is that a command like `logseq upsert tag --graph --id 180 --name "Project Renamed"` should rename tag `180`, but currently fails at option validation. + +## Testing Plan + +I will use `@test-driven-development` for all implementation. + +I will write all RED tests first in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` and `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` before any production edits. + +I will verify RED failures are behavior failures by asserting error codes or missing mutation ops rather than fixture or async setup problems. + +I will use `@clojure-debug` only if async transport stubs or db-worker test harness behavior is unclear. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current implementation baseline + +| Requirement | Current behavior | Gap | +| --- | --- | --- | +| `upsert tag --id --name ` renames an existing tag. | `invalid-options?` for `:upsert-tag` rejects mixed selectors in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. | Rename-by-id cannot be triggered. | +| `upsert tag --id ` stays supported. | `execute-upsert-tag` update mode validates id and returns `[id]` without mutation. | Must remain unchanged for compatibility. | +| Rename execution uses existing db-worker-node contracts. | db-worker-node already handles `:rename-page` via `:thread-api/apply-outliner-ops` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`. | CLI does not currently call rename op in tag update path. | + +## Target contract + +`upsert tag --name ` keeps create or idempotent-create semantics. + +`upsert tag --id ` keeps id-validation no-op semantics and returns the same id. + +`upsert tag --id --name ` renames the target tag to `` using the existing outliner rename op. + +`upsert tag --id --name ` must fail with `:upsert-id-not-found` or `:upsert-id-type-mismatch` when id is invalid or not a tag. + +`upsert tag --id --name ` must no-op when `` normalizes to the same `:block/name` as the target. + +`upsert tag --id --name ` must fail if `` belongs to another non-tag page with `:tag-name-conflict`. + +`upsert tag --id --name ` must fail if `` belongs to another existing tag to avoid ambiguous cross-tag merges. + +## Architecture sketch + +```text +CLI + logseq upsert tag --id 180 --name "Project Renamed" + -> /src/main/logseq/cli/commands.cljs finalize-command + -> /src/main/logseq/cli/command/upsert.cljs build-tag-action + -> /src/main/logseq/cli/command/upsert.cljs execute-upsert-tag + 1) ensure-tag-by-id! + 2) conflict lookup by target name + 3) transport/invoke :thread-api/apply-outliner-ops + [repo [[:rename-page [tag-uuid new-name]]] {}] + 4) pull by id and return same id +DB Worker + /src/main/frontend/worker/db_core.cljs + :rename-page handler -> outliner-core/save-block! with new title +``` + +## Detailed implementation plan + +1. Add a RED parse/build test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting `upsert tag --id 180 --name "Project Renamed"` is accepted and routed to `:upsert-tag`. +2. Add a RED action test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode tag action keeps `:id` and includes normalized `:name`. +3. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode with both id and name emits exactly one `:rename-page` op through `:thread-api/apply-outliner-ops`. +4. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting update-mode with id-only remains no-op and does not call `:thread-api/apply-outliner-ops`. +5. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting rename target conflict with non-tag page returns `:tag-name-conflict`. +6. Add a RED execute test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` asserting rename target conflict with another tag returns a dedicated conflict error code. +7. Add a RED integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a tag, fetches its id, runs `upsert tag --id --name `, and verifies the new title appears in `list tag`. +8. Add a RED integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` asserting the old name no longer appears in `list tag` after rename. +9. Run focused RED commands and confirm failures are expected contract failures. +10. Update `invalid-options?` for `:upsert-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to permit `--id` plus `--name` and keep invalid checks for empty or malformed names. +11. Update `build-tag-action` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` so update mode accepts optional `:name` and keeps create mode semantics unchanged. +12. Add helper logic in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to detect whether normalized rename target equals current tag name and skip mutation in that case. +13. Add helper logic in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` to pull by target name and detect conflicts with other entities before rename. +14. Update `execute-upsert-tag` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` update branch to call `:rename-page` when id and name are both provided. +15. Keep error handling in `execute-upsert-tag` aligned with existing `:upsert-id-not-found` and `:upsert-id-type-mismatch` contracts, and add one dedicated rename-conflict code for tag-to-tag collisions. +16. Update CLI reference docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document `upsert tag --id --name ` rename semantics and conflict behavior. +17. Run focused GREEN tests for `commands_test` and targeted integration tests. +18. Run `bb dev:test -v logseq.cli.commands-test` and `bb dev:test -v logseq.cli.integration-test` to confirm no regressions. +19. Run `bb dev:lint-and-test` as final verification. +20. Refactor duplicated tag-name normalization or conflict checks only after GREEN, then rerun focused tests. + +## Edge cases + +- `--name` with leading `#` in update mode should normalize exactly like create mode. +- `--name` that trims to blank should return `:invalid-options` and not hit db-worker. +- Rename to current name with different casing should follow normalized-name no-op behavior. +- Rename target that already exists as another tag should fail deterministically and not mutate either tag. +- Rename target that exists as a non-tag page should fail with `:tag-name-conflict`. +- Rename by id must preserve the original `:db/id` in command output. +- Id-mode not-found and type-mismatch errors must remain stable for scripts. + +## Verification commands and expected output + +| Command | Expected output | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test` | Upsert tag parse, build, and execute tests pass including rename-by-id and no-op id-only paths. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-tag-id-rename` | End-to-end rename-by-id test passes with renamed tag visible in list output. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-upsert-tag-id-rename-conflict` | Conflict behavior test passes and returns expected error code/message. | +| `bb dev:lint-and-test` | Full suite passes with exit code `0`. | + +## Testing Details + +The new tests verify user-observable behavior at parser, action, executor, and CLI integration levels. + +The tests assert command outputs, mutation calls, and list/query observable state instead of helper internals. + +The tests keep existing id-only no-op behavior covered so rename support does not regress current automation scripts. + +## Implementation Details + +- Modify tag option validation only in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs`. +- Keep db-worker-node thread API signatures unchanged in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs`. +- Reuse existing `:rename-page` outliner op instead of introducing a new op. +- Preserve create-by-name idempotency path for `upsert tag --name`. +- Preserve id-only validate path for `upsert tag --id`. +- Add deterministic rename conflict handling before invoking rename op. +- Keep error code stability for existing id lookup and type mismatch failures. +- Update CLI docs to reflect rename-by-id and no-op-by-id contracts. +- Follow `@test-driven-development` sequence strictly and use `@clojure-debug` only for harness issues. + +## Question + +Resolved: choose option 1. + +Rename-to-existing-tag returns a dedicated conflict error and must not be treated as success by returning the existing tag id. + +This prevents implicit merges and accidental retargeting. + +--- diff --git a/docs/agent-guide/047-logseq-cli-sync-command.md b/docs/agent-guide/047-logseq-cli-sync-command.md new file mode 100644 index 0000000000..945d496e5e --- /dev/null +++ b/docs/agent-guide/047-logseq-cli-sync-command.md @@ -0,0 +1,225 @@ +# Logseq CLI Sync Command Implementation Plan + +Goal: Add `logseq sync` subcommands to inspect and operate db-sync through existing db-worker-node APIs. + +Architecture: The CLI parser and executor will gain a dedicated sync command module that maps subcommands to `:thread-api/db-sync-*` calls via `/v1/invoke`. +Architecture: A small worker API addition will expose runtime sync status, and sync config commands will support headless token setup through CLI-managed config values. +Architecture: The design will reuse existing graph lock and repo binding behavior in `logseq.cli.server/ensure-server!` and `frontend.worker.db-worker-node/repo-error`. + +Tech Stack: ClojureScript, babashka.cli, promesa, db-worker-node HTTP API, frontend.worker.sync. + +Related: Builds on `docs/agent-guide/031-logseq-cli-doctor-command.md`, `docs/agent-guide/033-desktop-db-worker-node-backend.md`, and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +The current CLI exposes graph, server, doctor, list, upsert, remove, query, and show commands, but it does not expose db-sync control or observability. + +`frontend.worker.db-core` already exposes operational db-sync thread APIs such as `:thread-api/db-sync-start`, `:thread-api/db-sync-stop`, `:thread-api/db-sync-upload-graph`, and `:thread-api/db-sync-grant-graph-access`. + +`frontend.worker.db-worker-node` already routes these methods over `/v1/invoke` with repo-lock safety checks, so the missing piece is a CLI command surface and one read API for status inspection. + +This plan keeps scope tight by reusing current transport and server lifecycle code, and only adds new worker behavior where inspection data is currently unavailable. + +I will use @planning-documents for naming, @writing-plans for task granularity, @logseq-cli for CLI integration expectations, and @test-driven-development for implementation sequence. + +## Testing Plan + +I will add parser and action unit tests that fail first for new `sync` command help, option validation, and action shaping. + +I will add command execution tests that fail first and verify `logseq.cli.transport/invoke` receives the exact method names and argument shapes for each sync subcommand. + +I will add format tests that fail first and verify human output for `sync status` and action commands, while keeping JSON and EDN behavior unchanged. + +I will add worker API tests that fail first for the new sync inspection API exposed through `/v1/invoke`. + +I will add one CLI integration test that fails first and verifies an end-to-end `sync status` flow on a temp graph and a started db-worker-node process. + +I will run targeted tests after each behavior slice and then run `bb dev:lint-and-test` before final review. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Scope and CLI surface + +| CLI command | Purpose | Worker method | Repo required | +|---|---|---|---| +| `sync status [--graph ]` | View current db-sync runtime state and counters. | `:thread-api/db-sync-status` (new). | Yes. | +| `sync start [--graph ]` | Start db-sync websocket client for the graph. | `:thread-api/db-sync-start`. | Yes. | +| `sync stop [--graph ]` | Stop db-sync client for the running daemon. | `:thread-api/db-sync-stop`. | Yes, to target a graph daemon deterministically. | +| `sync upload [--graph ]` | Upload current graph snapshot and mark graph remote metadata. | `:thread-api/db-sync-upload-graph`. | Yes. | +| `sync download [--graph ]` | Download remote graph data and apply it to local graph storage. | `:thread-api/db-sync-download-graph` (new). | Yes. | +| `sync remote-graphs` | List remote graphs visible to current auth context. | `:thread-api/db-sync-list-remote-graphs` (new). | No. | +| `sync ensure-keys` | Ensure user RSA keys required by e2ee are present. | `:thread-api/db-sync-ensure-user-rsa-keys`. | No. | +| `sync grant-access --graph-id --email [--graph ]` | Grant encrypted graph key access to a target user email. | `:thread-api/db-sync-grant-graph-access`. | Yes. | +| `sync config set ` | Set one config value by key. | `:thread-api/set-db-sync-config`. | No. | +| `sync config get ` | Read one config value by key. | `:thread-api/get-db-sync-config` (new). | No. | +| `sync config unset ` | Remove one config value by key. | `:thread-api/set-db-sync-config`. | No. | + +The first release intentionally excludes asset download and raw kv import commands because they need more user-facing safety rails and payload tooling. + +`sync config set` supports `ws-url`, `http-base`, and `auth-token`, and `config set auth-token ` is the headless authentication entrypoint. + +`sync config get` and `sync config unset` reject unknown config keys. + +`sync status` will return normalized fields even when sync is not configured, so scripts can branch deterministically. + +`sync remote-graphs` and `sync download` require auth-token to be configured in headless mode. + +## Architecture and integration points + +```text +logseq sync + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs (parse/build/execute) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs (new) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs (ensure graph daemon) + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs (POST /v1/invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs (repo checks + invoke) + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs thread APIs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs and sync/crypt.cljs +``` + +Worker additions will be minimal, with no protocol changes to cloud endpoints. + +CLI additions will follow existing `graph` and `server` command module patterns for spec, `entries`, `build-action`, and `execute-*` helpers. + +## Implementation plan + +### Phase 1. Add failing parser and help tests. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that top-level help includes `sync` and `sync status`. +2. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `logseq sync` shows subgroup help like `server` and `graph`. +3. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `sync config set|get|unset` and `sync grant-access` show in sync group help. +4. Run `bb dev:test -v logseq.cli.commands-test/test-help-output` and confirm failure references missing `sync` command rows. + +### Phase 2. Add failing action and execution tests for sync command module. + +5. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` with failing tests for `build-action` graph requirement on `sync status`. +6. Add a failing test for `build-action` rejection when `sync config set` is missing name or value. +7. Add a failing test for `build-action` rejection when `sync grant-access` misses `--graph-id` or `--email`. +8. Add a failing test for `build-action` graph requirement on `sync download`. +9. Add a failing execution test that stubs `logseq.cli.server/ensure-server!` and `logseq.cli.transport/invoke` and expects `:thread-api/db-sync-start` with `[repo]`. +10. Add a failing execution test that expects `:thread-api/db-sync-stop` with `[]` and still routes through `ensure-server!` using selected repo. +11. Add a failing execution test that expects `:thread-api/db-sync-upload-graph` with `[repo]`. +12. Add a failing execution test that expects `:thread-api/db-sync-download-graph` with `[repo]`. +13. Add a failing execution test that expects `:thread-api/db-sync-list-remote-graphs` for `sync remote-graphs`. +14. Add a failing execution test that expects `:thread-api/db-sync-ensure-user-rsa-keys` without repo. +15. Add a failing execution test that expects `:thread-api/db-sync-grant-graph-access` with `[repo graph-id email]`. +16. Add a failing execution test that expects `:thread-api/get-db-sync-config` for `sync config get `. +17. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config set ` and payload merge behavior. +18. Add a failing execution test that expects `:thread-api/set-db-sync-config` for `sync config unset ` and key removal behavior. +19. Add a failing execution test that verifies `sync config set auth-token ` updates worker-consumable token config for headless mode. +20. Run `bb dev:test -v logseq.cli.command.sync-test` and confirm failures are only from missing sync implementation. + +### Phase 3. Implement CLI sync command wiring. + +21. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` with sync option specs, `entries`, `build-action`, and `execute-*` functions. +22. Register `sync-command/entries` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` command table. +23. Extend `finalize-command` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` with sync-specific required-option checks. +24. Extend single-token group help routing in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to include `sync`. +25. Extend `build-action` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to call `sync-command/build-action`. +26. Extend `execute` dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to route sync action types. +27. Add `sync` to top-level command grouping in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs`. +28. Run `bb dev:test -v logseq.cli.commands-test/test-parse-args-help` and `bb dev:test -v logseq.cli.command.sync-test` until green. + +### Phase 4. Add read-only worker APIs for sync inspection. + +29. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` accepts `thread-api/get-db-sync-config` without repo. +30. Add a failing worker test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `/v1/invoke` for `thread-api/db-sync-status` enforces repo and returns structured status. +31. Add `:thread-api/get-db-sync-config` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning current config map. +32. Add `:thread-api/db-sync-status` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` returning ws state, graph id, and sync counters for a repo. +33. Add or expose a small helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to compute status without requiring websocket side effects. +34. Add `:thread-api/db-sync-list-remote-graphs` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement cloud graph listing through worker sync HTTP helpers. +35. Add `:thread-api/db-sync-download-graph` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` and implement remote snapshot download plus local import flow. +36. Update worker sync auth token resolution so `sync config set auth-token ` is used in headless mode when state token is missing. +37. Register `:thread-api/get-db-sync-config` and `:thread-api/db-sync-list-remote-graphs` in `non-repo-methods` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`. +38. Run `bb dev:test -v frontend.worker.db-worker-node-test` and fix only sync-related regressions. + +### Phase 5. Add output formatting tests and implementation. + +39. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync status`. +40. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `sync remote-graphs`. +41. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of sync action commands such as start, upload, and download. +42. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` verifying token redaction for `sync config get auth-token` in human output. +43. Implement sync human formatters in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` with stable keys and token redaction. +44. Confirm JSON and EDN output behavior by running `bb dev:test -v logseq.cli.format-test`. + +### Phase 6. Add integration coverage and CLI docs. + +45. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a graph and runs `sync status` with `--output json`. +46. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config set auth-token` then `sync config get auth-token` behavior. +47. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync config unset auth-token`. +48. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync remote-graphs --output json`. +49. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` for `sync download --graph ` flow with mocked remote snapshot response. +50. Implement any missing glue for integration stability in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs`. +51. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` command docs with sync command examples and error behaviors. +52. Run `bb dev:test -v logseq.cli.integration-test` to verify end-to-end behavior. + +### Phase 7. Final verification and cleanup. + +53. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and confirm exit code `0`. +54. Run manual smoke commands with a temp graph and confirm both `--output human` and `--output json` are stable. +55. Review help text alignment and command ordering to match existing CLI aesthetics. + +## Edge cases and error handling + +`sync status` must return a valid map even when `:ws-url` is missing, with an explicit inactive state rather than throwing. + +`sync start` must keep current behavior where missing `ws-url` or missing graph uuid results in no crash and a deterministic status response. + +`sync grant-access` must surface cloud errors with existing `http-error` path and preserve status code and body context. + +`sync config get auth-token` must redact token values in human output while keeping full value available in JSON and EDN output for scripting. + +`sync config set auth-token ` must write to the config file selected by `--config` (default `~/logseq/cli.edn`) so headless auth survives daemon restarts. + +`sync remote-graphs` must return a deterministic empty list when user has no remote graphs instead of returning nil. + +`sync download` must fail fast when the target local graph is missing required auth or remote graph metadata, and must report a clear sync-specific error code. + +Repo mismatch and lock ownership behavior must remain enforced by db-worker-node and must not be bypassed in CLI command code. + +All new options must keep kebab-case keyword naming and avoid introducing `_` forms. + +## Verification commands + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.command.sync-test` | Sync command unit tests pass with no failures. | +| `bb dev:test -v logseq.cli.commands-test/test-help-output` | Help output includes `sync` group and subcommands. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Worker invoke tests pass including new sync read APIs. | +| `bb dev:test -v logseq.cli.format-test` | Human and structured output tests pass including sync formatters. | +| `bb dev:test -v logseq.cli.integration-test` | CLI integration tests pass for sync status and config flow. | +| `bb dev:lint-and-test` | Full lint and unit suite passes with exit code `0`. | +| `node ./dist/logseq.js sync status --graph demo --output json` | Returns `{"status":"ok","data":...}` with sync status fields. | +| `node ./dist/logseq.js sync remote-graphs --output json` | Returns remote graph list in structured output. | +| `node ./dist/logseq.js sync download --graph demo` | Downloads remote graph snapshot and imports it into local graph data. | +| `node ./dist/logseq.js sync config set auth-token ` | Sets headless auth token for db-sync API calls. | +| `node ./dist/logseq.js sync config get auth-token --output json` | Returns configured token value in structured output. | +| `node ./dist/logseq.js sync config unset auth-token` | Removes configured token and returns success message. | + +## Testing Details + +The new tests verify behavior at parser level, action-building level, transport payload level, worker invoke contract level, output formatting level, and end-to-end CLI invocation level. + +The tests assert external behavior such as command availability, returned status payloads, and worker method invocations, instead of asserting internal helper implementation details. + +## Implementation Details + +- Add new file `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` for sync command ownership. +- Keep `sync` command wiring inside existing dispatch points in `commands.cljs` and do not introduce a second dispatcher. +- Add core worker sync inspection APIs, `get-db-sync-config` and `db-sync-status`, and reuse existing `set-db-sync-config` for config writes. +- Add worker sync APIs for remote graph listing and graph download to support `sync remote-graphs` and `sync download`. +- Reuse `transport/invoke` with existing `direct-pass?` handling and default to transit mode. +- Keep `sync status` output fields stable for scripting, including `repo`, `graph-id`, `ws-state`, and pending counters. +- Keep human output terse and redact auth-token values. +- Update `command.core/top-level-summary` and group-help routing so `sync` behaves like existing command groups. +- Keep all new keyword names kebab-case and avoid shadowed local names such as `bytes`. +- Update `docs/cli/logseq-cli.md` with command list, examples, and expected error hints. +- Run full lint and tests after targeted green passes. + +## Question + +No open question. + +This plan adopts option A and includes `sync config set|get|unset` with `config set auth-token ` as the token setup path. + +--- diff --git a/docs/agent-guide/048-sync-download-start-reliability.md b/docs/agent-guide/048-sync-download-start-reliability.md new file mode 100644 index 0000000000..311fe77e04 --- /dev/null +++ b/docs/agent-guide/048-sync-download-start-reliability.md @@ -0,0 +1,179 @@ +# Sync Download And Sync Start Reliability Implementation Plan + +Goal: Validate and fix `logseq sync download` plus `logseq sync start` so download data is complete and start reaches a working sync session. + +Architecture: I will harden behavior with failing tests first at CLI, db-worker-node invoke boundary, and worker sync core, then apply minimal fixes in `logseq.cli.command.sync`, `frontend.worker.db_core`, and `frontend.worker.sync`. +Architecture: The start path will stop reporting false success by checking runtime status after start, and the download path will fail fast on incomplete or inconsistent snapshot import conditions. +Architecture: Manual verification will use a local untracked `cli.edn` built from the exact config block provided in this task. + +Tech Stack: ClojureScript, babashka test runner, db-worker-node daemon, frontend worker thread APIs, websocket plus HTTP sync APIs. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`, and relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md` and `docs/agent-guide/034-db-worker-node-owner-process-management.md`. + +## Problem statement + +`/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` currently reports `sync start` success after invoking `:thread-api/db-sync-start`, but it does not verify the websocket state becomes `:open`. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` skips start when config or graph-id is missing, and the CLI currently does not distinguish this from a successful start. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` enforces frame integrity in `finalize-framed-buffer`, but the current tests do not cover incomplete snapshot frame failure through the full download flow. + +`/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` imports downloaded rows and returns a summary map, but there is no end-to-end CLI integration test that confirms downloaded graph data is usable and consistent after import. + +This plan uses @planning-documents for naming, @writing-plans for granularity, @test-driven-development for sequence, and @clojure-debug when failures are not immediately obvious. + +## Testing Plan + +I will add CLI command tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` for start readiness polling, start timeout behavior, and download error propagation. + +I will add worker sync tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for incomplete framed snapshot behavior and download completeness invariants. + +I will add worker daemon invoke tests in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` to verify start and status interactions remain stable through `/v1/invoke`. + +I will add CLI integration tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` to verify `sync download` yields queryable graph data and `sync start` reaches `:open` within timeout in a deterministic mocked sync environment. + +I will run targeted tests after each behavior slice and then run `bb dev:lint-and-test` for regression safety. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Current integration map + +```text +logseq sync download/start + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs + -> remote sync HTTP + websocket endpoints +``` + +## Manual test configuration + +Create `/tmp/logseq-sync-cli.edn` with the exact EDN block provided in this task request. + +Keep this file local and untracked, and never commit the auth token to the repository. + +Use `--config /tmp/logseq-sync-cli.edn` for every manual `logseq sync` command in this plan. + +## Implementation plan + +### Phase 0. Baseline and reproducibility. + +1. Create `/tmp/logseq-sync-cli.edn` from the provided config block. +2. Run `bb dev:test -v logseq.cli.command.sync-test` to capture current CLI sync baseline. +3. Run `bb dev:test -v frontend.worker.db-sync-test` to capture current worker sync baseline. +4. Run `bb dev:test -v frontend.worker.db-worker-node-test` to capture current daemon invoke baseline. +5. Run `node ./dist/logseq.js sync remote-graphs --config /tmp/logseq-sync-cli.edn --output json` and record the graph-id and graph-name used for manual verification. + +### Phase 1. RED tests for sync start readiness semantics. + +6. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync start` polls `:thread-api/db-sync-status` and returns success only after `:ws-state` becomes `:open`. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync start` returns `:error` with diagnostic status when `:ws-state` never reaches `:open` before timeout. +8. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that start skipped conditions like missing `:ws-url` are surfaced as actionable CLI errors instead of success. +9. Run `bb dev:test -v logseq.cli.command.sync-test/test-execute-sync-start` and verify RED failures are behavior failures. + +### Phase 2. RED tests for download completeness and failure propagation. + +10. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that incomplete snapshot frames trigger `:db-sync/incomplete-snapshot-frame` behavior through download framing helpers. +11. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` that a valid download payload produces stable row batches that can be imported without truncation. +12. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync download` returns `:error` and preserves error code when `:thread-api/db-sync-download-graph-by-id` fails. +13. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that downloaded data is queryable after import and includes expected graph metadata. +14. Run `bb dev:test -v frontend.worker.db-sync-test` and `bb dev:test -v logseq.cli.integration-test` and verify RED failures are behavior failures. + +### Phase 3. GREEN implementation for sync start observability. + +15. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` to add a bounded wait helper that polls `:thread-api/db-sync-status` after `:thread-api/db-sync-start`. +16. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so `:sync-start` returns `{:status :ok}` only when status shows `:ws-state :open`. +17. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so timeout or skipped start returns `{:status :error}` with `:repo`, current `:ws-state`, and remediation hint. +18. If needed, update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` status payload to include one extra diagnostic field for recent start failures. +19. Re-run `bb dev:test -v logseq.cli.command.sync-test` until all start-related tests pass. + +### Phase 4. GREEN implementation for download completeness guarantees. + +20. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to keep strict framed payload validation and expose structured failure details used by CLI. +21. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` download/import path to assert required result fields before import and fail fast on invalid payload. +22. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` download branch to preserve worker error code and context in CLI output. +23. Re-run `bb dev:test -v frontend.worker.db-sync-test` and `bb dev:test -v logseq.cli.command.sync-test` until download-related tests pass. + +### Phase 5. REFACTOR and regression safety. + +24. Refactor only duplicated sync command helper code in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` without changing behavior. +25. Re-run `bb dev:test -v logseq.cli.command.sync-test` after refactor. +26. Re-run `bb dev:test -v frontend.worker.db-sync-test` after refactor. +27. Re-run `bb dev:test -v frontend.worker.db-worker-node-test` after refactor. + +### Phase 6. Manual end-to-end verification with provided cli.edn. + +28. Run `node ./dist/logseq.js sync download --graph --config /tmp/logseq-sync-cli.edn --output json` and record `graph-id`, `remote-tx`, and `row-count`. +29. Run `node ./dist/logseq.js q --graph --config /tmp/logseq-sync-cli.edn --output json '[:find (count ?e) :where [?e :block/uuid]]'` and confirm graph has non-zero entities. +30. Run `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` and capture pre-start `ws-state`. +31. Run `node ./dist/logseq.js sync start --graph --config /tmp/logseq-sync-cli.edn --output json` and verify command succeeds only when status reaches `:open`. +32. Run `node ./dist/logseq.js upsert block --graph --title "sync smoke $(date +%s)" --config /tmp/logseq-sync-cli.edn --output json` to create a local change. +33. Run `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` repeatedly until `pending-local` trends back to `0`. +34. If step 31 or step 33 fails, inspect db-worker-node log file under the graph repo directory and capture exact error code plus timestamp. + +### Phase 7. Final verification. + +35. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and ensure full suite is green. +36. Re-run manual step 31 one more time to verify no flakiness after full test run. +37. Document final behavior and known limitations in the PR summary. + +## Edge cases to cover explicitly + +`sync start` must return an actionable error when `:ws-url` is missing or empty. + +`sync start` must avoid false success when token exists but handshake never reaches `:open`. + +Repeated `sync start` calls must be idempotent and not create duplicate clients. + +`sync download` must fail clearly when remote graph is not found and must include graph-name in the error. + +`sync download` must fail clearly on incomplete framed snapshot payload. + +`sync download` must not leave a half-imported local graph when import throws mid-batch. + +`sync download` must preserve and report `graph-e2ee?` and remote graph-id in result payload. + +## Verification command matrix + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.command.sync-test` | Sync CLI tests pass, including start readiness and download error propagation. | +| `bb dev:test -v frontend.worker.db-sync-test` | Worker sync framing and download completeness tests pass. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Daemon invoke and status behavior remains green. | +| `bb dev:test -v logseq.cli.integration-test` | CLI integration verifies queryable data after download and start readiness path. | +| `node ./dist/logseq.js sync download --graph --config /tmp/logseq-sync-cli.edn --output json` | Returns `status: ok` with stable `graph-id`, `remote-tx`, and non-negative `row-count`. | +| `node ./dist/logseq.js sync start --graph --config /tmp/logseq-sync-cli.edn --output json` | Returns `status: ok` only when worker status reaches `ws-state: open`. | +| `node ./dist/logseq.js sync status --graph --config /tmp/logseq-sync-cli.edn --output json` | Shows non-negative queue counters and stable repo graph identifiers. | +| `bb dev:lint-and-test` | Full lint and unit suite passes with exit code `0`. | + +## Testing Details + +The test suite additions validate behavior from user-visible CLI result down to worker framed snapshot handling and daemon invoke boundary. + +The tests verify externally observable behavior such as command status, error code, websocket state progression, and queryable post-download graph data. + +The tests avoid checking private implementation details and focus on functional outcomes. + +## Implementation Details + +- Keep `sync` command API surface unchanged and improve readiness validation inside existing `:sync-start` execution path. +- Keep download protocol unchanged and improve payload validation plus error propagation semantics. +- Reuse `:thread-api/db-sync-status` as the source of readiness truth for CLI start. +- Keep all keyword names kebab-case and avoid introducing underscore keys. +- Ensure any new helper names avoid shadowed locals like `bytes`. +- Keep auth token handling in local config and never persist token values into repository files. +- 在 db-worker-node 中,如果是 desktop app 场景启动,token 读写保持原有代码路径。 +- 在 db-worker-node 中,如果是 cli 场景启动,读取 token 一律从 `cli.edn`(sync config)中获取。 +- 在 db-worker-node 中,如果是 cli 场景启动,写入 token 也走 `cli.edn`(sync config)路径。 +- Preserve existing db-worker-node repo lock and method access rules. +- Keep manual verification commands deterministic by pinning `--config /tmp/logseq-sync-cli.edn`. +- Use a fixed `sync start` wait timeout of `10000` ms with no CLI override option. + +## Question + +No open question. + +--- diff --git a/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md b/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md new file mode 100644 index 0000000000..940048ed24 --- /dev/null +++ b/docs/agent-guide/049-logseq-cli-graph-info-kv-display.md @@ -0,0 +1,193 @@ +# Graph Info Kv Display Implementation Plan + +Goal: Make `logseq graph info` display all persisted `:logseq.kv/<...>` values in a readable and script-friendly way without introducing new db-worker-node protocol endpoints. + +Architecture: Reuse existing `logseq-cli -> transport -> db-worker-node -> :thread-api/q` flow to query kv entities by `:db/ident` namespace `logseq.kv`. + +Architecture: Keep backward compatibility by preserving current summary fields while adding a structured ident-keyed kv map for JSON and EDN output plus a dedicated kv section in human output. + +Architecture: Follow `@test-driven-development` with failing tests first in command, formatter, and integration layers. + +Tech Stack: ClojureScript, Datascript query via existing `:thread-api/q`, db-worker-node HTTP transit transport, Logseq CLI formatter pipeline, Babashka test runner. + +Related: Builds on `docs/agent-guide/001-logseq-cli.md`. + +Related: Builds on `docs/agent-guide/007-logseq-cli-thread-api-and-command-split.md`. + +Related: Relates to `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +## Problem statement + +Current `execute-graph-info` only fetches `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version` via two `:thread-api/pull` calls. + +Current human output shows only three lines, which is too limited for diagnosing graph state during CLI usage. + +Current JSON and EDN outputs also omit other kv entries such as `:logseq.kv/db-type`, `:logseq.kv/graph-initial-schema-version`, and runtime metadata keys. + +The requested behavior is to show all `:logseq.kv/<...>` kv pairs in a way that is readable for humans and stable for machine consumers. + +The implementation should stay within the current db-worker-node design and avoid adding new RPC methods unless absolutely necessary. + +## Current and target data flow + +```text +Current. +logseq graph info + -> execute-graph-info + -> thread-api/pull (:graph-created-at) + -> thread-api/pull (:schema-version) + -> formatter renders 3 lines. + +Target. +logseq graph info + -> execute-graph-info + -> thread-api/q (query all :db/ident in namespace "logseq.kv" with :kv/value) + -> normalize + sort kv rows + -> data payload keeps summary fields + ident-keyed kv map + -> formatter renders summary + kv section (human) or structured kv payload (json/edn). +``` + +## Target behavior + +`graph info` keeps existing top summary lines so users still see graph name, created-at, and schema version immediately. + +`graph info` adds a complete kv section in human output, sorted by ident for deterministic reading. + +`graph info` adds a structured ident-keyed kv map field in result data for JSON and EDN output so scripts can parse all kv values directly. + +The kv field contains only persisted kv rows, which means keys without `:kv/value` in the DB are not synthesized. + +All output formats redact kv keys matching sensitive patterns like `token`, `secret`, or `password`. + +Human output truncates very long string values with explicit marker text. + +The implementation uses only existing `:thread-api/q` and transport invocation paths. + +## Testing Plan + +I will follow `@test-driven-development` and write all failing tests before implementation. + +I will add command-level tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` to validate `execute-graph-info` builds kv payload from query rows and preserves existing summary fields. + +I will extend formatting tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to verify human output includes a deterministic kv section and still keeps the existing summary lines. + +I will add JSON and EDN formatting assertions in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for the new structured kv field shape. + +I will add formatter tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for sensitive key redaction across human, JSON, and EDN outputs plus long string truncation in human output. + +I will add integration coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that creates a graph and asserts `graph info` returns non-empty `logseq.kv` data end-to-end through db-worker-node. + +I will run focused tests first and use `@clojure-debug` if any failure is non-obvious before changing implementation. + +I will run `bb dev:lint-and-test` as final gate. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1: Add failing tests for command payload shape. + +1. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that stubs `cli-server/ensure-server!` and `transport/invoke` and asserts `execute-graph-info` issues one `:thread-api/q` request for `logseq.kv` rows. +2. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that asserts result data still includes `:graph`, `:logseq.kv/graph-created-at`, and `:logseq.kv/schema-version`. +3. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/graph_test.cljs` that asserts kv rows are normalized into an ident-keyed map for machine output and deterministic sorted entries for human rendering. + +### Phase 2: Add failing tests for human output. + +4. Extend `test-human-output-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` to expect the existing summary lines plus a kv section. +5. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies kv rows are ordered lexicographically by ident. +6. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies sensitive kv keys are redacted in human output. +7. Add a failing formatter test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` that verifies long string kv values are truncated in human output. + +### Phase 3: Add failing tests for machine outputs and integration. + +8. Add a failing JSON output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting new kv map field is present and parseable. +9. Add a failing EDN output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting kv map preserves keyword idents and applies sensitive key redaction. +10. Add a failing JSON output test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` asserting sensitive keys are redacted in machine output. +11. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that runs `graph create` then `graph info` and asserts returned data includes multiple `:logseq.kv/...` keys. +12. Run focused tests and confirm failures are behavior gaps rather than fixture or server boot issues. + +### Phase 4: Implement command-side kv collection with existing db-worker-node API. + +13. Add a private datalog query constant in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/graph.cljs` that finds `?ident` and `?value` where ident namespace is `"logseq.kv"` and entity has `:kv/value`. +14. Update `execute-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/graph.cljs` to call `transport/invoke` once with `:thread-api/q` and query inputs. +15. Normalize query tuples into an ident-keyed kv map for machine output and keep a sorted kv entry list for deterministic human formatting. +16. Derive `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version` from the kv map to keep backward compatibility in `:data`. +17. Add the new structured kv map field to the response payload while keeping existing keys unchanged. + +### Phase 5: Implement formatter changes for reasonable display. + +18. Add a helper in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to format graph kv rows for human output with a clear header and deterministic ordering. +19. Update `format-graph-info` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` to render summary lines first and kv section second. +20. Add centralized masking for kv keys matching sensitive patterns like `token`, `secret`, or `password` so human, JSON, and EDN output share the same redaction behavior. +21. Add human output truncation for very long string values with an explicit marker. +22. Ensure human output remains readable when kv list is empty by showing an explicit empty state line. + +### Phase 6: Verify behavior and compatibility. + +23. Run `bb dev:test -v 'logseq.cli.command.graph-test'` and verify new command tests pass. +24. Run `bb dev:test -v 'logseq.cli.format-test'` and verify human, JSON, and EDN formatting tests pass. +25. Run `bb dev:test -v 'logseq.cli.integration-test'` and verify graph-info integration behavior across db-worker-node is green. +26. Run `bb dev:lint-and-test` and verify `0 failures, 0 errors`. +27. Review against `/Users/rcmerci/gh-repos/logseq/prompts/review.md` checklist before final submission. + +## Edge cases + +| Scenario | Expected behavior | +|---|---| +| Graph has only default kv rows. | Human output shows summary lines plus kv section with default keys. | +| Graph has additional future `:logseq.kv/*` keys. | All persisted keys are included automatically without code changes. | +| Graph has composite kv values such as map or vector. | Human output renders values safely as printable EDN strings. | +| Graph has boolean and UUID kv values. | JSON output serializes them in existing normalize-json behavior without crashes. | +| Kv key includes `token`, `secret`, or `password`. | Human, JSON, and EDN outputs all show redacted value. | +| Kv value is a very long string. | Human output shows truncated text with explicit marker. | +| Query returns no kv rows due to corrupted graph state. | Command still succeeds with summary placeholders and an explicit empty kv section. | +| Existing script parses `graph-created-at` and `schema-version` fields. | Backward compatible fields remain present in response data. | +| Human output consumers expect old first three lines. | Existing first three lines remain unchanged in order and meaning. | + +## Verification commands and expected outputs + +```bash +bb dev:test -v 'logseq.cli.command.graph-test' +bb dev:test -v 'logseq.cli.format-test' +bb dev:test -v 'logseq.cli.integration-test' +bb dev:lint-and-test +``` + +Each command should complete with `0 failures, 0 errors`. + +Integration output should include a non-empty kv field in `graph info` JSON payload. + +Human output should include `Graph`, `Created at`, and `Schema version` before the kv section. + +## Testing Details + +The tests verify behavior at command assembly, output formatting, and end-to-end runtime boundaries. + +The command tests validate query usage and payload compatibility instead of checking internal helper implementation details only. + +The format tests verify visible behavior for human readability and machine parseability for JSON and EDN outputs. + +The integration test verifies real db-worker-node invocation and confirms that kv data is actually surfaced through CLI transport. + +## Implementation Details + +- Reuse `:thread-api/q` and avoid introducing a new db-worker-node endpoint. +- Keep `execute-graph-info` backward compatible for existing top-level summary fields. +- Add one new ident-keyed kv map field for scripts instead of spreading arbitrary keys at top level. +- Keep kv ordering deterministic in CLI code to stabilize human snapshots and tests. +- Render complex values safely in human output with printable EDN formatting. +- Redact sensitive keys in human, JSON, and EDN output for `token`, `secret`, and `password` patterns. +- Truncate very long string values in human output only. +- Preserve existing output pipeline behavior for `:json`, `:edn`, and `:human`. +- Keep command parser and `graph info` CLI flags unchanged for this iteration. +- Follow `@test-driven-development` strictly and use `@clojure-debug` on unexpected failures. + +## Question + +Decision: The new structured kv field is a single ident-keyed map for machine output. + +Decision: Human, JSON, and EDN outputs must redact kv keys matching sensitive patterns like `token`, `secret`, or `password`. + +Decision: Human output must truncate very long string values with an explicit marker. + +--- diff --git a/docs/agent-guide/050-sync-download-create-empty-db.md b/docs/agent-guide/050-sync-download-create-empty-db.md new file mode 100644 index 0000000000..bcd6faf57f --- /dev/null +++ b/docs/agent-guide/050-sync-download-create-empty-db.md @@ -0,0 +1,188 @@ +# Sync Download Create Empty Db Implementation Plan + +Goal: Ensure `sync download` never writes `db-initial-data` before snapshot import by starting `db-worker-node` with an explicit empty-db mode. + +Architecture: Add a new db-worker startup flag `--create-empty-db` and plumb it through CLI server orchestration only for the `sync download` path. + +Architecture: When the flag is enabled, `db-worker-node` will call `:thread-api/create-or-open-db` with `{:datoms []}` during daemon startup so `frontend.worker.db-core/ /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> :thread-api/db-sync-download-graph-by-id import path +``` + +## Scope and non-goals + +In scope are CLI-to-daemon argument plumbing, db-worker startup behavior change behind a new flag, tests, and docs updates. + +Out of scope are sync protocol changes, snapshot payload format changes, and non-download command behavior changes. + +## Implementation plan + +### Phase 0. Baseline and failure capture. + +1. Run `bb dev:test -v logseq.db-worker.daemon-test` and record baseline. +2. Run `bb dev:test -v frontend.worker.db-worker-node-test` and record baseline. +3. Run `bb dev:test -v logseq.cli.server-test` and record baseline. +4. Run `bb dev:test -v logseq.cli.command.sync-test` and record baseline. + +### Phase 1. RED tests for daemon flag plumbing. + +5. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/daemon_test.cljs` that `spawn-server!` appends `--create-empty-db` when `:create-empty-db? true` is passed. +6. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/db_worker/daemon_test.cljs` that `spawn-server!` keeps default args unchanged when `:create-empty-db?` is absent or false. +7. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that `parse-args` recognizes `--create-empty-db` and maps it to `:create-empty-db? true`. +8. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that help output documents `--create-empty-db` as a startup option. +9. Run `bb dev:test -v logseq.db-worker.daemon-test` and `bb dev:test -v frontend.worker.db-worker-node-test` and confirm RED. + +### Phase 2. GREEN implementation for daemon flag plumbing. + +10. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` so `spawn-server!` accepts `:create-empty-db?` and conditionally appends `--create-empty-db`. +11. Keep process scanning in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/db_worker/daemon.cljs` compatible by ignoring unknown flags as today. +12. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `parse-args` to set `:create-empty-db? true` when `--create-empty-db` is present. +13. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `show-help!` to include one line for `--create-empty-db`. +14. Re-run `bb dev:test -v logseq.db-worker.daemon-test` and `bb dev:test -v frontend.worker.db-worker-node-test` and confirm green. + +### Phase 3. RED tests for sync download orchestration. + +15. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `:sync-download` calls `cli-server/ensure-server!` with `:create-empty-db? true`. +16. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/server_test.cljs` that `spawn-server!` forwards `:create-empty-db?` from `ensure-server-started!` config when spawning a new daemon. +17. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` that `sync download` path keeps existing `graph-exists` guard unchanged while still enabling create-empty mode on success paths. +18. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync download` fails fast with a dedicated error when the graph DB is not empty. +19. Run `bb dev:test -v logseq.cli.command.sync-test` and `bb dev:test -v logseq.cli.server-test` and confirm RED. + +### Phase 4. GREEN implementation for sync download orchestration. + +20. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so the `:sync-download` execution path enriches config with `:create-empty-db? true` before `invoke-global` and `invoke-with-repo`. +21. Add a DB emptiness guard in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` or worker thread-api path that explicitly validates target graph DB is empty before calling `:thread-api/db-sync-download-graph-by-id`. +22. Keep non-download sync actions in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` unchanged so default startup remains backward compatible. +23. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/server.cljs` and its internal `spawn-server!` wrapper to pass through `:create-empty-db?` to daemon spawn. +24. Re-run `bb dev:test -v logseq.cli.command.sync-test` and `bb dev:test -v logseq.cli.server-test` and confirm green. + +### Phase 5. RED tests for startup create-or-open behavior. + +25. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that startup invokes `thread-api/create-or-open-db` with `[repo {:datoms []}]` when `:create-empty-db? true`. +26. Add a failing test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` that default startup still invokes `[repo {}]` when flag is not set. +27. Add a failing behavior test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` or `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_worker_node_test.cljs` validating download bootstrap path does not depend on locally generated initial data. +28. Add a failing behavior test validating non-empty graph DB state is rejected before import starts. +29. Run targeted tests and confirm RED failures are behavior-based. + +### Phase 6. GREEN implementation for empty-db startup. + +30. Update `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` `start-daemon!` to pass startup opts `{:datoms []}` to `:thread-api/create-or-open-db` only when `:create-empty-db? true`. +31. Keep lock lifecycle and health endpoints unchanged in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs`. +32. Add a tiny helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_worker_node.cljs` if needed to keep startup opts construction testable. +33. Implement or expose a reusable DB emptiness check in worker/CLI path and return a stable error code for non-empty DB. +34. Re-run `bb dev:test -v frontend.worker.db-worker-node-test` and the selected download behavior test namespace until green. + +### Phase 7. Docs and regression. + +35. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document that `sync download` starts db-worker in create-empty mode and rejects non-empty DB state. +36. Update `/Users/rcmerci/gh-repos/logseq/docs/developers/desktop-db-worker-node.md` with a note that create-empty mode is CLI download bootstrap behavior and does not change desktop default startup. +37. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and ensure full suite passes. +38. Run final review checklist in `/Users/rcmerci/gh-repos/logseq/prompts/review.md` before merge. + +## Edge cases to cover explicitly + +| Scenario | Expected behavior | +|---|---| +| `sync download` for a brand-new graph. | Daemon starts with `--create-empty-db` and startup `create-or-open-db` uses `{:datoms []}`. | +| `sync start`, `sync upload`, and graph CRUD commands. | No `--create-empty-db` flag is used, and existing startup behavior stays unchanged. | +| Existing local graph (`graph-exists`). | Command fails before daemon startup as it does today. | +| Existing ready daemon lock for the same repo. | Command must validate DB emptiness before download and fail fast when DB is not empty. | +| Missing remote graph. | `remote-graph-not-found` response remains unchanged, and no behavior regression in error shape occurs. | +| Flag typo or absent flag. | Startup remains backward compatible with default `create-or-open-db` opts `{}`. | + +## Verification command matrix + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.db-worker.daemon-test` | Spawn arg and parser tests for `--create-empty-db` pass. | +| `bb dev:test -v frontend.worker.db-worker-node-test` | Startup invoke args and help text tests pass. | +| `bb dev:test -v logseq.cli.server-test` | CLI server pass-through tests for `:create-empty-db?` pass. | +| `bb dev:test -v logseq.cli.command.sync-test` | `sync download` ensures server with create-empty mode. | +| `bb dev:test -v logseq.cli.commands-test` | Existing `graph-exists` guard and command dispatch remain stable. | +| `bb dev:test -v logseq.cli.integration-test/test-cli-sync-download-and-start-readiness-with-mocked-sync` | Integration path remains green with download plus start readiness behavior. | +| `bb dev:lint-and-test` | Full lint and unit test suite exits with code `0`. | + +## Rollout notes + +This change is intentionally scoped to startup behavior for `sync download` and does not alter runtime sync protocol. + +If regressions appear, rollback is to remove `:create-empty-db?` wiring in sync command and keep daemon/parser support dormant until retried. + +## Testing Details + +Tests focus on externally visible behavior, namely which startup args are passed and whether startup create-or-open uses empty datoms in download bootstrap mode. + +Tests also verify `sync download` rejects non-empty graph DB state before any import side effect. + +Tests avoid asserting private implementation internals except where argument boundaries are the behavior contract. + +Integration coverage confirms existing sync download and sync start semantics remain stable after plumbing changes. + +## Implementation Details + +- Add `:create-empty-db?` option plumbing from sync command to CLI server spawn path. +- Add `--create-empty-db` process arg emission in daemon spawn helper. +- Parse `--create-empty-db` in db-worker-node entrypoint args. +- When create-empty is enabled, startup invoke payload uses `[repo {:datoms []}]`. +- Add a strict pre-download emptiness check and fail with a stable error code when DB is not empty. +- Keep default startup invoke payload `[repo {}]` for all other flows. +- Keep lock ownership and stale-lock cleanup behavior unchanged. +- Keep command-level `graph-exists` and `remote-graph-not-found` behaviors unchanged. +- Update CLI and developer docs with exact scope of the new flag. +- Run targeted tests first, then full `bb dev:lint-and-test`. +- Use `@test-driven-development` workflow and `@clojure-debug` when failures are non-obvious. + +## Question + +No open question. + +--- diff --git a/docs/agent-guide/051-logseq-cli-sync-upload-fix.md b/docs/agent-guide/051-logseq-cli-sync-upload-fix.md new file mode 100644 index 0000000000..e4c3c241ee --- /dev/null +++ b/docs/agent-guide/051-logseq-cli-sync-upload-fix.md @@ -0,0 +1,347 @@ +# Logseq CLI Sync Upload Fix Plan + +Goal: Fix `logseq sync upload` so a CLI-managed local graph can be uploaded successfully with the current `logseq-cli` + `db-worker-node` architecture, including the first upload of a graph that does not yet have a local `graph-id`. + +Architecture: Keep the CLI surface small and move the real fix into the worker-side sync layer so `:thread-api/db-sync-upload-graph` becomes self-sufficient: it should resolve or create the remote graph when needed, persist the resulting graph metadata locally, and then perform snapshot upload. + +Architecture: Remove the current error-swallowing behavior in worker upload so failures propagate back through `db-worker-node` and the CLI can report an actual error instead of a false success. + +Tech Stack: ClojureScript, `logseq-cli`, `db-worker-node`, `frontend.worker.sync`, db-sync HTTP endpoints, promesa, babashka test runner. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`. + +Related: Relates to `docs/cli/logseq-cli.md` and `docs/developers/desktop-db-worker-node.md`. + +## Problem statement + +With the current implementation, `logseq sync upload --graph ` is wired only as: + +- CLI `sync upload` +- `logseq.cli.command.sync/execute` +- `:thread-api/db-sync-upload-graph` +- `frontend.worker.sync/upload-graph!` + +That path assumes the local graph already has a usable remote `graph-id`. + +However, `frontend.worker.sync/upload-graph!` currently fails when either `http-base` or `graph-id` is missing: + +- `http-base` comes from CLI sync config and is present in the provided `cli.edn`. +- `graph-id` is read from local graph metadata via `get-graph-id`. +- For a fresh local graph that has never been uploaded or downloaded before, `graph-id` is typically missing. + +The desktop/UI flow does not have this problem because `frontend.handler.db_based.sync/` should: + +1. start or reuse the graph's `db-worker-node` +2. apply sync config to the worker +3. detect whether the local graph already has a remote `graph-id` +4. if missing, resolve or create the remote graph +5. persist the resolved/created graph id and sync metadata locally +6. upload the graph snapshot +7. return a real success result only when upload actually succeeds +8. return a real error with context when any step fails + +This should work both for: + +- an already-linked graph that has a local `graph-id` +- a fresh local graph being uploaded for the first time + +## Recommended design + +### Option A: Fix inside worker upload orchestration + +Preferred approach: make `frontend.worker.sync/upload-graph!` capable of ensuring remote graph identity before snapshot upload. + +This keeps the CLI simple and makes `:thread-api/db-sync-upload-graph` a complete unit of behavior. + +Suggested worker-side flow: + +1. Read local `graph-id`. +2. If present, continue with existing upload logic. +3. If absent, list remote graphs visible to the current auth context. +4. Match by canonical graph name. +5. If a matching remote graph exists, bind local graph to that `graph-id`. +6. If no matching remote graph exists, create one with the current graph name and schema version. +7. Persist returned graph metadata locally. +8. Continue snapshot upload using the resolved `graph-id`. + +This mirrors the desktop behavior conceptually while avoiding duplicated orchestration in CLI code. + +### Option B: Add CLI-specific preflight before invoking upload + +Alternative approach: keep worker upload low-level and add a new CLI-side preflight that creates or resolves the remote graph before invoking `:thread-api/db-sync-upload-graph`. + +I do not recommend this because it duplicates sync orchestration across CLI and desktop-adjacent code and makes the worker thread API less useful as a stable contract. + +## Scope + +### In scope + +- Fix first-time `sync upload` for CLI-managed graphs. +- Make worker upload propagate failures. +- Reuse existing remote graph APIs (`GET /graphs`, `POST /graphs`) instead of inventing a new cloud protocol. +- Add tests for first-time upload, already-linked upload, and failure propagation. +- Update CLI docs so `sync upload` behavior is explicit. + +### Out of scope + +- Changing the cloud snapshot upload protocol. +- Adding a new top-level CLI command for graph creation. +- Reworking websocket sync start/stop behavior. +- Large refactors unrelated to upload bootstrap. + +## Implementation plan + +### Phase 1. Add failing tests for the broken behavior + +1. Add a worker sync test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for first-time upload when local graph has no graph id. +2. Stub remote graph listing to return no match, stub graph creation to return a new id, and assert upload continues with that id. +3. Add a companion test where remote graph listing already contains the same graph name and assert upload reuses the existing graph id instead of creating a duplicate graph. +4. Add a failure test that simulates graph creation failure and asserts the promise rejects rather than resolving silently. +5. Add a failure test that simulates snapshot upload failure and asserts the CLI-visible result is an error, not success. +6. Add a CLI command execution/integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` or CLI integration tests that covers the real first-upload path, not just method wiring. + +### Phase 2. Move remote graph bootstrap into worker sync code + +7. Add a worker helper in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` to ensure a usable remote graph id before snapshot upload. +8. That helper should read the local graph name from repo name using the same canonicalization rules already used by CLI and desktop sync. +9. If local graph id exists, return it immediately. +10. If local graph id is missing, call existing remote graph list API. +11. If a graph with the same canonical name exists, reuse its `graph-id` and persist it locally. +12. If no match exists, call the existing graph creation API and persist the new `graph-id` locally. +13. Keep local metadata writes inside worker code so later sync commands (`sync start`, asset upload, etc.) see a consistent graph identity. + +### Phase 3. Reuse or extract desktop graph-creation logic + +14. Avoid leaving two divergent graph-creation implementations. +15. Extract the parts of `frontend.handler.db_based.sync/`. +3. Confirm the command returns success only after remote graph bootstrap + snapshot upload finish. +4. Run `logseq sync status --graph ` and confirm the graph now has stable remote metadata. +5. Re-run `logseq sync upload --graph ` and confirm it reuses the existing graph id. +6. Force a failing auth token and confirm the CLI returns an error instead of false success. +7. Force a server-side snapshot upload failure and confirm the CLI returns an error with useful context. + +## Acceptance criteria + +The fix is complete when all of the following are true: + +- `logseq sync upload --graph ` works for a fresh local graph with no prior `graph-id`. +- The command reuses an existing same-name remote graph when appropriate. +- Worker upload failures propagate to CLI callers and no longer appear as success. +- Desktop and CLI upload bootstrap logic no longer diverge in a risky way. +- Test coverage includes first-time upload and failure propagation. +- CLI docs describe first-time upload behavior and required config without exposing secrets. + +## Risks and mitigations + +### Risk: duplicate remote graphs + +If upload always creates a graph when local `graph-id` is missing, we may create duplicates. + +Mitigation: list remote graphs first and only create when there is no exact name match. + +### Risk: inconsistent E2EE defaults + +Fresh graph creation from CLI may accidentally create an unencrypted graph while desktop defaults to encrypted. + +Mitigation: make fresh-upload `graph-e2ee?` behavior explicit and test it. + +### Risk: hidden regressions in desktop flow + +If desktop and worker share more code, UI expectations could shift. + +Mitigation: keep only transport/bootstrap logic shared; keep UI state refresh and notifications in frontend handler code. + +## Suggested file touch list + +Primary implementation files: + +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/db_based/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` + +Primary test files: + +- `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` + +Primary docs: + +- `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` + +## Question + +No open question. The current implementation gap is clear enough to proceed with a worker-centered fix. diff --git a/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md b/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md new file mode 100644 index 0000000000..288acc2032 --- /dev/null +++ b/docs/agent-guide/052-logseq-cli-sync-upload-graph-uuid-alignment.md @@ -0,0 +1,172 @@ +# CLI Sync Upload Graph UUID Alignment Implementation Plan + +Goal: Make `logseq sync upload` persist local graph UUID metadata so CLI and web app upload flows leave the graph in the same sync-ready state. + +Architecture: Keep `:thread-api/db-sync-upload-graph` as the single upload contract used by both CLI and web app. +Architecture: Align identity persistence inside `frontend.worker.sync` so every resolved graph id is written to both client-op storage and graph KV metadata. +Architecture: Use web app graph identity persistence semantics from `frontend.handler.db_based.sync/` succeeds, local graph metadata must contain a stable `:logseq.kv/graph-uuid` value in graph KV and the same graph id in client-op storage. + +The persistence rule must be enforced by worker sync code so both CLI and web app callers inherit identical behavior. + +When upload finds graph id from client-op fallback, worker must backfill missing graph KV UUID before returning success. + +Repeated uploads must be idempotent and must not create a new graph id or rewrite to a different value unexpectedly. + +## Architecture and integration points + +```text +CLI path + logseq.cli.command.sync/execute-sync-upload + -> logseq.cli.transport/invoke POST /v1/invoke + -> frontend.worker.db_worker_node /v1/invoke + -> :thread-api/db-sync-upload-graph in frontend.worker.db_core + -> frontend.worker.sync/ frontend.worker.sync/upload-graph! + +Web app path + frontend.handler.db_based.sync/ state/ frontend.worker.sync/ frontend.worker.sync/upload-graph! + +Web app source-of-truth identity persistence reference + frontend.handler.db_based.sync/ ldb/transact! with :logseq.kv/graph-uuid +``` + +## Testing Plan + +I will add worker unit tests that fail first when upload identity resolution does not persist `:logseq.kv/graph-uuid` into the graph database. + +I will add worker unit tests that fail first for the three identity branches, which are graph id from remote create, graph id from remote name match, and graph id from client-op fallback. + +I will add regression assertions that verify the persisted graph UUID is readable through `logseq.db/get-graph-rtc-uuid` and matches `client-op/get-graph-uuid`. + +I will add a CLI-facing regression check that validates upload success is followed by graph info data containing `logseq.kv/graph-uuid` in real or staged integration coverage. + +I will run targeted tests after each micro-change, then run broad lint and test commands before final review. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Implementation plan + +### Phase 1. Add failing worker tests for UUID persistence parity. + +1. Add a failing assertion in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` for remote-create upload bootstrap to assert `ldb/get-graph-rtc-uuid` is set after `latest-remote-tx` +- `db-sync/*start-inflight-target` + +The test file would also rely heavily on: + +- `async done` +- nested `promesa` chains +- cleanup inside `p/finally` + +That structure would make it possible for a test to signal completion before all outer cleanup has fully restored shared state, especially when asynchronous callbacks or listeners are still active. + +## Proposed Changes + +### 1. Add a per-test fixture for global state reset + +A `:each` fixture in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` would reset the shared worker and db-sync atoms before and after every test. + +The fixture would reset at least: + +- `worker-state/*datascript-conns` +- `worker-state/*client-ops-conns` +- `worker-state/*db-sync-client` +- `worker-state/*db-sync-config` +- `db-sync/*repo->latest-remote-tx` +- `db-sync/*start-inflight-target` + +### 2. Review high-risk async tests + +If fixture-based isolation is not sufficient, the next step would be to tighten async structure in the tests most likely to leak state. + +Priority candidates would include: + +- upload identity tests +- snapshot download tests +- pull/ok ordering tests + +The goal would be to ensure that: + +- cleanup happens after the actual promise chain settles +- `done` is called only after cleanup boundaries are satisfied +- any listener or temporary override is fully restored before the next test starts + +## Why Production Code Is Less Likely To Be At Fault + +The current evidence would not strongly support a production db-sync bug because: + +1. the affected tests would pass when run alone +2. the failing test would change depending on suite shape +3. the observed incorrect values would match data seeded by neighboring tests +4. the expected production code path for incomplete snapshot handling already appears to raise the correct error in isolated execution + +Because of that, modifying production sync logic first would risk masking a test harness problem instead of fixing the actual instability. + +## Verification Plan + +After applying the test isolation changes, verification would include: + +1. `bb dev:test -v frontend.worker.db-sync-test` +2. `bb dev:lint-and-test` + +Success criteria would be: + +- individual failing tests still pass +- the full `frontend.worker.db-sync-test` namespace passes reliably +- the full `bb dev:lint-and-test` command no longer fails on these db-sync tests + +## Risks And Follow-ups + +If failures remain after the fixture reset, the next investigation would focus on: + +- leaked `d/listen!` listeners +- background tasks still running across tests +- `js/fetch` restoration timing +- promise chains whose cleanup is attached at the wrong level +- other shared atoms not yet covered by the fixture + +## Files + +Would modify: + +- `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/db_sync_test.cljs` + +Would not modify initially: + +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` +- `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/client_op.cljs` diff --git a/docs/agent-guide/055-logseq-cli-login-logout.md b/docs/agent-guide/055-logseq-cli-login-logout.md new file mode 100644 index 0000000000..978976febe --- /dev/null +++ b/docs/agent-guide/055-logseq-cli-login-logout.md @@ -0,0 +1,317 @@ +# Logseq CLI Login and Logout Implementation Plan + +Goal: Add `logseq login` and `logseq logout`, persist Cognito auth in `/Users/rcmerci/logseq/auth.json`, and remove `:auth-token` persistence from `/Users/rcmerci/logseq/cli.edn`. + +Architecture: The CLI will get a dedicated auth module that owns loopback OAuth login, token persistence, token refresh, and logout file cleanup. +Architecture: Sync commands will continue to pass `:auth-token` to db-sync at runtime, but that token will be resolved from `auth.json` instead of being edited through `sync config set|get|unset`. +Architecture: The flow will reuse existing Cognito constants and token refresh semantics from the frontend user code, while following the ECA browser plus localhost callback pattern for headless login. + +Tech Stack: ClojureScript, babashka.cli, Node.js HTTP server APIs, Cognito Hosted UI OAuth, Promesa, JSON file persistence. + +Related: Builds on `docs/agent-guide/047-logseq-cli-sync-command.md`, `docs/agent-guide/048-sync-download-start-reliability.md`, `docs/agent-guide/051-logseq-cli-sync-upload-fix.md`, and `docs/agent-guide/033-desktop-db-worker-node-backend.md`. + +## Problem statement + +The current CLI expects headless sync authentication to be provided as `:auth-token` inside `/Users/rcmerci/logseq/cli.edn`. + +That approach is awkward for users, leaks auth concerns into general CLI config, and does not provide a first-class login or logout flow. + +The current sync code path in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` still reads `:auth-token` directly from resolved config and writes it through `sync config set|get|unset`. + +The db-sync worker then consumes that runtime token through `worker-state/*db-sync-config` in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync.cljs` and `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/sync/crypt.cljs`. + +Logseq user management already uses AWS Cognito in the frontend app, with Cognito constants in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` and refresh-token logic in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs`. + +The ECA repository shows the closest CLI-friendly reference implementation for this feature, because it opens a browser, listens on localhost for the callback, exchanges the authorization code for tokens, and persists auth state locally. + +This plan keeps the existing db-sync runtime contract stable by continuing to inject `:auth-token` into the worker config map, while changing only how the CLI obtains and persists that token. + +I will use @planning-documents for naming, @writing-plans for task granularity, @test-driven-development for implementation order, and the current `logseq-cli` plus `db-worker-node` implementation as the baseline architecture. + +## Testing Plan + +I will add unit tests for auth file path resolution, JSON persistence, token refresh, and auth file deletion before adding implementation behavior. + +I will add command parser tests for the new `login` and `logout` commands before wiring them into the CLI. + +I will add execution tests for `login` and `logout` that verify browser launch, callback handling, Cognito token exchange, auth file writes, and logout cleanup. + +I will add sync command regression tests that fail first and verify `sync config set|get|unset` no longer accept `auth-token`. + +I will add config resolution tests that fail first and verify `resolve-config` no longer loads or persists `:auth-token` from `cli.edn`, while a new auth resolver loads the effective token from `auth.json`. + +I will add worker-facing tests that fail first and verify CLI sync runtime still injects an `:auth-token` into `worker-state/*db-sync-config` when `auth.json` exists. + +I will add integration tests that fail first and cover `login`, `logout`, and one authenticated sync command using a stubbed Cognito token exchange. + +I will run targeted tests after each slice, and I will finish with `bb dev:lint-and-test`. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Proposed CLI surface + +| Command | Purpose | Persistence effect | Notes | +|---|---|---|---| +| `logseq login` | Authenticate the current machine against Logseq cloud. | Creates or updates `/Users/rcmerci/logseq/auth.json`. | Opens browser and completes a localhost callback flow. | +| `logseq logout` | Remove locally persisted cloud auth for the CLI. | Deletes or clears `/Users/rcmerci/logseq/auth.json`. | Idempotent when the file does not exist. | +| `logseq sync config set ws-url|http-base|e2ee-password ` | Persist non-auth sync config. | Updates `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | +| `logseq sync config get ws-url|http-base|e2ee-password` | Read non-auth sync config. | Reads `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | +| `logseq sync config unset ws-url|http-base|e2ee-password` | Remove non-auth sync config. | Updates `/Users/rcmerci/logseq/cli.edn`. | `auth-token` is removed from this command. | + +The runtime db-sync config map will still contain `:auth-token` when the CLI invokes worker methods. + +Only the persistence source changes from `cli.edn` to `auth.json`. + +## Auth file design + +The new auth file will live at `/Users/rcmerci/logseq/auth.json` by default. + +The implementation should expose an internal override path for tests, but it should not add a new public CLI flag unless later required. + +The file format should be JSON rather than EDN so it is easy to inspect and delete manually. + +The file should contain enough information to refresh expired tokens without asking the user to log in again. + +A good starting shape is the following. + +```json +{ + "provider": "cognito", + "id-token": "", + "access-token": "", + "refresh-token": "", + "expires-at": 1735689600000, + "sub": "", + "email": "", + "updated-at": 1735686000000 +} +``` + +`id-token` should remain the value injected into db-sync as runtime `:auth-token`, because current CLI and worker behavior already assumes the sync token is the same value as `state/get-auth-id-token` on desktop. + +`refresh-token` is required so the CLI can refresh auth non-interactively before sync commands. + +`access-token` can be stored for parity with the existing frontend token model, even if the first CLI release does not consume it directly. + +The file should be written with restrictive permissions on Unix when feasible, and tests should verify the implementation does not fail on platforms where chmod semantics differ. + +## OAuth flow design + +The login flow should follow the ECA pattern more than the current browser app pattern. + +The CLI should start a temporary localhost callback server, generate a PKCE verifier and challenge, build a Cognito Hosted UI authorization URL, and then open the browser. + +The callback should validate `state`, read the authorization `code`, exchange it against Cognito `/oauth2/token`, persist the resulting tokens, and print a concise success result. + +A suggested flow is shown below. + +```text +logseq login + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs + -> start localhost callback server on an ephemeral port + -> build Cognito authorize URL with PKCE and redirect_uri http://127.0.0.1:/auth/callback + -> open browser with system launcher, or print URL fallback + -> receive code on callback server + -> POST code exchange to https:///oauth2/token + -> persist /Users/rcmerci/logseq/auth.json + -> future sync commands refresh token if needed and inject runtime :auth-token +``` + +The callback server should prefer an ephemeral port rather than a fixed port, because that avoids collisions during local development and test runs. + +The login result should include non-sensitive metadata such as email, subject, auth file path, and whether the token was freshly created or refreshed. + +The CLI should not print raw tokens in human output. + +## Refresh and runtime token resolution + +The current worker code does not refresh tokens in CLI-owned node mode. + +`frontend.worker.sync//oauth2/token` using `grant_type=refresh_token` and `client_id`. + +`/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` should stop reading `:auth-token` from resolved config and instead call the new auth helper when building the sync runtime config sent through `:thread-api/set-db-sync-config`. + +If no valid auth file exists, authenticated sync commands should return a dedicated error with a hint such as `Run logseq login first.`. + +## Implementation plan + +### Phase 1. Add failing tests for auth file helpers. + +1. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/auth_test.cljs`. +2. Add a failing test that the default auth path resolves to `/Users/rcmerci/logseq/auth.json`. +3. Add a failing test that writing auth data creates the parent directory when missing. +4. Add a failing test that reading a missing auth file returns `nil` instead of throwing. +5. Add a failing test that deleting auth data is idempotent when the file does not exist. +6. Add a failing test that expired `id-token` plus valid `refresh-token` triggers refresh and persists updated JSON. +7. Add a failing test that malformed JSON returns a stable `invalid-auth-file` error code. +8. Run `bb dev:test -v logseq.cli.auth-test` and confirm only auth helper tests fail. + +### Phase 2. Add failing parser and help tests for `login` and `logout`. + +9. Update `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` with a failing assertion that top-level help lists `login` and `logout`. +10. Add a failing assertion that `logseq login --help` and `logseq logout --help` produce command-specific help. +11. Add a failing assertion that `sync config set|get|unset` help no longer mentions `auth-token`. +12. Run `bb dev:test -v logseq.cli.commands-test` and confirm the failures reflect missing auth command wiring and old sync config text. + +### Phase 3. Add failing command execution tests for auth commands. + +13. Create `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/auth_test.cljs`. +14. Add a failing test that `login` starts a callback server and opens the browser with a Cognito authorize URL. +15. Add a failing test that `login` validates `state` before exchanging the authorization code. +16. Add a failing test that a successful code exchange writes `/Users/rcmerci/logseq/auth.json` through the auth persistence helper. +17. Add a failing test that `logout` removes the auth file and returns success when the file existed. +18. Add a failing test that `logout` still succeeds when the auth file was already absent. +19. Add a failing test that `login` returns a timeout error when no browser callback arrives. +20. Run `bb dev:test -v logseq.cli.command.auth-test` and confirm the tests fail for missing implementation only. + +### Phase 4. Implement auth helpers and OAuth plumbing. + +21. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs` for auth path resolution, JSON read and write helpers, token expiry checks, refresh logic, and auth file deletion. +22. Add `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs` for command entries, action builders, and command execution. +23. Implement PKCE helpers and loopback callback server support inside `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs`, unless the file becomes too large, in which case split transport helpers into `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth_oauth.cljs`. +24. Build the Cognito authorize URL using constants from `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs` so the CLI and app share the same cloud environment. +25. Implement the token exchange POST against `https:///oauth2/token` and map the response into the `auth.json` shape. +26. Implement refresh-token exchange using the same Cognito domain and client id. +27. Add best-effort browser launching for macOS and Linux, and always print the URL so the flow remains usable when auto-open fails. +28. Run `bb dev:test -v logseq.cli.auth-test` and `bb dev:test -v logseq.cli.command.auth-test` until green. + +### Phase 5. Wire `login` and `logout` into the CLI parser and help output. + +29. Register `auth-command/entries` in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`. +30. Extend `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs` so top-level summaries include `login` and `logout` in a dedicated auth section or the existing management section. +31. Extend action building and execute dispatch in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs` to route `:login` and `:logout`. +32. Re-run `bb dev:test -v logseq.cli.commands-test` until the new parser and help output tests pass. + +### Phase 6. Remove `:auth-token` persistence from `cli.edn`. + +33. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs` that `resolve-config` no longer returns `:auth-token` from file config. +34. Add a failing test that `update-config!` strips `:auth-token` from updates instead of persisting it. +35. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` so file-backed config excludes `:auth-token` while still preserving `ws-url`, `http-base`, `e2ee-password`, graph selection, and output settings. +36. Update any config-related tests that currently expect `:auth-token` round-tripping through `cli.edn`. +37. Run `bb dev:test -v logseq.cli.config-test` until green. + +### Phase 7. Update sync commands to resolve auth from `auth.json`. + +38. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs` that `sync config set|get|unset auth-token` is rejected as an unknown key. +39. Add a failing test that sync execution uses the auth helper to resolve a runtime `:auth-token` before invoking `:thread-api/set-db-sync-config`. +40. Add a failing test that missing auth file returns a `missing-auth` style error for authenticated sync operations such as `sync remote-graphs`. +41. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` so `config-key-map` removes `auth-token`. +42. Update sync execution to call the auth helper and merge the effective runtime token into the in-memory sync config map. +43. Update human-readable hints that currently say `Set sync config keys (ws-url/http-base/auth-token)` so they instead mention login plus the remaining sync config keys. +44. Run `bb dev:test -v logseq.cli.command.sync-test` until green. + +### Phase 8. Keep worker behavior stable while clarifying CLI ownership of auth. + +45. Add a failing regression test in `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` that CLI node mode still prefers the runtime `:auth-token` provided by sync config over renderer state. +46. Decide whether an additional worker test is needed to prove no worker-side changes are required beyond existing runtime behavior. +47. Make only the smallest worker change necessary, and prefer no production worker changes if CLI runtime injection remains sufficient. +48. Run `bb dev:test -v frontend.worker.sync.crypt-test` and any related worker namespace tests. + +### Phase 9. Update output formatting and docs. + +49. Add failing tests in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs` for human output of `login` and `logout`. +50. Remove or rewrite any format tests that currently mention `sync config get auth-token` redaction. +51. Update `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/format.cljs` so auth command output reports file path, email, and status without printing tokens. +52. Update `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to document `login`, `logout`, `auth.json`, and the reduced `sync config` key set. +53. Add one short troubleshooting section telling users to delete `/Users/rcmerci/logseq/auth.json` or run `logseq logout` when local auth becomes invalid. +54. Run `bb dev:test -v logseq.cli.format-test` after formatter updates. + +### Phase 10. Add integration coverage and perform final verification. + +55. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that simulates a successful `login` callback and verifies `auth.json` contents. +56. Add a failing integration test that `logout` removes the auth file. +57. Add a failing integration test that an authenticated sync command reads `auth.json`, refreshes if needed, and sends runtime `:auth-token` to the worker. +58. Stub browser opening and Cognito HTTP responses so the integration tests remain deterministic and offline. +59. Run `bb dev:test -v logseq.cli.integration-test` until the new coverage is green. +60. Run `bb dev:lint-and-test` from `/Users/rcmerci/gh-repos/logseq` and confirm exit code `0`. + +## File map + +| Area | Files to update or add | Reason | +|---|---|---| +| Auth persistence and OAuth flow | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth.cljs`, optionally `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/auth_oauth.cljs` | Own `auth.json`, login callback server, PKCE, refresh, and logout deletion. | +| CLI command wiring | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/commands.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/core.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/auth.cljs` | Expose `login` and `logout` and update help text. | +| Sync runtime auth | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/sync.cljs` | Replace file-backed `:auth-token` lookup with auth helper resolution. | +| Config cleanup | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/config.cljs` | Stop round-tripping `:auth-token` in `cli.edn`. | +| Cloud constants reuse | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/config.cljs`, `/Users/rcmerci/gh-repos/logseq/src/main/frontend/handler/user.cljs` | Reference existing Cognito endpoints and refresh semantics. | +| Tests | `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/auth_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/auth_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/config_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/command/sync_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/format_test.cljs`, `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`, and possibly `/Users/rcmerci/gh-repos/logseq/src/test/frontend/worker/sync/crypt_test.cljs` | Cover behavior before implementation. | +| User-facing docs | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` | Explain the new auth workflow and removal of `auth-token` from sync config commands. | + +## Edge cases and failure handling + +`logseq login` must fail cleanly when the localhost callback port cannot be opened. + +`logseq login` must reject callbacks with mismatched `state` or missing `code`. + +`logseq login` must surface Cognito code-exchange failures without writing a partial auth file. + +`logseq logout` must succeed even when `/Users/rcmerci/logseq/auth.json` is already missing. + +Malformed or partially written `auth.json` must produce a deterministic error code and an actionable hint rather than a generic JSON parse exception. + +Expired `id-token` with a valid `refresh-token` should auto-refresh before sync commands instead of forcing an immediate re-login. + +Expired `id-token` with an invalid or missing `refresh-token` should fail with a hint to run `logseq login`. + +`sync start`, `sync remote-graphs`, `sync upload`, `sync download`, `sync ensure-keys`, and `sync grant-access` should all use the same auth resolution path so behavior is consistent. + +The CLI must not print raw JWTs or refresh tokens in human output, logs, or test snapshots. + +The implementation should preserve the current worker contract by continuing to pass `:auth-token` in runtime sync config, because changing the worker key name would expand scope without user benefit. + +## Verification commands + +| Command | Expected outcome | +|---|---| +| `bb dev:test -v logseq.cli.auth-test` | Auth helper tests pass. | +| `bb dev:test -v logseq.cli.command.auth-test` | Login and logout command tests pass. | +| `bb dev:test -v logseq.cli.commands-test` | Help output includes `login` and `logout`, and sync config no longer mentions `auth-token`. | +| `bb dev:test -v logseq.cli.config-test` | `cli.edn` no longer persists `:auth-token`. | +| `bb dev:test -v logseq.cli.command.sync-test` | Sync command tests pass with auth resolved from `auth.json`. | +| `bb dev:test -v logseq.cli.integration-test` | End-to-end auth flow tests pass with stubs. | +| `bb dev:lint-and-test` | Full repo lint and test suite passes with exit code `0`. | +| `node ./dist/logseq.js login` | Opens browser or prints login URL, then writes `/Users/rcmerci/logseq/auth.json` after callback. | +| `node ./dist/logseq.js logout` | Removes local auth state and reports success. | +| `node ./dist/logseq.js sync remote-graphs --output json` | Reads auth from `auth.json` and returns remote graphs without requiring `sync config set auth-token`. | + +## Testing Details + +The tests focus on externally observable behavior, including CLI help text, browser launch requests, callback validation, auth file contents, refresh-on-expiry behavior, sync runtime payloads, and user-facing error hints. + +The tests should not assert private helper structure unless that structure is itself the behavior contract, such as the persisted JSON keys or the generated Cognito callback URL. + +## Implementation Details + +- Keep `auth-token` as an in-memory db-sync runtime key, but remove it from persistent `cli.edn` config and from `sync config` command parsing. +- Add a dedicated CLI auth module instead of scattering login and refresh logic across `sync.cljs` and `config.cljs`. +- Reuse Cognito constants from `frontend.config` so the CLI always targets the same environment as the app. +- Reuse the refresh grant semantics from `frontend.handler.user.cljs` instead of inventing a second refresh protocol. +- Prefer an ephemeral localhost callback port and PKCE-based authorization code flow. +- Always print a copyable login URL even when automatic browser opening succeeds. +- Make `logout` local and idempotent first, and treat remote Cognito session invalidation as optional future scope. +- Keep the worker contract stable and prefer solving expiration entirely in the CLI auth module. +- Add an internal test-only auth path override rather than a new public CLI flag. +- Update user-facing docs and error hints so `Run logseq login first.` becomes the primary recovery path. + +## Decision + +The implementation will use a loopback redirect URI such as `http://127.0.0.1:/auth/callback`. + +If the current Cognito app client does not yet allow that redirect URI pattern, updating the Cognito app-client configuration is part of the implementation prerequisite rather than a reason to change the CLI design. + +`logout` will only clear `/Users/rcmerci/logseq/auth.json` in the first release. + +Best-effort browser sign-out against the Cognito Hosted UI logout endpoint is explicitly out of scope for this iteration. + +`auth.json` will persist `id-token`, `access-token`, and `refresh-token`, plus non-sensitive metadata such as `expires-at`, `sub`, `email`, and `updated-at`. + +This keeps the first CLI implementation aligned with the existing frontend token model while still using `id-token` as the runtime `:auth-token` injected into db-sync. + +--- diff --git a/docs/agent-guide/056-graph-name-dir-encoding-alignment.md b/docs/agent-guide/056-graph-name-dir-encoding-alignment.md new file mode 100644 index 0000000000..b31aac8254 --- /dev/null +++ b/docs/agent-guide/056-graph-name-dir-encoding-alignment.md @@ -0,0 +1,377 @@ +# Align graph dir encoding between logseq-cli and desktop app + +## Summary + +Align `logseq-cli`, `db-worker-node`, and desktop app handling of `graph dir` / `graph-name` so special characters are encoded and decoded with one shared, reversible contract. + +The authoritative contract would be the existing `encode-graph-dir-name` / `decode-graph-dir-name` pair in `src/main/frontend/worker_common/util.cljc`, which is already used by `db-worker-node` and `logseq-cli` server-side graph directory resolution. + +This plan keeps user-facing graph names unchanged and only aligns their on-disk directory representation. + +## Background + +Current code paths do not agree on how a graph name maps to a graph directory on disk: + +- `db-worker-node` and `logseq-cli` server/runtime paths use a reversible graph-dir encoding. +- desktop app contains paths that join the raw graph name directly into a filesystem path. +- some Electron and CLI-adjacent helpers still use lossy `sanitize-db-name` behavior. +- shared graph discovery still contains legacy decoding logic for older naming conventions, but not the current reversible encoding. + +This mismatch becomes visible when graph names contain special characters such as `/`, `:`, `%`, `~`, or spaces. + +## Goals + +- Use one shared graph-dir encoding/decoding contract across CLI and desktop app. +- Preserve current user-facing graph-name semantics. +- Keep `logseq_db_` prefix canonicalization separate from graph-dir encoding. +- Define compatibility behavior for legacy graph directory names. +- Add tests that cover special-character graph names across all affected entry points. + +## Non-goals + +- Redesign the user-visible graph naming model. +- Change the existing `logseq_db_` display normalization rules. +- Remove all legacy compatibility in one step without an explicit migration strategy. + +## Current behavior + +### Shared reversible encoding already exists + +Authoritative implementation today: + +- `src/main/frontend/worker_common/util.cljc` + - `encode-graph-dir-name` + - `decode-graph-dir-name` + +Current behavior: + +1. `encodeURIComponent` is applied. +2. literal `~` is rewritten to `%7E`. +3. `%` is rewritten to `~`. +4. decoding reverses `~ -> %` and then applies `decodeURIComponent`. + +This gives a reversible filesystem-safe directory key without `/` or `\\` path separators. + +### db-worker-node follows the shared contract + +Relevant files: + +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/frontend/worker/platform/node.cljs` +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/graph_dir.cljs` + +Current behavior: + +- repo identity strips one leading `logseq_db_` to produce a graph-dir key. +- graph-dir key is encoded with `encode-graph-dir-name`. +- list-graphs decodes on-disk directory names back to graph-dir keys. +- worker log paths and lock paths are stored under the encoded graph directory. + +### CLI is partially aligned + +Relevant files: + +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/core.cljs` +- `src/main/logseq/cli/command/graph.cljs` +- `src/main/logseq/cli/common.cljs` +- `deps/cli/src/logseq/cli/common/graph.cljs` +- `deps/cli/src/logseq/cli/util.cljs` + +Current behavior: + +- `cli.server` already uses the same canonical graph-dir path contract as `db-worker-node`. +- graph display/input normalization strips or restores one `logseq_db_` prefix as needed. +- `unlink-graph!` still derives directory names with `sanitize-db-name`, which is lossy. +- shared discovery in `deps/cli` still decodes only older directory naming patterns such as `++` and `+3A+`. + +### Desktop app is not aligned + +Relevant files: + +- `src/electron/electron/utils.cljs` +- `src/electron/electron/db.cljs` +- `src/electron/electron/handler.cljs` +- `src/electron/electron/url.cljs` +- `src/main/frontend/config.cljs` + +Current behavior: + +- `electron.utils/get-graph-dir` joins the raw graph name into the graph path after db-prefix stripping. +- if the graph name contains `/`, the resulting path becomes nested directories. +- `electron.db` still uses `sanitize-db-name` in some db path creation logic. +- frontend local-dir helpers also treat graph name as a raw path segment. + +## Problem statement + +The same logical graph name can map to different on-disk paths depending on which subsystem touches it: + +- reversible encoded path in `db-worker-node` +- raw path join in Electron/frontend +- lossy underscore replacement in sanitize-based helpers +- legacy decode-only behavior in shared graph discovery + +As a result: + +- a graph may be listable but not removable +- a graph may be resolvable in CLI but not in desktop app +- a graph name containing `/` may accidentally create path nesting in one flow but not another +- existing tests do not enforce cross-subsystem parity + +## Proposed contract + +### 1. Separate graph identity from graph directory representation + +The plan would explicitly distinguish: + +- **graph-name / repo**: user-facing identifier, subject to existing `logseq_db_` canonicalization rules +- **graph-dir key**: graph-name with exactly one leading db prefix stripped +- **encoded graph-dir**: on-disk directory name produced only by `encode-graph-dir-name` + +This separation would make it clear that special-character handling belongs to the graph-dir layer, not the user-facing name layer. + +### 2. Make the db-worker-node contract authoritative + +The repository would standardize on: + +- `repo -> graph-dir key`: strip one leading `logseq_db_` +- `graph-dir key -> encoded graph-dir`: `encode-graph-dir-name` +- `encoded graph-dir -> graph-dir key`: `decode-graph-dir-name` + +Any code path that needs an on-disk db graph directory would route through this contract rather than reimplementing path logic. + +### 3. Keep user-visible graph names unchanged + +The plan would preserve current user-visible behavior: + +- CLI graph names remain prefix-free for display and config storage where already intended. +- desktop app continues to display logical graph names, not encoded directory names. +- URL-level graph identification continues to resolve to logical graph names, not on-disk encoded names. + +## Proposed code changes + +### A. Consolidate path-authoritative helpers + +Add or reuse one shared helper layer for: + +- converting repo to graph-dir key +- converting graph-dir key to encoded graph directory +- converting repo directly to on-disk graph directory path + +Target files likely involved: + +- `src/main/frontend/worker/graph_dir.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/electron/electron/utils.cljs` +- `src/main/frontend/config.cljs` +- `deps/cli/src/logseq/cli/util.cljs` + +Expected outcome: + +- no raw path join for logical graph names in path-authoritative code +- no duplicate graph-dir encoding implementations + +### B. Align Electron graph-dir resolution + +Replace raw graph path derivation in Electron with the shared encoded graph-dir contract. + +Target files: + +- `src/electron/electron/utils.cljs` +- `src/electron/electron/handler.cljs` +- `src/electron/electron/db.cljs` + +Expected outcome: + +- desktop app resolves the same on-disk graph dir as `db-worker-node` +- graph names containing `/`, `:`, `%`, `~`, or spaces behave predictably +- `sanitize-db-name` is no longer used for authoritative db graph-dir mapping + +### C. Align CLI remove/unlink behavior + +Update CLI removal/unlink flows to resolve graph directories via the same encoded contract used by list/start/lock behavior. + +Target file: + +- `src/main/logseq/cli/common.cljs` + +Expected outcome: + +- a graph that can be listed or switched to can also be removed through the same path mapping + +### D. Align shared graph discovery + +Update shared discovery helpers so current encoded graph dirs are decoded correctly, while preserving deliberate support for legacy names where needed. + +Target file: + +- `deps/cli/src/logseq/cli/common/graph.cljs` + +Expected outcome: + +- desktop/CLI discovery would recognize encoded graph dirs produced by current db-worker-node logic +- legacy decode branches would be explicitly documented as compatibility behavior + +### E. Audit frontend local-dir helpers + +Review helpers that expose graph-related directories to ensure they are either: + +- display-only helpers, or +- path-authoritative helpers using the shared encoded contract + +Target file: + +- `src/main/frontend/config.cljs` + +Expected outcome: + +- no ambiguous helper remains that appears safe for filesystem use while still using raw graph names + +## Compatibility and migration + +This plan should explicitly decide how to handle already-existing graph directories created by older logic. + +### Option 1: Read legacy names, write canonical encoded names + +Behavior: + +- discovery accepts legacy directory names and current encoded names +- all newly created or rewritten paths use the canonical encoded form +- optional one-time migration may rename legacy directories + +Pros: + +- safer rollout +- less risk of immediately losing access to existing graphs + +Cons: + +- mixed formats may coexist temporarily + +### Option 2: Auto-migrate on access + +Behavior: + +- when a legacy graph directory is detected, code renames it to the canonical encoded path before continuing + +Pros: + +- converges quickly to one format + +Cons: + +- higher operational risk +- rename behavior must be designed carefully for active workers and lock files + +### Option 3: Strict cutover + +Behavior: + +- only encoded graph dirs are supported after the change + +Pros: + +- simplest long-term contract + +Cons: + +- too risky without explicit migration tooling + +### Recommended direction + +Prefer **Option 1** for the first rollout: + +- read compatibility for legacy directory names +- canonical writes to encoded graph dirs +- add explicit migration follow-up only after parity tests pass + +## Test plan + +### Unit tests + +Extend or add tests for: + +- `src/test/frontend/worker/worker_common_util_test.cljs` +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `src/test/logseq/cli/common/graph_test.cljs` +- Electron-specific tests if available for graph-dir resolution + +### Special-character test matrix + +All subsystems should use the same examples: + +- `foo/bar` +- `a:b` +- `space name` +- `100% legit` +- `til~de` +- `mix/of:many %chars~here` + +### Behavior to verify + +1. encode/decode roundtrip is lossless +2. CLI list-graphs returns the same logical graph name that was encoded on disk +3. CLI switch/remove resolve the same graph directory +4. desktop app resolves the same graph directory as CLI/db-worker-node +5. graph names remain user-visible without encoded substitutions +6. legacy discovery behavior remains intentional and documented + +### Missing coverage today + +The repository currently appears to lack end-to-end parity tests for: + +- CLI create/switch/remove with special-character graph names +- Electron graph-name -> graph-dir resolution with special characters +- desktop and CLI agreement on one on-disk graph directory for the same logical graph + +## Rollout sequence + +1. Make the shared graph-dir contract explicit in code and docs. +2. Update Electron path-authoritative helpers to use encoded graph dirs. +3. Update CLI unlink/remove behavior to use the same mapping. +4. Update shared graph discovery for encoded graph dirs and legacy compatibility. +5. Add parity tests across worker, CLI, and desktop-related helpers. +6. Evaluate whether legacy directory migration should be a separate follow-up. + +## Risks + +- Existing graphs may already exist under lossy or raw directory naming rules. +- Desktop-specific compatibility code may rely on current path layout assumptions. +- URL/deeplink flows may resolve graph identifiers separately from filesystem mapping and should not accidentally expose encoded names to users. +- Removing `sanitize-db-name` from authoritative paths may surface hidden assumptions in older db bootstrap code. + +## Open questions + +1. Should legacy raw/sanitized graph directories remain writable, or only readable? +2. Should migration happen automatically, manually, or in a later dedicated change? +3. Which helper should become the single exported entry point for graph-name -> on-disk graph-dir path resolution? +4. Should `docs/cli/logseq-cli.md` be updated in the same change to clarify that on-disk graph directories are encoded, not always literal graph names? + +## Expected files to change in implementation + +Likely implementation targets: + +- `src/main/frontend/worker_common/util.cljc` +- `src/main/frontend/worker/graph_dir.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/common.cljs` +- `deps/cli/src/logseq/cli/common/graph.cljs` +- `deps/cli/src/logseq/cli/util.cljs` +- `src/electron/electron/utils.cljs` +- `src/electron/electron/db.cljs` +- `src/electron/electron/handler.cljs` +- `src/main/frontend/config.cljs` +- related tests under `src/test/` + +## Acceptance criteria + +This plan would be complete when: + +- one shared graph-dir encoding contract is identified as authoritative +- all affected subsystems and files are enumerated +- compatibility strategy for legacy graph directories is documented +- a concrete test matrix for special-character graph names is defined +- the plan preserves current user-facing graph-name semantics while aligning on-disk graph-dir behavior diff --git a/docs/agent-guide/057-cli-sync-download-realtime-progress.md b/docs/agent-guide/057-cli-sync-download-realtime-progress.md new file mode 100644 index 0000000000..83ad162da3 --- /dev/null +++ b/docs/agent-guide/057-cli-sync-download-realtime-progress.md @@ -0,0 +1,436 @@ +# 057: Make `logseq-cli sync download` stream realtime progress and support long-running downloads + +## Summary + +This document defines an execution-ready implementation plan for improving `logseq-cli sync download` for large graphs. + +Today, the CLI waits for a single `/v1/invoke` response and uses a short default timeout. This creates two bad outcomes for large graph downloads: + +1. the terminal is silent while the snapshot is downloading and importing, and +2. healthy long-running work may appear to fail because the CLI request timeout is too short. + +The implementation should reuse the existing sync log/event infrastructure already used by the app, instead of creating a separate CLI-only progress system. + +## Decision records (confirmed) + +The following decisions are fixed for implementation: + +1. `sync download` timeout strategy uses a command-level long-running policy (not a global timeout increase). +2. Progress logs are printed to `stdout`. +3. Add `--progress` option for `sync download`. +4. `--progress` defaults to `true` for human output. +5. For structured output modes (for example `json` / `edn`), progress is automatically disabled unless the user explicitly passes `--progress true`. +6. Download progress log emission should be unified into shared worker/sync code (no duplicate app-only emission path for the same milestones). + +## Problem statement + +`logseq-cli sync download` already uses the same db-worker-node and worker sync stack that powers app sync behavior, but it does not currently reuse the realtime event stream. + +Relevant current-state facts: + +- CLI sends a request to db-worker-node over `/v1/invoke` and waits for one final response. +- db-worker-node already exposes `/v1/events` as an SSE event stream. +- worker/app sync logic already emits `:rtc-log` events. +- app/desktop already displays those logs. +- CLI transport defaults to a `10000` ms timeout, which is not appropriate for a full graph snapshot download/import. + +The implementation goal is therefore not to invent progress tracking from scratch. The goal is to make CLI consume and display the same progress events, and to change timeout behavior so long-running downloads can complete. + +## Goals + +- Show realtime progress during `logseq-cli sync download`. +- Reuse the existing `:rtc-log` event model and db-worker-node SSE stream. +- Unify download progress log emission into shared worker/sync code for the full download flow. +- Add `--progress` to `sync download` with default behavior enabled for human output. +- Print progress to `stdout`. +- Automatically disable progress for structured output modes unless `--progress true` is explicitly set. +- Prevent large graph downloads from failing under the generic short CLI timeout path. +- Preserve the final command result semantics and existing validation behavior. + +## Non-goals + +- Redesign the sync protocol. +- Replace the app RTC log UI. +- Introduce a polling-based progress API. +- Change the existing success/failure meaning of `sync download`. +- Add a CLI-only event schema that diverges from app behavior. + +## Current implementation map + +### CLI entrypoints + +- `src/main/logseq/cli/command/sync.cljs` + - defines `sync download` + - resolves auth, starts db-worker-node, validates empty DB, invokes `:thread-api/db-sync-download-graph-by-id` +- `src/main/logseq/cli/transport.cljs` + - sends HTTP requests to db-worker-node `/v1/invoke` + - currently applies default timeout behavior +- `src/main/logseq/cli/format.cljs` + - formats final CLI output +- `src/main/logseq/cli/config.cljs` + - defines CLI defaults, including timeout +- `src/main/logseq/cli/command/core.cljs` + - defines global CLI options including `--timeout-ms` + +### db-worker-node / worker entrypoints + +- `src/main/frontend/worker/db_worker_node.cljs` + - serves `/v1/invoke` + - serves `/v1/events` as SSE +- `src/main/frontend/worker/db_core.cljs` + - handles `:thread-api/db-sync-download-graph-by-id` + - already emits import/decrypt/save-stage logs +- `src/main/frontend/worker/sync.cljs` + - performs remote snapshot download and sync data fetch +- `src/main/frontend/worker/sync/log_and_state.cljs` + - publishes `:rtc-log` events to connected clients + +### App-side log consumers and app-specific log emission + +- `src/main/frontend/handler/db_based/sync.cljs` + - currently emits useful early download messages such as: + - `Preparing graph snapshot download` + - `Start downloading graph snapshot, file size: ...` + - `Graph snapshot downloaded` +- `src/main/frontend/handler/worker.cljs` +- `src/main/frontend/handler/events.cljs` +- `src/main/frontend/handler/db_based/rtc_flows.cljs` +- `src/main/frontend/components/rtc/indicator.cljs` + +These app-side files are useful references because they show the desired end-user progress semantics. However, the actual shared event source should live in worker/shared sync code, not in app-only UI handlers. + +## Design constraints + +1. **One shared progress model** + - CLI and app should consume the same logical progress events. + - Avoid a separate CLI-only progress protocol. + +2. **Invoke result remains authoritative** + - Realtime logs improve visibility but must not replace final command success/failure semantics. + +3. **Long-running timeout behavior must be command-aware** + - `sync download` should not rely on the same timeout assumptions as short metadata requests. + +4. **Output compatibility matters** + - Streaming progress should not break final human or machine-readable command results. + +## Proposed implementation + +The implementation should be delivered in four phases. + +--- + +## Phase 1: Make timeout handling explicit for long-running `sync download` + +### Objective + +Remove the dependency on the generic short CLI request timeout for the long-running download/import invoke path. + +### Files + +- `src/main/logseq/cli/command/sync.cljs` +- `src/main/logseq/cli/transport.cljs` +- `src/main/logseq/cli/config.cljs` +- `src/main/logseq/cli/command/core.cljs` + +### Tasks + +1. Trace exactly how `:timeout-ms` flows from CLI options/config into `transport/invoke` for `sync download`. +2. Introduce command-specific timeout behavior for `sync download`. +3. Keep the timeout policy explicit in code, rather than relying on an accidental global default. +4. Preserve existing timeout behavior for short non-download CLI commands unless intentionally changed. + +### Recommended implementation direction + +Prefer a command-specific long-task timeout path over raising the global default for all CLI traffic. + +Good options include: + +- passing a much larger timeout only for the final `db-sync-download-graph-by-id` invoke, or +- introducing a dedicated long-running request helper for commands that are expected to take a long time. + +Do **not** solve this by silently changing all CLI requests to use a large default timeout. + +### Acceptance criteria + +- `sync download` no longer depends on the generic `10000` ms timeout for the full download/import request. +- Other CLI requests keep their current short-request behavior unless explicitly updated. +- The effective timeout policy is easy to understand from the command implementation. + +### Verification + +- Unit test or integration test proving that `sync download` can run longer than the old short timeout path. +- Regression test showing short requests are unchanged. + +--- + +## Phase 2: Unify all download-progress logs into shared worker/sync code + +### Objective + +Ensure the full download flow emits shared progress events from the same worker/sync path used by both app and CLI. + +### Files + +- `src/main/frontend/worker/sync.cljs` +- `src/main/frontend/worker/db_core.cljs` +- `src/main/frontend/worker/sync/log_and_state.cljs` +- `src/main/frontend/handler/db_based/sync.cljs` + +### Tasks + +1. Inventory all download-progress log emissions related to `sync download` across app handlers and worker code. +2. Move or extract all shared milestones into worker/shared sync code where the remote snapshot and import/decrypt flow actually executes. +3. Reuse the existing `:rtc.log/download` event family and existing `sub-type` semantics wherever possible. +4. Remove duplicate app-only emission for shared milestones, and keep app handlers as consumers of shared events. +5. Keep milestone wording stable enough to avoid unnecessary UI regression in existing app consumers. + +### Required shared milestones + +The worker/shared path should emit at least these human-meaningful milestones: + +- preparing snapshot download, +- snapshot download started, including file size when available, +- snapshot download completed, +- saving/import/decrypt progress, +- graph ready / download complete. + +The final wording may differ, but the milestones must cover both: + +- network download progress stages, and +- local import/decrypt stages. + +### Acceptance criteria + +- Shared download milestones are emitted from worker/sync code across the full flow (download + import/decrypt). +- There is no competing app-only emission path for the same shared milestones. +- CLI-triggered `db-sync-download-graph-by-id` produces the same family of download progress events as app-triggered flows. +- Existing app consumers can still display download progress using the shared event source. + +### Verification + +- Worker-level or sync-level tests for emitted `:rtc-log` events. +- Manual verification in app that download progress still appears after the refactor. + +--- + +## Phase 3: Add CLI support for db-worker-node SSE event consumption + +### Objective + +Allow the CLI to subscribe to `/v1/events` while a long-running invoke is in progress. + +### Files + +- `src/main/logseq/cli/transport.cljs` +- `src/main/logseq/cli/command/sync.cljs` +- optionally a new helper namespace under `src/main/logseq/cli/` if event-stream logic should be isolated +- reference implementation: + - `src/main/frontend/persist_db/remote.cljs` + - `src/main/frontend/persist_db/node.cljs` + +### Tasks + +1. Add a lightweight CLI-side SSE client for db-worker-node `/v1/events`. +2. Decode incoming event payloads into the same shape consumed elsewhere in the codebase. +3. Support subscription lifecycle management: + - connect before starting the long-running invoke, + - receive events during the invoke, + - close cleanly on success, error, timeout, or interruption. +4. Keep the event client generic enough that future CLI commands could reuse it if needed. + +### Important behavior + +- The SSE stream is for observability, not command truth. +- If SSE disconnects, the invoke result still determines command success/failure. +- If the invoke finishes successfully, the CLI must stop listening and finalize normally. + +### Acceptance criteria + +- CLI can consume db-worker-node `/v1/events` while a command is in flight. +- Incoming `:rtc-log` events are decoded correctly. +- Subscription cleanup is reliable across success and failure paths. + +### Verification + +- Tests for event decode behavior. +- Tests or integration coverage for stream setup/cleanup. +- Manual verification against a running db-worker-node. + +--- + +## Phase 4: Render download progress in `sync download` without breaking final output + +### Objective + +Display realtime download/import progress in the terminal while preserving final result compatibility. + +### Files + +- `src/main/logseq/cli/command/sync.cljs` +- `src/main/logseq/cli/format.cljs` +- any new CLI event/render helper added in Phase 3 +- `docs/cli/logseq-cli.md` + +### Tasks + +1. Add `--progress` option to `sync download` command handling. +2. Define default behavior: progress enabled for human-oriented output mode. +3. Define structured-output behavior: progress automatically disabled for structured modes unless explicitly overridden by `--progress true`. +4. Subscribe to the worker event stream before invoking `db-sync-download-graph-by-id` when progress is enabled. +5. Filter only the relevant download log events for the active graph. +6. Render progress messages in chronological order to `stdout`. +7. Preserve the final success/failure output contract. +8. Document the new behavior in CLI docs, including mode-dependent defaults and override rules. + +### Output policy + +The implementation must make a clear separation between: + +- streaming progress lines, and +- the final command result. + +Confirmed direction: + +- stream progress to `stdout` when progress is enabled, +- add `--progress` option for `sync download`, +- default `--progress` to `true` for human-oriented output, +- automatically disable progress for structured output modes unless the user explicitly passes `--progress true`, +- keep the final result formatter responsible for terminal success/failure summary semantics. + +### Filtering policy + +The command should filter progress events using enough context to avoid printing unrelated logs. + +At minimum, filtering should consider the active graph identity. If a more precise operation-level filter is available without major complexity, prefer it. + +### Acceptance criteria + +- Running `logseq-cli sync download` with human-oriented output prints realtime progress lines to `stdout` during download/import. +- `--progress false` suppresses progress streaming. +- Structured output modes auto-disable progress unless `--progress true` is explicitly provided. +- Final command output still reflects the authoritative invoke result. +- Structured output parsing is not broken under the default mode-dependent progress behavior. + +### Verification + +- Integration test or high-confidence manual test showing visible staged output. +- Verification that final success output still matches expected formatter behavior. +- Verification that failure cases still return the correct final error. + +--- + +## Concrete execution order + +Implement in this order: + +1. **Phase 1 first** so large downloads no longer die under the short timeout path. +2. **Phase 2 second** so the CLI has a complete shared event source to consume. +3. **Phase 3 third** to add CLI event subscription infrastructure. +4. **Phase 4 last** to wire the streaming logs into `sync download` and finalize output behavior. + +Do not start Phase 4 before Phase 2 is complete, or the CLI will only show partial progress from the existing worker import stage. + +## Testing plan + +### Unit / focused tests + +Add or update tests in the most appropriate existing test namespaces for: + +- CLI timeout behavior for `sync download` +- event decoding / event subscription lifecycle +- worker/shared sync download log emission +- filtering and rendering of relevant `:rtc-log` events + +Likely test locations: + +- `src/test/logseq/cli/command/sync_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/frontend/worker/db_sync_test.cljs` +- `src/test/frontend/handler/db_based/sync_test.cljs` + +The exact namespace choices may differ depending on existing test structure, but the coverage categories above are required. + +### Manual verification checklist + +1. Start a `sync download` against a graph large enough to produce visible staged progress. +2. Confirm the terminal shows early snapshot-download messages. +3. Confirm the terminal shows import/decrypt/save progress. +4. Confirm progress lines are emitted to `stdout` in human-oriented mode. +5. Confirm `--progress false` suppresses streaming progress output. +6. Confirm structured output mode auto-disables progress by default. +7. Confirm structured output mode prints progress only when explicitly using `--progress true`. +8. Confirm successful completion still depends on the invoke result. +9. Confirm a slow download no longer fails under the old short timeout path. +10. Confirm existing app download progress still works after shared-log refactoring. +11. Confirm non-empty DB validation and other preflight failures remain unchanged. + +## Risks and open questions + +### Risk 1: app and CLI may need slightly different rendering + +The event source should be shared, but rendering may differ by client. + +Mitigation: + +- share event emission, not presentation details. +- keep worker messages human-readable enough for both app and CLI. + +### Risk 2: progress events may not uniquely identify one operation + +If multiple operations or graphs are active, CLI could print unrelated logs. + +Mitigation: + +- filter by graph identity at minimum, +- add more precise filtering only if needed and justified by actual ambiguity. + +### Risk 3: output-mode compatibility + +Streaming logs can interfere with structured output modes. + +Mitigation: + +- enforce mode-dependent default behavior (`progress` auto-off for structured output unless explicitly enabled), +- verify human and machine-readable modes explicitly. + +### Resolved decision: timeout strategy + +`sync download` uses command-level long-running timeout handling instead of a global timeout increase. + +### Open question 1 + +Should the CLI event-stream helper remain local to `sync download`, or be introduced as a reusable CLI transport helper? + +Recommendation: + +- prefer a small reusable helper if the abstraction stays simple. + +## Out-of-scope follow-ups + +The following can be considered later and are **not required** for this plan: + +- byte-level progress bars, +- richer TUI formatting, +- resumable download semantics, +- generalized event streaming for all CLI commands. + +## Definition of done + +This work is complete when all of the following are true: + +- `logseq-cli sync download` displays realtime progress while a large graph is downloading/importing. +- The progress messages come from the shared worker/app event model, not a CLI-only ad hoc implementation. +- Download-progress milestones are unified in shared worker/sync code across the full flow. +- `sync download` supports `--progress`, with mode-dependent default behavior as documented. +- In structured output modes, progress is auto-disabled unless explicitly enabled. +- Large downloads are no longer constrained by the old generic short timeout path. +- Final command success/failure semantics remain intact. +- Relevant automated tests and CLI docs are updated. + +## Recommendation + +Execute this as a shared-event refactor plus a command-specific long-request timeout change. + +The key architecture decision is to make worker/shared sync code the source of truth for download progress, then let the CLI subscribe to db-worker-node events just like the app already does. This maximizes reuse, keeps progress semantics aligned across clients, and solves the two real UX issues together: silent long-running work and premature short-timeout failures. \ No newline at end of file diff --git a/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md b/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md new file mode 100644 index 0000000000..1ab5ad6f47 --- /dev/null +++ b/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md @@ -0,0 +1,226 @@ +# 058 — db-worker-node revision and CLI server list compatibility + +## Summary + +This plan proposes three related improvements to align `db-worker-node` runtime metadata with `logseq-cli` versioning: + +1. Add `--version` support to `db-worker-node`. +2. Persist `revision` in `db-worker.lock`. +3. Extend `logseq-cli server list` with a `REVISION` column and a compatibility warning when server revision differs from local CLI revision. + +The goal is to make version drift observable and debuggable in multi-process and multi-version environments. + +--- + +## Product decisions (locked) + +1. **Mismatch rule**: if revision/commit strings are not exactly equal, it is a mismatch. + - No normalization. + - No short-hash expansion. + - No suffix stripping. + +2. **Warning scope**: mismatch warning is shown **only in human output**. + - JSON output must not include human warning text. + +--- + +## Background + +Current behavior: + +- `logseq-cli --version` already prints build metadata including `Revision`. +- `db-worker-node` accepts operational flags (`--repo`, `--data-dir`, etc.) but does not expose `--version`. +- `db-worker.lock` does not include revision metadata. +- `logseq-cli server list` shows process/state fields (`GRAPH`, `STATUS`, `HOST`, `PORT`, `PID`, `OWNER`) but not revision. + +As a result, users cannot quickly determine whether a running DB worker matches the CLI binary currently used to manage it. + +--- + +## Goals + +- Expose `db-worker-node` revision consistently with CLI version semantics. +- Persist server revision in lock metadata for later discovery. +- Surface revision in `server list` output. +- Warn users when a listed server revision does not match local CLI revision (human output only). + +## Non-goals + +- No protocol change for worker RPC endpoints. +- No forced auto-restart or auto-migration for mismatched servers. +- No hard failure on mismatch (warning only). + +--- + +## Design + +### 1) Add `--version` to db-worker-node + +#### Behavior + +- `db-worker-node --version` prints version metadata and exits with code `0`. +- It must not require `--repo`. +- Output must include at least `Revision: `. +- Formatting should follow `logseq-cli --version` style for consistency. + +#### Implementation shape + +- Extend argument parsing in `db_worker_node.cljs` to recognize `--version`. +- Add an early return branch in `main` before repo validation. +- Introduce a worker-side version namespace using `goog-define` values injected at build time. + +### 2) Add `revision` to lock file + +#### Behavior + +- Lock creation writes `:revision`. +- Lock update preserves existing revision and remains backward compatible with old lock files. +- Reading old lock files without revision continues to work. + +#### Implementation shape + +- Add revision retrieval at lock write point. +- Extend `create-lock!` payload in `db_worker_node_lock.cljs`. +- Adjust `update-lock!` merge/preserve behavior. + +### 3) Show revision in `server list` and warn on mismatch + +#### Behavior + +- Add `REVISION` column to human table output. +- `--format json` includes per-server `revision` field. +- Mismatch warning text appears only in human output. +- Missing revision should render as `-` and should not crash formatting. + +#### Mismatch rule (exact string) + +Given: + +- local CLI revision = `cli-rev` +- server lock revision = `server-rev` + +Then: + +- mismatch if `cli-rev != server-rev` (exact string compare) +- no normalization at either side + +#### Implementation shape + +- Extend server discovery result in `logseq.cli.server/list-servers` to include `:revision`. +- Compute mismatch server set in command/data path (`server list` execute path), then pass structured mismatch info to formatter. +- Render warning block only in human formatter path. + +--- + +## Build metadata strategy + +Use the same metadata source philosophy as `logseq-cli`: + +- Revision source priority: + 1. `LOGSEQ_REVISION` env + 2. git describe output + 3. fallback `"dev"` + +- Build-time source priority: + 1. `LOGSEQ_BUILD_TIME` env + 2. current UTC timestamp + +For `db-worker-node`, inject closure defines through its shadow build target, analogous to existing CLI metadata injection. + +--- + +## Implementation plan (task list) + +1. Add worker version namespace and shadow metadata wiring. +2. Add `--version` parse + main dispatch branch in `db_worker_node.cljs`. +3. Add `:revision` field in `db-worker.lock` create/update path. +4. Extend `server list` data model with revision. +5. Extend table formatter with `REVISION` column. +6. Add mismatch detection (exact string compare) and pass structured mismatch data. +7. Render mismatch warning in human output only. +8. Add/update tests for all paths. +9. Update CLI documentation. + +--- + +## Testing plan + +### Unit tests + +- `db_worker_node_test.cljs` + - Add case: `--version` recognized and exits early without repo. + - Validate output includes `Revision`. + +- `db_worker_node_lock_test.cljs` + - Create lock includes `:revision`. + - Update lock preserves existing revision. + - Legacy lock without revision remains supported. + +- `format_test.cljs` + - Human `server list` table includes `REVISION` column. + - Missing revision renders `-`. + - Mismatch warning block appears in human output for exact-string mismatch. + - No warning block when revisions are equal. + +- `server_test.cljs` (or equivalent) + - Discovery output includes revision extracted from lock. + - Mismatch set computed with exact string comparison. + +- JSON output tests + - `server list --format json` contains per-server `revision`. + - JSON output does not contain human warning text. + +### Regression checks + +- Existing `logseq-cli --version` tests remain green. +- Existing server list owner/status formatting remains unchanged aside from the added `REVISION` column and optional warning block in human mode. + +--- + +## Rollout and compatibility + +- Backward compatible with existing lock files. +- New clients can read old locks (revision absent). +- Old clients can ignore new lock key (`revision`) if parser is permissive map decode. + +--- + +## Risks and mitigations + +- **Risk:** db-worker build target missing metadata hook. + - **Mitigation:** add explicit hook wiring and test fallback value (`dev`) in test target. + +- **Risk:** warning noise in mixed environments. + - **Mitigation:** warning remains informational and human-only. + +- **Risk:** flaky tests due to dynamic metadata. + - **Mitigation:** use deterministic test defines in shadow test config. + +--- + +## Acceptance criteria + +- `db-worker-node --version` prints revision and exits `0` without requiring repo args. +- `db-worker.lock` written by worker includes `revision`. +- `logseq-cli server list` (human) shows `REVISION` column. +- `logseq-cli server list` (human) prints mismatch warning when local CLI revision string is not exactly equal to server revision string. +- `logseq-cli server list --format json` includes server revision data and does not include human warning text. +- Added/updated tests cover happy path and backward-compatibility path. + +--- + +## File scope (expected) + +- `src/dev-cljs/shadow/hooks.clj` +- `shadow-cljs.edn` +- `src/main/frontend/worker/db_worker_node.cljs` +- `src/main/frontend/worker/db_worker_node_lock.cljs` +- `src/main/frontend/worker/version.cljs` (new) +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/server.cljs` +- `src/main/logseq/cli/format.cljs` +- `src/test/frontend/worker/db_worker_node_test.cljs` +- `src/test/frontend/worker/db_worker_node_lock_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `docs/cli/logseq-cli.md` diff --git a/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md b/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md new file mode 100644 index 0000000000..3afa5cf0c9 --- /dev/null +++ b/docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md @@ -0,0 +1,237 @@ +# 059 — Add doctor warning for server and CLI revision mismatch + +## Summary + +This plan adds a new `doctor` warning when a discovered db-worker server revision does not match the local `logseq-cli` revision. + +The warning must be actionable: for each mismatched server, `doctor` should tell the user to run: + +- `logseq server restart --graph ` + +This closes the current observability gap where `server list` already exposes revision mismatch, but `doctor` does not. + +--- + +## Product decisions (locked) + +1. **Mismatch rule is exact string compare**. + - `cli-revision != server-revision` means mismatch. + - No normalization, no hash shortening, no suffix stripping. + +2. **Severity is warning, not error**. + - Revision drift is diagnosable and recoverable. + - `doctor` should still return `:status :ok` at top-level transport with `:data {:status :warning ...}` behavior, consistent with existing warning checks. + +3. **`doctor` remains fail-fast for hard preconditions**. + - If script check fails, stop immediately (current behavior). + - If data-dir check fails, stop immediately (current behavior). + - Revision mismatch check runs only when server checks are reached. + +4. **Restart guidance is per affected graph/server**. + - The warning includes one restart command per mismatched graph. + - Command form is `logseq server restart --graph `. + +5. **Structured output must remain machine-friendly**. + - JSON/EDN outputs include structured mismatch check data. + - Human output includes readable warning text and restart guidance. + +--- + +## Background + +Current implementation already has the needed revision primitives: + +- CLI revision source: `logseq.cli.version/revision`. +- Server revision source: `db-worker.lock` revision surfaced by `logseq.cli.server/list-servers`. +- `server list` already computes and warns on mismatch in human output. + +Current `doctor` checks: + +1. `db-worker-script` +2. `data-dir` +3. `running-servers` readiness + +`doctor` does **not** currently compare server revision against CLI revision. + +--- + +## Goals + +- Add revision mismatch detection to `doctor` using existing revision sources. +- Provide actionable restart guidance for each mismatched graph. +- Keep mismatch semantics identical to existing `server list` behavior. +- Preserve existing `doctor` fail-fast behavior for script/data-dir errors. + +## Non-goals + +- No automatic server restart. +- No change to daemon ownership/permission rules. +- No new db-worker RPC endpoint for version info. +- No hard failure on mismatch. + +--- + +## User-facing behavior + +### Human output + +When mismatches exist, `doctor` includes a warning check (example shape): + +- `[warning] server-revision-mismatch - 2 servers use a different revision than this CLI` +- Followed by actionable per-graph commands, e.g.: + - `Run: logseq server restart --graph graph-a` + - `Run: logseq server restart --graph graph-b` + +If graph names include spaces, command examples should quote names in guidance output. + +### JSON and EDN output + +`doctor --output json|edn` keeps structured check payload, including mismatch details. + +Suggested check payload shape: + +- `:id :server-revision-mismatch` +- `:status :warning | :ok` +- `:code :doctor-server-revision-mismatch` (when warning) +- `:cli-revision ` +- `:servers [{:repo :revision :graph }]` +- `:message ` + +--- + +## Design + +### 1) Reuse one mismatch computation path + +Today, mismatch computation is a private helper in `logseq.cli.command.server`. + +Plan: + +- Move or extract mismatch computation into a shared helper (recommended in `logseq.cli.server`), so both `server list` and `doctor` use exactly the same comparison logic. +- Keep return shape stable enough to support both command paths. + +This prevents semantic drift between `server list` and `doctor`. + +### 2) Add a dedicated doctor check + +Add a new check in `logseq.cli.command.doctor`: + +- `check-server-revision-mismatch` + +Input: + +- local CLI revision (`logseq.cli.version/revision`) +- discovered servers from `list-servers` + +Behavior: + +- No mismatches -> check status `:ok` +- Any mismatch -> check status `:warning`, include mismatch metadata and restart guidance + +### 3) Execution flow in doctor + +Keep current order and fail-fast semantics for hard errors: + +1. `check-db-worker-script` (error stops execution) +2. `check-data-dir` (error stops execution) +3. `check-running-servers` +4. `check-server-revision-mismatch` (new) + +Final `doctor` status is: + +- `:ok` when all checks are `:ok` +- `:warning` when any warning check exists (`running-servers` and/or `server-revision-mismatch`) +- `:error` only for hard failures as today + +### 4) Restart guidance formatting strategy + +Two viable options: + +- **Option A (minimal formatter change):** put guidance directly into mismatch check `:message`. +- **Option B (cleaner long-term):** add a small `format-doctor` branch for `:server-revision-mismatch` to render command lines from structured fields. + +Recommended: **Option B** if it stays small, because it preserves clean machine payload while improving human readability and escaping/quoting behavior. + +### 5) Graph naming for restart commands + +Restart command uses `--graph`, not repo id. + +Plan: + +- Derive graph name via existing repo/graph conversion helper before rendering command guidance. +- Ensure guidance is aligned with user-facing graph names shown in CLI output. + +--- + +## Implementation plan (task list) + +1. Extract revision mismatch helper from `command/server.cljs` into shared location. +2. Update `server list` command to call the shared helper (behavior unchanged). +3. Add `check-server-revision-mismatch` to `command/doctor.cljs`. +4. Extend `execute-doctor` to append the new check and compute combined warning status. +5. Add per-graph restart guidance to human doctor output. +6. Keep JSON/EDN payload structured and stable. +7. Update CLI docs for `doctor` warning behavior and remediation command. + +--- + +## Testing plan + +### Unit tests + +- `src/test/logseq/cli/command/doctor_test.cljs` + - Adds warning check when one or more servers have mismatched revisions. + - Includes restart command guidance for each mismatched graph. + - Treats missing server revision as mismatch (exact compare behavior). + - Keeps existing fail-fast behavior for script/data-dir errors. + +- `src/test/logseq/cli/command/server_test.cljs` + - Verifies `server list` still uses exact-string mismatch semantics after helper extraction. + - Confirms human mismatch metadata remains attached when expected. + +- `src/test/logseq/cli/format_test.cljs` + - Human `doctor` output includes mismatch warning and restart guidance. + - JSON/EDN doctor outputs include structured mismatch fields. + +### Regression checks + +- Existing `running-servers` warning behavior is preserved. +- Existing `server list` mismatch warning behavior is preserved. +- `doctor` top-level status semantics remain unchanged (`ok|warning|error` through current response model). + +--- + +## Risks and mitigations + +- **Risk:** Duplicate warning noise (`running-servers` + mismatch) in one doctor run. + - **Mitigation:** Keep distinct check IDs/messages so users can tell readiness vs revision drift. + +- **Risk:** Restart command may fail with owner restrictions. + - **Mitigation:** Keep guidance explicit but non-blocking; existing error hint `server-owned-by-other` remains the fallback behavior. + +- **Risk:** Missing revision in legacy lock files increases warning count. + - **Mitigation:** Treat as mismatch by design; message should clearly indicate missing server revision value. + +--- + +## Acceptance criteria + +- `logseq doctor` emits a warning check when any discovered server revision differs from local CLI revision. +- Warning includes actionable restart guidance: `logseq server restart --graph ` for each mismatched graph. +- No mismatch warning when all discovered server revisions exactly equal CLI revision. +- `server list` mismatch behavior remains consistent with `doctor` (shared comparison semantics). +- `doctor --output json|edn` includes structured mismatch check data. + +--- + +## File scope (expected) + +- `src/main/logseq/cli/command/doctor.cljs` +- `src/main/logseq/cli/command/server.cljs` +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/format.cljs` (if dedicated doctor warning rendering is added) +- `src/test/logseq/cli/command/doctor_test.cljs` +- `src/test/logseq/cli/command/server_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `docs/cli/logseq-cli.md` +- `docs/agent-guide/059-cli-doctor-server-revision-mismatch-warning.md` diff --git a/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md b/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md new file mode 100644 index 0000000000..35f753f8b9 --- /dev/null +++ b/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md @@ -0,0 +1,197 @@ +# 060 — CLI graph list: mark legacy graph-dir and provide rename command + +## Summary + +`logseq-cli graph list` currently filters out graph directories that cannot be decoded by the current canonical graph-dir encoding. This behavior makes legacy graph directories invisible to users. + +This plan proposes to: + +1. Detect non-canonical (legacy) graph directories during graph listing. +2. Mark them as `legacy` in `graph list` output. +3. Show a warning when legacy entries exist. +4. Print a suggested shell rename command from legacy dir name to current canonical encoded dir name (only when graph-name can be derived reliably). + +## Product decisions (locked) + +1. Scope is `graph list` only. +2. Human output must show: + - regular graph items (existing behavior), + - legacy marker per legacy item, + - a warning section when at least one legacy graph is found, + - one rename command suggestion per legacy graph when derivation is reliable. +3. Structured outputs (`json` / `edn`) must include legacy metadata (not human-only text). +4. Suggested command target is POSIX shell (`mv`) with safe quoting. +5. If target encoded dir already exists, output should include conflict guidance instead of a blind rename suggestion. +6. If a legacy dir cannot be decoded to a reliable graph-name, show `legacy` marker and warning only; do not output a rename command. + +## Goals + +- Make legacy graph dirs visible in `logseq-cli graph list`. +- Provide actionable migration guidance with minimal user effort. +- Keep current canonical graph listing behavior unchanged for non-legacy entries. + +## Non-goals + +- Automatic rename/migration execution. +- Reworking graph storage layout. +- Adding new CLI subcommands in this iteration. + +## Current behavior (based on code) + +- `src/main/logseq/cli/server.cljs` `list-graphs` scans graph data dirs and decodes names via canonical decode. +- Entries that fail canonical decode are dropped. +- `src/main/logseq/cli/command/graph.cljs` `execute-graph-list` passes graph names to formatter. +- `src/main/logseq/cli/format.cljs` `format-graph-list` renders graph names but has no legacy path. + +Related encoding utilities: + +- Canonical encode/decode logic: + - `deps/common/src/logseq/common/graph_dir.cljs` + - `src/main/frontend/worker/db_worker_node_lock.cljs` +- Legacy token hints still exist in browser-side logic (`+3A+`, `++`) and can be used to define migration decoding behavior. + +## Design + +### 1) Data model for graph list results + +Introduce a richer graph-list item shape from CLI server layer: + +- Canonical item: + - `{:kind :canonical :graph-name :graph-dir }` +- Legacy item: + - `{:kind :legacy :legacy-dir :legacy-graph-name :target-graph-dir :conflict? }` +- Undecodable non-canonical item: + - `{:kind :legacy-undecodable :legacy-dir :reason }` + +Notes: + +- `legacy-graph-name` should be derived using a dedicated legacy decode path (fallbacks allowed). +- `target-graph-dir` should always be computed from `legacy-graph-name` through current canonical encoding. +- `:legacy-undecodable` entries must never produce rename commands. + +### 2) Legacy detection and derivation + +At graph discovery stage (`src/main/logseq/cli/server.cljs`): + +1. Keep current canonical decode attempt. +2. If canonical decode succeeds -> canonical item. +3. If canonical decode fails -> try legacy derivation: + - legacy token replacement decode path (e.g. `+3A+ -> :`, `++ -> /`) as compatibility rule, + - URI decode fallback if applicable. +4. If derivation yields a reliable name, classify as `:legacy`. +5. If no reliable derivation is possible, classify as `:legacy-undecodable` and include warning-only entry (no rename command). + +### 3) Command layer contract + +In `src/main/logseq/cli/command/graph.cljs`: + +- `execute-graph-list` should return: + - `:data` containing canonical + legacy + undecodable legacy metadata, + - `:human` warning payload when any legacy entries exist. + +This keeps formatter logic deterministic while preserving structured output for `json` and `edn`. + +### 4) Human formatter behavior + +In `src/main/logseq/cli/format.cljs`: + +- Extend graph list rendering to show legacy marker, for example: + - `- my/old/graph [legacy]` + - `- unknown-legacy-dir [legacy]` +- Add warning block when legacy entries are present. +- For each renameable legacy item, print a shell suggestion: + - `mv '/' '/'` +- For conflict entries (`target already exists`), print explicit conflict note and no direct `mv` command. +- For undecodable legacy entries, print explicit warning and no `mv` command. + +### 5) Shell quoting + +Current CLI arg quoting helper is not sufficient for robust shell copy/paste. + +Plan: + +- Add a dedicated POSIX single-quote escaping helper for rendered shell suggestions. +- Use it only in human formatting layer. + +## Output examples (human) + +Example list section: + +- `my/new/graph` +- `my/old/graph [legacy]` +- `strange-dir-name [legacy]` + +Warning section example: + +- `Warning: 2 legacy graph directories detected.` +- `Rename suggestion:` +- `mv '/path/to/data/my++old++graph' '/path/to/data/my~2Fold~2Fgraph'` +- `Warning: cannot derive graph name for legacy dir 'strange-dir-name'; rename command is not available.` + +Conflict example: + +- `Warning: target directory already exists for legacy graph 'my/old/graph'.` +- `Please rename manually after resolving the conflict.` + +## Test plan + +### Unit tests + +1. `src/test/logseq/cli/commands_test.cljs` + - verify `graph list` command data includes legacy and undecodable legacy entries. +2. `src/test/logseq/cli/format_test.cljs` + - verify human output marker `[legacy]`. + - verify warning block appears only when legacy exists. + - verify rename command rendering and shell quoting. + - verify undecodable legacy outputs warning only and no rename command. +3. `src/test/frontend/worker/graph_dir_test.cljs` + - extend coverage for canonical encode/decode + legacy derivation helper behavior. + +### Integration tests + +1. `src/test/logseq/cli/integration_test.cljs` + - seed canonical + legacy dirs in test data dir. + - assert `graph list` behavior for human/json/edn outputs. + - assert conflict message when target encoded dir already exists. + - assert undecodable legacy case emits warning without rename command. + +## Edge cases + +- Legacy dir cannot be decoded to a graph name. +- Canonical target dir already exists. +- Graph names containing shell-sensitive characters. +- Mixed directories that are not graph dirs (must avoid false positives). + +## Implementation plan + +1. Add legacy classification utilities and detailed graph list payload in CLI server layer. +2. Adapt `graph list` command contract to pass structured legacy information to formatter and machine outputs. +3. Extend human formatter with legacy marker, warning block, and safe rename suggestions. +4. Add/adjust tests for unit + integration coverage. +5. Validate no regression for all existing `graph list` output formats. + +## Acceptance criteria + +- `graph list` shows legacy entries instead of silently hiding them. +- Human output includes explicit legacy marker and warning. +- Human output includes safe rename command for renameable entries. +- Undecodable legacy entries are clearly reported with warning only (no rename command). +- Conflict scenarios are surfaced without unsafe rename suggestions. +- JSON/EDN outputs expose legacy metadata for automation. +- Existing canonical-only behavior remains stable when no legacy entries exist. + +## Affected files (planned) + +Would modify: + +- `src/main/logseq/cli/server.cljs` +- `src/main/logseq/cli/command/graph.cljs` +- `src/main/logseq/cli/format.cljs` +- `src/test/logseq/cli/commands_test.cljs` +- `src/test/logseq/cli/format_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/frontend/worker/graph_dir_test.cljs` + +Would create: + +- `docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md` diff --git a/docs/agent-guide/061-graph-dir-space-preserve-canonical.md b/docs/agent-guide/061-graph-dir-space-preserve-canonical.md new file mode 100644 index 0000000000..ed774cd0bc --- /dev/null +++ b/docs/agent-guide/061-graph-dir-space-preserve-canonical.md @@ -0,0 +1,202 @@ +# 061 — Preserve spaces in canonical graph directory names + +## Summary + +Change canonical graph directory encoding so spaces are preserved as literal spaces. + +- Before: graph-name `GRAPH one` -> graph dir `GRAPH~20one` +- After: graph-name `GRAPH one` -> graph dir `GRAPH one` + +All other special-character encode/decode behavior remains unchanged. + +This plan is based on the current `logseq-cli` and `db-worker-node` implementation, both of which already rely on shared graph-dir encoding utilities. + +## Product decision (locked) + +1. Canonical graph dir must preserve spaces (no `~20` for spaces). +2. Existing encoding/decoding rules for non-space special characters remain unchanged. +3. Existing directories using old space encodings (for example `~20` or `%20`) are treated as legacy-compatible inputs, not the canonical write format. +4. New writes/path derivations must use the new canonical format with literal spaces. + +## Goals + +- Align `logseq-cli` and `db-worker-node` on a canonical graph-dir format that keeps spaces as spaces. +- Keep current behavior for all non-space special characters. +- Preserve compatibility for existing legacy directory names during discovery/listing. +- Keep user-facing graph-name semantics unchanged. + +## Non-goals + +- Redesign graph naming. +- Change `logseq_db_` prefix handling rules. +- Introduce automatic on-disk migration in this iteration. +- Change unrelated path or lock semantics. + +## Current behavior (relevant paths) + +### Shared encoding contract + +Authoritative helpers: + +- `deps/common/src/logseq/common/graph_dir.cljs` + - `encode-graph-dir-name` + - `decode-graph-dir-name` + - `decode-legacy-graph-dir-name` + - `repo->graph-dir-key` + - `graph-dir-key->encoded-dir-name` + +Current behavior for space: + +1. `encodeURIComponent("GRAPH one")` -> `GRAPH%20one` +2. `%` is rewritten to `~` +3. output becomes `GRAPH~20one` + +### CLI usage + +- `src/main/logseq/cli/server.cljs` + - canonical/legacy classification for graph dirs (`classify-graph-dir`, `canonical-dir-name?`) +- `src/main/logseq/cli/command/graph.cljs` + - `graph list` payload construction +- `src/main/logseq/cli/format.cljs` + - legacy warning/rename suggestion rendering + +### db-worker-node usage + +- `src/main/frontend/worker/db_worker_node_lock.cljs` + - `repo-dir`, `lock-path`, `repo->graph-dir-key` +- `src/main/frontend/worker/platform/node.cljs` + - list/db path installation flows that resolve graph dir via shared helpers +- `src/main/frontend/worker/db_core.cljs` + - storage pool naming path uses graph-dir key/encoded dir conventions + +## Proposed behavior + +### Canonical encode/decode contract + +For graph-dir encoding: + +- Preserve literal spaces in output. +- Keep current reversible encode/decode behavior for all other special characters. +- Keep `~`/`%` safety rules unchanged. + +Examples: + +- `GRAPH one` -> `GRAPH one` (changed) +- `a/b` -> `a~2Fb` (unchanged) +- `x:y` -> `x~3Ay` (unchanged) +- `100% real` -> `100~25 real` (unchanged except space stays literal) + +Decoding expectations: + +- `GRAPH one` -> `GRAPH one` +- `GRAPH~20one` -> `GRAPH one` (legacy-compatible decode still works) + +### Canonical vs legacy classification in CLI + +`graph list` canonical check should reflect the new canonical encoding: + +- Directory `GRAPH one` is canonical. +- Directory `GRAPH~20one` is non-canonical (legacy) and should be surfaced as legacy with rename guidance targeting `GRAPH one`. +- Existing `%20` legacy directories remain legacy. + +## Design details + +### 1) Shared encoder update + +Update `encode-graph-dir-name` in `deps/common/src/logseq/common/graph_dir.cljs` so space is not rewritten into `~20`. + +Implementation constraint: + +- Do not alter non-space transformation behavior. +- Keep encode/decode reversibility for previously supported special characters. + +### 2) Keep decode compatibility + +`decode-graph-dir-name` remains compatibility-friendly so both old and new directory spellings decode to the same graph-name. + +No behavioral contraction should be introduced in decoding. + +### 3) CLI classification and guidance alignment + +Because canonical encoding changes, `src/main/logseq/cli/server.cljs` classification will naturally reclassify old `~20` dirs as legacy. + +Ensure `target-graph-dir` generation and formatter output use the new canonical output containing spaces. + +### 4) db-worker-node path outputs + +No separate encoding logic should be added. + +All db-worker-node path generation must continue to route through shared helpers so new canonical behavior applies consistently to: + +- graph repo dir +- lock path +- db path +- log path + +## Test plan + +### Update existing tests + +1. `src/test/frontend/worker/worker_common_util_test.cljs` + - Update roundtrip expectations for space-containing names to canonical literal-space output. +2. `src/test/logseq/cli/common/graph_test.cljs` + - Update graph-dir decode/list expectations from `space~20name` canonical assumptions to literal-space canonical behavior. +3. `src/test/logseq/cli/server_test.cljs` + - Update canonical vs legacy expectations: + - space directory canonical + - `~20`/`%20` variants legacy where applicable +4. `src/test/logseq/cli/integration_test.cljs` + - Update integration assertions for `graph list` legacy markers and rename targets. +5. `src/test/logseq/db/common_sqlite_test.cljs` + - Update encoded-dir path assertions for graphs with spaces. +6. `src/test/logseq/cli/common_test.cljs` + - Update unlink/move expectations when graph names contain spaces. +7. `src/test/logseq/cli/format_test.cljs` + - Ensure rename suggestion target renders literal-space canonical dir. + +### Add targeted coverage (if missing) + +- Roundtrip examples that mix spaces with other special characters (for example `A B/C:D%~E`) to prove only space behavior changed. +- Explicit assertion that non-space character transformations are unchanged. + +## Rollout and compatibility + +- No immediate forced migration. +- Legacy directories remain discoverable/readable via existing decode + CLI legacy classification. +- New writes and canonical suggestions converge toward literal-space directories. + +## Risks + +1. Hidden assumptions in tests that treat `~20` as canonical for spaces. +2. Any code path bypassing shared graph-dir helpers could diverge (must be checked during implementation). +3. Rename suggestion shell formatting with spaces must remain safely quoted. + +## Acceptance criteria + +1. For graph-name `GRAPH one`, canonical graph dir is exactly `GRAPH one`. +2. Non-space special-character encoding behavior is unchanged from current behavior. +3. CLI graph listing marks old space-encoded dirs (`~20`/`%20`) as legacy where relevant. +4. db-worker-node path derivation uses the new canonical space-preserving output via shared helpers. +5. Updated tests pass and demonstrate: + - new canonical space behavior, + - unchanged non-space behavior, + - legacy compatibility visibility. + +## Affected files (planned) + +Would modify: + +- `deps/common/src/logseq/common/graph_dir.cljs` +- `src/main/logseq/cli/server.cljs` (if classification adjustments are required) +- `src/main/logseq/cli/format.cljs` (if rename guidance formatting expectations need updates) +- `src/test/frontend/worker/worker_common_util_test.cljs` +- `src/test/logseq/cli/common/graph_test.cljs` +- `src/test/logseq/cli/server_test.cljs` +- `src/test/logseq/cli/integration_test.cljs` +- `src/test/logseq/db/common_sqlite_test.cljs` +- `src/test/logseq/cli/common_test.cljs` +- `src/test/logseq/cli/format_test.cljs` + +Would create: + +- `docs/agent-guide/061-graph-dir-space-preserve-canonical.md` diff --git a/docs/agent-guide/062-cli-list-default-sort-updated-at.md b/docs/agent-guide/062-cli-list-default-sort-updated-at.md new file mode 100644 index 0000000000..279190c799 --- /dev/null +++ b/docs/agent-guide/062-cli-list-default-sort-updated-at.md @@ -0,0 +1,164 @@ +# Logseq CLI List Default Updated-at Sort Implementation Plan + +Goal: Make `list page`, `list tag`, and `list property` behave as if `--sort updated-at` is provided when users do not pass `--sort`. + +Architecture: Keep the change in the CLI command layer so the existing db-worker-node API surface remains stable. +Architecture: Reuse current CLI-side sorting flow in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` where sorting already runs before offset and limit. +Architecture: Keep db-worker-node list providers in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` unchanged for this scope, and document that decision explicitly. + +Tech Stack: ClojureScript, babashka.cli option specs, Promesa, Datascript-backed db-worker-node thread-api, existing CLI integration test harness. + +Related: Builds on `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/004-logseq-cli-verb-subcommands.md` and `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/043-logseq-cli-tag-property-management.md`. + +## Problem statement + +Current list commands support `--sort` and `--order`, but default behavior is unsorted because no sort field is applied unless users pass `--sort`. + +In current implementation, `list` execution fetches items from db-worker-node and runs `apply-sort` only when `:sort` is present in options. + +This means default output order depends on entity scan order from db-worker-node list functions, which is not aligned with the desired product behavior. + +The requested behavior is a consistent default sort key of `updated-at` for page, tag, and property list subcommands. + +## Current behavior snapshot + +| Layer | File | Current behavior | +| --- | --- | --- | +| CLI command parsing and execution | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` | `:sort` is optional and defaults to nil, so `apply-sort` is skipped when `--sort` is absent. | +| db-worker-node list thread-api bridge | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` | `:thread-api/cli-list-pages`, `:thread-api/cli-list-tags`, and `:thread-api/cli-list-properties` return unsorted collections from shared list helpers. | +| Shared list helpers | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` | List helpers filter and shape items but do not apply sort by `updated-at`. | +| User docs | `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` | List command syntax documents `--sort` as optional but does not state default sort behavior. | + +## Scope and non-goals + +This plan changes default sort behavior for CLI commands `list page`, `list tag`, and `list property`. + +This plan does not add new db-worker-node thread-api methods. + +This plan does not push pagination or sorting into db-worker-node. + +This plan does not change field names or output schemas. + +## Proposed behavior + +When users run `logseq list page`, `logseq list tag`, or `logseq list property` without `--sort`, CLI should sort by `updated-at` using the existing list sorting pipeline. + +When users pass `--sort`, the explicit value must override the default. + +`--order` should continue to default to `asc` unless explicitly set to `desc`. + +When multiple entities have the same primary sort value, CLI should apply `:db/id` as the deterministic secondary sort key. + +`offset` and `limit` must still be applied after sorting. + +The behavior should be equivalent to explicitly passing `--sort updated-at` today, with deterministic tie-breaking by `:db/id`. + +## Integration overview + +```text +logseq list page + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs + - determine effective sort field (default updated-at) + - apply CLI-side sort/order + - apply offset/limit/fields + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/transport.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs + -> /Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs +``` + +## Testing Plan + +I will follow `@test-driven-development` and add tests before implementation changes. + +I will add unit tests for CLI list execution paths that verify default sorting is applied when `:sort` is missing. + +I will add unit tests that verify explicit `--sort` still overrides the new default. + +I will add integration tests that verify default output order matches explicit `--sort updated-at` output for page, tag, and property list commands. + +I will update docs assertions or command help expectations if any existing tests encode old behavior. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Detailed implementation plan + +1. Add a failing unit test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` for list-page execution where input items are intentionally out of updated-at order and no `:sort` is provided. +2. Assert in that test that returned item ids are ordered exactly as if `:sort` were `"updated-at"` with default `:order`. +3. Add a failing unit test for list-tag execution with no `:sort` and confirm default updated-at ordering is applied after tag item preparation. +4. Add a failing unit test for list-property execution with no `:sort` and confirm default updated-at ordering is applied after property item preparation. +5. Add a failing unit test that passes explicit `:sort "title"` and confirms explicit sort overrides default updated-at behavior. +6. Add a failing unit test that passes explicit `:order "desc"` without `:sort` and confirms order is applied to default updated-at sort key. +7. Add a failing integration test in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` that compares `list page` results against `list page --sort updated-at` for the same graph and asserts identical id sequences. +8. Add a failing integration test that compares `list tag` against `list tag --sort updated-at` and asserts identical id sequences. +9. Add a failing integration test that compares `list property` against `list property --sort updated-at` and asserts identical id sequences. +10. Run focused tests to verify they fail for the new default-sort behavior and not for unrelated setup issues. +11. Implement an `effective-sort` decision in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` so list commands use `"updated-at"` when `:sort` is absent. +12. Reuse the effective sort key for all three executors: `execute-list-page`, `execute-list-tag`, and `execute-list-property`. +13. Keep existing explicit sort validation and allowed sort fields unchanged. +14. Add deterministic tie-breaking by `:db/id` in CLI list sorting when primary sort values are equal. +15. Keep existing order default (`asc`) unchanged. +16. Update option descriptions in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` to clarify default sort behavior for users. +17. Update docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md` to state that list subcommands default to sorting by `updated-at`. +18. Run focused unit and integration tests again and confirm green. +19. Run `bb dev:test -v logseq.cli.commands-test` to confirm command-level regressions are not introduced. +20. Run `bb dev:test -v logseq.cli.integration-test` for list command scenarios impacted by ordering. +21. Run `bb dev:lint-and-test` for final confidence. + +## Edge cases to cover + +Entities with missing `:block/updated-at` should still be sortable without runtime errors. + +Multiple entities with equal `updated-at` values should be secondarily sorted by `:db/id` for deterministic output across repeated runs. + +`--fields` filtering should not remove `updated-at` before sorting is executed. + +`--offset` and `--limit` should continue to apply after sorting, not before sorting. + +`--sort` with any allowed non-time field should keep existing behavior and take precedence over the new default. + +`--order desc` without explicit `--sort` should now reverse default updated-at order. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.cli.commands-test/test-list-subcommand-parse` | Existing parse behavior remains valid and list options remain accepted. | +| `bb dev:test -v logseq.cli.commands-test` | New default-sort unit tests pass and no command-level regressions are introduced. | +| `bb dev:test -v logseq.cli.integration-test` | Integration checks for list default ordering pass against a real db-worker-node flow. | +| `bb dev:lint-and-test` | Full lint and unit suite pass with zero errors. | + +## Rollout and compatibility + +This is a behavior change in default ordering for three CLI list commands. + +Scripts that depended on previous implicit scan order may observe changed item order. + +Scripts that already pass explicit `--sort` remain unaffected. + +No db-worker-node API contract change is introduced in this scope. + +## Testing Details + +Tests verify user-visible command behavior by comparing result ordering between default list calls and explicit `--sort updated-at` calls. + +Tests validate override behavior so explicit sort fields still control final ordering. + +Tests validate order plus pagination interaction to ensure behavior consistency. + +## Implementation Details + +- Update default sort selection in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/list.cljs` for page, tag, and property list executors. +- Keep existing `apply-sort`, `apply-offset-limit`, and `apply-fields` sequencing unchanged. +- Extend CLI sort implementation to apply `:db/id` as secondary key when primary sort values are equal. +- Reuse existing `list-*-field-map` entries for `updated-at` so no new field mapping is introduced. +- Keep db-worker-node list handlers in `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` unchanged in this scope. +- Keep shared list helper behavior in `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/common/mcp/tools.cljs` unchanged in this scope. +- Add command-level unit coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs`. +- Add end-to-end list ordering coverage in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs`. +- Update user-facing list docs in `/Users/rcmerci/gh-repos/logseq/docs/cli/logseq-cli.md`. + +## Question + +No open questions. + +--- diff --git a/docs/agent-guide/063-logseq-cli-upsert-block-custom-many-property.md b/docs/agent-guide/063-logseq-cli-upsert-block-custom-many-property.md new file mode 100644 index 0000000000..585741ee09 --- /dev/null +++ b/docs/agent-guide/063-logseq-cli-upsert-block-custom-many-property.md @@ -0,0 +1,164 @@ +# Logseq CLI Upsert Block Custom Many Property Reliability Plan + +Goal: Fix `logseq upsert block --update-properties` so custom public properties with `type=default` and `cardinality=many` can reliably persist multiple values on blocks. + +Architecture: Keep CLI command shape stable and preserve `upsert property` / `upsert block` UX. +Architecture: Apply the core behavior fix in outliner property write logic (`:batch-set-property`) because db-worker-node forwards ops without property-value normalization. +Architecture: Add end-to-end regression coverage in CLI integration tests and low-level behavioral coverage in outliner property tests. + +Tech Stack: ClojureScript, Datascript, Promesa, Logseq CLI command layer, db-worker-node thread-api, outliner ops (`deps/outliner`), existing CLI integration test harness. + +Related: +- `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/044-logseq-cli-upsert-block-page.md` +- `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/045-logseq-cli-property-type-and-upsert-option-unification.md` +- `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/043-logseq-cli-tag-property-management.md` + +## Problem statement + +A reproducible CLI flow currently fails or behaves inconsistently when assigning multiple values to a custom property on a block: + +1. `graph create` +2. `upsert property --name "Reproducible steps" --type default --cardinality many --public true` +3. `upsert block --update-properties '{"Reproducible steps" ["Step 1" "Step 2" "Step 3"]}'` + +Observed behavior: +- String-vector payload can fail with a generic CLI `http request failed` during `:batch-set-property`. +- A numeric-id vector payload may report `ok` but still not materialize expected property datoms on the target block. + +Expected behavior: +- The target block should persist exactly three values for that custom property. +- CLI should provide deterministic success/failure semantics for both title-based values and id-based values. + +## Current implementation snapshot + +| Layer | File | Current behavior | +| --- | --- | --- | +| CLI upsert command wiring | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/upsert.cljs` | `upsert block` (create/update paths) resolves properties and emits `[:batch-set-property [block-ids k v {}]]`. | +| CLI property parse/resolve | `/Users/rcmerci/gh-repos/logseq/src/main/logseq/cli/command/add.cljs` | `allow-non-built-in? true` path supports custom properties and many values. | +| db-worker-node bridge | `/Users/rcmerci/gh-repos/logseq/src/main/frontend/worker/db_core.cljs` | `:thread-api/apply-outliner-ops` forwards ops to outliner; no property-value normalization. | +| Outliner op execution | `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/op.cljs` | `:batch-set-property` delegates to `outliner-property/batch-set-property!`. | +| Property value conversion | `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/property.cljs` | `convert-ref-property-value` is scalar-centric and has incomplete/implicit handling for custom ref-many collection values. | + +## Scope and non-goals + +This plan fixes custom property many-value persistence for block updates through CLI upsert/update flows. + +This plan includes both string-input and id-input value shapes for custom `default` + `many` properties. + +This plan does not redesign property schema semantics. + +This plan does not change CLI command flags or argument names. + +This plan does not add new thread-api methods. + +## Proposed behavior + +For custom public properties (`:user.property/*`) with `:db.cardinality/many` and ref-capable types (including `:default`): + +- `--update-properties '{"Name" ["Step 1" "Step 2" "Step 3"]}'` should persist three values on the block. +- `--update-properties '{"Name" [180 181 182]}'` should persist three values on the block when ids are valid entities. +- `--update-properties '{"Name" []}'` should clear the property from the target block. +- Duplicate input values should be preserved at input semantics level (CLI/outliner must not proactively dedupe user input before write). +- Result must be observable via Datascript query as expected datoms for that block/property. + +Failure cases should return explicit error payloads (invalid values, invalid ids, schema mismatch), not silent no-op behavior. + +## Root-cause hypothesis + +Primary hypothesis: +- `batch-set-property!` in outliner currently applies ref conversion in a way that is optimized for scalar values and does not consistently normalize collection values for custom ref-many properties. +- `convert-ref-property-value` treats collection values as a special case only when all elements are integers; string collection conversion is not explicit/element-wise for many mode. +- This causes unstable behavior in CLI flows that legitimately pass many values. + +Secondary hypothesis: +- Success responses can occur even when final value conversion does not produce a valid persisted property value set for the target block. + +## Testing plan (TDD first) + +I will follow `@test-driven-development` and add/adjust tests before implementation changes. + +### Outliner-level tests (RED first) + +Add tests in: +`/Users/rcmerci/gh-repos/logseq/deps/outliner/test/logseq/outliner/property_test.cljs` + +1. Add failing test: custom property (`:user.property/reproducible-steps`) with `:default + many`, call `batch-set-property!` with string vector values, assert three persisted values. +2. Add failing test: same property, call `batch-set-property!` with numeric id vector, assert three persisted values. +3. Add failing test: mixed/invalid value set returns explicit error and does not partially persist. +4. Add regression assertion that built-in many properties (for example tags/page-tags-like behavior) remain unchanged. + +### CLI integration tests (RED first) + +Add tests in: +`/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` + +5. Add failing end-to-end test: + - create graph + - upsert custom property (`default + many + public`) + - upsert block with string-vector update-properties + - query and assert exactly three values on target block. +6. Add failing end-to-end test for id-vector values. +7. Ensure assertion helper is many-aware (do not rely on scalar-only query helpers). + +### Optional command-level guard tests + +8. If needed, add command-level tests in: + - `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/commands_test.cljs` + to verify many-value payload survives parse/build-action shape without lossy normalization. + +## Detailed implementation plan + +1. In `/Users/rcmerci/gh-repos/logseq/deps/outliner/src/logseq/outliner/property.cljs`, introduce explicit many-aware ref conversion helper (for example, scalar conversion + collection mapping wrapper). +2. Keep existing scalar conversion semantics as baseline for non-many properties. +3. In `batch-set-property!`, compute cardinality from property entity and branch conversion flow: + - one: current scalar conversion path + - many: normalize to collection and convert each element deterministically. +4. Ensure string inputs for custom default properties create/reuse property value entities element-by-element. +5. Ensure integer/ref inputs for many mode are validated and preserved element-by-element. +6. Preserve validation (`throw-error-if-invalid-property-value`) after conversion and before tx construction. +7. Ensure self-reference protection for ref values also works in many mode (check each resolved element against block id when applicable). +8. Keep transaction generation centralized via `build-property-value-tx-data` and avoid introducing duplicate write paths. +9. Add explicit error context when conversion fails in many mode (property id, incoming value shape) to reduce generic `http request failed` surface area. +10. Re-run newly added outliner tests and iterate until green. +11. Re-run CLI integration tests and confirm end-to-end pass for both string-vector and id-vector payloads. +12. Confirm existing upsert flows for built-in properties still pass. + +## Edge cases to cover + +- Empty vector value for many property should clear the property on the target block. +- Duplicate values in input vector should be preserved at input semantics level (no proactive dedupe in CLI/outliner conversion path). +- Mixed-type vectors (`["Step 1" 181]`) for default many property. +- Invalid entity ids in id-vector input. +- Non-public property update attempts. +- Property just upserted in same flow followed immediately by block update. + +## Verification commands + +| Command | Expected result | +| --- | --- | +| `bb dev:test -v logseq.outliner.property-test` | New many-value conversion tests pass and no outliner regressions. | +| `bb dev:test -v logseq.cli.integration-test` | New CLI e2e tests for custom many property pass. | +| `bb dev:test -v logseq.cli.commands-test` | Command parse/build behavior remains stable. | +| `bb dev:lint-and-test` | Full lint/test suite remains green. | + +## Acceptance criteria + +1. CLI flow for custom `default + many + public` property with string values persists all values on target block. +2. CLI flow with id values persists all values on target block. +3. Empty vector updates (`[]`) clear the target property from the block. +4. Duplicate input values are preserved at input semantics level (conversion path does not proactively dedupe). +5. Datascript query confirms expected count and value set semantics for the target block property. +6. No regression for built-in property update flows. +7. Failures return actionable error details instead of generic silent/no-op outcomes. + +## Rollout and compatibility + +This is a behavior correctness fix, not a CLI API change. + +Existing scripts using current flags remain valid. + +Behavioral change is limited to making many-value persistence for custom properties deterministic and correct. + +## Question + +No open design question for this phase. diff --git a/docs/agent-guide/064-logseq-cli-integration-test-shell-refactor.md b/docs/agent-guide/064-logseq-cli-integration-test-shell-refactor.md new file mode 100644 index 0000000000..556b85cf47 --- /dev/null +++ b/docs/agent-guide/064-logseq-cli-integration-test-shell-refactor.md @@ -0,0 +1,175 @@ +# Logseq CLI Integration Test Shell Refactor Implementation Plan + +Goal: Build a new `cli-e2e` test suite that refactors `logseq.cli.integration-test` into babashka-driven shell e2e tests for compiled `logseq-cli` and `db-worker-node`, excluding `sync`, `login`, and `logout`, with full in-scope subcommand coverage and key-option coverage. + +Architecture: Use a declarative command case manifest in `cli-e2e/` and execute every case via explicit shell command strings so test execution is transparent and copy-pasteable. +Architecture: Run a build preflight that compiles `logseq-cli` and `db-worker-node` artifacts before every e2e run, then run cases against those compiled artifacts only. +Architecture: Enforce completeness with a coverage checker that fails when any in-scope subcommand or any declared key option is not covered by at least one case. + +Tech Stack: Babashka, babashka.cli, babashka.process, EDN manifests, Node.js, shadow-cljs build targets `logseq-cli` and `db-worker-node`, existing Logseq CLI and db-worker-node runtime. + +Related: Relates to `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/053-logseq-cli-async-test-isolation.md`. +Related: Relates to `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/058-db-worker-node-revision-and-cli-server-list.md`. +Related: Relates to `/Users/rcmerci/gh-repos/logseq/docs/agent-guide/060-cli-graph-list-legacy-graph-dir-rename-command.md`. + +## Problem statement + +The current `src/test/logseq/cli/integration_test.cljs` suite mostly calls `logseq.cli.main/run!` directly instead of executing `logseq-cli` as a shell command. +This makes command-line behavior, quoting, piping, process lifecycle, and compiled artifact boundaries under-tested. +The current suite also relies heavily on `with-redefs` for `sync` and auth flows, which hides process-level integration issues. +The suite location and runtime are tied to `shadow-cljs :test`, while the new requirement is an independent `cli-e2e/` babashka flow that tests compiled binaries. +The new requirement also asks for in-scope subcommand completeness with key option coverage enforcement, which is not explicitly guaranteed today. + +## Current implementation snapshot + +| Area | Current file | Current behavior | Gap against requirement | +| --- | --- | --- | --- | +| CLI integration tests | `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` | Runs in cljs test bundle and invokes `run!` directly for most tests. | Not fully shell-first and not isolated under `cli-e2e/`. | +| CLI compiled artifact | `/Users/rcmerci/gh-repos/logseq/static/logseq-cli.js` | Generated by `shadow-cljs` target `logseq-cli`. | Not currently enforced as precondition in integration test flow. | +| db-worker runtime used by CLI server lifecycle | `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js` | Spawned by `logseq.cli.server/db-worker-script-path`. | Must be rebuilt before e2e to ensure latest runtime. | +| Existing integration runner entrypoint | `/Users/rcmerci/gh-repos/logseq/package.json` (`cljs:run-integration-test`) | Runs `node static/tests.js -r '^(logseq.cli.integration-test).*'`. | Keeps tests inside cljs test harness instead of babashka shell runner. | + +## In-scope command and key option inventory + +The e2e suite target excludes every `sync *` command and excludes `login` and `logout`. +The suite treats global options and command options as separate coverage buckets. +The option inventory below is the key-option scope for this phase. + +| Scope | Commands | Options to cover | +| --- | --- | --- | +| Global | Any in-scope command invocation | `--help`, `--version`, `--config`, `--graph`, `--data-dir`, `--output`, `--verbose`. | +| graph | `graph list`, `graph create`, `graph switch`, `graph remove`, `graph validate`, `graph info`, `graph export`, `graph import` | `--fix`, `--type`, `--file`, `--input`. | +| list | `list page`, `list tag`, `list property` | `--expand`, `--fields`, `--limit`, `--offset`, `--sort`, `--order`, `--journal-only`, `--with-properties`, `--with-extends`, `--with-classes`, `--with-type`. | +| upsert | `upsert block`, `upsert page`, `upsert tag`, `upsert property` | `--id`, `--uuid`, `--target-id`, `--target-uuid`, `--target-page`, `--pos`, `--content`, `--blocks-file`, `--status`, `--update-tags`, `--update-properties`, `--remove-tags`, `--remove-properties`, `--page`, `--name`, `--type`, `--cardinality`, `--hide`, `--public`. | +| remove | `remove block`, `remove page`, `remove tag`, `remove property` | `--id`, `--uuid`, `--name`. | +| query | `query`, `query list` | `--query`, `--name`, `--inputs`. | +| show | `show` | `--id`, `--uuid`, `--page`, `--linked-references`, `--level`, stdin `--id` input path. | +| server | `server list`, `server status`, `server start`, `server stop`, `server restart` | Command-specific graph use through `--graph`. | +| doctor | `doctor` | `--dev-script`. | +| completion | `completion` | Positional `zsh` or `bash`, and `--shell`. | + +## Testing Plan + +I will follow `@test-driven-development` for this refactor and write failing tests first in the new `cli-e2e` harness before implementation logic. +I will keep the tests behavior-first by asserting process exit code, stdout or stderr payload, file-system side effects, and db state changes after shell commands. + +I will add a failing manifest coverage test that enumerates all in-scope subcommands and key options and fails when any item is missing from executed cases. +I will add failing runner tests that assert commands are executed as explicit shell strings and that each case logs the exact command line. +I will add failing preflight tests that assert build preflight runs before cases and that required compiled artifacts exist. +I will add failing e2e cases for each in-scope command and each key option from the inventory table, using positive and validation-error scenarios. +I will add regression tests for piping and stdin (`query | show --id`) to preserve shell-native workflows. +I will add tests that verify db-worker-node lifecycle commands (`server start/status/restart/stop`) operate against compiled runtime. + +NOTE: I will write *all* tests before I add any implementation behavior. + +## Target layout and integration points + +The new harness will live under `/Users/rcmerci/gh-repos/logseq/cli-e2e/`. +The existing `src/test/logseq/cli/integration_test.cljs` namespace will be reduced or split so in-scope command responsibilities move to the new harness. + +```text ++----------------------+ shell exec +------------------------------+ +| cli-e2e bb runner | -----------------------> | node static/logseq-cli.js | +| (bb + EDN manifests) | +--------------+---------------+ ++----------+-----------+ | + | | spawns + | preflight build v + | +------------------------------+ + +---------------------------------------> | dist/db-worker-node.js | + | via logseq.cli.server | + +------------------------------+ +``` + +## Detailed implementation plan + +1. Create `/Users/rcmerci/gh-repos/logseq/cli-e2e/` with `bb.edn`, `README.md`, and minimal source directories. +2. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/bb.edn` tasks `build`, `test`, and `list-cases`. +3. Use `@clojure-babashka-cli` patterns in `bb.edn` to parse `--case`, `--include`, `--skip-build`, and `--verbose`. +4. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/paths.clj` to resolve absolute repo-root aware paths. +5. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/shell.clj` with a single shell execution utility that records exact command strings. +6. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/preflight.clj` to run build preflight commands. +7. Implement preflight commands as `clojure -M:cljs compile logseq-cli db-worker-node` followed by `pnpm db-worker-node:compile:bundle`. +8. Add preflight artifact existence checks for `/Users/rcmerci/gh-repos/logseq/static/logseq-cli.js`, `/Users/rcmerci/gh-repos/logseq/static/db-worker-node.js`, `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node.js`, and `/Users/rcmerci/gh-repos/logseq/dist/db-worker-node-assets.json`. +9. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/spec/non_sync_inventory.edn` to declare required command and key-option coverage excluding sync, login, and logout. +10. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/spec/non_sync_cases.edn` to declare all shell cases with `:cmd`, `:expect`, and `:covers`. +11. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/coverage.clj` that computes missing command and key-option coverage from manifests. +12. Write a failing test that coverage checker reports missing entries before cases are added. +13. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/fixtures.clj` for temp data-dir, temp config, and graph seed helpers. +14. Implement graph seed helper using explicit shell commands only, not direct namespace calls. +15. Add base cases for global key-option coverage in `/Users/rcmerci/gh-repos/logseq/cli-e2e/spec/non_sync_cases.edn`. +16. Add base cases for every `graph` subcommand and its options, including negative validation paths. +17. Add base cases for every `list` subcommand and key list options with deterministic assertions. +18. Add base cases for every `upsert` subcommand and key options, including selector conflict errors and file-based block insertion. +19. Add base cases for every `remove` subcommand and key options, including uuid and name resolution behavior. +20. Add base cases for `query` and `query list`, including `--query`, `--name`, and `--inputs` paths. +21. Add base cases for `show` including `--id`, `--uuid`, `--page`, `--linked-references`, `--level`, and stdin id input. +22. Add base cases for `server` commands and assert lifecycle transitions through `server status`. +23. Add base cases for `doctor --dev-script` and assert readable script check behavior. +24. Add base cases for `completion` positional and `--shell` usage and assert shell snippet markers. +25. Add case-runner assertions for stdout parsing in `json`, `edn`, and `human` output modes. +26. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/src/logseq/cli/e2e/report.clj` to print pass or fail summary with missing coverage details. +27. Add `/Users/rcmerci/gh-repos/logseq/cli-e2e/README.md` with setup, run commands, and expected artifacts. +28. Add root task wiring in `/Users/rcmerci/gh-repos/logseq/bb.edn` for `dev:cli-e2e` delegating to `cli-e2e/bb.edn`. +29. Add package script wiring in `/Users/rcmerci/gh-repos/logseq/package.json` for `cli:e2e` and `cli:e2e:skip-build`. +30. Move or slim in-scope tests out of `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` and keep only compatibility coverage that still belongs in cljs unit scope. +31. Add a deprecation note in `/Users/rcmerci/gh-repos/logseq/src/test/logseq/cli/integration_test.cljs` pointing engineers to `cli-e2e/`. +32. Run the full new `cli-e2e` suite with build preflight enabled and confirm zero missing coverage. +33. Run a focused subset using `--case` to validate developer ergonomics and rerun speed. +34. Run `bb dev:lint-and-test` to confirm the migration does not break existing unit test workflows. + +## Verification commands and expected outcomes + +| Command | Expected outcome | +| --- | --- | +| `bb -f /Users/rcmerci/gh-repos/logseq/cli-e2e/bb.edn build` | Compiles CLI and db-worker artifacts and passes artifact existence checks. | +| `bb -f /Users/rcmerci/gh-repos/logseq/cli-e2e/bb.edn test` | Runs all in-scope cases and prints zero missing command or key-option coverage. | +| `bb -f /Users/rcmerci/gh-repos/logseq/cli-e2e/bb.edn test --case show-stdin-id` | Runs one targeted case with exact shell command output shown. | +| `bb dev:cli-e2e` | Delegates to `cli-e2e` full run from repository root. | +| `bb dev:lint-and-test` | Existing lint and unit suites remain green after migration. | + +## Edge cases to include in case manifests + +1. Graph names and data-dir paths that include spaces and legacy-encoded names. +2. `show --id` stdin behavior with empty stdin and invalid EDN input. +3. Boolean option coercion for `--with-type`, `--linked-references`, `--hide`, and `--public`. +4. Mutually exclusive selectors such as `upsert page --id` with `--page` and `remove block --id` with `--uuid`. +5. Output format assertions for `--output human`, `--output json`, and `--output edn`. +6. `query` argument validation for `--query` and `--name` conflict and invalid `--inputs` EDN. +7. `completion` unsupported shell error path. +8. `server` command behavior when the target graph server is missing or already running. +9. `doctor --dev-script` when static script exists and when path is unreadable in a controlled failure case. +10. Ensure excluded commands (`sync`, `login`, `logout`) are rejected by coverage checker if accidentally added to in-scope inventory. + +## Clarifications and assumptions captured in implementation + +1. Key option coverage means each in-scope key option must appear in at least one behaviorally asserted case and not merely in help text. +2. Combinatorial explosion is controlled by one-option-at-least-once coverage plus targeted interaction cases for known conflicts. +3. Sync commands plus `login` and `logout` are explicitly excluded from this plan and remain covered by other test scopes. +4. The canonical runtime under test for CLI-managed server lifecycle is `dist/db-worker-node.js`, so bundle build is part of preflight. +5. The new suite is the source of truth for shell e2e behavior, while cljs namespace tests remain unit and component level. + +## Testing Details + +The new tests validate real behavior at process boundaries by invoking compiled binaries through shell commands and asserting outputs, files, and db effects. +Coverage checks verify every declared in-scope subcommand and key option has at least one asserted case. +Failure output will include missing coverage entries and the exact failing shell command for fast diagnosis. + +## Implementation Details + +- Use `cli-e2e/spec/non_sync_inventory.edn` as the single source of required in-scope command and key-option coverage. +- Use `cli-e2e/spec/non_sync_cases.edn` as declarative executable test cases with explicit `:covers`. +- Use one shared shell executor to keep command logging and error reporting consistent. +- Build preflight must run by default and may be skipped only with explicit `--skip-build`. +- Keep all test setup through shell commands to satisfy shell-first requirement. +- Use temp dirs per case to avoid cross-test contamination. +- Keep `login` and `logout` out of this suite by explicit inventory exclusions and checker guards. +- Preserve at least one pipeline test that uses stdout to stdin command chaining. +- Keep root task wiring minimal and avoid coupling `cli-e2e` to existing cljs test runner. +- Document command examples and troubleshooting in `cli-e2e/README.md`. + +## Question + +Do you want `login` and `logout` to stay permanently out of `cli-e2e`, or should we plan a separate optional suite for them under a dedicated `:manual` or `:network` tag? + + +--- diff --git a/docs/agent-guide/065-logseq-cli-show-ref-id-footer.md b/docs/agent-guide/065-logseq-cli-show-ref-id-footer.md new file mode 100644 index 0000000000..97d1448de6 --- /dev/null +++ b/docs/agent-guide/065-logseq-cli-show-ref-id-footer.md @@ -0,0 +1,166 @@ +# Logseq CLI Show Referenced Entity IDs in Footer Plan + +Goal: Add a footer section to `logseq show` human output that lists referenced entities and their IDs when block content contains `[[]]`. + +Architecture: Keep the existing tree rendering unchanged and append a post-tree summary section (`Referenced Entities`) generated from refs discovered in the shown tree (and linked references when enabled). + +Tech Stack: ClojureScript, `logseq.cli.command.show`, db-worker-node thread-api pull/query via existing transport. + +Related: +- `docs/agent-guide/021-logseq-cli-reference-uuid-rewrite.md` +- `docs/agent-guide/024-logseq-cli-show-updates.md` +- `docs/agent-guide/010-logseq-cli-show-linked-references.md` + +## Problem Statement + +Current `show` output rewrites `[[]]` to readable labels, but users cannot see the stable entity ID for each reference in human output. + +When reviewing large trees or debugging graph links, users need a deterministic ID mapping without polluting every inline block line. + +The selected UX is a dedicated footer section (not inline expansion) so the tree remains readable. + +## Proposed UX + +### Human output (new footer section) + +Keep current tree output exactly as-is, then append: + +```text +Referenced Entities (2) +181 -> First child line A\nline B for wrapping +179 -> Root task for show command +``` + +Behavior: +- Section appears only when at least one reference is present. +- Order is by first appearance in rendered traversal order. +- ID should be `:db/id` (human-facing stable numeric id in CLI context). +- Label should prefer `:block/title`, then `:block/name`, then UUID string fallback. + +### Option behavior + +Provide an explicit option so users can disable the footer when needed: +- `--ref-id-footer` (boolean, default `true`) + +When `true` (default), append the footer section. +When `false`, do not render the footer section. + +Because this changes default human output, update related snapshots/tests accordingly and document the behavior in release notes. + +## Scope + +In scope: +- Human output for `show` command only. +- Single target and multi-id target modes. +- Respect existing `--linked-references` behavior for reference discovery source. + +Out of scope: +- JSON/EDN schema changes. +- Inline replacement format changes. +- New reference graph traversal semantics. + +## Design Details + +### Reference discovery source + +Build the footer map from already available tree data: +1. Traverse `:root` tree nodes and gather references from text fields (`:block/title`, `:block/name`, `:block/content`). +2. If linked references are enabled, include linked reference block texts too. +3. Reuse existing UUID extraction and label fetch behavior where possible. + +### UUID to entity resolution + +Current logic already fetches labels for UUID refs. Extend resolution to also fetch `:db/id` for each referenced UUID: +- Pull selector should include `:db/id`, `:block/uuid`, `:block/title`, `:block/name`. +- Build map: `uuid-lowercase -> {:id :label