mirror of
https://github.com/logseq/logseq.git
synced 2026-05-18 01:42:19 +00:00
Merge 'master' into refactor/vite-migration
This commit is contained in:
Mega Yu
139
.agents/skills/logseq-cli-maintenance/SKILL.md
Normal file
139
.agents/skills/logseq-cli-maintenance/SKILL.md
Normal file
@@ -0,0 +1,139 @@
|
||||
---
|
||||
name: logseq-cli-maintenance
|
||||
description: Improve readability, consistency, and long-term maintainability of Logseq CLI-related code, command flows, and tests.
|
||||
---
|
||||
|
||||
# logseq-cli-maintenance
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this skill when working on Logseq CLI-related code and you want to improve:
|
||||
|
||||
- Readability
|
||||
- Maintainability
|
||||
- Consistency across command modules
|
||||
- Long-term development ergonomics
|
||||
|
||||
This skill focuses on practical refactoring and guardrails, without changing behavior unless explicitly requested.
|
||||
|
||||
---
|
||||
|
||||
## When to use
|
||||
|
||||
Trigger this skill when tasks include any of the following:
|
||||
|
||||
1. “Refactor Logseq CLI code”
|
||||
2. “Clean up command implementation”
|
||||
3. “Improve readability/structure of CLI logic”
|
||||
4. “Make CLI easier to extend and test”
|
||||
5. “Reduce duplication in argument parsing/output handling”
|
||||
|
||||
Do **not** use this skill for pure feature delivery unless the request also asks for code quality improvements.
|
||||
|
||||
---
|
||||
|
||||
## Core maintenance principles
|
||||
|
||||
### 1) Separate concerns clearly
|
||||
|
||||
Keep these responsibilities isolated:
|
||||
|
||||
- **Input parsing** (args/options/env)
|
||||
- **Validation** (schema/rules/errors)
|
||||
- **Execution** (domain logic)
|
||||
- **Presentation** (stdout/stderr formatting, exit codes)
|
||||
|
||||
A command handler should orchestrate these steps, not mix all logic in one large function.
|
||||
|
||||
### 2) Prefer small, composable functions
|
||||
|
||||
- Keep functions single-purpose.
|
||||
- Use descriptive names that explain intent.
|
||||
- Extract repeated logic into shared helpers.
|
||||
- Avoid deep nesting; use early returns for invalid states.
|
||||
|
||||
### 3) Keep command contracts explicit
|
||||
|
||||
For each command/subcommand, make explicit:
|
||||
|
||||
- Required and optional args
|
||||
- Defaults
|
||||
- Validation constraints
|
||||
- Output shape and error format
|
||||
- Exit code semantics
|
||||
|
||||
### 4) Make errors actionable
|
||||
|
||||
- Show concise error messages with next-step hints.
|
||||
- Keep error wording consistent across commands.
|
||||
- Distinguish user errors (invalid input) from internal errors.
|
||||
|
||||
### 5) Standardize CLI output
|
||||
|
||||
- Use one style for success, warning, and error output.
|
||||
- Ensure machine-readable mode (if supported) is stable and documented.
|
||||
- Avoid hidden format drift across subcommands.
|
||||
|
||||
### 6) Preserve behavior while refactoring
|
||||
|
||||
When request is maintenance-focused, avoid feature changes.
|
||||
If behavior must change, call it out explicitly and add tests.
|
||||
|
||||
---
|
||||
|
||||
## Suggested workflow
|
||||
|
||||
1. **Map current command flow**
|
||||
- Locate parse → validate → execute → print boundaries.
|
||||
2. **Identify maintenance hotspots**
|
||||
- Long functions, duplicated parsing/output logic, hidden side effects.
|
||||
3. **Refactor in small, reviewable steps**
|
||||
- One concern per change.
|
||||
4. **Add/update tests around behavior contracts**
|
||||
- Especially for error cases and output format.
|
||||
5. **Run relevant CLI tests and linters**
|
||||
- Confirm no regressions.
|
||||
6. **Run relevant cmds in logseq-cli**
|
||||
- Use logseq-cli skill
|
||||
- Confirm no regressions.
|
||||
7. **Document extension points**
|
||||
- Show where future subcommands/options should be added.
|
||||
|
||||
---
|
||||
|
||||
## Refactoring checklist
|
||||
|
||||
Use this checklist before finishing:
|
||||
|
||||
- [ ] Command entrypoint is concise and easy to scan.
|
||||
- [ ] Parsing/validation/execution/output are separated.
|
||||
- [ ] Repeated logic is extracted to shared helpers.
|
||||
- [ ] Names are intention-revealing.
|
||||
- [ ] Error messages are consistent and actionable.
|
||||
- [ ] Output and exit code behavior are tested.
|
||||
- [ ] No behavior changes were introduced unintentionally.
|
||||
- [ ] `bb lint:large-vars` passes for touched vars.
|
||||
- [ ] Existing `^:large-vars/cleanup-todo` annotations were removed when possible.
|
||||
- [ ] New structure is easy for future contributors to extend.
|
||||
|
||||
---
|
||||
|
||||
## Common anti-patterns to remove
|
||||
|
||||
- God-function command handlers
|
||||
- Implicit defaults scattered across files
|
||||
- Inconsistent option names/aliases
|
||||
- Silent failures or ambiguous exit codes
|
||||
- Copy-pasted output formatting logic
|
||||
- Mixing domain logic directly with terminal I/O
|
||||
|
||||
---
|
||||
|
||||
## Deliverable expectations
|
||||
|
||||
When applying this skill, produce:
|
||||
|
||||
1. Cleaner structure with equivalent behavior (unless requested otherwise)
|
||||
2. Focused tests for command contracts
|
||||
3. Brief rationale for key refactors
|
||||
4. Clear future extension path for new CLI commands/options
|
||||
87
.agents/skills/logseq-cli/SKILL.md
Normal file
87
.agents/skills/logseq-cli/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: logseq-cli
|
||||
description: Operate the current Logseq command-line interface to inspect or modify graphs, pages, blocks, tags, and properties; run Datascript queries; show page/block trees; manage graphs; and manage db-worker-node servers. Use when a request involves running `logseq` commands or interpreting CLI output.
|
||||
---
|
||||
|
||||
# Logseq CLI
|
||||
|
||||
## Overview
|
||||
|
||||
Use `logseq` to inspect and edit graph entities, run Datascript queries, and control graph/server lifecycle.
|
||||
|
||||
## Quick start
|
||||
|
||||
- Run `logseq --help` to see top-level commands and global flags.
|
||||
- Run `logseq <command> --help` to see command-specific options.
|
||||
- Use `--graph` to target a specific graph.
|
||||
- Omit `--output` for human output. Set `--output json` or `--output edn` only when machine-readable output is required.
|
||||
|
||||
## Command groups (from `logseq --help`)
|
||||
|
||||
- Graph inspect/edit:
|
||||
- `list node`, `list page`, `list tag`, `list property`, `list task`, `list asset`
|
||||
- `upsert block`, `upsert page`, `upsert tag`, `upsert property`, `upsert task`, `upsert assert`
|
||||
- `remove block`, `remove page`, `remove tag`, `remove property`
|
||||
- `query`, `query list`, `show`, `search`
|
||||
- Graph management: `graph list|create|switch|remove|validate|info|export|import|backup`
|
||||
- Server management: `server list|cleanup|start|stop|restart`
|
||||
- Diagnostics: `doctor`, `debug`
|
||||
|
||||
## Global options
|
||||
|
||||
- `--config` Path to `cli.edn` (default `~/logseq/cli.edn`)
|
||||
- `--graph` Graph name
|
||||
- `--data-dir` Path to db-worker data dir (default `~/logseq/graphs`)
|
||||
- `--timeout-ms` Request timeout in ms (default `10000`)
|
||||
- `--output` Output format (`human`, `json`, `edn`)
|
||||
- `--verbose` Enable verbose debug logging to stderr
|
||||
|
||||
## Command option policy
|
||||
|
||||
- Do not memorize or hardcode command options in this skill.
|
||||
- Before running any command, always check live options with:
|
||||
- `logseq <command> --help`
|
||||
- `logseq <command> <subcommand> --help`
|
||||
|
||||
## Examples policy
|
||||
|
||||
- Do not maintain long static command examples in this skill.
|
||||
- Use `logseq example` as the source of truth for runnable examples.
|
||||
- Before proposing runnable commands, always inspect live examples with:
|
||||
- `logseq example`
|
||||
- `logseq example <command-or-prefix...>`
|
||||
- `logseq example <command-or-prefix...> --help`
|
||||
- Prefer exact selectors when possible (for example, `logseq example upsert page`).
|
||||
- Use prefix selectors when grouped examples are needed (for example, `logseq example upsert`).
|
||||
- Replace placeholder ids/uuids in retrieved examples with real entities from the target graph.
|
||||
- Use `logseq list ...`, `logseq show ...`, or `logseq query ...` first to discover valid ids/uuids.
|
||||
- For graph transfer flows, keep `graph export --file` and `graph import --input` paths consistent.
|
||||
|
||||
|
||||
## Tag association semantics
|
||||
|
||||
- For block or page tag association, prefer explicit CLI tag options such as `--update-tags` and `--remove-tags`.
|
||||
- Do not treat writing `#TagName` inside `--content` as equivalent guidance to explicit tag association.
|
||||
- `upsert block` supports `--update-tags` in both create mode and update mode.
|
||||
- `--update-tags` expects an EDN vector.
|
||||
- Tag values may be tag title/name strings, db/id, UUID, or `:db/ident` values.
|
||||
- String tag values may include a leading `#`, but they should still be passed inside `--update-tags` rather than embedded in content as a substitute for association.
|
||||
- If the user asks to tag a block or page, prefer explicit tag association over embedding hashtags in content.
|
||||
- Tags must already exist and be public. If needed, create the tag first with `upsert tag --name "<TagName>"`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- `--content "Summary #AI-GENERATED"` is not the same guidance as `--update-tags '["AI-GENERATED"]'`.
|
||||
- Do not pass `--update-tags` as a comma-separated string. Use an EDN vector.
|
||||
- Do not assume a hashtag in block text will replace the need for explicit tag association when the user asks for a tagged block.
|
||||
- If tag association fails, verify the tag exists and is public before retrying.
|
||||
|
||||
## Tips
|
||||
|
||||
- `query list` returns both built-ins and `custom-queries` from `cli.edn`.
|
||||
- `show --id` accepts either one db/id or an EDN vector of ids.
|
||||
- `remove block --id` also accepts one db/id or an EDN vector.
|
||||
- `upsert block` enters update mode when `--id` or `--uuid` is provided.
|
||||
- Always verify command flags with `logseq --help` and `logseq <...> --help` before execution.
|
||||
- If `logseq` reports that it doesn’t have read/write permission for data-dir, then add read/write permission for data-dir in the agent’s config.
|
||||
- In sandboxed environments, `graph create` may print a process-scan warning to stderr; if command status is `ok`, the graph is still created.
|
||||
108
.agents/skills/logseq-debug-workflow/SKILL.md
Normal file
108
.agents/skills/logseq-debug-workflow/SKILL.md
Normal file
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: logseq-debug-workflow
|
||||
description: Debug Logseq bugs with the right runtime, concrete before/after evidence, and end-to-end reproduction steps.
|
||||
---
|
||||
|
||||
# Logseq Debug Workflow
|
||||
|
||||
Use for any Logseq bug investigation.
|
||||
|
||||
## Core rule
|
||||
|
||||
Treat this as a debugging workflow, not a code-only change.
|
||||
|
||||
Before claiming a fix, you must:
|
||||
|
||||
1. Choose the correct *runtime repl* and say why:
|
||||
- `:app` for renderer, DOM, UI, frontend state
|
||||
- `:electron` for main-process, BrowserWindow, IPC, app config
|
||||
- `:db-worker-node` for worker DB behavior, worker IPC, queries, transactions
|
||||
- CLI for `logseq` command behavior
|
||||
2. Reproduce the bug in *runtime REPL* before editing code.
|
||||
3. Capture concrete evidence from that runtime: REPL output, CLI output, logs, or a failing test that matches the real runtime path.
|
||||
4. Check relevant logs early.
|
||||
5. Apply the smallest justified fix.
|
||||
6. Re-run the same reproduction flow after the fix and capture evidence again.
|
||||
7. Include restart/reload/reopen verification when the bug involves settings, startup, persistence, window creation, or cross-process behavior.
|
||||
|
||||
## Do not conclude early
|
||||
|
||||
Do **not** say the bug is fixed if any of these is missing:
|
||||
|
||||
- pre-fix reproduction evidence
|
||||
- relevant log evidence, or an explicit statement that checked logs had nothing useful
|
||||
- post-fix evidence from the same runtime/path
|
||||
- required end-to-end lifecycle verification
|
||||
|
||||
Unit tests alone are **not** enough when this skill applies, unless the bug is truly unit-level and you explicitly justify that.
|
||||
|
||||
If the environment blocks full verification, report:
|
||||
|
||||
- the blocker
|
||||
- what you tried
|
||||
- which evidence is still missing
|
||||
- the strongest partial evidence you have
|
||||
|
||||
## Debugging tools
|
||||
|
||||
### General
|
||||
|
||||
- Add labeled `prn` checkpoints when helpful.
|
||||
- Use small REPL/eval checks.
|
||||
- Use targeted tests to confirm behavior.
|
||||
- Inspect only relevant inputs, branches, transformed values, outputs, and errors.
|
||||
- For async flows, inspect both sides of the async boundary.
|
||||
|
||||
### Logseq REPL
|
||||
|
||||
check `logseq-repl` skill.
|
||||
|
||||
### Logs
|
||||
|
||||
Logs are evidence. Check them early for Electron, CLI, worker, IPC, async, or persistence issues.
|
||||
|
||||
Common locations:
|
||||
- `tmp/desktop-app-repl/desktop-electron.log` (logseq-repl skill)
|
||||
- `tmp/logseq-repl/shared-shadow-watch.log` (logseq-repl skill)
|
||||
- `~/Library/Logs/Logseq/main.log`
|
||||
- `~/Library/Logs/Logseq/main.old.log`
|
||||
- `~/Library/Application Support/Logseq/configs.edn`
|
||||
- graph-local `db-worker-node-<timestamp>.log`
|
||||
|
||||
### Logseq CLI
|
||||
|
||||
Before using `logseq`, load `logseq-cli` skill.
|
||||
|
||||
## Required final output
|
||||
|
||||
The final response must include these sections or an equivalent structure:
|
||||
|
||||
1. **Runtime chosen** — which *runtime REPL* you used and why
|
||||
2. **Pre-fix reproduction** — reproduce bug in *runtime REPL*, exact steps and evidence
|
||||
3. **Root cause** — concrete cause and relevant files/flow
|
||||
4. **Fix applied** — short description of the change
|
||||
5. **Post-fix verification** — same steps again with new evidence
|
||||
6. **Additional verification** — tests/checks run, and what was not verified
|
||||
7. **Gaps or blockers** — any missing evidence and why
|
||||
|
||||
## Quick checklist
|
||||
|
||||
Before ending, make sure the answer is yes to all:
|
||||
|
||||
- Did I reproduce the bug before fixing it?
|
||||
- Did I show evidence, not just claim reproduction?
|
||||
- Did I inspect relevant logs?
|
||||
- Did I verify in the correct runtime?
|
||||
- Did I rerun the same scenario after the fix?
|
||||
- Did I include both before and after evidence in the final output?
|
||||
- Did I avoid claiming completion if required evidence is missing?
|
||||
|
||||
## Verification reminders
|
||||
|
||||
- Never run tests, lint, build, or E2E verification in the background.
|
||||
- Check logs before trusting REPL/CLI output alone.
|
||||
- For performance bugs, compare `--profile` before/after on the same graph and command.
|
||||
- For CLI bugs, reuse the same `--graph`, `--data-dir`, and output mode.
|
||||
- For REPL debugging, verify against the intended runtime, not a stale one.
|
||||
|
||||
Common checks: `bb dev:test -v <namespace/testcase-name>`, `bb dev:lint-and-test`, `bb dev:cli-e2e`.
|
||||
@@ -74,7 +74,10 @@ If the English text matches but the meaning differs, create a new key and follow
|
||||
### Rule 3: English source lives in `en.edn`
|
||||
|
||||
- Add new English source text to `src/resources/dicts/en.edn`.
|
||||
- Add non-English entries only when you are also providing actual translations.
|
||||
- **When introducing a new key for the first time, you must also add the
|
||||
Simplified Chinese (`zh-CN`) translation in the same change.** English and
|
||||
`zh-CN` are the two required locales for any new key.
|
||||
- Add other non-English entries only when you are also providing actual translations.
|
||||
- When renaming or removing keys, update affected locale files so stale keys do
|
||||
not remain behind.
|
||||
- Do not copy English into non-English locale files just to fill gaps. Tongue
|
||||
|
||||
205
.agents/skills/logseq-repl/SKILL.md
Normal file
205
.agents/skills/logseq-repl/SKILL.md
Normal file
@@ -0,0 +1,205 @@
|
||||
---
|
||||
name: logseq-repl
|
||||
description: Start and coordinate Logseq development REPL workflows for the Desktop renderer `:app`, Electron main-process `:electron`, and `:db-worker-node` runtimes through one unified workflow.
|
||||
---
|
||||
|
||||
# Logseq REPL Workflow
|
||||
|
||||
Use this skill when the user needs a Logseq development REPL for:
|
||||
|
||||
- Desktop renderer `:app`
|
||||
- Electron main process `:electron`
|
||||
- `:db-worker-node`
|
||||
- any combination of those runtimes
|
||||
|
||||
The workflow uses one shared state directory: `<repo>/tmp/logseq-repl/`.
|
||||
|
||||
## Scripts
|
||||
|
||||
Start everything with the default Logseq data root (`$LOGSEQ_CLI_ROOT_DIR` or `~/logseq`):
|
||||
|
||||
```bash
|
||||
./scripts/start-repl.sh --repo demo
|
||||
```
|
||||
|
||||
Start everything with an explicit Logseq data root:
|
||||
|
||||
```bash
|
||||
./scripts/start-repl.sh --repo demo --root-dir ~/logseq
|
||||
```
|
||||
|
||||
Clean up everything:
|
||||
|
||||
```bash
|
||||
./scripts/cleanup-repl.sh
|
||||
```
|
||||
|
||||
Verify all REPL targets after startup:
|
||||
|
||||
```bash
|
||||
./scripts/verify-repls.sh
|
||||
```
|
||||
|
||||
`start-repl.sh` starts:
|
||||
|
||||
1. shared `pnpm watch`
|
||||
2. Desktop dev app via `pnpm dev-electron-app`
|
||||
3. `db-worker-node` via `node ./static/db-worker-node.js --repo <name> --root-dir <path> --owner-source cli`
|
||||
|
||||
`start-repl.sh` is a small shell wrapper around `start-repl.py`. The Python script verifies that `:app`, `:electron`, and `:db-worker-node` runtimes are live, runs `verify-repls.sh` to connect to each target REPL and print a small result, then prints attach commands. It does not attach an interactive REPL by itself and exits after verification.
|
||||
|
||||
`cleanup-repl.sh` stops all workflow-managed processes without requiring a target selection. It also removes legacy state files from older split workflows and attempts to stop repo-owned listeners on the standard REPL ports.
|
||||
|
||||
`verify-repls.sh` connects to `app`, `electron`, and `db-worker-node` with `pnpm exec shadow-cljs cljs-repl`, evaluates one target-specific expression in each REPL, and prints the output.
|
||||
|
||||
## Standard Flow
|
||||
|
||||
Before starting or attaching:
|
||||
|
||||
```bash
|
||||
./scripts/cleanup-repl.sh
|
||||
```
|
||||
|
||||
Then start all runtimes:
|
||||
|
||||
```bash
|
||||
./scripts/start-repl.sh --repo demo
|
||||
```
|
||||
|
||||
Use `--root-dir <path>` if the target graph lives outside `$LOGSEQ_CLI_ROOT_DIR` or `~/logseq`.
|
||||
|
||||
Attach only to the target you need:
|
||||
|
||||
```bash
|
||||
pnpm exec shadow-cljs cljs-repl app
|
||||
pnpm exec shadow-cljs cljs-repl electron
|
||||
pnpm exec shadow-cljs cljs-repl db-worker-node
|
||||
```
|
||||
|
||||
## Runtime Selection
|
||||
|
||||
- Need DOM, UI, renderer state, or page rendering? Use `:app`.
|
||||
- Need Electron main process APIs, menus, window lifecycle, or main-process filesystem behavior? Use `:electron`.
|
||||
- Need Node worker behavior or db-worker-node code paths? Use `:db-worker-node`.
|
||||
|
||||
Runtime reminders:
|
||||
|
||||
- `:app` = Electron renderer
|
||||
- `:electron` = Electron main process
|
||||
- `:db-worker` = browser worker
|
||||
- `:db-worker-node` = Node worker
|
||||
|
||||
## Readiness Model
|
||||
|
||||
Keep these separate:
|
||||
|
||||
1. watch alive: a `shadow-cljs` server or `pnpm watch` process exists
|
||||
2. build ready: the target build completed successfully
|
||||
3. runtime attached: a live JS runtime is connected for `:app`, `:electron`, or `:db-worker-node`
|
||||
|
||||
If runtime count is `0`, do not attach yet. Fix runtime startup first.
|
||||
|
||||
Check runtime counts:
|
||||
|
||||
```bash
|
||||
pnpm exec shadow-cljs clj-eval "(do (require '[shadow.cljs.devtools.api :as api]) (println {:app (count (api/repl-runtimes :app)) :electron (count (api/repl-runtimes :electron)) :db-worker-node (count (api/repl-runtimes :db-worker-node))}))"
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `:app > 0` means a Desktop renderer runtime is attached
|
||||
- `:electron > 0` means an Electron main-process runtime is attached
|
||||
- `:db-worker-node > 0` means a worker-node runtime is attached
|
||||
- `0` means not ready, even if watch/build logs look healthy
|
||||
|
||||
## Logs
|
||||
|
||||
Look here first:
|
||||
|
||||
- `<repo>/tmp/logseq-repl/shared-shadow-watch.log`
|
||||
- `<repo>/tmp/logseq-repl/desktop-electron.log`
|
||||
- `<repo>/tmp/logseq-repl/db-worker-node.log`
|
||||
|
||||
## Port Audit
|
||||
|
||||
After cleanup, verify standard ports if startup still reports conflicts:
|
||||
|
||||
```bash
|
||||
lsof -nP -iTCP:8701 -sTCP:LISTEN
|
||||
lsof -nP -iTCP:3001 -sTCP:LISTEN
|
||||
lsof -nP -iTCP:3002 -sTCP:LISTEN
|
||||
lsof -nP -iTCP:9630 -sTCP:LISTEN
|
||||
lsof -nP -iTCP:9631 -sTCP:LISTEN
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- no listeners: clean enough to continue
|
||||
- listeners after cleanup: resolve the external conflict first
|
||||
- listeners only after startup: expected if owned by the workflow
|
||||
|
||||
## Non-Interactive Verification Examples
|
||||
|
||||
Desktop `:app`:
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | pnpm exec shadow-cljs cljs-repl app
|
||||
(prn {:runtime :app :document? (some? js/document) :title (.-title js/document)})
|
||||
:cljs/quit
|
||||
EOF
|
||||
```
|
||||
|
||||
Electron `:electron`:
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | pnpm exec shadow-cljs cljs-repl electron
|
||||
(prn {:runtime :electron :process? (some? js/process) :type (.-type js/process)})
|
||||
:cljs/quit
|
||||
EOF
|
||||
```
|
||||
|
||||
`db-worker-node`:
|
||||
|
||||
```bash
|
||||
cat <<'EOF' | pnpm exec shadow-cljs cljs-repl db-worker-node
|
||||
(prn {:runtime :db-worker-node :process? (some? js/process) :platform (.-platform js/process)})
|
||||
:cljs/quit
|
||||
EOF
|
||||
```
|
||||
|
||||
## Editor Attach Helpers
|
||||
|
||||
```clojure
|
||||
(shadow.user/cljs-repl)
|
||||
(shadow.user/electron-repl)
|
||||
(shadow.user/worker-node-repl)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Failure triage order:
|
||||
|
||||
1. inspect `tmp/logseq-repl/shared-shadow-watch.log`
|
||||
2. inspect `tmp/logseq-repl/desktop-electron.log`
|
||||
3. inspect `tmp/logseq-repl/db-worker-node.log`
|
||||
4. inspect standard port listeners with `lsof`
|
||||
5. inspect runtime counts with `shadow.cljs.devtools.api/repl-runtimes`
|
||||
|
||||
Common cases:
|
||||
|
||||
- `No available JS runtime`: the build may be ready, but the runtime has not connected. Check runtime counts before retrying attach.
|
||||
- multiple `:app` runtimes: close browser dev app instances so only the Desktop renderer remains.
|
||||
- ports already in use after cleanup: another Logseq/shadow-cljs dev session is still running.
|
||||
- `db-worker-node` repo mismatch: rerun `start-repl.sh --repo <name>`; it restarts the worker runtime for the requested repo.
|
||||
|
||||
## Recommended Response Pattern
|
||||
|
||||
When helping a user connect to a REPL:
|
||||
|
||||
1. identify whether they need `:app`, `:electron`, `:db-worker-node`, or a combination
|
||||
2. run `cleanup-repl.sh`
|
||||
3. if standard ports remain occupied, resolve that conflict first
|
||||
4. run `start-repl.sh --repo <name>`
|
||||
5. verify runtime counts if attach fails
|
||||
6. attach to the matching build or helper
|
||||
7. run `cleanup-repl.sh` when finished
|
||||
176
.agents/skills/logseq-repl/scripts/cleanup-repl.sh
Executable file
176
.agents/skills/logseq-repl/scripts/cleanup-repl.sh
Executable file
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DEFAULT_REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
|
||||
REPO_ROOT="${REPO_ROOT:-$DEFAULT_REPO_ROOT}"
|
||||
FORCE_KILL=0
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Stop all processes started by start-repl.sh.
|
||||
|
||||
Usage:
|
||||
cleanup-repl.sh [options]
|
||||
|
||||
Options:
|
||||
--repo-root <path> Logseq repository root (default: auto-detect from script location)
|
||||
--force Use SIGKILL if a process does not stop gracefully
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--repo-root)
|
||||
shift
|
||||
REPO_ROOT="${1:?missing value for --repo-root}"
|
||||
;;
|
||||
--force)
|
||||
FORCE_KILL=1
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
LOG_DIR="$REPO_ROOT/tmp/logseq-repl"
|
||||
LEGACY_DESKTOP_LOG_DIR="$REPO_ROOT/tmp/desktop-app-repl"
|
||||
LEGACY_DB_LOG_DIR="$REPO_ROOT/tmp/db-worker-node-repl"
|
||||
|
||||
is_running_pid() {
|
||||
local pid="$1"
|
||||
[[ "$pid" =~ ^[0-9]+$ ]] && kill -0 "$pid" 2>/dev/null
|
||||
}
|
||||
|
||||
read_pid() {
|
||||
local file="$1"
|
||||
if [[ -f "$file" ]]; then
|
||||
tr -d '[:space:]' < "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
process_group_id() {
|
||||
local pid="$1"
|
||||
ps -o pgid= -p "$pid" 2>/dev/null | tr -d '[:space:]'
|
||||
}
|
||||
|
||||
current_process_group_id() {
|
||||
process_group_id "$$"
|
||||
}
|
||||
|
||||
signal_process_group() {
|
||||
local signal="$1"
|
||||
local pid="$2"
|
||||
|
||||
local current_pgid pgid
|
||||
pgid="$(process_group_id "$pid" || true)"
|
||||
current_pgid="$(current_process_group_id || true)"
|
||||
|
||||
if [[ -n "${pgid:-}" && "$pgid" =~ ^[0-9]+$ && "$pgid" != "${current_pgid:-}" ]]; then
|
||||
kill "-$signal" "-$pgid" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
kill "-$signal" "$pid" 2>/dev/null || true
|
||||
}
|
||||
|
||||
stop_by_pid_file() {
|
||||
local pid_file="$1"
|
||||
local label="$2"
|
||||
|
||||
local pid
|
||||
pid="$(read_pid "$pid_file" || true)"
|
||||
|
||||
if [[ -z "${pid:-}" ]]; then
|
||||
rm -f "$pid_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if ! is_running_pid "$pid"; then
|
||||
echo "$label: process already stopped (pid=$pid)"
|
||||
rm -f "$pid_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "$label: stopping pid=$pid"
|
||||
signal_process_group TERM "$pid"
|
||||
|
||||
local i
|
||||
for ((i=0; i<10; i++)); do
|
||||
if ! is_running_pid "$pid"; then
|
||||
echo "$label: stopped"
|
||||
rm -f "$pid_file"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
if [[ "$FORCE_KILL" -eq 1 ]]; then
|
||||
echo "$label: force killing pid=$pid"
|
||||
signal_process_group KILL "$pid"
|
||||
sleep 1
|
||||
fi
|
||||
|
||||
if is_running_pid "$pid"; then
|
||||
echo "$label: failed to stop pid=$pid" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "$label: stopped"
|
||||
rm -f "$pid_file"
|
||||
}
|
||||
|
||||
repo_owns_pid() {
|
||||
local pid="$1"
|
||||
local cwd
|
||||
cwd="$(lsof -nP -a -p "$pid" -d cwd 2>/dev/null | awk 'NR > 1 {print $NF; exit}' || true)"
|
||||
[[ "$cwd" == "$REPO_ROOT" ]]
|
||||
}
|
||||
|
||||
stop_repo_port_listener() {
|
||||
local port="$1"
|
||||
local pid
|
||||
|
||||
while read -r pid; do
|
||||
[[ -n "${pid:-}" ]] || continue
|
||||
if is_running_pid "$pid" && repo_owns_pid "$pid"; then
|
||||
echo "port $port listener: stopping pid=$pid"
|
||||
signal_process_group TERM "$pid"
|
||||
fi
|
||||
done < <(lsof -nP -tiTCP:"$port" -sTCP:LISTEN 2>/dev/null || true)
|
||||
}
|
||||
|
||||
if [[ ! -d "$LOG_DIR" && ! -d "$LEGACY_DESKTOP_LOG_DIR" && ! -d "$LEGACY_DB_LOG_DIR" ]]; then
|
||||
echo "State directory not found: $LOG_DIR"
|
||||
echo "Nothing to clean up."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
stop_by_pid_file "$LOG_DIR/db-worker-node.pid" "db-worker-node"
|
||||
stop_by_pid_file "$LOG_DIR/desktop-electron.pid" "desktop-electron"
|
||||
stop_by_pid_file "$LOG_DIR/shared-shadow-watch.pid" "shadow-cljs watch"
|
||||
|
||||
stop_by_pid_file "$LEGACY_DB_LOG_DIR/db-worker-node.pid" "legacy db-worker-node"
|
||||
stop_by_pid_file "$LEGACY_DB_LOG_DIR/shadow-db-worker-node.pid" "legacy shadow-cljs watch"
|
||||
stop_by_pid_file "$LEGACY_DESKTOP_LOG_DIR/desktop-electron.pid" "legacy desktop-electron"
|
||||
stop_by_pid_file "$LEGACY_DESKTOP_LOG_DIR/shadow-watch.pid" "legacy shadow-cljs watch"
|
||||
|
||||
rm -f "$LOG_DIR/db-worker-node.repo" \
|
||||
"$LOG_DIR/db-worker-node.options.json" \
|
||||
"$LEGACY_DB_LOG_DIR/db-worker-node.repo"
|
||||
|
||||
for port in 8701 3001 3002 9630 9631; do
|
||||
stop_repo_port_listener "$port"
|
||||
done
|
||||
|
||||
rm -f "$LOG_DIR"/*.pid "$LEGACY_DB_LOG_DIR"/*.pid "$LEGACY_DESKTOP_LOG_DIR"/*.pid 2>/dev/null || true
|
||||
|
||||
echo "Cleanup done."
|
||||
74
.agents/skills/logseq-repl/scripts/common.sh
Normal file
74
.agents/skills/logseq-repl/scripts/common.sh
Normal file
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
logseq_repl_is_running_pid() {
|
||||
local pid="$1"
|
||||
[[ "$pid" =~ ^[0-9]+$ ]] && kill -0 "$pid" 2>/dev/null
|
||||
}
|
||||
|
||||
logseq_repl_read_pid() {
|
||||
local file="$1"
|
||||
if [[ -f "$file" ]]; then
|
||||
tr -d '[:space:]' < "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
logseq_repl_port_check_line() {
|
||||
local port="$1"
|
||||
lsof -nP -iTCP:"$port" -sTCP:LISTEN 2>/dev/null || true
|
||||
}
|
||||
|
||||
logseq_repl_audit_standard_ports() {
|
||||
local had_conflict=0
|
||||
local ports=(8701 3001 3002 9630 9631)
|
||||
local port output
|
||||
|
||||
for port in "${ports[@]}"; do
|
||||
output="$(logseq_repl_port_check_line "$port")"
|
||||
if [[ -n "$output" ]]; then
|
||||
had_conflict=1
|
||||
echo "Port $port is already listening:" >&2
|
||||
echo "$output" >&2
|
||||
echo >&2
|
||||
fi
|
||||
done
|
||||
|
||||
return "$had_conflict"
|
||||
}
|
||||
|
||||
logseq_repl_require_clean_standard_ports() {
|
||||
if logseq_repl_audit_standard_ports; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "Error: standard Logseq REPL ports are still occupied after cleanup." >&2
|
||||
echo "Resolve the external conflict first, then retry." >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
logseq_repl_runtime_count() {
|
||||
local repo_root="$1"
|
||||
local build_name="$2"
|
||||
local output
|
||||
|
||||
pushd "$repo_root" >/dev/null
|
||||
if ! output="$(pnpm exec shadow-cljs clj-eval "(do (require '[shadow.cljs.devtools.api :as api]) (println (count (api/repl-runtimes :$build_name))))" 2>&1)"; then
|
||||
popd >/dev/null
|
||||
echo "Error: failed to inspect :$build_name runtimes." >&2
|
||||
echo "--- clj-eval output ---" >&2
|
||||
echo "$output" >&2
|
||||
echo "-----------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
popd >/dev/null
|
||||
|
||||
local runtime_count
|
||||
runtime_count="$(printf '%s\n' "$output" | awk '/^[0-9]+$/{n=$0} END{if (n != "") print n; else exit 1}')" || {
|
||||
echo "Error: could not parse :$build_name runtime count." >&2
|
||||
echo "--- clj-eval output ---" >&2
|
||||
echo "$output" >&2
|
||||
echo "-----------------------" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
printf '%s\n' "$runtime_count"
|
||||
}
|
||||
374
.agents/skills/logseq-repl/scripts/start-repl.py
Executable file
374
.agents/skills/logseq-repl/scripts/start-repl.py
Executable file
@@ -0,0 +1,374 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
DEFAULT_REPO_ROOT = SCRIPT_DIR.parents[3]
|
||||
STANDARD_PORTS = (8701, 3001, 3002, 9630, 9631)
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(line_buffering=True)
|
||||
sys.stderr.reconfigure(line_buffering=True)
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="start-repl.sh",
|
||||
description="Start the unified Logseq REPL workflow.",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=(
|
||||
"This starts shared pnpm watch, the Desktop dev app, and the\n"
|
||||
"db-worker-node runtime, verifies the REPL targets, and exits."
|
||||
),
|
||||
)
|
||||
parser.add_argument("--repo", default=os.environ.get("DB_REPO", "demo"),
|
||||
help="Graph repo name passed to db-worker-node (default: demo)")
|
||||
parser.add_argument("--root-dir", default=os.environ.get("LOGSEQ_CLI_ROOT_DIR", str(Path.home() / "logseq")),
|
||||
help="Logseq data root passed to db-worker-node (default: $LOGSEQ_CLI_ROOT_DIR or ~/logseq)")
|
||||
parser.add_argument("--repo-root", default=os.environ.get("REPO_ROOT", str(DEFAULT_REPO_ROOT)),
|
||||
help="Logseq repository root (default: auto-detect from script location)")
|
||||
parser.add_argument("extra_node_args", nargs=argparse.REMAINDER,
|
||||
help="Arguments after -- are passed to db-worker-node")
|
||||
args = parser.parse_args()
|
||||
if args.extra_node_args and args.extra_node_args[0] == "--":
|
||||
args.extra_node_args = args.extra_node_args[1:]
|
||||
return args
|
||||
|
||||
|
||||
def require_command(name):
|
||||
if subprocess.run(["/usr/bin/env", "sh", "-c", f"command -v {name} >/dev/null 2>&1"]).returncode != 0:
|
||||
raise SystemExit(f"Error: {name} not found in PATH")
|
||||
|
||||
|
||||
def read_pid(path):
|
||||
try:
|
||||
text = path.read_text().strip()
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
return int(text) if re.fullmatch(r"\d+", text) else None
|
||||
|
||||
|
||||
def is_running(pid):
|
||||
if not pid:
|
||||
return False
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def write_pid(path, pid):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(f"{pid}\n")
|
||||
|
||||
|
||||
def wait_for_patterns(path, timeout, patterns, all_required=True):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if path.exists():
|
||||
text = path.read_text(errors="replace")
|
||||
if all_required and all(pattern in text for pattern in patterns):
|
||||
return True
|
||||
if not all_required and any(pattern in text for pattern in patterns):
|
||||
return True
|
||||
time.sleep(1)
|
||||
return False
|
||||
|
||||
|
||||
def process_group_kwargs():
|
||||
if os.name == "nt":
|
||||
return {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP}
|
||||
return {"start_new_session": True}
|
||||
|
||||
|
||||
def start_process(repo_root, log_path, command):
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
log_file = log_path.open("wb")
|
||||
return subprocess.Popen(
|
||||
command,
|
||||
cwd=repo_root,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
**process_group_kwargs(),
|
||||
)
|
||||
|
||||
|
||||
def port_listener_pid(port):
|
||||
try:
|
||||
output = subprocess.check_output(
|
||||
["lsof", "-nP", f"-iTCP:{port}", "-sTCP:LISTEN", "-t"],
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
except (FileNotFoundError, subprocess.CalledProcessError):
|
||||
return None
|
||||
for line in output.splitlines():
|
||||
if line.strip().isdigit():
|
||||
return int(line.strip())
|
||||
return None
|
||||
|
||||
|
||||
def has_managed_processes(paths):
|
||||
return any(is_running(read_pid(path)) for path in paths)
|
||||
|
||||
|
||||
def read_json(path):
|
||||
try:
|
||||
return json.loads(path.read_text())
|
||||
except (FileNotFoundError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
def write_json(path, payload):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def require_clean_ports():
|
||||
had_conflict = False
|
||||
for port in STANDARD_PORTS:
|
||||
pid = port_listener_pid(port)
|
||||
if pid:
|
||||
had_conflict = True
|
||||
print(f"Port {port} is already listening (pid={pid})", file=sys.stderr)
|
||||
if had_conflict:
|
||||
print("Error: standard Logseq REPL ports are still occupied after cleanup.", file=sys.stderr)
|
||||
print("Resolve the external conflict first, then retry.", file=sys.stderr)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def ensure_shadow_watch(repo_root, shadow_pid_file, shadow_log):
|
||||
pid = read_pid(shadow_pid_file)
|
||||
if is_running(pid):
|
||||
print(f"Reusing shared shadow-cljs watch (pid={pid})")
|
||||
return pid
|
||||
|
||||
print("Starting shared shadow-cljs watch via pnpm watch ...")
|
||||
process = start_process(repo_root, shadow_log, ["pnpm", "watch"])
|
||||
write_pid(shadow_pid_file, process.pid)
|
||||
time.sleep(1)
|
||||
|
||||
if process.poll() is not None:
|
||||
raise SystemExit(f"Error: pnpm watch exited early. Check {shadow_log}")
|
||||
|
||||
if not wait_for_patterns(shadow_log, 180, [
|
||||
"[:electron] Build completed.",
|
||||
"[:app] Build completed.",
|
||||
"[:db-worker-node] Build completed.",
|
||||
]):
|
||||
raise SystemExit(f"Error: pnpm watch did not finish the expected builds in time. Check {shadow_log}")
|
||||
|
||||
listener_pid = port_listener_pid(8701)
|
||||
if listener_pid and is_running(listener_pid):
|
||||
write_pid(shadow_pid_file, listener_pid)
|
||||
pid = listener_pid
|
||||
else:
|
||||
pid = process.pid
|
||||
|
||||
print("shadow-cljs watch builds are ready")
|
||||
return pid
|
||||
|
||||
|
||||
def ensure_desktop_app(repo_root, desktop_pid_file, desktop_log):
|
||||
pid = read_pid(desktop_pid_file)
|
||||
if is_running(pid):
|
||||
print(f"Reusing Desktop dev app (pid={pid})")
|
||||
return pid
|
||||
|
||||
print("Starting Desktop dev app via pnpm dev-electron-app ...")
|
||||
process = start_process(repo_root, desktop_log, ["pnpm", "dev-electron-app"])
|
||||
write_pid(desktop_pid_file, process.pid)
|
||||
time.sleep(1)
|
||||
|
||||
if process.poll() is not None:
|
||||
raise SystemExit(f"Error: pnpm dev-electron-app exited early. Check {desktop_log}")
|
||||
|
||||
if not wait_for_patterns(desktop_log, 120, ["shadow-cljs - #", "Logseq App("], all_required=False):
|
||||
raise SystemExit(f"Error: Desktop dev app did not report startup in time. Check {desktop_log}")
|
||||
|
||||
print(f"Desktop dev app is running (pid={process.pid})")
|
||||
return process.pid
|
||||
|
||||
|
||||
def ensure_db_worker_node(repo_root, db_pid_file, db_options_file, db_log, repo, root_dir, extra_args):
|
||||
pid = read_pid(db_pid_file)
|
||||
startup_options = {
|
||||
"repo": repo,
|
||||
"root_dir": str(root_dir),
|
||||
"owner_source": "cli",
|
||||
"extra_args": list(extra_args),
|
||||
}
|
||||
existing_options = read_json(db_options_file)
|
||||
|
||||
if is_running(pid):
|
||||
if existing_options == startup_options:
|
||||
print(f"Reusing db-worker-node runtime (pid={pid}, repo={repo}, root-dir={root_dir})")
|
||||
return pid
|
||||
print(f"Stopping existing db-worker-node (pid={pid}) due to startup option mismatch")
|
||||
terminate_pid(pid)
|
||||
time.sleep(1)
|
||||
|
||||
print(f"Starting db-worker-node (repo={repo}, root-dir={root_dir}) ...")
|
||||
process = start_process(
|
||||
repo_root,
|
||||
db_log,
|
||||
[
|
||||
"node",
|
||||
"./static/db-worker-node.js",
|
||||
"--repo",
|
||||
repo,
|
||||
"--root-dir",
|
||||
str(root_dir),
|
||||
"--owner-source",
|
||||
"cli",
|
||||
*extra_args,
|
||||
],
|
||||
)
|
||||
write_pid(db_pid_file, process.pid)
|
||||
write_json(db_options_file, startup_options)
|
||||
time.sleep(1)
|
||||
|
||||
if process.poll() is not None:
|
||||
raise SystemExit(f"Error: db-worker-node exited early. Check {db_log}")
|
||||
|
||||
print(f"db-worker-node is running (pid={process.pid}, repo={repo}, root-dir={root_dir})")
|
||||
return process.pid
|
||||
|
||||
|
||||
def runtime_count(repo_root, build_name):
|
||||
form = f"(do (require '[shadow.cljs.devtools.api :as api]) (println (count (api/repl-runtimes :{build_name}))))"
|
||||
proc = subprocess.run(
|
||||
["pnpm", "exec", "shadow-cljs", "clj-eval", form],
|
||||
cwd=repo_root,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise SystemExit(
|
||||
f"Error: failed to inspect :{build_name} runtimes.\n"
|
||||
f"--- clj-eval output ---\n{proc.stdout}{proc.stderr}\n-----------------------"
|
||||
)
|
||||
matches = re.findall(r"^(\d+)$", proc.stdout + proc.stderr, flags=re.MULTILINE)
|
||||
if not matches:
|
||||
raise SystemExit(
|
||||
f"Error: could not parse :{build_name} runtime count.\n"
|
||||
f"--- clj-eval output ---\n{proc.stdout}{proc.stderr}\n-----------------------"
|
||||
)
|
||||
return int(matches[-1])
|
||||
|
||||
|
||||
def wait_for_runtime_count(repo_root, build_name, expected, timeout):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
count = runtime_count(repo_root, build_name)
|
||||
if expected == "exactly-one":
|
||||
if count == 1:
|
||||
print(f"Detected exactly one live :{build_name} runtime")
|
||||
return
|
||||
if count != 0:
|
||||
raise SystemExit(f"Error: Expected exactly one live :{build_name} runtime, found {count}.")
|
||||
elif expected == "nonzero" and count != 0:
|
||||
print(f"Detected live :{build_name} runtime count: {count}")
|
||||
return
|
||||
time.sleep(1)
|
||||
|
||||
if expected == "exactly-one":
|
||||
raise SystemExit(f"Error: Expected exactly one live :{build_name} runtime, found 0 after waiting.")
|
||||
raise SystemExit(f"Error: expected a live :{build_name} runtime, but runtime count stayed 0.")
|
||||
|
||||
|
||||
def verify_repls(repo_root):
|
||||
subprocess.run([str(SCRIPT_DIR / "verify-repls.sh"), "--repo-root", str(repo_root)], check=True)
|
||||
|
||||
|
||||
def print_summary(shadow_log, desktop_log, db_log, shadow_pid_file, desktop_pid_file, db_pid_file):
|
||||
print()
|
||||
print("Logs:")
|
||||
print(f" shared shadow-cljs: {shadow_log}")
|
||||
print(f" desktop-app: {desktop_log}")
|
||||
print(f" db-worker-node: {db_log}")
|
||||
print("PID files:")
|
||||
print(f" {shadow_pid_file}")
|
||||
print(f" {desktop_pid_file}")
|
||||
print(f" {db_pid_file}")
|
||||
print()
|
||||
print("Attach commands:")
|
||||
print(" pnpm exec shadow-cljs cljs-repl app")
|
||||
print(" pnpm exec shadow-cljs cljs-repl electron")
|
||||
print(" pnpm exec shadow-cljs cljs-repl db-worker-node")
|
||||
print()
|
||||
print("Startup complete. Attach to the needed REPL manually.")
|
||||
|
||||
|
||||
def terminate_pid(pid):
|
||||
if not pid:
|
||||
return
|
||||
try:
|
||||
if os.name == "nt":
|
||||
os.kill(pid, signal.CTRL_BREAK_EVENT)
|
||||
else:
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
repo_root = Path(args.repo_root).resolve()
|
||||
db_root_dir = Path(args.root_dir).expanduser().resolve()
|
||||
if not repo_root.is_dir():
|
||||
raise SystemExit(f"Error: repo root not found: {repo_root}")
|
||||
|
||||
require_command("pnpm")
|
||||
require_command("node")
|
||||
|
||||
log_dir = repo_root / "tmp" / "logseq-repl"
|
||||
legacy_desktop_dir = repo_root / "tmp" / "desktop-app-repl"
|
||||
legacy_db_dir = repo_root / "tmp" / "db-worker-node-repl"
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
shadow_pid_file = log_dir / "shared-shadow-watch.pid"
|
||||
desktop_pid_file = log_dir / "desktop-electron.pid"
|
||||
db_pid_file = log_dir / "db-worker-node.pid"
|
||||
db_options_file = log_dir / "db-worker-node.options.json"
|
||||
shadow_log = log_dir / "shared-shadow-watch.log"
|
||||
desktop_log = log_dir / "desktop-electron.log"
|
||||
db_log = log_dir / "db-worker-node.log"
|
||||
|
||||
managed_pid_files = [
|
||||
shadow_pid_file,
|
||||
desktop_pid_file,
|
||||
db_pid_file,
|
||||
legacy_desktop_dir / "shadow-watch.pid",
|
||||
legacy_desktop_dir / "desktop-electron.pid",
|
||||
legacy_db_dir / "shadow-db-worker-node.pid",
|
||||
legacy_db_dir / "db-worker-node.pid",
|
||||
]
|
||||
|
||||
if not has_managed_processes(managed_pid_files) and not require_clean_ports():
|
||||
return 1
|
||||
|
||||
ensure_shadow_watch(repo_root, shadow_pid_file, shadow_log)
|
||||
ensure_desktop_app(repo_root, desktop_pid_file, desktop_log)
|
||||
ensure_db_worker_node(repo_root, db_pid_file, db_options_file, db_log, args.repo, db_root_dir, args.extra_node_args)
|
||||
wait_for_runtime_count(repo_root, "app", "exactly-one", 60)
|
||||
wait_for_runtime_count(repo_root, "electron", "nonzero", 60)
|
||||
wait_for_runtime_count(repo_root, "db-worker-node", "nonzero", 60)
|
||||
verify_repls(repo_root)
|
||||
print_summary(shadow_log, desktop_log, db_log, shadow_pid_file, desktop_pid_file, db_pid_file)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
5
.agents/skills/logseq-repl/scripts/start-repl.sh
Executable file
5
.agents/skills/logseq-repl/scripts/start-repl.sh
Executable file
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
exec python3 "$SCRIPT_DIR/start-repl.py" "$@"
|
||||
86
.agents/skills/logseq-repl/scripts/verify-repls.sh
Executable file
86
.agents/skills/logseq-repl/scripts/verify-repls.sh
Executable file
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DEFAULT_REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
|
||||
REPO_ROOT="${REPO_ROOT:-$DEFAULT_REPO_ROOT}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Verify that all Logseq CLJS REPL targets are usable.
|
||||
|
||||
Usage:
|
||||
verify-repls.sh [options]
|
||||
|
||||
Options:
|
||||
--repo-root <path> Logseq repository root (default: auto-detect from script location)
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--repo-root)
|
||||
shift
|
||||
REPO_ROOT="${1:?missing value for --repo-root}"
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
if [[ ! -d "$REPO_ROOT" ]]; then
|
||||
echo "Error: repo root not found: $REPO_ROOT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v pnpm >/dev/null 2>&1; then
|
||||
echo "Error: pnpm not found in PATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
verify_target() {
|
||||
local target="$1"
|
||||
local form="$2"
|
||||
|
||||
echo "Checking :$target ..."
|
||||
|
||||
local repl_output
|
||||
pushd "$REPO_ROOT" >/dev/null
|
||||
if ! repl_output="$(printf '%s\n' "$form" | pnpm exec shadow-cljs cljs-repl "$target" 2>&1)"; then
|
||||
popd >/dev/null
|
||||
echo "Error: REPL verification failed for :$target." >&2
|
||||
echo "--- :$target output ---" >&2
|
||||
echo "$repl_output" >&2
|
||||
echo "-----------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
popd >/dev/null
|
||||
|
||||
if [[ "$repl_output" != *"shadow-cljs - connected to server"* ]]; then
|
||||
echo "Error: REPL verification did not connect for :$target." >&2
|
||||
echo "--- :$target output ---" >&2
|
||||
echo "$repl_output" >&2
|
||||
echo "-----------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "--- :$target result ---"
|
||||
echo "$repl_output"
|
||||
echo "-----------------------"
|
||||
echo "REPL verification passed for :$target"
|
||||
}
|
||||
|
||||
echo "Verifying CLJS REPL targets ..."
|
||||
verify_target app "(prn {:runtime :app :document? (some? js/document)})"
|
||||
verify_target electron "(prn {:runtime :electron :process? (some? js/process) :type (some-> js/process .-type)})"
|
||||
verify_target db-worker-node "(prn {:runtime :db-worker-node :process? (some? js/process) :platform (some-> js/process .-platform)})"
|
||||
echo "All CLJS REPL targets verified."
|
||||
83
.agents/skills/logseq-repl/tests/test-lib.sh
Normal file
83
.agents/skills/logseq-repl/tests/test-lib.sh
Normal file
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
fail() {
|
||||
echo "FAIL: $*" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
assert_file_exists() {
|
||||
local path="$1"
|
||||
[[ -e "$path" ]] || fail "expected file to exist: $path"
|
||||
}
|
||||
|
||||
assert_not_exists() {
|
||||
local path="$1"
|
||||
[[ ! -e "$path" ]] || fail "expected path to be absent: $path"
|
||||
}
|
||||
|
||||
assert_contains() {
|
||||
local needle="$1"
|
||||
local haystack_file="$2"
|
||||
if ! grep -Fq "$needle" "$haystack_file"; then
|
||||
echo "Expected to find: $needle" >&2
|
||||
echo "--- file: $haystack_file ---" >&2
|
||||
cat "$haystack_file" >&2
|
||||
echo "----------------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
assert_not_contains_text() {
|
||||
local needle="$1"
|
||||
local file="$2"
|
||||
if grep -Fq "$needle" "$file"; then
|
||||
echo "Did not expect to find: $needle" >&2
|
||||
echo "--- file: $file ---" >&2
|
||||
cat "$file" >&2
|
||||
echo "--------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
assert_not_matches() {
|
||||
local pattern="$1"
|
||||
local file="$2"
|
||||
if grep -Eq "$pattern" "$file"; then
|
||||
echo "Did not expect regex match: $pattern" >&2
|
||||
echo "--- file: $file ---" >&2
|
||||
cat "$file" >&2
|
||||
echo "--------------------" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
assert_equals() {
|
||||
local expected="$1"
|
||||
local actual="$2"
|
||||
[[ "$expected" == "$actual" ]] || fail "expected '$expected' but got '$actual'"
|
||||
}
|
||||
|
||||
portable_path_pattern() {
|
||||
local slash='/'
|
||||
local windows_drive='[A-Za-z]:\\[^[:space:]]+'
|
||||
printf '%s' "${slash}Users${slash}[^[:space:]]+|${slash}home${slash}[^[:space:]]+|${windows_drive}"
|
||||
}
|
||||
|
||||
run_test() {
|
||||
local name="$1"
|
||||
local fn="$2"
|
||||
|
||||
local status
|
||||
set +e
|
||||
(set -e; "$fn")
|
||||
status=$?
|
||||
set -e
|
||||
|
||||
if [[ "$status" -eq 0 ]]; then
|
||||
echo "PASS: $name"
|
||||
PASS_COUNT=$((PASS_COUNT + 1))
|
||||
else
|
||||
echo "FAIL: $name" >&2
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
fi
|
||||
}
|
||||
438
.agents/skills/logseq-repl/tests/test-logseq-repl.sh
Executable file
438
.agents/skills/logseq-repl/tests/test-logseq-repl.sh
Executable file
@@ -0,0 +1,438 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
TEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILL_DIR="$(cd "$TEST_DIR/.." && pwd)"
|
||||
START_SCRIPT="$SKILL_DIR/scripts/start-repl.sh"
|
||||
START_PY_SCRIPT="$SKILL_DIR/scripts/start-repl.py"
|
||||
CLEANUP_SCRIPT="$SKILL_DIR/scripts/cleanup-repl.sh"
|
||||
VERIFY_SCRIPT="$SKILL_DIR/scripts/verify-repls.sh"
|
||||
SKILL_FILE="$SKILL_DIR/SKILL.md"
|
||||
ORIGINAL_PATH="$PATH"
|
||||
|
||||
# shellcheck source=./test-lib.sh
|
||||
source "$TEST_DIR/test-lib.sh"
|
||||
|
||||
PASS_COUNT=0
|
||||
FAIL_COUNT=0
|
||||
|
||||
create_fake_env() {
|
||||
TEST_ROOT="$(mktemp -d)"
|
||||
REPO_ROOT="$TEST_ROOT/repo"
|
||||
DB_ROOT_DIR="$TEST_ROOT/logseq-root"
|
||||
BIN_DIR="$TEST_ROOT/bin"
|
||||
CMD_LOG="$TEST_ROOT/commands.log"
|
||||
|
||||
mkdir -p "$REPO_ROOT/static" "$DB_ROOT_DIR" "$BIN_DIR"
|
||||
DB_ROOT_DIR="$(cd "$DB_ROOT_DIR" && pwd -P)"
|
||||
: > "$CMD_LOG"
|
||||
|
||||
cat > "$BIN_DIR/pnpm" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
echo "pnpm $*" >> "$FAKE_CMD_LOG"
|
||||
|
||||
case "${1:-}" in
|
||||
watch)
|
||||
echo "shadow-cljs - nREPL server started on port 8701"
|
||||
echo "[:electron] Build completed."
|
||||
echo "[:app] Build completed."
|
||||
echo "[:db-worker-node] Build completed."
|
||||
while true; do sleep 1; done
|
||||
;;
|
||||
dev-electron-app)
|
||||
echo "17:12:00.841 Logseq App(2.0.1) Starting..."
|
||||
echo "shadow-cljs - #6 ready!"
|
||||
while true; do sleep 1; done
|
||||
;;
|
||||
exec)
|
||||
shift
|
||||
if [[ "${1:-}" != "shadow-cljs" ]]; then
|
||||
echo "Unexpected pnpm exec command: $*" >&2
|
||||
exit 1
|
||||
fi
|
||||
shift
|
||||
|
||||
input=""
|
||||
if [[ "${1:-}" == "cljs-repl" ]] && [[ -p /dev/stdin ]]; then
|
||||
input="$(cat)"
|
||||
fi
|
||||
|
||||
if [[ "${1:-}" == "clj-eval" ]]; then
|
||||
echo "pnpm-exec-clj-eval shadow-cljs $*" >> "$FAKE_CMD_LOG"
|
||||
echo "shadow-cljs - connected to server"
|
||||
case "$*" in
|
||||
*"repl-runtimes :app"*) echo "${FAKE_APP_RUNTIME_COUNT:-1}" ;;
|
||||
*"repl-runtimes :electron"*) echo "${FAKE_ELECTRON_RUNTIME_COUNT:-1}" ;;
|
||||
*"repl-runtimes :db-worker-node"*) echo "${FAKE_DB_RUNTIME_COUNT:-1}" ;;
|
||||
*) echo "0" ;;
|
||||
esac
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${1:-}" == "cljs-repl" ]]; then
|
||||
echo "pnpm-exec-repl ${2:-missing}" >> "$FAKE_CMD_LOG"
|
||||
echo "shadow-cljs - connected to server"
|
||||
|
||||
if [[ "$input" == *":cljs/quit"* ]]; then
|
||||
echo "verification must not send :cljs/quit" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
case "${2:-}" in
|
||||
app)
|
||||
if [[ "$input" != *":runtime :app"* ]]; then
|
||||
echo "unexpected app verification form" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo 'cljs.user=> {:runtime :app, :document? true}'
|
||||
;;
|
||||
electron)
|
||||
if [[ "$input" != *":runtime :electron"* ]]; then
|
||||
echo "unexpected electron verification form" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo 'cljs.user=> {:runtime :electron, :process? true}'
|
||||
;;
|
||||
db-worker-node)
|
||||
if [[ "$input" != *":runtime :db-worker-node"* ]]; then
|
||||
echo "unexpected db-worker-node verification form" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo 'cljs.user=> {:runtime :db-worker-node, :process? true}'
|
||||
;;
|
||||
*)
|
||||
echo "Unexpected cljs-repl target: ${2:-}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "cljs.user=>"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Unexpected shadow-cljs command: $*" >&2
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
echo "Unexpected pnpm command: $*" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
EOF
|
||||
|
||||
cat > "$BIN_DIR/node" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
echo "node $*" >> "$FAKE_CMD_LOG"
|
||||
|
||||
repo=""
|
||||
root_dir=""
|
||||
owner_source=""
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
./static/db-worker-node.js)
|
||||
shift
|
||||
;;
|
||||
--repo)
|
||||
repo="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--root-dir)
|
||||
root_dir="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--owner-source)
|
||||
owner_source="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ -z "$repo" ]]; then
|
||||
echo "repo is required" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "$root_dir" ]]; then
|
||||
echo "root-dir is required" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$owner_source" != "cli" ]]; then
|
||||
echo "owner-source cli is required" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "shadow-cljs - #6 ready!"
|
||||
while true; do sleep 1; done
|
||||
EOF
|
||||
|
||||
cat > "$BIN_DIR/lsof" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
exit 1
|
||||
EOF
|
||||
|
||||
cat > "$BIN_DIR/python3" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
exec "$(command -v python3)" "\$@"
|
||||
EOF
|
||||
|
||||
chmod +x "$BIN_DIR/pnpm" "$BIN_DIR/node" "$BIN_DIR/lsof" "$BIN_DIR/python3"
|
||||
|
||||
export PATH="$BIN_DIR:$ORIGINAL_PATH"
|
||||
export FAKE_CMD_LOG="$CMD_LOG"
|
||||
export FAKE_APP_RUNTIME_COUNT="${FAKE_APP_RUNTIME_COUNT:-1}"
|
||||
export FAKE_ELECTRON_RUNTIME_COUNT="${FAKE_ELECTRON_RUNTIME_COUNT:-1}"
|
||||
export FAKE_DB_RUNTIME_COUNT="${FAKE_DB_RUNTIME_COUNT:-1}"
|
||||
}
|
||||
|
||||
cleanup_fake_env() {
|
||||
if [[ -n "${REPO_ROOT:-}" ]]; then
|
||||
local pid_file pid
|
||||
for pid_file in "$REPO_ROOT"/tmp/logseq-repl/*.pid; do
|
||||
[[ -e "$pid_file" ]] || continue
|
||||
pid="$(tr -d '[:space:]' < "$pid_file")"
|
||||
if [[ "$pid" =~ ^[0-9]+$ ]]; then
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
PATH="$ORIGINAL_PATH"
|
||||
unset FAKE_CMD_LOG FAKE_APP_RUNTIME_COUNT FAKE_ELECTRON_RUNTIME_COUNT FAKE_DB_RUNTIME_COUNT || true
|
||||
|
||||
if [[ -n "${TEST_ROOT:-}" && -d "$TEST_ROOT" ]]; then
|
||||
rm -rf "$TEST_ROOT"
|
||||
fi
|
||||
}
|
||||
|
||||
scripts_exist_test() {
|
||||
assert_file_exists "$START_SCRIPT"
|
||||
assert_file_exists "$START_PY_SCRIPT"
|
||||
assert_file_exists "$CLEANUP_SCRIPT"
|
||||
assert_file_exists "$VERIFY_SCRIPT"
|
||||
}
|
||||
|
||||
start_launches_all_repl_processes_without_attaching_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1
|
||||
|
||||
assert_contains "Verifying CLJS REPL targets ..." "$TEST_ROOT/start.log"
|
||||
assert_contains "REPL verification passed for :app" "$TEST_ROOT/start.log"
|
||||
assert_contains "REPL verification passed for :electron" "$TEST_ROOT/start.log"
|
||||
assert_contains "REPL verification passed for :db-worker-node" "$TEST_ROOT/start.log"
|
||||
assert_contains "Startup complete. Attach to the needed REPL manually." "$TEST_ROOT/start.log"
|
||||
assert_contains "pnpm exec shadow-cljs cljs-repl app" "$TEST_ROOT/start.log"
|
||||
assert_contains "pnpm exec shadow-cljs cljs-repl electron" "$TEST_ROOT/start.log"
|
||||
assert_contains "pnpm exec shadow-cljs cljs-repl db-worker-node" "$TEST_ROOT/start.log"
|
||||
assert_contains "pnpm watch" "$CMD_LOG"
|
||||
assert_contains "pnpm dev-electron-app" "$CMD_LOG"
|
||||
assert_contains "node ./static/db-worker-node.js --repo demo --root-dir $DB_ROOT_DIR --owner-source cli" "$CMD_LOG"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.log"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.log"
|
||||
assert_file_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.log"
|
||||
}
|
||||
|
||||
verify_script_checks_all_targets_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
bash "$VERIFY_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/verify.log" 2>&1
|
||||
|
||||
assert_contains "Verifying CLJS REPL targets ..." "$TEST_ROOT/verify.log"
|
||||
assert_contains "REPL verification passed for :app" "$TEST_ROOT/verify.log"
|
||||
assert_contains "REPL verification passed for :electron" "$TEST_ROOT/verify.log"
|
||||
assert_contains "REPL verification passed for :db-worker-node" "$TEST_ROOT/verify.log"
|
||||
assert_contains "All CLJS REPL targets verified." "$TEST_ROOT/verify.log"
|
||||
assert_contains "pnpm-exec-repl app" "$CMD_LOG"
|
||||
assert_contains "pnpm-exec-repl electron" "$CMD_LOG"
|
||||
assert_contains "pnpm-exec-repl db-worker-node" "$CMD_LOG"
|
||||
}
|
||||
|
||||
verify_script_fails_when_target_repl_fails_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
cat > "$BIN_DIR/pnpm" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
echo "pnpm $*" >> "$FAKE_CMD_LOG"
|
||||
|
||||
if [[ "${1:-}" == "exec" && "${2:-}" == "shadow-cljs" && "${3:-}" == "cljs-repl" && "${4:-}" == "electron" ]]; then
|
||||
echo "No available JS runtime" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "shadow-cljs - connected to server"
|
||||
echo "cljs.user=> {:ok true}"
|
||||
EOF
|
||||
chmod +x "$BIN_DIR/pnpm"
|
||||
|
||||
if bash "$VERIFY_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/verify.log" 2>&1; then
|
||||
fail "expected verify script to fail when one target REPL fails"
|
||||
fi
|
||||
|
||||
assert_contains "Error: REPL verification failed for :electron." "$TEST_ROOT/verify.log"
|
||||
assert_contains "No available JS runtime" "$TEST_ROOT/verify.log"
|
||||
}
|
||||
|
||||
start_reuses_all_running_processes_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/first.log" 2>&1
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/second.log" 2>&1
|
||||
|
||||
assert_equals "1" "$(grep -c '^pnpm watch$' "$CMD_LOG")"
|
||||
assert_equals "1" "$(grep -c '^pnpm dev-electron-app$' "$CMD_LOG")"
|
||||
assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $DB_ROOT_DIR --owner-source cli" "$CMD_LOG")"
|
||||
assert_contains "Reusing shared shadow-cljs watch" "$TEST_ROOT/second.log"
|
||||
assert_contains "Reusing Desktop dev app" "$TEST_ROOT/second.log"
|
||||
assert_contains "Reusing db-worker-node runtime" "$TEST_ROOT/second.log"
|
||||
}
|
||||
|
||||
start_restarts_db_worker_when_root_dir_changes_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
local first_root second_root
|
||||
first_root="$TEST_ROOT/logseq-root-a"
|
||||
second_root="$TEST_ROOT/logseq-root-b"
|
||||
mkdir -p "$first_root" "$second_root"
|
||||
first_root="$(cd "$first_root" && pwd -P)"
|
||||
second_root="$(cd "$second_root" && pwd -P)"
|
||||
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$first_root" --repo demo > "$TEST_ROOT/first.log" 2>&1
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$second_root" --repo demo > "$TEST_ROOT/second.log" 2>&1
|
||||
|
||||
assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $first_root --owner-source cli" "$CMD_LOG")"
|
||||
assert_equals "1" "$(grep -Fc "node ./static/db-worker-node.js --repo demo --root-dir $second_root --owner-source cli" "$CMD_LOG")"
|
||||
assert_contains "Stopping existing db-worker-node" "$TEST_ROOT/second.log"
|
||||
}
|
||||
|
||||
start_fails_when_app_runtime_is_ambiguous_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
export FAKE_APP_RUNTIME_COUNT=2
|
||||
|
||||
if bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1; then
|
||||
fail "expected start script to fail when more than one :app runtime exists"
|
||||
fi
|
||||
|
||||
assert_contains "Expected exactly one live :app runtime" "$TEST_ROOT/start.log"
|
||||
}
|
||||
|
||||
cleanup_stops_all_repl_processes_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
bash "$START_SCRIPT" --repo-root "$REPO_ROOT" --root-dir "$DB_ROOT_DIR" --repo demo > "$TEST_ROOT/start.log" 2>&1
|
||||
|
||||
local watch_pid desktop_pid db_pid
|
||||
watch_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid")"
|
||||
desktop_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid")"
|
||||
db_pid="$(tr -d '[:space:]' < "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid")"
|
||||
|
||||
bash "$CLEANUP_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/cleanup.log" 2>&1
|
||||
|
||||
if kill -0 "$watch_pid" 2>/dev/null; then
|
||||
fail "expected shared watch to stop"
|
||||
fi
|
||||
|
||||
if kill -0 "$desktop_pid" 2>/dev/null; then
|
||||
fail "expected Desktop dev app to stop"
|
||||
fi
|
||||
|
||||
if kill -0 "$db_pid" 2>/dev/null; then
|
||||
fail "expected db-worker-node to stop"
|
||||
fi
|
||||
|
||||
assert_not_exists "$REPO_ROOT/tmp/logseq-repl/shared-shadow-watch.pid"
|
||||
assert_not_exists "$REPO_ROOT/tmp/logseq-repl/desktop-electron.pid"
|
||||
assert_not_exists "$REPO_ROOT/tmp/logseq-repl/db-worker-node.pid"
|
||||
assert_contains "Cleanup done." "$TEST_ROOT/cleanup.log"
|
||||
}
|
||||
|
||||
cleanup_removes_legacy_state_files_test() {
|
||||
create_fake_env
|
||||
trap cleanup_fake_env RETURN
|
||||
|
||||
mkdir -p "$REPO_ROOT/tmp/desktop-app-repl" "$REPO_ROOT/tmp/db-worker-node-repl" "$REPO_ROOT/tmp/logseq-repl"
|
||||
sleep 30 &
|
||||
local legacy_desktop_pid=$!
|
||||
sleep 30 &
|
||||
local legacy_db_pid=$!
|
||||
echo "$legacy_desktop_pid" > "$REPO_ROOT/tmp/desktop-app-repl/desktop-electron.pid"
|
||||
echo "$legacy_db_pid" > "$REPO_ROOT/tmp/db-worker-node-repl/db-worker-node.pid"
|
||||
echo "$legacy_db_pid" > "$REPO_ROOT/tmp/db-worker-node-repl/shadow-db-worker-node.pid"
|
||||
|
||||
bash "$CLEANUP_SCRIPT" --repo-root "$REPO_ROOT" > "$TEST_ROOT/cleanup.log" 2>&1
|
||||
|
||||
if kill -0 "$legacy_desktop_pid" 2>/dev/null; then
|
||||
fail "expected legacy desktop pid to stop"
|
||||
fi
|
||||
|
||||
if kill -0 "$legacy_db_pid" 2>/dev/null; then
|
||||
fail "expected legacy db-worker pid to stop"
|
||||
fi
|
||||
|
||||
assert_not_exists "$REPO_ROOT/tmp/desktop-app-repl/desktop-electron.pid"
|
||||
assert_not_exists "$REPO_ROOT/tmp/db-worker-node-repl/db-worker-node.pid"
|
||||
assert_not_exists "$REPO_ROOT/tmp/db-worker-node-repl/shadow-db-worker-node.pid"
|
||||
}
|
||||
|
||||
help_and_docs_describe_unified_scripts_test() {
|
||||
local temp_dir start_help cleanup_help
|
||||
temp_dir="$(mktemp -d)"
|
||||
start_help="$temp_dir/start-help.txt"
|
||||
cleanup_help="$temp_dir/cleanup-help.txt"
|
||||
|
||||
bash "$START_SCRIPT" --help > "$start_help"
|
||||
bash "$CLEANUP_SCRIPT" --help > "$cleanup_help"
|
||||
|
||||
assert_contains "start-repl.sh" "$start_help"
|
||||
assert_contains "start-repl.py" "$SKILL_FILE"
|
||||
assert_contains "cleanup-repl.sh" "$cleanup_help"
|
||||
assert_contains "start-repl.sh" "$SKILL_FILE"
|
||||
assert_contains "cleanup-repl.sh" "$SKILL_FILE"
|
||||
assert_contains "verify-repls.sh" "$SKILL_FILE"
|
||||
assert_not_contains_text "start-desktop-app-repl.sh" "$SKILL_FILE"
|
||||
assert_not_contains_text "start-db-worker-node-repl.sh" "$SKILL_FILE"
|
||||
assert_not_contains_text "cleanup-desktop-app-repl.sh" "$SKILL_FILE"
|
||||
assert_not_contains_text "cleanup-db-worker-node-repl.sh" "$SKILL_FILE"
|
||||
|
||||
rm -rf "$temp_dir"
|
||||
}
|
||||
|
||||
run_test "scripts exist" scripts_exist_test
|
||||
run_test "start launches all REPL processes without attaching" start_launches_all_repl_processes_without_attaching_test
|
||||
run_test "verify script checks all targets" verify_script_checks_all_targets_test
|
||||
run_test "verify script fails when target REPL fails" verify_script_fails_when_target_repl_fails_test
|
||||
run_test "start reuses all running processes" start_reuses_all_running_processes_test
|
||||
run_test "start restarts db-worker when root-dir changes" start_restarts_db_worker_when_root_dir_changes_test
|
||||
run_test "start fails when app runtime is ambiguous" start_fails_when_app_runtime_is_ambiguous_test
|
||||
run_test "cleanup stops all REPL processes" cleanup_stops_all_repl_processes_test
|
||||
run_test "cleanup removes legacy state files" cleanup_removes_legacy_state_files_test
|
||||
run_test "help and docs describe unified scripts" help_and_docs_describe_unified_scripts_test
|
||||
|
||||
echo
|
||||
if [[ "$FAIL_COUNT" -gt 0 ]]; then
|
||||
echo "$FAIL_COUNT test(s) failed; $PASS_COUNT passed." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "All $PASS_COUNT test(s) passed."
|
||||
@@ -60,6 +60,10 @@ frontend.ui/_emoji-init-data
|
||||
frontend.worker.rtc.op-mem-layer/_sync-loop-canceler
|
||||
;; Used by shadow.cljs
|
||||
frontend.worker.db-worker/init
|
||||
;; Used by shadow.cljs (node entrypoint)
|
||||
frontend.worker.db-worker-node/main
|
||||
;; CLI entrypoint (shadow-cljs :node-script)
|
||||
logseq.cli.main/main
|
||||
;; Future use?
|
||||
frontend.worker.rtc.hash/hash-blocks
|
||||
;; Repl fn
|
||||
|
||||
13
.github/workflows/build.yml
vendored
13
.github/workflows/build.yml
vendored
@@ -62,6 +62,7 @@ jobs:
|
||||
uses: DeLaGuardo/setup-clojure@13.5
|
||||
with:
|
||||
cli: ${{ env.CLOJURE_VERSION }}
|
||||
bb: ${{ env.BABASHKA_VERSION }}
|
||||
|
||||
- name: Clojure cache
|
||||
uses: actions/cache@v4
|
||||
@@ -84,11 +85,21 @@ jobs:
|
||||
run: clojure -M:test compile test
|
||||
|
||||
- name: Run ClojureScript query tests against basic query type
|
||||
run: DB_QUERY_TYPE=basic node static/tests.js -r frontend.db.query-dsl-test
|
||||
run: DB_QUERY_TYPE=basic pnpm cljs:run-test -r frontend.db.query-dsl-test
|
||||
|
||||
- name: Run ClojureScript tests
|
||||
run: pnpm cljs:run-test
|
||||
|
||||
- name: Setup CLI integration test
|
||||
# logseq-cli.js needed just for test-cli-query-human-output-pipes-to-show
|
||||
run: clojure -M:cljs compile logseq-cli && pnpm db-worker-node:release:bundle
|
||||
|
||||
- name: Run ClojureScript integration tests
|
||||
run: pnpm cljs:run-integration-test
|
||||
|
||||
- name: Run CLI E2E tests
|
||||
run: bb dev:cli-e2e --verbose --skip-build
|
||||
|
||||
lint:
|
||||
runs-on: ubuntu-22.04
|
||||
|
||||
|
||||
3
.github/workflows/deps-graph-parser.yml
vendored
3
.github/workflows/deps-graph-parser.yml
vendored
@@ -83,9 +83,6 @@ jobs:
|
||||
- name: Fetch pnpm deps
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Run ClojureScript tests
|
||||
run: clojure -M:test
|
||||
|
||||
- name: Run nbb-logseq tests
|
||||
run: pnpm test
|
||||
|
||||
|
||||
11
.gitignore
vendored
11
.gitignore
vendored
@@ -16,6 +16,7 @@ node_modules/
|
||||
static/**
|
||||
tmp
|
||||
cljs-test-runner-out
|
||||
.tmp/
|
||||
|
||||
.cpcache/
|
||||
/src/gen
|
||||
@@ -44,6 +45,7 @@ resources/electron.js
|
||||
/libs/dist/
|
||||
charlie/
|
||||
.vscode
|
||||
/.claude
|
||||
/.preprocessor-cljs
|
||||
docker
|
||||
android/app/src/main/assets/capacitor.plugin.json
|
||||
@@ -79,3 +81,12 @@ clj-e2e/e2e-dump
|
||||
.dir-locals.el
|
||||
.projectile
|
||||
deps/db-sync/data
|
||||
*.map
|
||||
/dist/db-worker-node.js
|
||||
/dist/build/
|
||||
/dist/db-worker-node-assets.json
|
||||
/dist/*.wasm
|
||||
/dist/cljs-runtime/
|
||||
/.agent-shell/
|
||||
|
||||
clj-e2e/.clj-kondo/imports
|
||||
30
AGENTS.md
30
AGENTS.md
@@ -1,20 +1,20 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Project Structure & Module Organization
|
||||
- `src/` is the main codebase.
|
||||
- `src/main/` contains core application logic.
|
||||
- `src/main/mobile/` is the mobile app code.
|
||||
- `src/main/frontend/components/` houses UI components.
|
||||
- `src/main/frontend/worker/` holds webworker code, including RTC in `src/main/frontend/worker/rtc/`.
|
||||
- `src/electron/` is Electron-specific code.
|
||||
- `src/test/` contains unit tests.
|
||||
- `deps/` contains internal dependencies/modules.
|
||||
- `clj-e2e/` contains end-to-end tests.
|
||||
|
||||
## Build, Test, and Development Commands
|
||||
- `bb dev:lint-and-test` runs linters and unit tests.
|
||||
- `bb dev:test -v <namespace/testcase-name>` runs a single unit test (example: `bb dev:test -v logseq.some-test/foo`).
|
||||
- E2E tests live in `clj-e2e/`; run them from that directory if needed.
|
||||
- App E2E tests live in `clj-e2e/`; run from that directory with `bb test` (or `bb -f clj-e2e/bb.edn test` from repo root).
|
||||
- CLI E2E tests live in `cli-e2e/`; run with `bb -f cli-e2e/bb.edn test --skip-build` (or `bb -f cli-e2e/bb.edn build` first when needed).
|
||||
- If a request says only “e2e”, clarify whether it targets `clj-e2e/` or `cli-e2e/` before planning changes.
|
||||
|
||||
## Error handling and compatibility
|
||||
- When modifying code, first consider removing compatibility layers rather than extending them.
|
||||
- Prefer fail-fast over fallback.
|
||||
- Do not add backward compatibility unless explicitly requested.
|
||||
- Do not introduce default values to mask invalid state.
|
||||
- Do not silently recover from programmer errors.
|
||||
- Keep one clear code path whenever possible.
|
||||
- Internal code may assume well-formed inputs from controlled callers.
|
||||
|
||||
## Coding Style & Naming Conventions
|
||||
- ClojureScript keywords are defined via `logseq.common.defkeywords/defkeyword`; use existing keywords and add new ones in the shared definitions.
|
||||
@@ -29,12 +29,16 @@
|
||||
- Name tests after their namespaces; use `-v` to target a specific test case.
|
||||
- Run lint/tests before submitting PRs; keep changes green.
|
||||
|
||||
## *IMPORTANT*: Always respect directory-specific AGENTS.md based on file path
|
||||
- when editing code in a specific directory, you must recursively read `AGENTS.md` files up the directory tree, and `AGENTS.md` in subdirectories takes precedence over the root-level one
|
||||
|
||||
## Commit & Pull Request Guidelines
|
||||
- Commit subjects are short and imperative; optional scope prefixes appear (e.g., `fix:` or `enhance(rtc):`).
|
||||
- PRs should describe the behavior change, link relevant issues, and note any test coverage added or skipped.
|
||||
|
||||
## Agent-Specific Notes
|
||||
- Project-specific skills live under `.agents/skills/`; load `.agents/skills/logseq-i18n/SKILL.md` for i18n/localization/hardcoded UI text tasks.
|
||||
- Use repo-local skills discovered under `.agents/skills/`; load the matching `SKILL.md` before editing files or proposing changes.
|
||||
- **i18n (mandatory)**: Always load `.agents/skills/logseq-i18n/SKILL.md` before any change that adds, edits, or removes user-facing UI text, regardless of whether other skills also apply.
|
||||
- Review notes live in `prompts/review.md`; check them when preparing changes.
|
||||
- DB-sync feature guide for AI agents: `docs/agent-guide/db-sync/db-sync-guide.md`.
|
||||
- DB-sync protocol reference: `docs/agent-guide/db-sync/protocol.md`.
|
||||
|
||||
@@ -169,6 +169,10 @@ If you want to set up a development environment for the Logseq web or desktop ap
|
||||
|
||||
In addition to these guides, you can also find other helpful resources in the [docs/](docs/) folder, such as the [Guide for Contributing to Translations](docs/contributing-to-translations.md), the [Docker Web App Guide](docs/docker-web-app-guide.md) and the [mobile development guide](docs/develop-logseq-on-mobile.md)
|
||||
|
||||
### 🧰 Logseq CLI (Node)
|
||||
|
||||
Logseq CLI documentation is maintained in `docs/cli/logseq-cli.md`.
|
||||
|
||||
## ✨ Inspiration
|
||||
|
||||
Logseq is inspired by several unique tools and projects, including [Roam Research](https://roamresearch.com/), [Org Mode](https://orgmode.org/), [TiddlyWiki](https://tiddlywiki.com/), [Workflowy](https://workflowy.com/), and [Cuekeeper](https://github.com/talex5/cuekeeper).
|
||||
|
||||
19
bb.edn
19
bb.edn
@@ -68,6 +68,7 @@
|
||||
(run '-dev:publishing-dev {:parallel true})
|
||||
(run '-dev:publishing-release))}
|
||||
|
||||
;; legacy cli
|
||||
dev:cli
|
||||
{:doc "Run CLI with current deps/db code. Commands with JS deps are not usable e.g. mcp-server"
|
||||
:task (apply shell {:dir "deps/db"}
|
||||
@@ -192,6 +193,24 @@
|
||||
dev:gen-malli-kondo-config
|
||||
logseq.tasks.dev/gen-malli-kondo-config
|
||||
|
||||
dev:db-worker-node
|
||||
{:doc "Compile and start db-worker-node (pass-through args forwarded to node)"
|
||||
:task (do
|
||||
(shell "clojure" "-M:cljs" "compile" "db-worker-node")
|
||||
(apply shell "node" "./static/db-worker-node.js" *command-line-args*))}
|
||||
|
||||
dev:cli-e2e
|
||||
{:doc "Run shell-first CLI end-to-end tests"
|
||||
:task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "test" *command-line-args*)}
|
||||
|
||||
dev:cli-e2e-cleanup
|
||||
{:doc "Run shell-first CLI end-to-end cleanup"
|
||||
:task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "cleanup" *command-line-args*)}
|
||||
|
||||
dev:cli-e2e-sync
|
||||
{:doc "Run shell-first CLI sync end-to-end tests"
|
||||
:task (apply shell {:shutdown nil} "bb" "-f" "cli-e2e/bb.edn" "test-sync" *command-line-args*)}
|
||||
|
||||
lint:dev
|
||||
logseq.tasks.dev.lint/dev
|
||||
|
||||
|
||||
38
cli-e2e/AGENTS.md
Normal file
38
cli-e2e/AGENTS.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# cli-e2e
|
||||
|
||||
Shell-first end-to-end tests for logseq CLI.
|
||||
|
||||
## Test cli-e2e itself
|
||||
- Run internal cli-e2e harness unit tests: `bb unit-test`
|
||||
|
||||
## cleanup first
|
||||
- Execute cleanup: terminate stale db-worker-node processes, terminate stale db-sync listeners on port `18080`, and remove cli-e2e temp roots: `bb cleanup`
|
||||
- Preview cleanup actions only (no kill/delete): `bb cleanup --dry-run`
|
||||
|
||||
## Run non-sync suite
|
||||
- List declared non-sync case ids: `bb list-cases`
|
||||
- Run non-sync cases with build preflight unless `--skip-build` is provided: `bb test`
|
||||
- `bb test --help` for options
|
||||
- Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test --skip-build --jobs 4`
|
||||
- Parallelism is case-scoped only; each case still runs setup/main/cleanup sequentially in the existing ephemeral shell model
|
||||
|
||||
## Run sync suite
|
||||
- List declared sync case ids: `bb list-sync-cases`
|
||||
- Run sync cases with build preflight by default: `bb test-sync`
|
||||
- Add `--skip-build` only when you intentionally want to reuse existing artifacts
|
||||
- `bb test-sync --help` for options
|
||||
- Increase case-level parallelism with `--jobs N` (default: `4`), for example: `bb test-sync --jobs 4`
|
||||
- Parallelism is case-scoped only; each sync case still runs its own setup/main/cleanup sequentially
|
||||
- The local db-sync server is shared per suite and starts once before cases begin, then stops once after all cases complete
|
||||
- Each sync case gets its own isolated temp root, data directories, home directory, auth copy, and generated config files
|
||||
- Configure sync E2EE password: `--e2ee-password <value>` (default: `11111`)
|
||||
- Run only sync MVP case: `bb test-sync --case sync-upload-download-mvp`
|
||||
|
||||
### Sync suite prerequisites
|
||||
- CLI auth file must exist at `~/logseq/auth.json` (generated by `logseq login`).
|
||||
- Sync suite starts/stops one shared local db-sync server per suite run.
|
||||
- Required build artifact for local db-sync server:
|
||||
- `deps/db-sync/worker/dist/node-adapter.js`
|
||||
- If artifact is missing, build it before running sync suite:
|
||||
- `pnpm --dir deps/db-sync build:node-adapter`
|
||||
|
||||
1
cli-e2e/README.md
Symbolic link
1
cli-e2e/README.md
Symbolic link
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
47
cli-e2e/bb.edn
Normal file
47
cli-e2e/bb.edn
Normal file
@@ -0,0 +1,47 @@
|
||||
{:paths ["src" "test"]
|
||||
:deps {cheshire/cheshire {:mvn/version "5.12.0"}}
|
||||
:tasks
|
||||
{:requires ([babashka.cli :as cli]
|
||||
[logseq.cli.e2e.main :as main]
|
||||
[logseq.cli.e2e.test-runner :as test-runner])
|
||||
:init (def cli-opts
|
||||
(cli/parse-opts
|
||||
*command-line-args*
|
||||
{:alias {:i :include
|
||||
:h :help}
|
||||
:spec {:jobs {:default 4}}
|
||||
:coerce {:include []
|
||||
:help :boolean
|
||||
:dry-run :boolean
|
||||
:skip-build :boolean
|
||||
:verbose :boolean
|
||||
:timings :boolean
|
||||
:jobs :long}}))
|
||||
|
||||
unit-test
|
||||
{:doc "Run internal cli-e2e harness unit tests"
|
||||
:task (test-runner/run! cli-opts)}
|
||||
|
||||
build
|
||||
{:doc "Compile required CLI artifacts and verify expected outputs exist"
|
||||
:task (main/build! cli-opts)}
|
||||
|
||||
list-cases
|
||||
{:doc "List declared non-sync cli-e2e case ids"
|
||||
:task (main/list-cases! cli-opts)}
|
||||
|
||||
list-sync-cases
|
||||
{:doc "List declared sync cli-e2e case ids"
|
||||
:task (main/list-sync-cases! cli-opts)}
|
||||
|
||||
test
|
||||
{:doc "Run non-sync cli-e2e cases with build preflight unless --skip-build is provided"
|
||||
:task (main/test! cli-opts)}
|
||||
|
||||
test-sync
|
||||
{:doc "Run sync cli-e2e cases with build preflight unless --skip-build is provided; use --jobs for case-level parallelism"
|
||||
:task (main/test-sync! cli-opts)}
|
||||
|
||||
cleanup
|
||||
{:doc "Terminate cli-e2e db-worker-node processes, terminate stale db-sync listeners, and remove cli-e2e temp roots"
|
||||
:task (main/cleanup! cli-opts)}}}
|
||||
181
cli-e2e/scripts/compare_graph_queries.py
Normal file
181
cli-e2e/scripts/compare_graph_queries.py
Normal file
@@ -0,0 +1,181 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare normalized query payloads between two cli graph contexts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
IGNORED_KEYS = {
|
||||
"db/id",
|
||||
"db/created-at",
|
||||
"db/updated-at",
|
||||
"block/uuid",
|
||||
"block/updated-at",
|
||||
"block/created-at",
|
||||
}
|
||||
|
||||
|
||||
def fail(message: str, **context: object) -> None:
|
||||
payload = {"status": "error", "message": message}
|
||||
if context:
|
||||
payload["context"] = context
|
||||
print(json.dumps(payload), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def normalize(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
normalized = {}
|
||||
for key in sorted(value.keys(), key=str):
|
||||
key_str = str(key)
|
||||
if key_str in IGNORED_KEYS:
|
||||
continue
|
||||
normalized[key_str] = normalize(value[key])
|
||||
return normalized
|
||||
if isinstance(value, list):
|
||||
normalized_list = [normalize(item) for item in value]
|
||||
try:
|
||||
return sorted(normalized_list, key=lambda item: json.dumps(item, sort_keys=True))
|
||||
except TypeError:
|
||||
return normalized_list
|
||||
return value
|
||||
|
||||
|
||||
def run_query(cli_path: Path, config_path: Path, root_dir: Path, graph: str, query: str) -> Dict[str, Any]:
|
||||
command = [
|
||||
"node",
|
||||
str(cli_path),
|
||||
"--root-dir",
|
||||
str(root_dir),
|
||||
"--config",
|
||||
str(config_path),
|
||||
"--output",
|
||||
"json",
|
||||
"query",
|
||||
"--graph",
|
||||
graph,
|
||||
"--query",
|
||||
query,
|
||||
]
|
||||
|
||||
result = subprocess.run(command, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
fail(
|
||||
"query command failed",
|
||||
command=command,
|
||||
exit=result.returncode,
|
||||
stdout=result.stdout,
|
||||
stderr=result.stderr,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = json.loads(result.stdout)
|
||||
except json.JSONDecodeError as error:
|
||||
fail(
|
||||
"query command did not return valid JSON",
|
||||
command=command,
|
||||
stdout=result.stdout,
|
||||
stderr=result.stderr,
|
||||
detail=str(error),
|
||||
)
|
||||
|
||||
if payload.get("status") != "ok":
|
||||
fail("query command returned non-ok status", command=command, payload=payload)
|
||||
|
||||
data = payload.get("data") or {}
|
||||
return {
|
||||
"payload": payload,
|
||||
"result": data.get("result"),
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Compare normalized query payloads between two cli contexts")
|
||||
parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js")
|
||||
parser.add_argument("--graph", required=True)
|
||||
parser.add_argument("--query", required=True, action="append")
|
||||
parser.add_argument("--config-a", required=True)
|
||||
parser.add_argument("--root-dir-a", required=True)
|
||||
parser.add_argument("--config-b", required=True)
|
||||
parser.add_argument("--root-dir-b", required=True)
|
||||
parser.add_argument("--require-result", action="store_true")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
|
||||
cli_path = Path(args.cli).expanduser().resolve()
|
||||
if not cli_path.exists():
|
||||
fail("cli entry file does not exist", cli=str(cli_path))
|
||||
|
||||
queries = args.query
|
||||
left_config = Path(args.config_a).expanduser().resolve()
|
||||
left_root_dir = Path(args.root_dir_a).expanduser().resolve()
|
||||
right_config = Path(args.config_b).expanduser().resolve()
|
||||
right_root_dir = Path(args.root_dir_b).expanduser().resolve()
|
||||
|
||||
normalized_results = {}
|
||||
|
||||
for query in queries:
|
||||
left = run_query(
|
||||
cli_path,
|
||||
left_config,
|
||||
left_root_dir,
|
||||
args.graph,
|
||||
query,
|
||||
)
|
||||
right = run_query(
|
||||
cli_path,
|
||||
right_config,
|
||||
right_root_dir,
|
||||
args.graph,
|
||||
query,
|
||||
)
|
||||
|
||||
left_result = left["result"]
|
||||
right_result = right["result"]
|
||||
|
||||
if args.require_result and (left_result is None or right_result is None):
|
||||
fail(
|
||||
"query result is empty",
|
||||
query=query,
|
||||
left_result=left_result,
|
||||
right_result=right_result,
|
||||
)
|
||||
|
||||
left_normalized = normalize(left_result)
|
||||
right_normalized = normalize(right_result)
|
||||
|
||||
if left_normalized != right_normalized:
|
||||
fail(
|
||||
"normalized query results differ",
|
||||
query=query,
|
||||
left_result=left_normalized,
|
||||
right_result=right_normalized,
|
||||
left_payload=left["payload"],
|
||||
right_payload=right["payload"],
|
||||
)
|
||||
|
||||
normalized_results[query] = left_normalized
|
||||
|
||||
payload_key = "result" if len(normalized_results) == 1 else "results"
|
||||
payload_value = next(iter(normalized_results.values())) if len(normalized_results) == 1 else normalized_results
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
payload_key: payload_value,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
290
cli-e2e/scripts/db_sync_server.py
Normal file
290
cli-e2e/scripts/db_sync_server.py
Normal file
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Manage local db-sync server process for cli-e2e sync suite."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import ctypes
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
DEFAULT_COGNITO_ISSUER = "https://cognito-idp.us-east-2.amazonaws.com/us-east-2_kAqZcxIeM"
|
||||
DEFAULT_COGNITO_CLIENT_ID = "1qi1uijg8b6ra70nejvbptis0q"
|
||||
|
||||
|
||||
def fail(message: str, **context: object) -> None:
|
||||
payload = {"status": "error", "message": message}
|
||||
if context:
|
||||
payload["context"] = context
|
||||
print(json.dumps(payload), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def load_pid(pid_file: Path) -> Optional[int]:
|
||||
if not pid_file.exists():
|
||||
return None
|
||||
raw = pid_file.read_text(encoding="utf-8").strip()
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def process_running(pid: int) -> bool:
|
||||
if pid <= 0:
|
||||
return False
|
||||
if os.name == "nt":
|
||||
kernel32 = ctypes.windll.kernel32
|
||||
access = 0x00100000 | 0x1000 # SYNCHRONIZE | PROCESS_QUERY_LIMITED_INFORMATION
|
||||
handle = kernel32.OpenProcess(access, False, pid)
|
||||
if handle:
|
||||
kernel32.CloseHandle(handle)
|
||||
return True
|
||||
return ctypes.get_last_error() == 5
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
else:
|
||||
return True
|
||||
|
||||
|
||||
def terminate_process(pid: int, force: bool) -> bool:
|
||||
if pid <= 0:
|
||||
return True
|
||||
if os.name == "nt":
|
||||
cmd = ["taskkill", "/PID", str(pid), "/T"]
|
||||
if force:
|
||||
cmd.append("/F")
|
||||
result = subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
|
||||
return result.returncode == 0 or not process_running(pid)
|
||||
try:
|
||||
os.kill(pid, signal.SIGKILL if force else signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
return True
|
||||
except PermissionError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def parse_json_file(path: Path) -> Dict[str, Any]:
|
||||
if not path.exists():
|
||||
fail("sync auth file is missing", auth_path=str(path), hint="Run `logseq login` first.")
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as error:
|
||||
fail("sync auth file is invalid JSON", auth_path=str(path), detail=str(error))
|
||||
if not isinstance(payload, dict):
|
||||
fail("sync auth file must be a JSON object", auth_path=str(path))
|
||||
return payload
|
||||
|
||||
|
||||
def decode_jwt_claims(token: str) -> Optional[Dict[str, Any]]:
|
||||
if not isinstance(token, str):
|
||||
return None
|
||||
parts = token.split(".")
|
||||
if len(parts) != 3:
|
||||
return None
|
||||
payload = parts[1]
|
||||
padded = payload + "=" * ((4 - (len(payload) % 4)) % 4)
|
||||
try:
|
||||
decoded = base64.urlsafe_b64decode(padded.encode("utf-8")).decode("utf-8")
|
||||
claims = json.loads(decoded)
|
||||
except (ValueError, UnicodeDecodeError, json.JSONDecodeError):
|
||||
return None
|
||||
return claims if isinstance(claims, dict) else None
|
||||
|
||||
|
||||
def auth_cognito_from_auth_file(auth_path: Path) -> Dict[str, str]:
|
||||
payload = parse_json_file(auth_path)
|
||||
token = payload.get("id-token") or payload.get("access-token")
|
||||
claims = decode_jwt_claims(token) if isinstance(token, str) else None
|
||||
|
||||
issuer = claims.get("iss") if isinstance(claims, dict) else None
|
||||
client_id = None
|
||||
if isinstance(claims, dict):
|
||||
client_id = claims.get("aud") or claims.get("client_id")
|
||||
|
||||
return {
|
||||
"issuer": issuer if isinstance(issuer, str) and issuer else "",
|
||||
"client_id": client_id if isinstance(client_id, str) and client_id else "",
|
||||
}
|
||||
|
||||
|
||||
def wait_health(base_url: str, timeout_s: float, interval_s: float) -> bool:
|
||||
deadline = time.time() + timeout_s
|
||||
url = base_url.rstrip("/") + "/health"
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
with opener.open(url, timeout=2) as response:
|
||||
if response.status == 200:
|
||||
return True
|
||||
except (urllib.error.URLError, TimeoutError, socket.timeout):
|
||||
pass
|
||||
time.sleep(interval_s)
|
||||
return False
|
||||
|
||||
|
||||
def start_server(args: argparse.Namespace) -> None:
|
||||
repo_root = Path(args.repo_root).expanduser().resolve()
|
||||
entry = repo_root / "deps" / "db-sync" / "worker" / "dist" / "node-adapter.js"
|
||||
if not entry.exists():
|
||||
fail("db-sync node adapter build artifact is missing", entry=str(entry), hint="Run: yarn --cwd deps/db-sync build:node-adapter")
|
||||
|
||||
pid_file = Path(args.pid_file).expanduser().resolve()
|
||||
log_file = Path(args.log_file).expanduser().resolve()
|
||||
data_dir = Path(args.data_dir).expanduser().resolve()
|
||||
|
||||
pid_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
log_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
data_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
existing_pid = load_pid(pid_file)
|
||||
if existing_pid and process_running(existing_pid):
|
||||
fail("db-sync server already running", pid=existing_pid, pid_file=str(pid_file))
|
||||
|
||||
auth_path = Path(args.auth_path).expanduser().resolve() if args.auth_path else None
|
||||
auth_derived = auth_cognito_from_auth_file(auth_path) if auth_path else {"issuer": "", "client_id": ""}
|
||||
|
||||
issuer = args.cognito_issuer or auth_derived.get("issuer") or DEFAULT_COGNITO_ISSUER
|
||||
client_id = args.cognito_client_id or auth_derived.get("client_id") or DEFAULT_COGNITO_CLIENT_ID
|
||||
jwks_url = args.cognito_jwks_url or f"{issuer}/.well-known/jwks.json"
|
||||
|
||||
env = os.environ.copy()
|
||||
env.update(
|
||||
{
|
||||
"DB_SYNC_PORT": str(args.port),
|
||||
"DB_SYNC_DATA_DIR": str(data_dir),
|
||||
"COGNITO_ISSUER": issuer,
|
||||
"COGNITO_CLIENT_ID": client_id,
|
||||
"COGNITO_JWKS_URL": jwks_url,
|
||||
# CLI e2e sync suite should remain runnable without outbound internet.
|
||||
"DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS": "true",
|
||||
}
|
||||
)
|
||||
|
||||
with log_file.open("a", encoding="utf-8") as stream:
|
||||
stream.write(f"\n=== cli-e2e db-sync start {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n")
|
||||
stream.flush()
|
||||
process = subprocess.Popen(
|
||||
["node", str(entry)],
|
||||
stdout=stream,
|
||||
stderr=stream,
|
||||
cwd=str(repo_root),
|
||||
env=env,
|
||||
start_new_session=True,
|
||||
)
|
||||
|
||||
base_url = f"http://{args.host}:{args.port}"
|
||||
if not wait_health(base_url, args.startup_timeout_s, args.poll_interval_s):
|
||||
terminate_process(process.pid, force=False)
|
||||
fail(
|
||||
"db-sync server failed health check before timeout",
|
||||
base_url=base_url,
|
||||
pid=process.pid,
|
||||
log_file=str(log_file),
|
||||
)
|
||||
|
||||
pid_file.write_text(f"{process.pid}\n", encoding="utf-8")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
"pid": process.pid,
|
||||
"pid_file": str(pid_file),
|
||||
"log_file": str(log_file),
|
||||
"base_url": base_url,
|
||||
"data_dir": str(data_dir),
|
||||
"cognito_issuer": issuer,
|
||||
"cognito_client_id": client_id,
|
||||
"auth_path": str(auth_path) if auth_path else None,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def stop_server(args: argparse.Namespace) -> None:
|
||||
pid_file = Path(args.pid_file).expanduser().resolve()
|
||||
pid = load_pid(pid_file)
|
||||
|
||||
if pid is None:
|
||||
print(json.dumps({"status": "ok", "stopped": False, "reason": "pid-file-missing-or-empty", "pid_file": str(pid_file)}))
|
||||
return
|
||||
|
||||
if not process_running(pid):
|
||||
try:
|
||||
pid_file.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
print(json.dumps({"status": "ok", "stopped": False, "reason": "process-not-running", "pid": pid, "pid_file": str(pid_file)}))
|
||||
return
|
||||
|
||||
if not terminate_process(pid, force=False):
|
||||
fail("db-sync server stop failed", pid=pid, signal="SIGTERM", pid_file=str(pid_file))
|
||||
deadline = time.time() + args.shutdown_timeout_s
|
||||
while time.time() < deadline:
|
||||
if not process_running(pid):
|
||||
pid_file.unlink(missing_ok=True)
|
||||
print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGTERM", "pid": pid, "pid_file": str(pid_file)}))
|
||||
return
|
||||
time.sleep(args.poll_interval_s)
|
||||
|
||||
if not terminate_process(pid, force=True):
|
||||
fail("db-sync server force stop failed", pid=pid, signal="SIGKILL", pid_file=str(pid_file))
|
||||
pid_file.unlink(missing_ok=True)
|
||||
print(json.dumps({"status": "ok", "stopped": True, "signal": "SIGKILL", "pid": pid, "pid_file": str(pid_file)}))
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Manage db-sync local server for cli-e2e sync tests")
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
start = subparsers.add_parser("start", help="Start server and wait for /health")
|
||||
start.add_argument("--repo-root", required=True)
|
||||
start.add_argument("--pid-file", required=True)
|
||||
start.add_argument("--log-file", required=True)
|
||||
start.add_argument("--data-dir", required=True)
|
||||
start.add_argument("--host", default="127.0.0.1")
|
||||
start.add_argument("--port", type=int, default=8080)
|
||||
start.add_argument("--startup-timeout-s", type=float, default=25.0)
|
||||
start.add_argument("--poll-interval-s", type=float, default=0.5)
|
||||
start.add_argument("--auth-path", default="~/logseq/auth.json")
|
||||
start.add_argument("--cognito-issuer")
|
||||
start.add_argument("--cognito-client-id")
|
||||
start.add_argument("--cognito-jwks-url")
|
||||
|
||||
stop = subparsers.add_parser("stop", help="Stop server if running")
|
||||
stop.add_argument("--pid-file", required=True)
|
||||
stop.add_argument("--shutdown-timeout-s", type=float, default=10.0)
|
||||
stop.add_argument("--poll-interval-s", type=float, default=0.25)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_parser().parse_args()
|
||||
if args.command == "start":
|
||||
start_server(args)
|
||||
elif args.command == "stop":
|
||||
stop_server(args)
|
||||
else:
|
||||
fail("unknown command", command=args.command)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
81
cli-e2e/scripts/prepare_sync_config.py
Normal file
81
cli-e2e/scripts/prepare_sync_config.py
Normal file
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare per-case CLI config for sync e2e tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def fail(message: str, **context: object) -> None:
|
||||
payload = {"status": "error", "message": message}
|
||||
if context:
|
||||
payload["context"] = context
|
||||
print(json.dumps(payload), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def read_auth(auth_path: Path) -> dict:
|
||||
if not auth_path.exists():
|
||||
fail("sync auth file is missing", auth_path=str(auth_path), hint="Run `logseq login` first.")
|
||||
try:
|
||||
payload = json.loads(auth_path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError as error:
|
||||
fail("sync auth file is invalid JSON", auth_path=str(auth_path), detail=str(error))
|
||||
|
||||
has_token = any(payload.get(key) for key in ("refresh-token", "id-token", "access-token"))
|
||||
if not has_token:
|
||||
fail(
|
||||
"sync auth file does not contain usable tokens",
|
||||
auth_path=str(auth_path),
|
||||
required_any_of=["refresh-token", "id-token", "access-token"],
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def write_config(output_path: Path, http_base: str, ws_url: str) -> None:
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = "\n".join(
|
||||
[
|
||||
"{",
|
||||
" :output-format :json",
|
||||
f' :http-base "{http_base}"',
|
||||
f' :ws-url "{ws_url}"',
|
||||
"}",
|
||||
"",
|
||||
]
|
||||
)
|
||||
output_path.write_text(payload, encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Prepare cli.edn for sync e2e")
|
||||
parser.add_argument("--output", required=True)
|
||||
parser.add_argument("--auth-path", default="~/logseq/auth.json")
|
||||
parser.add_argument("--http-base", required=True)
|
||||
parser.add_argument("--ws-url", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
auth_path = Path(args.auth_path).expanduser().resolve()
|
||||
_auth = read_auth(auth_path)
|
||||
|
||||
output_path = Path(args.output).expanduser().resolve()
|
||||
write_config(output_path, args.http_base, args.ws_url)
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
"auth_path": str(auth_path),
|
||||
"config_path": str(output_path),
|
||||
"http_base": args.http_base,
|
||||
"ws_url": args.ws_url,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
480
cli-e2e/scripts/random_bidirectional_block_ops.py
Normal file
480
cli-e2e/scripts/random_bidirectional_block_ops.py
Normal file
@@ -0,0 +1,480 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run randomized bidirectional block operations on two synced graph peers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
|
||||
class CliCommandError(RuntimeError):
|
||||
"""Raised when a CLI command does not complete successfully."""
|
||||
|
||||
def __init__(self, message: str, *, context: Dict[str, Any]) -> None:
|
||||
super().__init__(message)
|
||||
self.context = context
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClientContext:
|
||||
name: str
|
||||
config: Path
|
||||
root_dir: Path
|
||||
|
||||
|
||||
def fail(message: str, **context: object) -> None:
|
||||
payload = {"status": "error", "message": message}
|
||||
if context:
|
||||
payload["context"] = context
|
||||
print(json.dumps(payload), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run_cli_json(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
args: List[str],
|
||||
) -> Dict[str, Any]:
|
||||
command = [
|
||||
"node",
|
||||
str(cli_path),
|
||||
"--root-dir",
|
||||
str(client.root_dir),
|
||||
"--config",
|
||||
str(client.config),
|
||||
"--output",
|
||||
"json",
|
||||
*args,
|
||||
"--graph",
|
||||
graph,
|
||||
]
|
||||
result = subprocess.run(command, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
raise CliCommandError(
|
||||
"cli command exited with non-zero status",
|
||||
context={
|
||||
"client": client.name,
|
||||
"command": command,
|
||||
"exit": result.returncode,
|
||||
"stdout": result.stdout,
|
||||
"stderr": result.stderr,
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
payload = json.loads(result.stdout)
|
||||
except json.JSONDecodeError as error:
|
||||
raise CliCommandError(
|
||||
"cli command did not return valid json",
|
||||
context={
|
||||
"client": client.name,
|
||||
"command": command,
|
||||
"stdout": result.stdout,
|
||||
"stderr": result.stderr,
|
||||
"detail": str(error),
|
||||
},
|
||||
) from error
|
||||
|
||||
if payload.get("status") != "ok":
|
||||
raise CliCommandError(
|
||||
"cli command returned non-ok status",
|
||||
context={
|
||||
"client": client.name,
|
||||
"command": command,
|
||||
"payload": payload,
|
||||
},
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def page_block_ids(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
) -> List[int]:
|
||||
query = (
|
||||
"[:find [?e ...] "
|
||||
":where "
|
||||
f"[?p :block/title {json.dumps(page_title)}] "
|
||||
"[?e :block/page ?p] "
|
||||
"[?e :block/uuid]]"
|
||||
)
|
||||
payload = run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=["query", "--query", query],
|
||||
)
|
||||
result = (payload.get("data") or {}).get("result")
|
||||
if not isinstance(result, list):
|
||||
return []
|
||||
output: List[int] = []
|
||||
for item in result:
|
||||
try:
|
||||
output.append(int(item))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return output
|
||||
|
||||
|
||||
def upsert_page(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
) -> None:
|
||||
run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=["upsert", "page", "--page", page_title],
|
||||
)
|
||||
|
||||
|
||||
def create_block(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
content: str,
|
||||
ids: List[int],
|
||||
rng: random.Random,
|
||||
) -> None:
|
||||
if ids and rng.random() < 0.6:
|
||||
target_id = str(rng.choice(ids))
|
||||
run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=[
|
||||
"upsert",
|
||||
"block",
|
||||
"--target-id",
|
||||
target_id,
|
||||
"--pos",
|
||||
"first-child",
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
return
|
||||
|
||||
run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=[
|
||||
"upsert",
|
||||
"block",
|
||||
"--target-page",
|
||||
page_title,
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def move_block(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
ids: List[int],
|
||||
rng: random.Random,
|
||||
) -> bool:
|
||||
if not ids:
|
||||
return False
|
||||
source_id = str(rng.choice(ids))
|
||||
run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=[
|
||||
"upsert",
|
||||
"block",
|
||||
"--id",
|
||||
source_id,
|
||||
"--target-page",
|
||||
page_title,
|
||||
"--pos",
|
||||
"last-child",
|
||||
],
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def delete_block(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
ids: List[int],
|
||||
rng: random.Random,
|
||||
) -> bool:
|
||||
if len(ids) <= 2:
|
||||
return False
|
||||
source_id = str(rng.choice(ids))
|
||||
run_cli_json(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
args=["remove", "block", "--id", source_id],
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def feasible(operation: str, ids: List[int]) -> bool:
|
||||
if operation == "create":
|
||||
return True
|
||||
if operation == "move":
|
||||
return len(ids) >= 1
|
||||
if operation == "delete":
|
||||
return len(ids) > 2
|
||||
return False
|
||||
|
||||
|
||||
def choose_operation(op_counts: Dict[str, int], ids: List[int], rng: random.Random) -> str:
|
||||
for required in ("create", "move", "delete"):
|
||||
if op_counts.get(required, 0) == 0 and feasible(required, ids):
|
||||
return required
|
||||
|
||||
candidates = ["create", "create", "move"]
|
||||
if feasible("delete", ids):
|
||||
candidates.append("delete")
|
||||
if feasible("move", ids):
|
||||
candidates.append("move")
|
||||
return rng.choice(candidates)
|
||||
|
||||
|
||||
def apply_operation(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
op_counts: Dict[str, int],
|
||||
round_index: int,
|
||||
rng: random.Random,
|
||||
) -> None:
|
||||
last_error: Dict[str, Any] | None = None
|
||||
for attempt in range(1, 7):
|
||||
ids = page_block_ids(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
page_title=page_title,
|
||||
)
|
||||
operation = choose_operation(op_counts, ids, rng)
|
||||
content = f"{client.name}-rnd-{round_index:03d}-{rng.randint(100000, 999999)}"
|
||||
|
||||
try:
|
||||
if operation == "create":
|
||||
create_block(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
page_title=page_title,
|
||||
content=content,
|
||||
ids=ids,
|
||||
rng=rng,
|
||||
)
|
||||
op_counts["create"] = op_counts.get("create", 0) + 1
|
||||
return
|
||||
|
||||
if operation == "move":
|
||||
moved = move_block(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
page_title=page_title,
|
||||
ids=ids,
|
||||
rng=rng,
|
||||
)
|
||||
if moved:
|
||||
op_counts["move"] = op_counts.get("move", 0) + 1
|
||||
return
|
||||
continue
|
||||
|
||||
if operation == "delete":
|
||||
deleted = delete_block(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
ids=ids,
|
||||
rng=rng,
|
||||
)
|
||||
if deleted:
|
||||
op_counts["delete"] = op_counts.get("delete", 0) + 1
|
||||
return
|
||||
continue
|
||||
except CliCommandError as error:
|
||||
last_error = {
|
||||
"attempt": attempt,
|
||||
"operation": operation,
|
||||
"context": error.context,
|
||||
}
|
||||
continue
|
||||
|
||||
fail(
|
||||
"failed to apply random operation after retries",
|
||||
client=client.name,
|
||||
round_index=round_index,
|
||||
last_error=last_error,
|
||||
)
|
||||
|
||||
|
||||
def ensure_non_empty_page(
|
||||
*,
|
||||
cli_path: Path,
|
||||
graph: str,
|
||||
client: ClientContext,
|
||||
page_title: str,
|
||||
rng: random.Random,
|
||||
) -> None:
|
||||
ids = page_block_ids(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
page_title=page_title,
|
||||
)
|
||||
if ids:
|
||||
return
|
||||
content = f"{client.name}-reseed-{rng.randint(100000, 999999)}"
|
||||
create_block(
|
||||
cli_path=cli_path,
|
||||
graph=graph,
|
||||
client=client,
|
||||
page_title=page_title,
|
||||
content=content,
|
||||
ids=[],
|
||||
rng=rng,
|
||||
)
|
||||
|
||||
|
||||
PROFILE_DEFAULT_ROUNDS = {
|
||||
"default": 40,
|
||||
"high-stress": 100,
|
||||
}
|
||||
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run randomized bidirectional block operations on two synced graph clients"
|
||||
)
|
||||
parser.add_argument("--cli", required=True, help="Path to static/logseq-cli.js")
|
||||
parser.add_argument("--graph", required=True)
|
||||
parser.add_argument("--config-a", required=True)
|
||||
parser.add_argument("--root-dir-a", required=True)
|
||||
parser.add_argument("--config-b", required=True)
|
||||
parser.add_argument("--root-dir-b", required=True)
|
||||
parser.add_argument("--page", required=True)
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
choices=sorted(PROFILE_DEFAULT_ROUNDS.keys()),
|
||||
default="default",
|
||||
help="Execution profile controlling default stress level",
|
||||
)
|
||||
parser.add_argument("--rounds-per-client", type=int, default=None)
|
||||
parser.add_argument("--seed", type=int, default=424242)
|
||||
args = parser.parse_args()
|
||||
if args.rounds_per_client is None:
|
||||
args.rounds_per_client = PROFILE_DEFAULT_ROUNDS[args.profile]
|
||||
return args
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
cli_path = Path(args.cli).expanduser().resolve()
|
||||
if not cli_path.exists():
|
||||
fail("cli path does not exist", cli=str(cli_path))
|
||||
|
||||
rng = random.Random(args.seed)
|
||||
client_a = ClientContext(
|
||||
name="a",
|
||||
config=Path(args.config_a).expanduser().resolve(),
|
||||
root_dir=Path(args.root_dir_a).expanduser().resolve(),
|
||||
)
|
||||
client_b = ClientContext(
|
||||
name="b",
|
||||
config=Path(args.config_b).expanduser().resolve(),
|
||||
root_dir=Path(args.root_dir_b).expanduser().resolve(),
|
||||
)
|
||||
clients = [client_a, client_b]
|
||||
|
||||
for client in clients:
|
||||
upsert_page(
|
||||
cli_path=cli_path,
|
||||
graph=args.graph,
|
||||
client=client,
|
||||
page_title=args.page,
|
||||
)
|
||||
|
||||
# Seed both peers so move/delete have available targets from the beginning.
|
||||
for seed_round in range(3):
|
||||
for client in clients:
|
||||
create_block(
|
||||
cli_path=cli_path,
|
||||
graph=args.graph,
|
||||
client=client,
|
||||
page_title=args.page,
|
||||
content=f"{client.name}-seed-{seed_round}-{rng.randint(100000, 999999)}",
|
||||
ids=[],
|
||||
rng=rng,
|
||||
)
|
||||
|
||||
op_stats: Dict[str, Dict[str, int]] = {
|
||||
client.name: {"create": 0, "move": 0, "delete": 0} for client in clients
|
||||
}
|
||||
|
||||
for round_index in range(args.rounds_per_client):
|
||||
for client in clients:
|
||||
apply_operation(
|
||||
cli_path=cli_path,
|
||||
graph=args.graph,
|
||||
client=client,
|
||||
page_title=args.page,
|
||||
op_counts=op_stats[client.name],
|
||||
round_index=round_index,
|
||||
rng=rng,
|
||||
)
|
||||
|
||||
for client in clients:
|
||||
ensure_non_empty_page(
|
||||
cli_path=cli_path,
|
||||
graph=args.graph,
|
||||
client=client,
|
||||
page_title=args.page,
|
||||
rng=rng,
|
||||
)
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
"graph": args.graph,
|
||||
"page": args.page,
|
||||
"rounds_per_client": args.rounds_per_client,
|
||||
"seed": args.seed,
|
||||
"stats": op_stats,
|
||||
"total_operations": args.rounds_per_client * len(clients),
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
302
cli-e2e/scripts/wait_sync_status.py
Normal file
302
cli-e2e/scripts/wait_sync_status.py
Normal file
@@ -0,0 +1,302 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Poll `logseq sync status` until queues settle and tx converges."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def fail(message: str, **context: object) -> None:
|
||||
payload = {"status": "error", "message": message}
|
||||
if context:
|
||||
payload["context"] = context
|
||||
print(json.dumps(payload), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def parse_int(value: Any) -> int:
|
||||
if value is None:
|
||||
return 0
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def status_command(args: argparse.Namespace) -> list[str]:
|
||||
existing = getattr(args, "status_command", None)
|
||||
if existing is not None:
|
||||
return existing
|
||||
|
||||
command = [
|
||||
"node",
|
||||
str(Path(args.cli).expanduser().resolve()),
|
||||
"--root-dir",
|
||||
str(Path(args.root_dir).expanduser().resolve()),
|
||||
"--config",
|
||||
str(Path(args.config).expanduser().resolve()),
|
||||
"--output",
|
||||
"json",
|
||||
"sync",
|
||||
"status",
|
||||
"--graph",
|
||||
args.graph,
|
||||
]
|
||||
args.status_command = command
|
||||
return command
|
||||
|
||||
|
||||
|
||||
def run_status(args: argparse.Namespace) -> Dict[str, Any]:
|
||||
command = status_command(args)
|
||||
|
||||
result = subprocess.run(command, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
fail(
|
||||
"sync status command failed",
|
||||
command=command,
|
||||
exit=result.returncode,
|
||||
stdout=result.stdout,
|
||||
stderr=result.stderr,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = json.loads(result.stdout)
|
||||
except json.JSONDecodeError as error:
|
||||
fail(
|
||||
"sync status command did not return valid JSON",
|
||||
command=command,
|
||||
stdout=result.stdout,
|
||||
stderr=result.stderr,
|
||||
detail=str(error),
|
||||
)
|
||||
|
||||
if payload.get("status") != "ok":
|
||||
fail("sync status returned non-ok status", payload=payload)
|
||||
|
||||
return payload
|
||||
|
||||
|
||||
def pending_counts(status_payload: Dict[str, Any]) -> Dict[str, int]:
|
||||
data = status_payload.get("data") or {}
|
||||
return {
|
||||
"pending-local": parse_int(data.get("pending-local")),
|
||||
"pending-asset": parse_int(data.get("pending-asset")),
|
||||
"pending-server": parse_int(data.get("pending-server")),
|
||||
}
|
||||
|
||||
|
||||
def all_settled(counts: Dict[str, int]) -> bool:
|
||||
required_keys = ("pending-local", "pending-asset", "pending-server")
|
||||
return all(counts.get(key, 0) == 0 for key in required_keys)
|
||||
|
||||
|
||||
def parse_required_int(value: Any) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, str) and value.strip() == "":
|
||||
return None
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, (int, float)):
|
||||
return int(value)
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def tx_sync_status(status_payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
data = status_payload.get("data") or {}
|
||||
local_tx = parse_required_int(data.get("local-tx"))
|
||||
remote_tx = parse_required_int(data.get("remote-tx"))
|
||||
synced = (
|
||||
local_tx is not None
|
||||
and remote_tx is not None
|
||||
and local_tx == remote_tx
|
||||
)
|
||||
return {
|
||||
"local-tx": local_tx,
|
||||
"remote-tx": remote_tx,
|
||||
"synced": synced,
|
||||
}
|
||||
|
||||
|
||||
def tx_deltas(
|
||||
tx_status: Dict[str, Any],
|
||||
baseline_tx: Dict[str, Any],
|
||||
) -> Dict[str, int | None]:
|
||||
local_tx = tx_status.get("local-tx")
|
||||
remote_tx = tx_status.get("remote-tx")
|
||||
baseline_local_tx = baseline_tx.get("local-tx")
|
||||
baseline_remote_tx = baseline_tx.get("remote-tx")
|
||||
|
||||
local_tx_delta = (
|
||||
None
|
||||
if local_tx is None or baseline_local_tx is None
|
||||
else local_tx - baseline_local_tx
|
||||
)
|
||||
remote_tx_delta = (
|
||||
None
|
||||
if remote_tx is None or baseline_remote_tx is None
|
||||
else remote_tx - baseline_remote_tx
|
||||
)
|
||||
|
||||
return {
|
||||
"local-tx-delta": local_tx_delta,
|
||||
"remote-tx-delta": remote_tx_delta,
|
||||
}
|
||||
|
||||
|
||||
def min_tx_delta_reached(
|
||||
tx_status: Dict[str, Any],
|
||||
baseline_tx: Dict[str, Any],
|
||||
min_tx_delta: int,
|
||||
) -> bool:
|
||||
if min_tx_delta <= 0:
|
||||
return True
|
||||
|
||||
deltas = tx_deltas(tx_status, baseline_tx)
|
||||
local_tx_delta = deltas.get("local-tx-delta")
|
||||
remote_tx_delta = deltas.get("remote-tx-delta")
|
||||
if local_tx_delta is None or remote_tx_delta is None:
|
||||
return False
|
||||
|
||||
return local_tx_delta >= min_tx_delta and remote_tx_delta >= min_tx_delta
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Wait for sync status to settle"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cli",
|
||||
required=True,
|
||||
help="Path to static/logseq-cli.js",
|
||||
)
|
||||
parser.add_argument("--root-dir", required=True)
|
||||
parser.add_argument("--config", required=True)
|
||||
parser.add_argument("--graph", required=True)
|
||||
parser.add_argument("--timeout-s", type=float, default=120.0)
|
||||
parser.add_argument("--interval-s", type=float, default=1.0)
|
||||
parser.add_argument(
|
||||
"--min-tx-delta",
|
||||
type=int,
|
||||
default=0,
|
||||
help=(
|
||||
"Require synced tx delta to be >= this value: "
|
||||
"(new-tx - old-tx) >= min-tx-delta "
|
||||
"(default: 0)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--baseline-tx",
|
||||
type=int,
|
||||
default=None,
|
||||
help=(
|
||||
"Optional explicit baseline tx used as old-tx for both local "
|
||||
"and remote; if omitted, first observed tx values "
|
||||
"are used"
|
||||
),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.min_tx_delta < 0:
|
||||
invalid_min_tx_delta = args.min_tx_delta
|
||||
fail(
|
||||
"min-tx-delta must be non-negative",
|
||||
min_tx_delta=invalid_min_tx_delta,
|
||||
)
|
||||
|
||||
started = time.time()
|
||||
deadline = started + args.timeout_s
|
||||
last_payload: Dict[str, Any] | None = None
|
||||
baseline_tx: Dict[str, Any] | None = (
|
||||
{
|
||||
"local-tx": args.baseline_tx,
|
||||
"remote-tx": args.baseline_tx,
|
||||
}
|
||||
if args.baseline_tx is not None
|
||||
else None
|
||||
)
|
||||
|
||||
while time.time() < deadline:
|
||||
payload = run_status(args)
|
||||
last_payload = payload
|
||||
data = payload.get("data") or {}
|
||||
last_error = data.get("last-error")
|
||||
|
||||
if last_error is not None:
|
||||
fail("sync status reports last-error", payload=payload)
|
||||
|
||||
counts = pending_counts(payload)
|
||||
tx_status = tx_sync_status(payload)
|
||||
if baseline_tx is None:
|
||||
baseline_tx = {
|
||||
"local-tx": tx_status.get("local-tx"),
|
||||
"remote-tx": tx_status.get("remote-tx"),
|
||||
}
|
||||
|
||||
deltas = tx_deltas(tx_status, baseline_tx)
|
||||
if (
|
||||
all_settled(counts)
|
||||
and tx_status["synced"]
|
||||
and min_tx_delta_reached(
|
||||
tx_status,
|
||||
baseline_tx,
|
||||
args.min_tx_delta,
|
||||
)
|
||||
):
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
"elapsed_s": round(time.time() - started, 3),
|
||||
"counts": counts,
|
||||
"tx": {
|
||||
"local-tx": tx_status["local-tx"],
|
||||
"remote-tx": tx_status["remote-tx"],
|
||||
},
|
||||
"tx-delta": deltas,
|
||||
"baseline-tx": baseline_tx,
|
||||
"min_tx_delta": args.min_tx_delta,
|
||||
"payload": payload,
|
||||
}
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
time.sleep(max(args.interval_s, 0.0))
|
||||
|
||||
last_tx_status = tx_sync_status(last_payload or {})
|
||||
last_baseline_tx = baseline_tx or {"local-tx": None, "remote-tx": None}
|
||||
fail(
|
||||
(
|
||||
"sync status polling timed out before queues settled, tx synced, "
|
||||
"and min tx delta reached"
|
||||
),
|
||||
timeout_s=args.timeout_s,
|
||||
min_tx_delta=args.min_tx_delta,
|
||||
baseline_tx=last_baseline_tx,
|
||||
last_payload=last_payload,
|
||||
last_counts=pending_counts(last_payload or {}),
|
||||
last_tx=last_tx_status,
|
||||
last_tx_delta=tx_deltas(
|
||||
last_tx_status,
|
||||
last_baseline_tx,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
1100
cli-e2e/spec/non_sync_cases.edn
Normal file
1100
cli-e2e/spec/non_sync_cases.edn
Normal file
File diff suppressed because it is too large
Load Diff
165
cli-e2e/spec/non_sync_inventory.edn
Normal file
165
cli-e2e/spec/non_sync_inventory.edn
Normal file
@@ -0,0 +1,165 @@
|
||||
{:excluded-command-prefixes ["sync" "login" "logout"]
|
||||
:scopes
|
||||
{:global
|
||||
{:options ["--help"
|
||||
"--version"
|
||||
"--config"
|
||||
"--graph"
|
||||
"--root-dir"
|
||||
"--output"
|
||||
"--verbose"]}
|
||||
|
||||
:graph
|
||||
{:commands ["graph list"
|
||||
"graph create"
|
||||
"graph switch"
|
||||
"graph remove"
|
||||
"graph validate"
|
||||
"graph info"
|
||||
"graph export"
|
||||
"graph import"
|
||||
"graph backup list"
|
||||
"graph backup create"
|
||||
"graph backup restore"
|
||||
"graph backup remove"]
|
||||
:options ["--fix"
|
||||
"--type"
|
||||
"--file"
|
||||
"--input"
|
||||
"--name"
|
||||
"--src"
|
||||
"--dst"]}
|
||||
|
||||
:list
|
||||
{:commands ["list page"
|
||||
"list tag"
|
||||
"list property"
|
||||
"list task"
|
||||
"list node"
|
||||
"list asset"]
|
||||
:options ["--expand"
|
||||
"--fields"
|
||||
"--limit"
|
||||
"--offset"
|
||||
"--sort"
|
||||
"--order"
|
||||
"--status"
|
||||
"--priority"
|
||||
"--content"
|
||||
"--tags"
|
||||
"--properties"
|
||||
"--journal-only"
|
||||
"--with-properties"
|
||||
"--with-extends"
|
||||
"--with-classes"
|
||||
"--with-type"]}
|
||||
|
||||
:upsert
|
||||
{:commands ["upsert block"
|
||||
"upsert page"
|
||||
"upsert task"
|
||||
"upsert asset"
|
||||
"upsert tag"
|
||||
"upsert property"]
|
||||
:options ["--id"
|
||||
"--uuid"
|
||||
"--target-id"
|
||||
"--target-uuid"
|
||||
"--target-page"
|
||||
"--pos"
|
||||
"--content"
|
||||
"--path"
|
||||
"--blocks-file"
|
||||
"--status"
|
||||
"--priority"
|
||||
"--scheduled"
|
||||
"--deadline"
|
||||
"--no-status"
|
||||
"--no-priority"
|
||||
"--no-scheduled"
|
||||
"--no-deadline"
|
||||
"--update-tags"
|
||||
"--update-properties"
|
||||
"--remove-tags"
|
||||
"--remove-properties"
|
||||
"--page"
|
||||
"--name"
|
||||
"--type"
|
||||
"--cardinality"
|
||||
"--hide"
|
||||
"--public"]}
|
||||
|
||||
:remove
|
||||
{:commands ["remove block"
|
||||
"remove page"
|
||||
"remove tag"
|
||||
"remove property"]
|
||||
:options ["--id"
|
||||
"--uuid"
|
||||
"--name"]}
|
||||
|
||||
:query
|
||||
{:commands ["query"
|
||||
"query list"]
|
||||
:options ["--query"
|
||||
"--name"
|
||||
"--inputs"]}
|
||||
|
||||
:search
|
||||
{:commands ["search block"
|
||||
"search page"
|
||||
"search property"
|
||||
"search tag"]
|
||||
:options ["--content"]}
|
||||
|
||||
:show
|
||||
{:commands ["show"]
|
||||
:options ["--id"
|
||||
"--uuid"
|
||||
"--page"
|
||||
"--linked-references"
|
||||
"--level"
|
||||
"stdin:--id"]}
|
||||
|
||||
:debug
|
||||
{:commands ["debug pull"]
|
||||
:options ["--id"
|
||||
"--uuid"
|
||||
"--ident"]}
|
||||
|
||||
:example
|
||||
{:commands ["example list"
|
||||
"example list page"
|
||||
"example upsert"
|
||||
"example upsert page"
|
||||
"example remove"
|
||||
"example remove page"
|
||||
"example query"
|
||||
"example query list"
|
||||
"example search"
|
||||
"example search block"
|
||||
"example show"]
|
||||
:options []}
|
||||
|
||||
:server
|
||||
{:commands ["server list"
|
||||
"server cleanup"
|
||||
"server start"
|
||||
"server stop"
|
||||
"server restart"]
|
||||
:options ["--graph"]}
|
||||
|
||||
:doctor
|
||||
{:commands ["doctor"]
|
||||
:options ["--dev-script"]}
|
||||
|
||||
:completion
|
||||
{:commands ["completion"]
|
||||
:options ["bash"
|
||||
"zsh"
|
||||
"--shell"]}
|
||||
|
||||
:skill
|
||||
{:commands ["skill show"
|
||||
"skill install"]
|
||||
:options ["--global"]}}}
|
||||
171
cli-e2e/spec/sync_cases.edn
Normal file
171
cli-e2e/spec/sync_cases.edn
Normal file
@@ -0,0 +1,171 @@
|
||||
{:templates
|
||||
{:sync/base
|
||||
{:cleanup
|
||||
["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
|
||||
"{{cli}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json server stop --graph {{graph-arg}}"],
|
||||
:tags [:sync],
|
||||
:vars
|
||||
{:sync-port "18080",
|
||||
:sync-http-base "http://127.0.0.1:18080",
|
||||
:sync-ws-url "ws://127.0.0.1:18080/sync/%s",
|
||||
:home-dir "{{tmp-dir}}/home",
|
||||
:auth-path "{{tmp-dir}}/home/logseq/auth.json",
|
||||
:cli-home "HOME='{{tmp-dir}}/home' {{cli}}"},
|
||||
:setup
|
||||
["mkdir -p '{{tmp-dir}}/graphs-b'"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"]},
|
||||
:sync/common
|
||||
{:extends :sync/base,
|
||||
:cmds
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync download --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"],
|
||||
:expect
|
||||
{:exit 0,
|
||||
:stdout-json-paths
|
||||
{[:status] "ok",
|
||||
[:data :pending-local] 0,
|
||||
[:data :pending-asset] 0,
|
||||
[:data :pending-server] 0,
|
||||
[:data :last-error] nil}},
|
||||
:covers
|
||||
{:commands ["sync upload" "sync download" "sync status"],
|
||||
:options {:global ["--config" "--graph" "--root-dir" "--output"]}}}}
|
||||
:cases
|
||||
[{:tags [:happy-path :bootstrap :a-to-b],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncBootstrapHome >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncBootstrapHome --content '{{marker-content}}' >/dev/null"],
|
||||
:cmds
|
||||
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-bootstrap-upload-download-a-to-b",
|
||||
:graph "sync-e2e-bootstrap-a-to-b",
|
||||
:vars {:marker-content "sync-happy-bootstrap-a-to-b-marker"}}
|
||||
{:tags [:happy-path :incremental :a-to-b],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncIncrementalHome >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{seed-marker}}' >/dev/null"],
|
||||
:cmds
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncIncrementalHome --content '{{incremental-marker}}' >/dev/null"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{incremental-marker}}\")] ]' --require-result"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-incremental-update-a-to-b",
|
||||
:graph "sync-e2e-incremental-a-to-b",
|
||||
:vars
|
||||
{:seed-marker "sync-happy-incremental-seed-marker",
|
||||
:incremental-marker "sync-happy-incremental-update-marker"}}
|
||||
{:tags [:happy-path :bidirectional :roundtrip],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncRoundtripAHome >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripAHome --content '{{marker-a}}' >/dev/null"],
|
||||
:cmds
|
||||
["{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert page --graph {{graph-arg}} --page SyncRoundtripBHome >/dev/null"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json upsert block --graph {{graph-arg}} --target-page SyncRoundtripBHome --content '{{marker-b}}' >/dev/null"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 120 --interval-s 1"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-b}}\")] ]' --require-result"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-bidirectional-roundtrip",
|
||||
:graph "sync-e2e-bidirectional-roundtrip",
|
||||
:vars
|
||||
{:marker-a "sync-happy-roundtrip-a-seed-marker",
|
||||
:marker-b "sync-happy-roundtrip-b-origin-marker"}}
|
||||
{:tags [:happy-path :multi-batch :a-to-b],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchHome >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchHome --content '{{seed-marker}}' >/dev/null"],
|
||||
:cmds
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchOne >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchOne --content '{{batch-marker-1}}' >/dev/null"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchTwo >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchTwo --content '{{batch-marker-2}}' >/dev/null"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncMultiBatchThree >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncMultiBatchThree --content '{{batch-marker-3}}' >/dev/null"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-1}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-2}}\")] ]' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{batch-marker-3}}\")] ]' --require-result"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-multi-batch-operations",
|
||||
:graph "sync-e2e-multi-batch-operations",
|
||||
:vars
|
||||
{:seed-marker "sync-happy-multi-batch-seed-marker",
|
||||
:batch-marker-3 "sync-happy-multi-batch-marker-3",
|
||||
:batch-marker-1 "sync-happy-multi-batch-marker-1",
|
||||
:batch-marker-2 "sync-happy-multi-batch-marker-2"}}
|
||||
{:tags [:negative :upload :duplicate],
|
||||
:extends :sync/base,
|
||||
:setup
|
||||
["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}} >/dev/null"],
|
||||
:cmds
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync upload --graph {{graph-arg}}"],
|
||||
:expect
|
||||
{:exit 1,
|
||||
:stdout-json-paths
|
||||
{[:status] "error",
|
||||
[:error :code] "graph-already-exists",
|
||||
[:error :context :graph-name] "{{graph}}"},
|
||||
:stdout-contains ["delete it before uploading again"]},
|
||||
:id "sync-upload-rejects-duplicate-remote-graph",
|
||||
:graph "sync-e2e-upload-rejects-duplicate-remote-graph"}
|
||||
{:tags [:happy-path :steady-state :status],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page SyncSteadyStateHome >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page SyncSteadyStateHome --content '{{marker-content}}' >/dev/null"],
|
||||
:cmds
|
||||
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{marker-content}}\")] ]' --require-result"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 30 --interval-s 1"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-status-steady-state",
|
||||
:graph "sync-e2e-status-steady-state",
|
||||
:vars {:marker-content "sync-happy-steady-state-marker"}}
|
||||
{:tags [:stress :bidirectional :random :block-ops],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null"],
|
||||
:cmds
|
||||
["python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile default --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 180 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-random-bidirectional-block-ops",
|
||||
:graph "sync-e2e-random-bidirectional-block-ops",
|
||||
:vars
|
||||
{:random-seed "424242",
|
||||
:random-page "SyncRandomOpsHome",
|
||||
:rounds-per-client "40"}}
|
||||
{:tags [:stress :offline :bidirectional :random :block-ops],
|
||||
:extends :sync/common,
|
||||
:setup
|
||||
["{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert page --graph {{graph-arg}} --page '{{random-page}}' >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json upsert block --graph {{graph-arg}} --target-page '{{random-page}}' --content '{{seed-marker}}' >/dev/null"],
|
||||
:cmds
|
||||
["python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find ?title :where [?b :block/title ?title] [(= ?title \"{{seed-marker}}\")] ]' --require-result"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync stop --graph {{graph-arg}}"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync stop --graph {{graph-arg}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/random_bidirectional_block_ops.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --page '{{random-page}}' --profile high-stress --rounds-per-client {{rounds-per-client}} --seed {{random-seed}}"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync start --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{root-dir}}' --config '{{config-path}}' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/wait_sync_status.py' --cli '{{repo-root}}/static/logseq-cli.js' --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --graph '{{graph}}' --timeout-s 240 --interval-s 1 --min-tx-delta 1 --baseline-tx 0"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/compare_graph_queries.py' --cli '{{repo-root}}/static/logseq-cli.js' --graph '{{graph}}' --config-a '{{config-path}}' --root-dir-a '{{root-dir}}' --config-b '{{tmp-dir}}/cli-b.edn' --root-dir-b '{{tmp-dir}}/graphs-b' --query '[:find (pull ?b [:block/uuid :block/title :block/order {:block/parent [:block/uuid]}]) :where [?p :block/title \"{{random-page}}\"] [?b :block/page ?p] [?b :block/uuid]]' --require-result"
|
||||
"{{cli-home}} --root-dir '{{tmp-dir}}/graphs-b' --config '{{tmp-dir}}/cli-b.edn' --output json sync status --graph {{graph-arg}}"],
|
||||
:id "sync-offline-random-bidirectional-block-ops",
|
||||
:graph "sync-e2e-offline-random-bidirectional-block-ops",
|
||||
:vars
|
||||
{:seed-marker "sync-offline-random-seed-marker",
|
||||
:random-seed "989898",
|
||||
:random-page "SyncOfflineRandomOpsHome",
|
||||
:rounds-per-client "40"}}]}
|
||||
13
cli-e2e/spec/sync_inventory.edn
Normal file
13
cli-e2e/spec/sync_inventory.edn
Normal file
@@ -0,0 +1,13 @@
|
||||
{:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes
|
||||
{:global
|
||||
{:options ["--config"
|
||||
"--graph"
|
||||
"--root-dir"
|
||||
"--output"]}
|
||||
|
||||
:sync
|
||||
{:commands ["sync upload"
|
||||
"sync download"
|
||||
"sync status"]
|
||||
:options []}}}
|
||||
184
cli-e2e/src/logseq/cli/e2e/cleanup.clj
Normal file
184
cli-e2e/src/logseq/cli/e2e/cleanup.clj
Normal file
@@ -0,0 +1,184 @@
|
||||
(ns logseq.cli.e2e.cleanup
|
||||
(:require [babashka.fs :as fs]
|
||||
[clojure.java.shell :as java-shell]
|
||||
[clojure.string :as string]))
|
||||
|
||||
(def ^:private cli-e2e-temp-prefix "logseq-cli-e2e-")
|
||||
(def ^:private db-sync-default-port 18080)
|
||||
|
||||
(defn- parse-ps-line
|
||||
[line]
|
||||
(when-let [[_ pid-str command] (re-matches #"\s*(\d+)\s+(.*)" (or line ""))]
|
||||
{:pid (Long/parseLong pid-str)
|
||||
:command command}))
|
||||
|
||||
(defn- cli-e2e-db-worker-command?
|
||||
[command]
|
||||
(and (re-find #"db-worker-node(?:\.js)?\b" command)
|
||||
(re-find #"logseq-cli-e2e-" command)))
|
||||
|
||||
(defn list-cli-e2e-db-worker-pids
|
||||
([]
|
||||
(list-cli-e2e-db-worker-pids {}))
|
||||
([{:keys [shell-fn]
|
||||
:or {shell-fn java-shell/sh}}]
|
||||
(let [{:keys [exit out err]} (shell-fn "ps" "-ax" "-o" "pid=" "-o" "command=")]
|
||||
(when-not (zero? exit)
|
||||
(throw (ex-info "Unable to scan processes"
|
||||
{:exit exit
|
||||
:err err})))
|
||||
(->> (string/split-lines (or out ""))
|
||||
(keep parse-ps-line)
|
||||
(filter (fn [{:keys [command]}]
|
||||
(cli-e2e-db-worker-command? command)))
|
||||
(mapv :pid)))))
|
||||
|
||||
(defn- pid-alive?
|
||||
[pid]
|
||||
(zero? (:exit (java-shell/sh "kill" "-0" (str pid)))))
|
||||
|
||||
(defn- kill-pid!
|
||||
[pid]
|
||||
(try
|
||||
(if-not (pid-alive? pid)
|
||||
:not-found
|
||||
(do
|
||||
(java-shell/sh "kill" "-TERM" (str pid))
|
||||
(Thread/sleep 100)
|
||||
(if-not (pid-alive? pid)
|
||||
:killed
|
||||
(do
|
||||
(java-shell/sh "kill" "-KILL" (str pid))
|
||||
(Thread/sleep 50)
|
||||
(if (pid-alive? pid)
|
||||
:failed
|
||||
:killed)))))
|
||||
(catch Exception _
|
||||
:failed)))
|
||||
|
||||
(defn cleanup-db-worker-processes!
|
||||
([]
|
||||
(cleanup-db-worker-processes! {}))
|
||||
([{:keys [dry-run list-pids-fn kill-pid-fn]}]
|
||||
(let [list-pids-fn (or list-pids-fn list-cli-e2e-db-worker-pids)
|
||||
kill-pid-fn (or kill-pid-fn kill-pid!)
|
||||
found-pids (vec (list-pids-fn))]
|
||||
(if dry-run
|
||||
{:dry-run? true
|
||||
:found-pids found-pids
|
||||
:would-kill-pids found-pids
|
||||
:killed-pids []
|
||||
:failed-pids []}
|
||||
(let [{:keys [killed-pids failed-pids]}
|
||||
(reduce (fn [acc pid]
|
||||
(if (= :failed (kill-pid-fn pid))
|
||||
(update acc :failed-pids conj pid)
|
||||
(update acc :killed-pids conj pid)))
|
||||
{:killed-pids []
|
||||
:failed-pids []}
|
||||
found-pids)]
|
||||
{:dry-run? false
|
||||
:found-pids found-pids
|
||||
:would-kill-pids []
|
||||
:killed-pids killed-pids
|
||||
:failed-pids failed-pids})))))
|
||||
|
||||
(defn- parse-long-safe
|
||||
[value]
|
||||
(try
|
||||
(Long/parseLong value)
|
||||
(catch Exception _
|
||||
nil)))
|
||||
|
||||
(defn list-cli-e2e-db-sync-port-pids
|
||||
([]
|
||||
(list-cli-e2e-db-sync-port-pids {}))
|
||||
([{:keys [shell-fn port]
|
||||
:or {shell-fn java-shell/sh
|
||||
port db-sync-default-port}}]
|
||||
(let [{:keys [exit out err]} (shell-fn "lsof" "-nP" (str "-iTCP:" port) "-sTCP:LISTEN")
|
||||
out-lines (->> (string/split-lines (or out ""))
|
||||
(map string/trim)
|
||||
(remove string/blank?)
|
||||
vec)]
|
||||
(when (and (not (zero? exit))
|
||||
(or (not (string/blank? err))
|
||||
(seq out-lines)))
|
||||
(throw (ex-info "Unable to scan db-sync server port listeners"
|
||||
{:exit exit
|
||||
:err err
|
||||
:port port})))
|
||||
(->> out-lines
|
||||
(filter #(re-find (re-pattern (str ":" port "\\b")) %))
|
||||
(keep (fn [line]
|
||||
(some-> (string/split line #"\s+")
|
||||
(nth 1 nil)
|
||||
parse-long-safe)))
|
||||
distinct
|
||||
vec))))
|
||||
|
||||
(defn cleanup-db-sync-port-processes!
|
||||
([]
|
||||
(cleanup-db-sync-port-processes! {}))
|
||||
([{:keys [dry-run list-pids-fn kill-pid-fn]
|
||||
:as _opts}]
|
||||
(let [list-pids-fn (or list-pids-fn list-cli-e2e-db-sync-port-pids)
|
||||
kill-pid-fn (or kill-pid-fn kill-pid!)
|
||||
found-pids (vec (list-pids-fn))]
|
||||
(if dry-run
|
||||
{:dry-run? true
|
||||
:found-pids found-pids
|
||||
:would-kill-pids found-pids
|
||||
:killed-pids []
|
||||
:failed-pids []}
|
||||
(let [{:keys [killed-pids failed-pids]}
|
||||
(reduce (fn [acc pid]
|
||||
(if (= :failed (kill-pid-fn pid))
|
||||
(update acc :failed-pids conj pid)
|
||||
(update acc :killed-pids conj pid)))
|
||||
{:killed-pids []
|
||||
:failed-pids []}
|
||||
found-pids)]
|
||||
{:dry-run? false
|
||||
:found-pids found-pids
|
||||
:would-kill-pids []
|
||||
:killed-pids killed-pids
|
||||
:failed-pids failed-pids})))))
|
||||
|
||||
(defn- list-cli-e2e-temp-roots
|
||||
[tmp-root]
|
||||
(->> (fs/list-dir tmp-root)
|
||||
(filter fs/directory?)
|
||||
(filter #(string/starts-with? (fs/file-name %) cli-e2e-temp-prefix))
|
||||
(mapv str)))
|
||||
|
||||
(defn cleanup-temp-roots!
|
||||
([]
|
||||
(cleanup-temp-roots! {}))
|
||||
([{:keys [dry-run tmp-root list-dirs-fn delete-dir-fn]
|
||||
:or {tmp-root (System/getProperty "java.io.tmpdir")
|
||||
delete-dir-fn fs/delete-tree}}]
|
||||
(let [list-dirs-fn (or list-dirs-fn
|
||||
#(list-cli-e2e-temp-roots tmp-root))
|
||||
found-dirs (vec (list-dirs-fn))]
|
||||
(if dry-run
|
||||
{:dry-run? true
|
||||
:found-dirs found-dirs
|
||||
:would-remove-dirs found-dirs
|
||||
:removed-dirs []
|
||||
:failed-dirs []}
|
||||
(let [{:keys [removed-dirs failed-dirs]}
|
||||
(reduce (fn [acc dir]
|
||||
(try
|
||||
(delete-dir-fn dir)
|
||||
(update acc :removed-dirs conj dir)
|
||||
(catch Exception _
|
||||
(update acc :failed-dirs conj dir))))
|
||||
{:removed-dirs []
|
||||
:failed-dirs []}
|
||||
found-dirs)]
|
||||
{:dry-run? false
|
||||
:found-dirs found-dirs
|
||||
:would-remove-dirs []
|
||||
:removed-dirs removed-dirs
|
||||
:failed-dirs failed-dirs})))))
|
||||
120
cli-e2e/src/logseq/cli/e2e/coverage.clj
Normal file
120
cli-e2e/src/logseq/cli/e2e/coverage.clj
Normal file
@@ -0,0 +1,120 @@
|
||||
(ns logseq.cli.e2e.coverage
|
||||
(:require [clojure.set :as set]
|
||||
[clojure.string :as string]))
|
||||
|
||||
(defn- normalize-commands
|
||||
[commands]
|
||||
(->> commands
|
||||
(map str)
|
||||
(remove string/blank?)
|
||||
distinct
|
||||
sort
|
||||
vec))
|
||||
|
||||
(defn- normalize-options
|
||||
[options]
|
||||
(->> options
|
||||
(map str)
|
||||
(remove string/blank?)
|
||||
distinct
|
||||
sort
|
||||
vec))
|
||||
|
||||
(defn- excluded-command?
|
||||
[excluded-prefixes command]
|
||||
(let [head (first (string/split (str command) #" "))]
|
||||
(contains? (set excluded-prefixes) head)))
|
||||
|
||||
(defn validate-inventory!
|
||||
[{:keys [excluded-command-prefixes scopes] :as inventory}]
|
||||
(let [invalid-commands (->> scopes
|
||||
vals
|
||||
(mapcat :commands)
|
||||
(filter #(excluded-command? excluded-command-prefixes %))
|
||||
normalize-commands)
|
||||
invalid-scopes (->> scopes
|
||||
(keep (fn [[scope {:keys [commands]}]]
|
||||
(when (some #(excluded-command? excluded-command-prefixes %) commands)
|
||||
scope)))
|
||||
sort
|
||||
vec)]
|
||||
(when (seq invalid-commands)
|
||||
(throw (ex-info "Excluded commands cannot appear in cli-e2e inventory"
|
||||
{:invalid-commands invalid-commands
|
||||
:invalid-scopes invalid-scopes
|
||||
:inventory inventory})))
|
||||
inventory))
|
||||
|
||||
(defn validate-cases!
|
||||
[{:keys [excluded-command-prefixes] :as inventory} cases]
|
||||
(let [invalid-cases (->> cases
|
||||
(filter (fn [{:keys [covers]}]
|
||||
(some #(excluded-command? excluded-command-prefixes %)
|
||||
(get covers :commands []))))
|
||||
(mapv :id))]
|
||||
(when (seq invalid-cases)
|
||||
(throw (ex-info "Excluded commands cannot be covered by cli-e2e cases"
|
||||
{:invalid-case-ids invalid-cases
|
||||
:inventory inventory})))
|
||||
cases))
|
||||
|
||||
(defn command->scope
|
||||
[{:keys [scopes]}]
|
||||
(into {}
|
||||
(mapcat (fn [[scope {:keys [commands]}]]
|
||||
(map (fn [command]
|
||||
[command scope])
|
||||
commands)))
|
||||
scopes))
|
||||
|
||||
(defn required-commands
|
||||
[{:keys [scopes]}]
|
||||
(->> scopes
|
||||
vals
|
||||
(mapcat :commands)
|
||||
normalize-commands))
|
||||
|
||||
(defn covered-commands
|
||||
[cases]
|
||||
(->> cases
|
||||
(mapcat #(get-in % [:covers :commands]))
|
||||
normalize-commands))
|
||||
|
||||
(defn required-options-by-scope
|
||||
[{:keys [scopes]}]
|
||||
(into {}
|
||||
(map (fn [[scope {:keys [options]}]]
|
||||
[scope (normalize-options options)]))
|
||||
scopes))
|
||||
|
||||
(defn covered-options-by-scope
|
||||
[cases]
|
||||
(reduce (fn [acc {:keys [covers]}]
|
||||
(reduce-kv (fn [acc' scope options]
|
||||
(update acc' scope (fnil into #{}) options))
|
||||
acc
|
||||
(get covers :options {})))
|
||||
{}
|
||||
cases))
|
||||
|
||||
(defn coverage-report
|
||||
[inventory cases]
|
||||
(validate-inventory! inventory)
|
||||
(validate-cases! inventory cases)
|
||||
(let [required-commands* (set (required-commands inventory))
|
||||
covered-commands* (set (covered-commands cases))
|
||||
covered-options* (covered-options-by-scope cases)
|
||||
required-options* (required-options-by-scope inventory)]
|
||||
{:missing-commands (->> (set/difference required-commands* covered-commands*)
|
||||
normalize-commands)
|
||||
:missing-options (into {}
|
||||
(map (fn [[scope options]]
|
||||
[scope (->> (set/difference (set options)
|
||||
(get covered-options* scope #{}))
|
||||
normalize-options)]))
|
||||
required-options*)}))
|
||||
|
||||
(defn complete?
|
||||
[{:keys [missing-commands missing-options]}]
|
||||
(and (empty? missing-commands)
|
||||
(every? empty? (vals missing-options))))
|
||||
467
cli-e2e/src/logseq/cli/e2e/main.clj
Normal file
467
cli-e2e/src/logseq/cli/e2e/main.clj
Normal file
@@ -0,0 +1,467 @@
|
||||
(ns logseq.cli.e2e.main
|
||||
(:require [clojure.set :as set]
|
||||
[logseq.cli.e2e.cleanup :as cleanup]
|
||||
[logseq.cli.e2e.coverage :as coverage]
|
||||
[logseq.cli.e2e.manifests :as manifests]
|
||||
[logseq.cli.e2e.preflight :as preflight]
|
||||
[logseq.cli.e2e.report :as report]
|
||||
[logseq.cli.e2e.runner :as runner]
|
||||
[logseq.cli.e2e.shell :as shell]
|
||||
[logseq.cli.e2e.sync-fixture :as sync-fixture])
|
||||
(:import (java.util.concurrent Executors LinkedBlockingQueue TimeUnit)))
|
||||
|
||||
(defn select-cases
|
||||
[cases {:keys [case include]}]
|
||||
(cond
|
||||
case
|
||||
(filterv #(= case (:id %)) cases)
|
||||
|
||||
(seq include)
|
||||
(let [include-tags (set (map keyword include))]
|
||||
(filterv (fn [{:keys [tags]}]
|
||||
(not-empty (set/intersection include-tags (set tags))))
|
||||
cases))
|
||||
|
||||
:else
|
||||
(vec cases)))
|
||||
|
||||
(def default-suite :non-sync)
|
||||
(def default-jobs 1)
|
||||
(def default-cli-jobs 4)
|
||||
|
||||
(defn- suite-from-opts
|
||||
[opts]
|
||||
(or (:suite opts) default-suite))
|
||||
|
||||
(defn- elapsed-ms
|
||||
[started-at]
|
||||
(long (/ (- (System/nanoTime) started-at) 1000000)))
|
||||
|
||||
(defn- format-duration
|
||||
[started-at]
|
||||
(format "%.2fs" (/ (double (- (System/nanoTime) started-at)) 1000000000.0)))
|
||||
|
||||
(defn- positive-jobs
|
||||
[jobs]
|
||||
(let [jobs (or jobs default-jobs)]
|
||||
(when-not (and (integer? jobs) (pos? jobs))
|
||||
(throw (ex-info "--jobs must be a positive integer"
|
||||
{:jobs jobs})))
|
||||
jobs))
|
||||
|
||||
(defn run-selected-cases!
|
||||
[selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}]
|
||||
(let [total (count selected-cases)
|
||||
_ (positive-jobs jobs)]
|
||||
(reduce (fn [acc [idx case]]
|
||||
(let [index (inc idx)
|
||||
started-at (System/nanoTime)]
|
||||
(when on-case-start
|
||||
(on-case-start {:index index
|
||||
:total total
|
||||
:case case}))
|
||||
(try
|
||||
(let [result (run-case case {:run-command run-command
|
||||
:detailed-log? detailed-log?
|
||||
:timings? timings?})
|
||||
payload {:index index
|
||||
:total total
|
||||
:case case
|
||||
:result result
|
||||
:elapsed-ms (elapsed-ms started-at)}]
|
||||
(when on-case-success
|
||||
(on-case-success payload))
|
||||
(conj acc result))
|
||||
(catch Exception error
|
||||
(when on-case-failure
|
||||
(on-case-failure {:index index
|
||||
:total total
|
||||
:case case
|
||||
:error error
|
||||
:elapsed-ms (elapsed-ms started-at)}))
|
||||
(throw error)))))
|
||||
[]
|
||||
(map-indexed vector selected-cases))))
|
||||
|
||||
(defn run-selected-cases-in-parallel!
|
||||
[selected-cases run-case run-command {:keys [on-case-start on-case-success on-case-failure detailed-log? timings? jobs]}]
|
||||
(let [total (count selected-cases)
|
||||
jobs (positive-jobs jobs)
|
||||
executor (Executors/newFixedThreadPool jobs)
|
||||
completions (LinkedBlockingQueue.)]
|
||||
(try
|
||||
(doseq [[idx case] (map-indexed vector selected-cases)]
|
||||
(let [index (inc idx)]
|
||||
(when on-case-start
|
||||
(on-case-start {:index index
|
||||
:total total
|
||||
:case case}))
|
||||
(.submit executor
|
||||
^Runnable
|
||||
(fn []
|
||||
(let [started-at (System/nanoTime)]
|
||||
(.put completions
|
||||
(try
|
||||
(let [result (run-case case {:run-command run-command
|
||||
:detailed-log? detailed-log?
|
||||
:timings? timings?})]
|
||||
{:index index
|
||||
:total total
|
||||
:case case
|
||||
:result result
|
||||
:elapsed-ms (elapsed-ms started-at)})
|
||||
(catch Exception error
|
||||
{:index index
|
||||
:total total
|
||||
:case case
|
||||
:error error
|
||||
:elapsed-ms (elapsed-ms started-at)}))))))))
|
||||
(loop [remaining total
|
||||
results []
|
||||
failure nil]
|
||||
(if (zero? remaining)
|
||||
(do
|
||||
(when failure
|
||||
(throw failure))
|
||||
(->> results
|
||||
(sort-by :index)
|
||||
(mapv :result)))
|
||||
(let [payload (.take completions)]
|
||||
(if-let [error (:error payload)]
|
||||
(do
|
||||
(when on-case-failure
|
||||
(on-case-failure payload))
|
||||
(recur (dec remaining) results (or failure error)))
|
||||
(do
|
||||
(when on-case-success
|
||||
(on-case-success payload))
|
||||
(recur (dec remaining) (conj results payload) failure))))))
|
||||
(finally
|
||||
(.shutdown executor)
|
||||
(.awaitTermination executor 1 TimeUnit/MINUTES)))))
|
||||
|
||||
(defn run!
|
||||
[{:keys [inventory cases skip-build run-command jobs]
|
||||
:as opts}]
|
||||
(let [run-command (or run-command shell/run!)
|
||||
run-case (or (:run-case opts) runner/run-case!)
|
||||
suite (suite-from-opts opts)
|
||||
sync-suite? (= suite :sync)
|
||||
jobs (positive-jobs jobs)
|
||||
targeted-run? (or (:case opts) (seq (:include opts)))
|
||||
on-preflight-start (:on-preflight-start opts)
|
||||
on-preflight-complete (:on-preflight-complete opts)
|
||||
on-cases-ready (:on-cases-ready opts)]
|
||||
(when on-preflight-start
|
||||
(on-preflight-start {:skip-build skip-build}))
|
||||
(let [preflight-result (preflight/run! {:skip-build skip-build
|
||||
:run-command run-command})
|
||||
inventory (or inventory (manifests/load-inventory suite))
|
||||
cases (or cases (manifests/load-cases suite))]
|
||||
(when on-preflight-complete
|
||||
(on-preflight-complete preflight-result))
|
||||
(let [selected-cases (select-cases cases opts)
|
||||
coverage-result (when-not targeted-run?
|
||||
(coverage/coverage-report inventory selected-cases))]
|
||||
(when (and coverage-result
|
||||
(not (coverage/complete? coverage-result)))
|
||||
(throw (ex-info "Missing coverage"
|
||||
{:coverage coverage-result
|
||||
:message (report/format-missing-coverage coverage-result)})))
|
||||
(when on-cases-ready
|
||||
(on-cases-ready {:total (count selected-cases)
|
||||
:targeted-run? targeted-run?}))
|
||||
(let [suite-context (when sync-suite?
|
||||
(sync-fixture/before-suite! {:run-command run-command}))
|
||||
sync-context (if suite-context
|
||||
(assoc suite-context :e2ee-password (:e2ee-password opts))
|
||||
suite-context)
|
||||
run-case* (if sync-suite?
|
||||
(fn [case case-opts]
|
||||
(run-case (sync-fixture/prepare-case case sync-context)
|
||||
case-opts))
|
||||
run-case)]
|
||||
(try
|
||||
{:status :ok
|
||||
:cases selected-cases
|
||||
:coverage coverage-result
|
||||
:results ((if (> jobs 1)
|
||||
run-selected-cases-in-parallel!
|
||||
run-selected-cases!)
|
||||
selected-cases
|
||||
run-case*
|
||||
run-command
|
||||
(assoc opts :jobs jobs))}
|
||||
(finally
|
||||
(when suite-context
|
||||
(sync-fixture/after-suite! suite-context {:run-command run-command})))))))))
|
||||
(defn build!
|
||||
[opts]
|
||||
(preflight/run! opts))
|
||||
|
||||
(defn list-cases!
|
||||
[opts]
|
||||
(doseq [{:keys [id]} (manifests/load-cases (suite-from-opts opts))]
|
||||
(println id)))
|
||||
|
||||
(defn list-sync-cases!
|
||||
[opts]
|
||||
(list-cases! (assoc opts :suite :sync)))
|
||||
|
||||
(defn- print-cleanup-help!
|
||||
[]
|
||||
(println "Usage: bb -f cli-e2e/bb.edn cleanup [options]")
|
||||
(println)
|
||||
(println "Options:")
|
||||
(println " -h, --help Show this help and exit")
|
||||
(println " --dry-run Scan and report only; do not kill/delete")
|
||||
(println)
|
||||
(println "Cleanups performed:")
|
||||
(println " - Terminate cli-e2e db-worker-node processes")
|
||||
(println " - Terminate db-sync server listeners on port 18080")
|
||||
(println " - Remove cli-e2e temp roots")
|
||||
(flush))
|
||||
|
||||
(defn cleanup!
|
||||
[opts]
|
||||
(if (:help opts)
|
||||
(do
|
||||
(print-cleanup-help!)
|
||||
{:status :help})
|
||||
(let [dry-run? (boolean (:dry-run opts))
|
||||
cleanup-opts (cond-> {}
|
||||
dry-run? (assoc :dry-run true))
|
||||
processes (cleanup/cleanup-db-worker-processes! cleanup-opts)
|
||||
db-sync-port-processes (cleanup/cleanup-db-sync-port-processes! cleanup-opts)
|
||||
temp-roots (cleanup/cleanup-temp-roots! cleanup-opts)]
|
||||
(println "==> Running cli-e2e cleanup")
|
||||
(if dry-run?
|
||||
(do
|
||||
(println (format "[dry-run] db-worker-node processes: found %d, would kill %d"
|
||||
(count (:found-pids processes))
|
||||
(count (:would-kill-pids processes))))
|
||||
(println (format "[dry-run] db-sync server processes (port 18080): found %d, would kill %d"
|
||||
(count (:found-pids db-sync-port-processes))
|
||||
(count (:would-kill-pids db-sync-port-processes))))
|
||||
(println (format "[dry-run] temp roots: found %d, would remove %d"
|
||||
(count (:found-dirs temp-roots))
|
||||
(count (:would-remove-dirs temp-roots)))))
|
||||
(do
|
||||
(println (format "db-worker-node processes: found %d, killed %d, failed %d"
|
||||
(count (:found-pids processes))
|
||||
(count (:killed-pids processes))
|
||||
(count (:failed-pids processes))))
|
||||
(println (format "db-sync server processes (port 18080): found %d, killed %d, failed %d"
|
||||
(count (:found-pids db-sync-port-processes))
|
||||
(count (:killed-pids db-sync-port-processes))
|
||||
(count (:failed-pids db-sync-port-processes))))
|
||||
(println (format "temp roots: found %d, removed %d, failed %d"
|
||||
(count (:found-dirs temp-roots))
|
||||
(count (:removed-dirs temp-roots))
|
||||
(count (:failed-dirs temp-roots))))))
|
||||
(flush)
|
||||
{:status :ok
|
||||
:dry-run? dry-run?
|
||||
:processes processes
|
||||
:db-sync-port-processes db-sync-port-processes
|
||||
:temp-roots temp-roots})))
|
||||
|
||||
(defn- print-failure-details!
|
||||
[error]
|
||||
(let [data (ex-data error)]
|
||||
(println (str " reason: " (.getMessage error)))
|
||||
(when-let [cmd (:cmd data)]
|
||||
(println (str " cmd: " cmd)))
|
||||
(when-let [stream (:stream data)]
|
||||
(println (str " stream: " stream)))
|
||||
(when-let [snippet (:snippet data)]
|
||||
(println (str " snippet: " snippet)))
|
||||
(flush)))
|
||||
|
||||
(defn- format-step-label
|
||||
[{:keys [phase step-index step-total]}]
|
||||
(let [phase-name (name (or phase :command))]
|
||||
(format "%s %d/%d" phase-name (or step-index 1) (or step-total 1))))
|
||||
|
||||
(defn- print-case-timings!
|
||||
[timings]
|
||||
(when (seq timings)
|
||||
(println " step timings:")
|
||||
(doseq [{:keys [elapsed-ms status cmd] :as timing} timings]
|
||||
(println (format " - [%-12s] %6dms (%s) $ %s"
|
||||
(format-step-label timing)
|
||||
elapsed-ms
|
||||
(name (or status :ok))
|
||||
cmd)))
|
||||
(flush)))
|
||||
|
||||
(defn- print-slow-steps!
|
||||
[all-step-timings]
|
||||
(when (seq all-step-timings)
|
||||
(println "Slow steps (top 10):")
|
||||
(doseq [{:keys [case-id elapsed-ms cmd] :as timing}
|
||||
(->> all-step-timings
|
||||
(sort-by :elapsed-ms >)
|
||||
(take 10))]
|
||||
(println (format " - %-45s [%-12s] %6dms $ %s"
|
||||
case-id
|
||||
(format-step-label timing)
|
||||
elapsed-ms
|
||||
cmd)))
|
||||
(flush)))
|
||||
|
||||
(defn- progress-prefix
|
||||
[{:keys [parallel? index total]} symbol]
|
||||
(if parallel?
|
||||
(str symbol " ")
|
||||
(format "[%d/%d] %s " index total symbol)))
|
||||
|
||||
(defn- print-test-help!
|
||||
[command-name suite]
|
||||
(let [sync-suite? (= suite :sync)]
|
||||
(println (str "Usage: bb -f cli-e2e/bb.edn " command-name " [options]"))
|
||||
(println)
|
||||
(println "Options:")
|
||||
(println " -h, --help Show this help and exit")
|
||||
(println " --skip-build Skip build preflight steps")
|
||||
(println " -i, --include TAG Run only cases with matching tag (repeatable)")
|
||||
(println " --case ID Run a single case by id")
|
||||
(println (format " --jobs N %s (Default: %d)"
|
||||
(if sync-suite?
|
||||
"Run up to N sync cases in parallel"
|
||||
"Run up to N non-sync cases in parallel")
|
||||
default-cli-jobs))
|
||||
(when sync-suite?
|
||||
(println " --e2ee-password VALUE E2EE password for sync commands (Default: 11111)"))
|
||||
(println " --verbose Enable verbose output")
|
||||
(println " --timings Print per-step timings and slow-step summary")
|
||||
(println)
|
||||
(println "Examples:")
|
||||
(if sync-suite?
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name))
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build")))
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name
|
||||
(if sync-suite?
|
||||
" --jobs 4"
|
||||
" --skip-build --jobs 4")))
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name " -i smoke"))
|
||||
(if sync-suite?
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name " --case sync-upload-download-mvp"))
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name " --skip-build --case global-help")))
|
||||
(when sync-suite?
|
||||
(println (str " bb -f cli-e2e/bb.edn " command-name " --e2ee-password 'my-secret'")))
|
||||
(flush)))
|
||||
|
||||
(defn- test-suite!
|
||||
[opts {:keys [suite command-name]
|
||||
:or {suite default-suite
|
||||
command-name "test"}}]
|
||||
(let [suite (or suite default-suite)
|
||||
opts (assoc opts :suite suite)]
|
||||
(if (:help opts)
|
||||
(do
|
||||
(print-test-help! command-name suite)
|
||||
{:status :help})
|
||||
(let [started-at (System/nanoTime)
|
||||
passed (atom 0)
|
||||
failed (atom 0)
|
||||
total-count (atom 0)
|
||||
timings? (boolean (:timings opts))
|
||||
all-step-timings (atom [])
|
||||
detailed-case-log? (some? (:case opts))
|
||||
parallel? (> (positive-jobs (:jobs opts)) 1)
|
||||
base-run-command (or (:run-command opts) shell/run!)
|
||||
run-command (if detailed-case-log?
|
||||
(fn [{:keys [cmd phase step-index step-total] :as command-opts}]
|
||||
(let [prefix (case phase
|
||||
:setup (format " [setup %d/%d]" step-index step-total)
|
||||
:main (format " [main %d/%d]" (or step-index 1) (or step-total 1))
|
||||
:cleanup (format " [cleanup %d/%d]" step-index step-total)
|
||||
" [command]")]
|
||||
(println (str prefix " $ " cmd))
|
||||
(flush)
|
||||
(base-run-command (assoc command-opts :stream-output? true))))
|
||||
base-run-command)]
|
||||
(println "==> Running cli-e2e cases")
|
||||
(when detailed-case-log?
|
||||
(println (format "==> Detailed case logging enabled (--case %s)" (:case opts))))
|
||||
(when timings?
|
||||
(println "==> Step timing enabled (--timings)"))
|
||||
(flush)
|
||||
(try
|
||||
(run! (assoc opts
|
||||
:run-command run-command
|
||||
:detailed-log? detailed-case-log?
|
||||
:timings? timings?
|
||||
:on-preflight-start (fn [_]
|
||||
(println "==> Build preflight: running...")
|
||||
(flush))
|
||||
:on-preflight-complete (fn [{:keys [status]}]
|
||||
(println (case status
|
||||
:skipped "==> Build preflight: skipped (--skip-build)"
|
||||
"==> Build preflight: completed"))
|
||||
(flush))
|
||||
:on-cases-ready (fn [{:keys [total]}]
|
||||
(reset! total-count total)
|
||||
(println (format "==> Prepared %d case(s), starting execution" total))
|
||||
(flush))
|
||||
:on-case-start (fn [{:keys [index total case]}]
|
||||
(println (str (progress-prefix {:parallel? parallel?
|
||||
:index index
|
||||
:total total}
|
||||
"▶")
|
||||
(:id case)))
|
||||
(flush))
|
||||
:on-case-success (fn [{:keys [index total result elapsed-ms]}]
|
||||
(swap! passed inc)
|
||||
(println (format "%s%s (%dms)"
|
||||
(progress-prefix {:parallel? parallel?
|
||||
:index index
|
||||
:total total}
|
||||
"✓")
|
||||
(:id result)
|
||||
elapsed-ms))
|
||||
(when timings?
|
||||
(let [case-timings (vec (:timings result))]
|
||||
(swap! all-step-timings into
|
||||
(map #(assoc % :case-id (:id result)) case-timings))
|
||||
(print-case-timings! case-timings)))
|
||||
(flush))
|
||||
:on-case-failure (fn [{:keys [index total case error elapsed-ms]}]
|
||||
(swap! failed inc)
|
||||
(println (format "%s%s (%dms)"
|
||||
(progress-prefix {:parallel? parallel?
|
||||
:index index
|
||||
:total total}
|
||||
"✗")
|
||||
(:id case)
|
||||
elapsed-ms))
|
||||
(print-failure-details! error)
|
||||
(when timings?
|
||||
(let [case-timings (vec (:timings (ex-data error)))]
|
||||
(swap! all-step-timings into
|
||||
(map #(assoc % :case-id (:id case)) case-timings))
|
||||
(print-case-timings! case-timings))))))
|
||||
(println (format "Summary: %d passed, %d failed" @passed @failed))
|
||||
(println (str "Selected cases: " @total-count))
|
||||
(println (str "Duration: " (format-duration started-at)))
|
||||
(when timings?
|
||||
(print-slow-steps! @all-step-timings))
|
||||
(catch Exception error
|
||||
(let [failed-count (max 1 @failed)]
|
||||
(println (format "Summary: %d passed, %d failed" @passed failed-count))
|
||||
(println (str "Selected cases: " (max @total-count failed-count)))
|
||||
(println (str "Duration: " (format-duration started-at)))
|
||||
(when timings?
|
||||
(print-slow-steps! @all-step-timings)))
|
||||
(throw error)))))))
|
||||
|
||||
(defn test!
|
||||
[opts]
|
||||
(test-suite! opts {:suite :non-sync
|
||||
:command-name "test"}))
|
||||
|
||||
(defn test-sync!
|
||||
[opts]
|
||||
(test-suite! opts {:suite :sync
|
||||
:command-name "test-sync"}))
|
||||
237
cli-e2e/src/logseq/cli/e2e/manifests.clj
Normal file
237
cli-e2e/src/logseq/cli/e2e/manifests.clj
Normal file
@@ -0,0 +1,237 @@
|
||||
(ns logseq.cli.e2e.manifests
|
||||
(:require [clojure.edn :as edn]
|
||||
[logseq.cli.e2e.paths :as paths]))
|
||||
|
||||
(def suite->manifest-files
|
||||
{:non-sync {:inventory "non_sync_inventory.edn"
|
||||
:cases "non_sync_cases.edn"}
|
||||
:sync {:inventory "sync_inventory.edn"
|
||||
:cases "sync_cases.edn"}})
|
||||
|
||||
(def default-suite :non-sync)
|
||||
|
||||
(def ^:private append-merge-keys
|
||||
#{:setup :cmds :cleanup :tags})
|
||||
|
||||
(def ^:private deep-merge-keys
|
||||
#{:vars :covers :expect})
|
||||
|
||||
(defn read-edn-file
|
||||
[path]
|
||||
(edn/read-string (slurp path)))
|
||||
|
||||
(defn- normalize-suite
|
||||
[suite]
|
||||
(let [suite' (cond
|
||||
(nil? suite) default-suite
|
||||
(keyword? suite) suite
|
||||
(string? suite) (keyword suite)
|
||||
:else suite)]
|
||||
(when-not (contains? suite->manifest-files suite')
|
||||
(throw (ex-info "Unknown cli-e2e suite"
|
||||
{:suite suite
|
||||
:known-suites (sort (keys suite->manifest-files))})))
|
||||
suite'))
|
||||
|
||||
(defn- manifest-file
|
||||
[suite kind]
|
||||
(get-in suite->manifest-files [(normalize-suite suite) kind]))
|
||||
|
||||
(defn load-inventory
|
||||
([]
|
||||
(load-inventory nil))
|
||||
([suite]
|
||||
(read-edn-file (paths/spec-path (manifest-file suite :inventory)))))
|
||||
|
||||
(defn- normalize-extends
|
||||
[extends]
|
||||
(let [extends' (cond
|
||||
(nil? extends) []
|
||||
(keyword? extends) [extends]
|
||||
(vector? extends) extends
|
||||
:else
|
||||
(throw (ex-info "Invalid :extends value in cli-e2e manifest"
|
||||
{:extends extends
|
||||
:expected "keyword | vector<keyword> | nil"})))
|
||||
invalid-entries (remove keyword? extends')]
|
||||
(when (seq invalid-entries)
|
||||
(throw (ex-info "Invalid :extends entries in cli-e2e manifest"
|
||||
{:extends extends
|
||||
:invalid-entries (vec invalid-entries)
|
||||
:expected "keyword | vector<keyword> | nil"})))
|
||||
extends'))
|
||||
|
||||
(defn- parse-manifest
|
||||
[manifest-data]
|
||||
(if-not (map? manifest-data)
|
||||
(throw (ex-info "Invalid cli-e2e manifest format"
|
||||
{:manifest-type (type manifest-data)
|
||||
:expected "{:templates {...} :cases [...]}"}))
|
||||
(let [templates (or (:templates manifest-data) {})
|
||||
cases (:cases manifest-data)]
|
||||
(when-not (map? templates)
|
||||
(throw (ex-info "Invalid cli-e2e manifest :templates format"
|
||||
{:templates templates
|
||||
:expected "map"})))
|
||||
(when-not (vector? cases)
|
||||
(throw (ex-info "Invalid cli-e2e manifest :cases format"
|
||||
{:cases cases
|
||||
:expected "vector"})))
|
||||
(doseq [case cases]
|
||||
(when-not (map? case)
|
||||
(throw (ex-info "Invalid cli-e2e case format"
|
||||
{:case case
|
||||
:expected "map"}))))
|
||||
{:templates templates
|
||||
:cases cases})))
|
||||
|
||||
(defn- reachable-template-ids
|
||||
[templates roots]
|
||||
(loop [stack (vec roots)
|
||||
visited #{}]
|
||||
(if-let [template-id (peek stack)]
|
||||
(if (contains? visited template-id)
|
||||
(recur (pop stack) visited)
|
||||
(let [template (get templates template-id)
|
||||
parent-ids (if template
|
||||
(normalize-extends (:extends template))
|
||||
[])]
|
||||
(recur (into (pop stack) parent-ids)
|
||||
(conj visited template-id))))
|
||||
visited)))
|
||||
|
||||
(defn- lint-manifest!
|
||||
[{:keys [templates cases]}]
|
||||
(let [template-ids (set (keys templates))
|
||||
template-refs (mapcat (fn [[template-id template]]
|
||||
(map (fn [target]
|
||||
{:type :invalid-extends
|
||||
:source-type :template
|
||||
:source template-id
|
||||
:target target})
|
||||
(normalize-extends (:extends template))))
|
||||
templates)
|
||||
case-refs (mapcat (fn [[index case]]
|
||||
(map (fn [target]
|
||||
{:type :invalid-extends
|
||||
:source-type :case
|
||||
:source (or (:id case)
|
||||
(str "case#" (inc index)))
|
||||
:target target})
|
||||
(normalize-extends (:extends case))))
|
||||
(map-indexed vector cases))
|
||||
invalid-extends-issues (->> (concat template-refs case-refs)
|
||||
(filter (fn [{:keys [target]}]
|
||||
(not (contains? template-ids target)))))
|
||||
duplicate-id-issues (->> cases
|
||||
(keep :id)
|
||||
frequencies
|
||||
(filter (fn [[_ count]] (> count 1)))
|
||||
(sort-by first)
|
||||
(map (fn [[case-id count]]
|
||||
{:type :duplicate-case-id
|
||||
:id case-id
|
||||
:count count})))
|
||||
roots (->> cases
|
||||
(mapcat #(normalize-extends (:extends %)))
|
||||
distinct)
|
||||
used-template-ids (reachable-template-ids templates roots)
|
||||
unused-template-issues (->> (keys templates)
|
||||
(remove used-template-ids)
|
||||
sort
|
||||
(map (fn [template-id]
|
||||
{:type :unused-template
|
||||
:template template-id})))
|
||||
issues (vec (concat invalid-extends-issues
|
||||
duplicate-id-issues
|
||||
unused-template-issues))]
|
||||
(when (seq issues)
|
||||
(throw (ex-info "cli-e2e manifest lint failed"
|
||||
{:issues issues})))))
|
||||
|
||||
(defn- as-seq
|
||||
[value]
|
||||
(cond
|
||||
(nil? value) []
|
||||
(sequential? value) value
|
||||
:else [value]))
|
||||
|
||||
(defn- deep-merge-maps
|
||||
[left right]
|
||||
(merge-with (fn [left-val right-val]
|
||||
(if (and (map? left-val)
|
||||
(map? right-val))
|
||||
(deep-merge-maps left-val right-val)
|
||||
right-val))
|
||||
(or left {})
|
||||
(or right {})))
|
||||
|
||||
(defn- merge-entry
|
||||
[parent child]
|
||||
(let [all-keys (set (concat (keys parent) (keys child)))]
|
||||
(reduce (fn [acc key]
|
||||
(let [parent-val (get parent key)
|
||||
child-val (get child key)]
|
||||
(assoc acc
|
||||
key
|
||||
(cond
|
||||
(contains? append-merge-keys key)
|
||||
(if (contains? child key)
|
||||
(vec (concat (as-seq parent-val)
|
||||
(as-seq child-val)))
|
||||
(vec (as-seq parent-val)))
|
||||
|
||||
(contains? deep-merge-keys key)
|
||||
(if (contains? child key)
|
||||
(deep-merge-maps parent-val child-val)
|
||||
parent-val)
|
||||
|
||||
(contains? child key)
|
||||
child-val
|
||||
|
||||
:else
|
||||
parent-val))))
|
||||
{}
|
||||
all-keys)))
|
||||
|
||||
(defn- resolve-template
|
||||
[templates template-id stack]
|
||||
(when (some #{template-id} stack)
|
||||
(throw (ex-info "Circular template inheritance detected in cli-e2e manifest"
|
||||
{:template template-id
|
||||
:cycle (conj (vec stack) template-id)})))
|
||||
(let [template (get templates template-id)]
|
||||
(when-not template
|
||||
(throw (ex-info "Unknown template in cli-e2e manifest"
|
||||
{:template template-id
|
||||
:known-templates (sort (keys templates))})))
|
||||
(let [parent-ids (normalize-extends (:extends template))
|
||||
parent-values (map #(resolve-template templates % (conj stack template-id))
|
||||
parent-ids)]
|
||||
(reduce merge-entry
|
||||
{}
|
||||
(concat parent-values
|
||||
[(dissoc template :extends)])))))
|
||||
|
||||
(defn- expand-manifest-cases
|
||||
[{:keys [templates cases]}]
|
||||
(mapv (fn [case]
|
||||
(let [parent-ids (normalize-extends (:extends case))
|
||||
parent-values (map #(resolve-template templates % [])
|
||||
parent-ids)]
|
||||
(reduce merge-entry
|
||||
{}
|
||||
(concat parent-values
|
||||
[(dissoc case :extends)]))))
|
||||
cases))
|
||||
|
||||
(defn load-cases
|
||||
([]
|
||||
(load-cases nil))
|
||||
([suite]
|
||||
(let [manifest-data (-> (manifest-file suite :cases)
|
||||
paths/spec-path
|
||||
read-edn-file
|
||||
parse-manifest)]
|
||||
(lint-manifest! manifest-data)
|
||||
(expand-manifest-cases manifest-data))))
|
||||
48
cli-e2e/src/logseq/cli/e2e/paths.clj
Normal file
48
cli-e2e/src/logseq/cli/e2e/paths.clj
Normal file
@@ -0,0 +1,48 @@
|
||||
(ns logseq.cli.e2e.paths
|
||||
(:require [babashka.fs :as fs]))
|
||||
|
||||
(defn- ancestors
|
||||
[path]
|
||||
(take-while some? (iterate fs/parent path)))
|
||||
|
||||
(defn- find-parent-named
|
||||
[path name]
|
||||
(some (fn [candidate]
|
||||
(when (= name (fs/file-name candidate))
|
||||
candidate))
|
||||
(ancestors (fs/canonicalize path))))
|
||||
|
||||
(def ^:private cli-e2e-root-path
|
||||
(or (some-> *file*
|
||||
(find-parent-named "cli-e2e")
|
||||
str)
|
||||
(throw (ex-info "Unable to locate cli-e2e root"
|
||||
{:file *file*}))))
|
||||
|
||||
(defn cli-e2e-root
|
||||
[]
|
||||
cli-e2e-root-path)
|
||||
|
||||
(defn repo-root
|
||||
[]
|
||||
(str (fs/parent cli-e2e-root-path)))
|
||||
|
||||
(defn repo-path
|
||||
[& segments]
|
||||
(str (apply fs/path (repo-root) segments)))
|
||||
|
||||
(defn cli-e2e-path
|
||||
[& segments]
|
||||
(str (apply fs/path (cli-e2e-root) segments)))
|
||||
|
||||
(defn spec-path
|
||||
[filename]
|
||||
(cli-e2e-path "spec" filename))
|
||||
|
||||
(defn required-artifacts
|
||||
[]
|
||||
[(repo-path "static" "logseq-cli.js")
|
||||
(repo-path "static" "db-worker-node.js")
|
||||
(repo-path "dist" "db-worker-node.js")
|
||||
(repo-path "dist" "db-worker-node-assets.json")
|
||||
(repo-path "deps" "db-sync" "worker" "dist" "node-adapter.js")])
|
||||
37
cli-e2e/src/logseq/cli/e2e/preflight.clj
Normal file
37
cli-e2e/src/logseq/cli/e2e/preflight.clj
Normal file
@@ -0,0 +1,37 @@
|
||||
(ns logseq.cli.e2e.preflight
|
||||
(:require [babashka.fs :as fs]
|
||||
[logseq.cli.e2e.paths :as paths]
|
||||
[logseq.cli.e2e.shell :as shell]))
|
||||
|
||||
(def build-plan
|
||||
[{:cmd "clojure -M:cljs compile logseq-cli"}
|
||||
{:cmd "pnpm db-worker-node:compile:bundle"}
|
||||
{:cmd "pnpm --dir deps/db-sync build:node-adapter"}])
|
||||
|
||||
(defn missing-artifacts
|
||||
([]
|
||||
(missing-artifacts (paths/required-artifacts) fs/exists?))
|
||||
([artifacts file-exists?]
|
||||
(->> artifacts
|
||||
(remove file-exists?)
|
||||
vec)))
|
||||
|
||||
(defn run!
|
||||
[{:keys [skip-build run-command file-exists?]
|
||||
:or {run-command shell/run!
|
||||
file-exists? fs/exists?}}]
|
||||
(if skip-build
|
||||
{:status :skipped
|
||||
:commands []
|
||||
:missing-artifacts []}
|
||||
(let [artifacts (paths/required-artifacts)]
|
||||
(doseq [{:keys [cmd]} build-plan]
|
||||
(run-command {:cmd cmd
|
||||
:dir (paths/repo-root)}))
|
||||
(let [missing-after-build (missing-artifacts artifacts file-exists?)]
|
||||
(when (seq missing-after-build)
|
||||
(throw (ex-info "Build preflight completed but required artifacts are missing"
|
||||
{:missing-artifacts missing-after-build})))
|
||||
{:status :ok
|
||||
:commands build-plan
|
||||
:missing-artifacts []}))))
|
||||
18
cli-e2e/src/logseq/cli/e2e/report.clj
Normal file
18
cli-e2e/src/logseq/cli/e2e/report.clj
Normal file
@@ -0,0 +1,18 @@
|
||||
(ns logseq.cli.e2e.report
|
||||
(:require [clojure.string :as string]))
|
||||
|
||||
(defn format-missing-coverage
|
||||
[{:keys [missing-commands missing-options]}]
|
||||
(str
|
||||
"Missing coverage\n"
|
||||
(when (seq missing-commands)
|
||||
(str "Commands:\n"
|
||||
(string/join "\n" (map #(str "- " %) missing-commands))
|
||||
"\n"))
|
||||
(string/join
|
||||
"\n"
|
||||
(keep (fn [[scope options]]
|
||||
(when (seq options)
|
||||
(str (name scope) " options:\n"
|
||||
(string/join "\n" (map #(str "- " %) options)))))
|
||||
missing-options))))
|
||||
280
cli-e2e/src/logseq/cli/e2e/runner.clj
Normal file
280
cli-e2e/src/logseq/cli/e2e/runner.clj
Normal file
@@ -0,0 +1,280 @@
|
||||
(ns logseq.cli.e2e.runner
|
||||
(:require [babashka.fs :as fs]
|
||||
[cheshire.core :as json]
|
||||
[clojure.edn :as edn]
|
||||
[clojure.string :as string]
|
||||
[logseq.cli.e2e.paths :as paths]
|
||||
[logseq.cli.e2e.shell :as shell]))
|
||||
|
||||
(defn shell-escape
|
||||
[value]
|
||||
(let [text (str value)]
|
||||
(if (re-find #"[^\w@%+=:,./-]" text)
|
||||
(str "'" (string/replace text #"'" "'\"'\"'") "'")
|
||||
text)))
|
||||
|
||||
(def template-pattern #"\{\{([^}]+)\}\}")
|
||||
(def ^:private e2e-env {"CLI_E2E_TEST" "1"
|
||||
"NO_COLOR" "1"})
|
||||
|
||||
(defn- render-string
|
||||
[template context]
|
||||
(string/replace template
|
||||
template-pattern
|
||||
(fn [[_ key]]
|
||||
(let [lookup-key (keyword (string/trim key))]
|
||||
(when-not (contains? context lookup-key)
|
||||
(throw (ex-info "Missing case template value"
|
||||
{:template template
|
||||
:key lookup-key
|
||||
:context context})))
|
||||
(str (get context lookup-key))))))
|
||||
|
||||
(defn- render-value
|
||||
[value context]
|
||||
(cond
|
||||
(string? value) (render-string value context)
|
||||
(map? value) (into {}
|
||||
(map (fn [[k v]]
|
||||
[k (render-value v context)]))
|
||||
value)
|
||||
(vector? value) (mapv #(render-value % context) value)
|
||||
(seq? value) (doall (map #(render-value % context) value))
|
||||
:else value))
|
||||
|
||||
(defn render-case
|
||||
[case context]
|
||||
(render-value case context))
|
||||
|
||||
(defn- ensure-contains!
|
||||
[id cmd output snippets stream]
|
||||
(doseq [snippet (or snippets [])]
|
||||
(let [normalize-pathish-text (fn [text]
|
||||
(-> (str text)
|
||||
(string/replace "\\\\" "\\")
|
||||
(string/replace "\\" "/")
|
||||
(string/replace "\r" "")))
|
||||
matches? (or (string/includes? output snippet)
|
||||
(string/includes? (normalize-pathish-text output)
|
||||
(normalize-pathish-text snippet)))]
|
||||
(when-not matches?
|
||||
(throw (ex-info (str stream " did not contain expected text")
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:snippet snippet
|
||||
:output output
|
||||
:stream stream}))))))
|
||||
|
||||
(defn- ensure-not-contains!
|
||||
[id cmd output snippets stream]
|
||||
(doseq [snippet (or snippets [])]
|
||||
(when (string/includes? output snippet)
|
||||
(throw (ex-info (str stream " contained forbidden text")
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:snippet snippet
|
||||
:output output
|
||||
:stream stream})))))
|
||||
|
||||
(defn- ensure-equals!
|
||||
[id cmd actual expected label]
|
||||
(when (and (some? expected)
|
||||
(not= expected actual))
|
||||
(throw (ex-info (str label " did not match")
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:expected expected
|
||||
:actual actual}))))
|
||||
|
||||
(defn- ensure-json-paths!
|
||||
[id cmd output expected-paths]
|
||||
(when expected-paths
|
||||
(let [payload (json/parse-string output true)]
|
||||
(doseq [[path expected] expected-paths]
|
||||
(let [actual (get-in payload path)]
|
||||
(when-not (= expected actual)
|
||||
(throw (ex-info "stdout JSON path did not match"
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:path path
|
||||
:expected expected
|
||||
:actual actual
|
||||
:payload payload}))))))))
|
||||
|
||||
(defn- ensure-edn-paths!
|
||||
[id cmd output expected-paths]
|
||||
(when expected-paths
|
||||
(let [payload (edn/read-string output)]
|
||||
(doseq [[path expected] expected-paths]
|
||||
(let [actual (get-in payload path)]
|
||||
(when-not (= expected actual)
|
||||
(throw (ex-info "stdout EDN path did not match"
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:path path
|
||||
:expected expected
|
||||
:actual actual
|
||||
:payload payload}))))))))
|
||||
|
||||
(defn assert-result!
|
||||
[{:keys [id expect]} {:keys [cmd exit out err] :as result}]
|
||||
(when-let [expected-exit (:exit expect)]
|
||||
(when-not (= expected-exit exit)
|
||||
(throw (ex-info "Unexpected exit code"
|
||||
{:id id
|
||||
:cmd cmd
|
||||
:expected expected-exit
|
||||
:actual exit
|
||||
:result result}))))
|
||||
(ensure-equals! id cmd out (:stdout-equals expect) "stdout")
|
||||
(ensure-equals! id cmd err (:stderr-equals expect) "stderr")
|
||||
(ensure-contains! id cmd out (:stdout-contains expect) "stdout")
|
||||
(ensure-contains! id cmd err (:stderr-contains expect) "stderr")
|
||||
(ensure-not-contains! id cmd out (:stdout-not-contains expect) "stdout")
|
||||
(ensure-not-contains! id cmd err (:stderr-not-contains expect) "stderr")
|
||||
(ensure-json-paths! id cmd out (:stdout-json-paths expect))
|
||||
(ensure-edn-paths! id cmd out (:stdout-edn-paths expect))
|
||||
nil)
|
||||
|
||||
(defn- default-context
|
||||
[case context]
|
||||
(let [tmp-dir (or (:tmp-dir context)
|
||||
(str (fs/create-temp-dir {:prefix (str "logseq-cli-e2e-" (:id case) "-")})))
|
||||
root-dir (or (:root-dir context)
|
||||
tmp-dir)
|
||||
config-path (or (:config-path context)
|
||||
(str (fs/path root-dir "cli.edn")))
|
||||
export-path (or (:export-path context)
|
||||
(str (fs/path tmp-dir "graph-export.edn")))
|
||||
graph (or (:graph context)
|
||||
(:graph case)
|
||||
(str "cli-e2e-" (:id case)))]
|
||||
(fs/create-dirs (fs/path root-dir "graphs"))
|
||||
(spit config-path "{:output-format :json}\n")
|
||||
{:tmp-dir tmp-dir
|
||||
:tmp-dir-arg (shell-escape tmp-dir)
|
||||
:root-dir root-dir
|
||||
:root-dir-arg (shell-escape root-dir)
|
||||
:config-path config-path
|
||||
:config-path-arg (shell-escape config-path)
|
||||
:export-path export-path
|
||||
:export-path-arg (shell-escape export-path)
|
||||
:repo-root (paths/repo-root)
|
||||
:repo-root-arg (shell-escape (paths/repo-root))
|
||||
:cli (str "node " (shell-escape (paths/repo-path "static" "logseq-cli.js")))
|
||||
:graph graph
|
||||
:graph-arg (shell-escape graph)}))
|
||||
|
||||
(defn- case-context
|
||||
[case {:keys [context]}]
|
||||
(merge (default-context case context)
|
||||
context
|
||||
(:vars case)))
|
||||
|
||||
(defn- run-command!
|
||||
[command context {:keys [run-command stdin allow-failure phase step-index step-total case-id]}]
|
||||
(run-command {:cmd (render-string command context)
|
||||
:dir (paths/repo-root)
|
||||
:env e2e-env
|
||||
:stdin (some-> stdin (render-string context))
|
||||
:phase phase
|
||||
:step-index step-index
|
||||
:step-total step-total
|
||||
:case-id case-id
|
||||
:throw? (not allow-failure)}))
|
||||
|
||||
(defn- elapsed-ms
|
||||
[started-at]
|
||||
(long (/ (- (System/nanoTime) started-at) 1000000)))
|
||||
|
||||
(defn- run-step!
|
||||
[timings command context {:keys [timings? phase step-index step-total]
|
||||
:as run-opts}]
|
||||
(if-not timings?
|
||||
(run-command! command context run-opts)
|
||||
(let [started-at (System/nanoTime)]
|
||||
(try
|
||||
(let [result (run-command! command context run-opts)]
|
||||
(swap! timings conj {:phase phase
|
||||
:step-index step-index
|
||||
:step-total step-total
|
||||
:cmd (:cmd result)
|
||||
:elapsed-ms (elapsed-ms started-at)
|
||||
:status :ok})
|
||||
result)
|
||||
(catch Exception error
|
||||
(swap! timings conj {:phase phase
|
||||
:step-index step-index
|
||||
:step-total step-total
|
||||
:cmd (or (:cmd (ex-data error))
|
||||
(render-string command context))
|
||||
:elapsed-ms (elapsed-ms started-at)
|
||||
:status :failed})
|
||||
(throw error))))))
|
||||
|
||||
(defn run-case!
|
||||
[case {:keys [run-command detailed-log? timings?]
|
||||
:or {run-command shell/run!}
|
||||
:as opts}]
|
||||
(let [context (case-context case opts)
|
||||
rendered (render-case case context)
|
||||
case-id (:id rendered)
|
||||
timings? (boolean timings?)
|
||||
timings (atom [])
|
||||
cleanup-commands (vec (:cleanup rendered))
|
||||
setup-commands (vec (:setup rendered))
|
||||
main-commands (vec (:cmds rendered))
|
||||
cleanup! (fn []
|
||||
(doseq [[idx command] (map-indexed vector cleanup-commands)]
|
||||
(try
|
||||
(run-step! timings command context {:run-command run-command
|
||||
:timings? timings?
|
||||
:allow-failure true
|
||||
:phase (when (or detailed-log? timings?) :cleanup)
|
||||
:step-index (inc idx)
|
||||
:step-total (count cleanup-commands)
|
||||
:case-id case-id})
|
||||
(catch Exception _
|
||||
nil))))]
|
||||
(try
|
||||
(doseq [[idx command] (map-indexed vector setup-commands)]
|
||||
(run-step! timings command context {:run-command run-command
|
||||
:timings? timings?
|
||||
:phase (when (or detailed-log? timings?) :setup)
|
||||
:step-index (inc idx)
|
||||
:step-total (count setup-commands)
|
||||
:case-id case-id}))
|
||||
(let [main-total (count main-commands)
|
||||
_ (when (zero? main-total)
|
||||
(throw (ex-info "Missing case commands"
|
||||
{:id case-id
|
||||
:case rendered})))
|
||||
result (reduce (fn [_ [idx command]]
|
||||
(let [last-step? (= idx (dec main-total))]
|
||||
(run-step! timings command context {:run-command run-command
|
||||
:timings? timings?
|
||||
:stdin (when last-step? (:stdin rendered))
|
||||
:allow-failure last-step?
|
||||
:phase (when (or detailed-log? timings?) :main)
|
||||
:step-index (inc idx)
|
||||
:step-total main-total
|
||||
:case-id case-id})))
|
||||
nil
|
||||
(map-indexed vector main-commands))]
|
||||
(assert-result! rendered result)
|
||||
(cleanup!)
|
||||
(cond-> {:id case-id
|
||||
:status :ok
|
||||
:cmd (:cmd result)
|
||||
:result result
|
||||
:context context}
|
||||
timings? (assoc :timings @timings)))
|
||||
(catch Exception error
|
||||
(cleanup!)
|
||||
(if timings?
|
||||
(throw (ex-info (.getMessage error)
|
||||
(assoc (or (ex-data error) {})
|
||||
:timings @timings
|
||||
:case-id case-id)
|
||||
error))
|
||||
(throw error))))))
|
||||
67
cli-e2e/src/logseq/cli/e2e/shell.clj
Normal file
67
cli-e2e/src/logseq/cli/e2e/shell.clj
Normal file
@@ -0,0 +1,67 @@
|
||||
(ns logseq.cli.e2e.shell
|
||||
(:require [babashka.process :as process])
|
||||
(:import [java.io ByteArrayOutputStream OutputStream]))
|
||||
|
||||
(defn- utf8-string
|
||||
[^ByteArrayOutputStream payload]
|
||||
(.toString payload "UTF-8"))
|
||||
|
||||
(defn- capture-stream
|
||||
[target stream-output?]
|
||||
(let [payload (ByteArrayOutputStream.)]
|
||||
{:payload payload
|
||||
:stream (if stream-output?
|
||||
(proxy [OutputStream] []
|
||||
(write
|
||||
([byte]
|
||||
(.write payload byte)
|
||||
(.write target byte)
|
||||
(.flush target))
|
||||
([bytes offset length]
|
||||
(.write payload bytes offset length)
|
||||
(.write target bytes offset length)
|
||||
(.flush target)))
|
||||
(flush []
|
||||
(.flush target))
|
||||
(close []
|
||||
(.flush target)))
|
||||
payload)}))
|
||||
|
||||
(defn- default-executor
|
||||
[cmd {:keys [dir extra-env in stream-output?]}]
|
||||
(let [{out-payload :payload
|
||||
out-stream :stream} (capture-stream System/out stream-output?)
|
||||
{err-payload :payload
|
||||
err-stream :stream} (capture-stream System/err stream-output?)
|
||||
result @(process/process {:continue true
|
||||
:dir dir
|
||||
:extra-env extra-env
|
||||
:in in
|
||||
:out out-stream
|
||||
:err err-stream}
|
||||
"bash"
|
||||
"-c"
|
||||
cmd)]
|
||||
{:exit (:exit result 0)
|
||||
:out (utf8-string out-payload)
|
||||
:err (utf8-string err-payload)}))
|
||||
|
||||
(defn run!
|
||||
[{:keys [cmd dir env stdin executor throw? stream-output?]
|
||||
:or {throw? true}}]
|
||||
(let [runner (or executor default-executor)
|
||||
result (runner cmd {:dir dir
|
||||
:extra-env env
|
||||
:in stdin
|
||||
:stream-output? stream-output?})
|
||||
exit (:exit result 0)
|
||||
payload {:cmd cmd
|
||||
:dir dir
|
||||
:exit exit
|
||||
:out (:out result "")
|
||||
:err (:err result "")}]
|
||||
(when (and throw?
|
||||
(not (zero? exit)))
|
||||
(throw (ex-info "Shell command failed"
|
||||
payload)))
|
||||
payload))
|
||||
132
cli-e2e/src/logseq/cli/e2e/sync_fixture.clj
Normal file
132
cli-e2e/src/logseq/cli/e2e/sync_fixture.clj
Normal file
@@ -0,0 +1,132 @@
|
||||
(ns logseq.cli.e2e.sync-fixture
|
||||
(:require [babashka.fs :as fs]
|
||||
[clojure.string :as string]
|
||||
[logseq.cli.e2e.paths :as paths]
|
||||
[logseq.cli.e2e.runner :as runner]
|
||||
[logseq.cli.e2e.shell :as shell]))
|
||||
|
||||
(def default-sync-port "18080")
|
||||
(def default-e2ee-password "11111")
|
||||
|
||||
(def ^:private heavy-setup-patterns
|
||||
[#"^mkdir -p '\{\{tmp-dir\}\}/home/logseq'$"
|
||||
#"^cp ~/logseq/auth\.json\b"
|
||||
#"prepare_sync_config\.py"
|
||||
#"db_sync_server\.py'? start"])
|
||||
|
||||
(def ^:private heavy-cleanup-patterns
|
||||
[#"db_sync_server\.py'? stop"])
|
||||
|
||||
(defn- shell-quote
|
||||
[value]
|
||||
(runner/shell-escape value))
|
||||
|
||||
(defn- heavy-command?
|
||||
[command patterns]
|
||||
(boolean (some #(re-find % command) patterns)))
|
||||
|
||||
(defn- case-local-setup-prefix
|
||||
[]
|
||||
["mkdir -p '{{tmp-dir}}/home/logseq'"
|
||||
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"])
|
||||
|
||||
(def ^:private case-local-resource-markers
|
||||
["{{cli-home}}"
|
||||
"{{config-path}}"
|
||||
"{{config-path-arg}}"
|
||||
"{{tmp-dir}}/cli-b.edn"
|
||||
"{{auth-path}}"
|
||||
"{{home-dir}}"])
|
||||
|
||||
(defn- requires-case-local-resources?
|
||||
[command]
|
||||
(boolean (some #(string/includes? command %) case-local-resource-markers)))
|
||||
|
||||
(defn- suite-user-keys-bootstrap-command
|
||||
[suite-tmp-dir]
|
||||
(let [lock-dir (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.lock")))
|
||||
done-file (shell-quote (str (fs/path suite-tmp-dir "user-rsa-keys.ready")))]
|
||||
(format (str "LOCK_DIR=%s; DONE_FILE=%s; "
|
||||
"if [ -f \"$DONE_FILE\" ]; then exit 0; fi; "
|
||||
"while ! mkdir \"$LOCK_DIR\" 2>/dev/null; do [ -f \"$DONE_FILE\" ] && exit 0; sleep 0.1; done; "
|
||||
"trap 'rmdir \"$LOCK_DIR\" 2>/dev/null || true' EXIT; "
|
||||
"if [ ! -f \"$DONE_FILE\" ]; then "
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}} --upload-keys >/dev/null && touch \"$DONE_FILE\"; "
|
||||
"fi")
|
||||
lock-dir
|
||||
done-file)))
|
||||
|
||||
(defn before-suite!
|
||||
[{:keys [run-command sync-port]
|
||||
:or {run-command shell/run!
|
||||
sync-port default-sync-port}}]
|
||||
(let [sync-port (str sync-port)
|
||||
suite-tmp-dir (str (fs/create-temp-dir {:prefix "logseq-cli-e2e-sync-suite-"}))
|
||||
db-sync-pid-file (str (fs/path suite-tmp-dir "db-sync-server.pid"))
|
||||
db-sync-log-file (str (fs/path suite-tmp-dir "db-sync-server.log"))
|
||||
db-sync-root-dir (str (fs/path suite-tmp-dir "db-sync-server-data"))
|
||||
sync-http-base (str "http://127.0.0.1:" sync-port)
|
||||
sync-ws-url (str "ws://127.0.0.1:" sync-port "/sync/%s")
|
||||
auth-path (str (fs/path (System/getProperty "user.home") "logseq" "auth.json"))
|
||||
start-db-sync-cmd (format "python3 %s start --repo-root %s --pid-file %s --log-file %s --data-dir %s --port %s --startup-timeout-s 60 --auth-path %s"
|
||||
(shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py"))
|
||||
(shell-quote (paths/repo-root))
|
||||
(shell-quote db-sync-pid-file)
|
||||
(shell-quote db-sync-log-file)
|
||||
(shell-quote db-sync-root-dir)
|
||||
sync-port
|
||||
(shell-quote auth-path))]
|
||||
(run-command {:cmd start-db-sync-cmd
|
||||
:dir (paths/repo-root)})
|
||||
{:suite-tmp-dir suite-tmp-dir
|
||||
:db-sync-pid-file db-sync-pid-file
|
||||
:db-sync-log-file db-sync-log-file
|
||||
:db-sync-root-dir db-sync-root-dir
|
||||
:sync-port sync-port
|
||||
:sync-http-base sync-http-base
|
||||
:sync-ws-url sync-ws-url}))
|
||||
|
||||
(defn prepare-case
|
||||
[case {:keys [suite-tmp-dir sync-port sync-http-base sync-ws-url e2ee-password]}]
|
||||
(let [e2ee-password (or e2ee-password default-e2ee-password)
|
||||
setup-commands (vec (:setup case))
|
||||
insertion-point? (fn [command]
|
||||
(or (heavy-command? command heavy-setup-patterns)
|
||||
(requires-case-local-resources? command)))
|
||||
leading-setup (->> setup-commands
|
||||
(take-while #(not (insertion-point? %)))
|
||||
vec)
|
||||
trailing-setup (->> setup-commands
|
||||
(drop (count leading-setup))
|
||||
(remove #(heavy-command? % heavy-setup-patterns))
|
||||
vec)
|
||||
bootstrap-setup (when suite-tmp-dir
|
||||
[(suite-user-keys-bootstrap-command suite-tmp-dir)])
|
||||
cleanup' (->> (:cleanup case)
|
||||
(remove #(heavy-command? % heavy-cleanup-patterns))
|
||||
vec)]
|
||||
(-> case
|
||||
(update :vars merge {:sync-port sync-port
|
||||
:sync-http-base sync-http-base
|
||||
:sync-ws-url sync-ws-url
|
||||
:e2ee-password e2ee-password
|
||||
:e2ee-password-arg (shell-quote e2ee-password)})
|
||||
(assoc :setup (vec (concat leading-setup
|
||||
(case-local-setup-prefix)
|
||||
bootstrap-setup
|
||||
trailing-setup)))
|
||||
(assoc :cleanup cleanup'))))
|
||||
|
||||
(defn after-suite!
|
||||
[{:keys [db-sync-pid-file]}
|
||||
{:keys [run-command]
|
||||
:or {run-command shell/run!}}]
|
||||
(when (and (string? db-sync-pid-file)
|
||||
(not (string/blank? db-sync-pid-file)))
|
||||
(run-command {:cmd (format "python3 %s stop --pid-file %s"
|
||||
(shell-quote (paths/repo-path "cli-e2e" "scripts" "db_sync_server.py"))
|
||||
(shell-quote db-sync-pid-file))
|
||||
:dir (paths/repo-root)
|
||||
:throw? false})))
|
||||
23
cli-e2e/src/logseq/cli/e2e/test_runner.clj
Normal file
23
cli-e2e/src/logseq/cli/e2e/test_runner.clj
Normal file
@@ -0,0 +1,23 @@
|
||||
(ns logseq.cli.e2e.test-runner
|
||||
(:require [clojure.test :as test]))
|
||||
|
||||
(def test-namespaces
|
||||
'[logseq.cli.e2e.coverage-test
|
||||
logseq.cli.e2e.preflight-test
|
||||
logseq.cli.e2e.paths-test
|
||||
logseq.cli.e2e.shell-test
|
||||
logseq.cli.e2e.runner-test
|
||||
logseq.cli.e2e.cleanup-test
|
||||
logseq.cli.e2e.manifests-test
|
||||
logseq.cli.e2e.sync-fixture-test
|
||||
logseq.cli.e2e.main-test])
|
||||
|
||||
(defn run!
|
||||
[_opts]
|
||||
(doseq [ns-sym test-namespaces]
|
||||
(require ns-sym))
|
||||
(let [{:keys [fail error]} (apply test/run-tests test-namespaces)]
|
||||
(when (pos? (+ fail error))
|
||||
(throw (ex-info "cli-e2e unit tests failed"
|
||||
{:fail fail
|
||||
:error error})))))
|
||||
149
cli-e2e/test/logseq/cli/e2e/cleanup_test.clj
Normal file
149
cli-e2e/test/logseq/cli/e2e/cleanup_test.clj
Normal file
@@ -0,0 +1,149 @@
|
||||
(ns logseq.cli.e2e.cleanup-test
|
||||
(:require [babashka.fs :as fs]
|
||||
[clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.cleanup :as cleanup]))
|
||||
|
||||
(deftest list-cli-e2e-db-worker-pids-filters-processes
|
||||
(let [shell-fn (fn [& _]
|
||||
{:exit 0
|
||||
:out (str " 101 node /repo/dist/db-worker-node.js --repo graph-a --root-dir /tmp/logseq-cli-e2e-graph-a-123/graphs --owner-source cli\n"
|
||||
" 202 node /repo/dist/db-worker-node.js --repo production --root-dir /tmp/production-graphs --owner-source cli\n"
|
||||
" 303 node /repo/static/logseq-cli.js graph list\n"
|
||||
" 404 /usr/bin/python3 background.py\n"
|
||||
" 505 node /repo/static/db-worker-node.js --repo graph-b --root-dir /private/tmp/logseq-cli-e2e-graph-b-999/graphs --owner-source cli\n")
|
||||
:err ""})]
|
||||
(is (= [101 505]
|
||||
(cleanup/list-cli-e2e-db-worker-pids {:shell-fn shell-fn}))))
|
||||
|
||||
(testing "throws when ps invocation fails"
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Unable to scan processes"
|
||||
(cleanup/list-cli-e2e-db-worker-pids {:shell-fn (fn [& _]
|
||||
{:exit 1
|
||||
:out ""
|
||||
:err "permission denied"})})))))
|
||||
|
||||
(deftest cleanup-db-worker-processes-separates-killed-and-failed
|
||||
(let [killed (atom [])
|
||||
result (cleanup/cleanup-db-worker-processes! {:list-pids-fn (fn [] [11 22 33])
|
||||
:kill-pid-fn (fn [pid]
|
||||
(swap! killed conj pid)
|
||||
(if (= pid 22)
|
||||
:failed
|
||||
:killed))})]
|
||||
(is (= [11 22 33] (:found-pids result)))
|
||||
(is (= [11 33] (:killed-pids result)))
|
||||
(is (= [22] (:failed-pids result)))
|
||||
(is (= [11 22 33] @killed))))
|
||||
|
||||
(deftest cleanup-db-worker-processes-dry-run-does-not-kill
|
||||
(let [killed (atom [])
|
||||
result (cleanup/cleanup-db-worker-processes! {:dry-run true
|
||||
:list-pids-fn (fn [] [11 22])
|
||||
:kill-pid-fn (fn [pid]
|
||||
(swap! killed conj pid)
|
||||
:killed)})]
|
||||
(is (= [11 22] (:found-pids result)))
|
||||
(is (true? (:dry-run? result)))
|
||||
(is (= [11 22] (:would-kill-pids result)))
|
||||
(is (= [] (:killed-pids result)))
|
||||
(is (= [] (:failed-pids result)))
|
||||
(is (= [] @killed))))
|
||||
|
||||
(deftest list-cli-e2e-db-sync-port-pids-filters-port-18080
|
||||
(let [shell-fn (fn [& _]
|
||||
{:exit 0
|
||||
:out (str "COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\n"
|
||||
"node 111 me 13u IPv6 0x0 0t0 TCP *:18080 (LISTEN)\n"
|
||||
"python 222 me 13u IPv4 0x0 0t0 TCP 127.0.0.1:18080 (LISTEN)\n"
|
||||
"node 333 me 13u IPv6 0x0 0t0 TCP *:18081 (LISTEN)\n")
|
||||
:err ""})]
|
||||
(is (= [111 222]
|
||||
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn shell-fn}))))
|
||||
|
||||
(testing "returns empty when no listener exists"
|
||||
(is (= []
|
||||
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _]
|
||||
{:exit 1
|
||||
:out ""
|
||||
:err ""})}))))
|
||||
|
||||
(testing "throws when lsof invocation fails"
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Unable to scan db-sync server port listeners"
|
||||
(cleanup/list-cli-e2e-db-sync-port-pids {:shell-fn (fn [& _]
|
||||
{:exit 1
|
||||
:out ""
|
||||
:err "permission denied"})})))))
|
||||
|
||||
(deftest cleanup-db-sync-port-processes-separates-killed-and-failed
|
||||
(let [killed (atom [])
|
||||
result (cleanup/cleanup-db-sync-port-processes! {:list-pids-fn (fn [] [44 55])
|
||||
:kill-pid-fn (fn [pid]
|
||||
(swap! killed conj pid)
|
||||
(if (= pid 55)
|
||||
:failed
|
||||
:killed))})]
|
||||
(is (= [44 55] (:found-pids result)))
|
||||
(is (= [44] (:killed-pids result)))
|
||||
(is (= [55] (:failed-pids result)))
|
||||
(is (= [44 55] @killed))))
|
||||
|
||||
(deftest cleanup-db-sync-port-processes-dry-run-does-not-kill
|
||||
(let [killed (atom [])
|
||||
result (cleanup/cleanup-db-sync-port-processes! {:dry-run true
|
||||
:list-pids-fn (fn [] [44])
|
||||
:kill-pid-fn (fn [pid]
|
||||
(swap! killed conj pid)
|
||||
:killed)})]
|
||||
(is (= [44] (:found-pids result)))
|
||||
(is (true? (:dry-run? result)))
|
||||
(is (= [44] (:would-kill-pids result)))
|
||||
(is (= [] (:killed-pids result)))
|
||||
(is (= [] (:failed-pids result)))
|
||||
(is (= [] @killed))))
|
||||
|
||||
(deftest cleanup-temp-roots-removes-only-cli-e2e-temp-roots
|
||||
(let [tmp-root (fs/create-temp-dir {:prefix "cleanup-e2e-test-"})
|
||||
matching-root (fs/path tmp-root "logseq-cli-e2e-case-a-123")
|
||||
another-matching-root (fs/path tmp-root "logseq-cli-e2e-case-b-456")
|
||||
non-matching-root (fs/path tmp-root "other-case")]
|
||||
(try
|
||||
(fs/create-dirs (fs/path matching-root "graphs"))
|
||||
(fs/create-dirs (fs/path matching-root "home" "logseq"))
|
||||
(fs/create-dirs (fs/path another-matching-root "graphs"))
|
||||
(fs/create-dirs (fs/path non-matching-root "graphs"))
|
||||
|
||||
(let [result (cleanup/cleanup-temp-roots! {:tmp-root (str tmp-root)})
|
||||
expected-dirs (sort [(str matching-root)
|
||||
(str another-matching-root)])]
|
||||
(is (= expected-dirs
|
||||
(sort (:found-dirs result))))
|
||||
(is (= expected-dirs
|
||||
(sort (:removed-dirs result))))
|
||||
(is (empty? (:failed-dirs result)))
|
||||
(is (false? (fs/exists? matching-root)))
|
||||
(is (false? (fs/exists? another-matching-root)))
|
||||
(is (true? (fs/exists? non-matching-root))))
|
||||
(finally
|
||||
(fs/delete-tree tmp-root)))))
|
||||
|
||||
(deftest cleanup-temp-roots-dry-run-does-not-delete
|
||||
(let [deleted (atom [])
|
||||
result (cleanup/cleanup-temp-roots! {:dry-run true
|
||||
:list-dirs-fn (fn [] ["/tmp/logseq-cli-e2e-a"
|
||||
"/tmp/logseq-cli-e2e-b"])
|
||||
:delete-dir-fn (fn [dir]
|
||||
(swap! deleted conj dir))})]
|
||||
(is (= ["/tmp/logseq-cli-e2e-a"
|
||||
"/tmp/logseq-cli-e2e-b"]
|
||||
(:found-dirs result)))
|
||||
(is (true? (:dry-run? result)))
|
||||
(is (= ["/tmp/logseq-cli-e2e-a"
|
||||
"/tmp/logseq-cli-e2e-b"]
|
||||
(:would-remove-dirs result)))
|
||||
(is (= [] (:removed-dirs result)))
|
||||
(is (= [] (:failed-dirs result)))
|
||||
(is (= [] @deleted))))
|
||||
97
cli-e2e/test/logseq/cli/e2e/compare_graph_queries_test.py
Normal file
97
cli-e2e/test/logseq/cli/e2e/compare_graph_queries_test.py
Normal file
@@ -0,0 +1,97 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "compare_graph_queries.py"
|
||||
spec = importlib.util.spec_from_file_location("compare_graph_queries", MODULE_PATH)
|
||||
compare_graph_queries = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(compare_graph_queries)
|
||||
|
||||
|
||||
def test_parse_args_supports_repeated_queries(monkeypatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
compare_graph_queries.sys,
|
||||
"argv",
|
||||
[
|
||||
"compare_graph_queries.py",
|
||||
"--cli",
|
||||
"/tmp/logseq-cli.js",
|
||||
"--graph",
|
||||
"demo",
|
||||
"--config-a",
|
||||
"/tmp/a.edn",
|
||||
"--root-dir-a",
|
||||
"/tmp/a",
|
||||
"--config-b",
|
||||
"/tmp/b.edn",
|
||||
"--root-dir-b",
|
||||
"/tmp/b",
|
||||
"--query",
|
||||
"[:find ?x]",
|
||||
"--query",
|
||||
"[:find ?y]",
|
||||
],
|
||||
)
|
||||
|
||||
args = compare_graph_queries.parse_args()
|
||||
|
||||
assert args.root_dir_a == "/tmp/a"
|
||||
assert args.root_dir_b == "/tmp/b"
|
||||
assert args.query == ["[:find ?x]", "[:find ?y]"]
|
||||
|
||||
|
||||
def test_main_batches_multiple_queries_in_one_process(tmp_path: Path) -> None:
|
||||
left_queries = []
|
||||
right_queries = []
|
||||
printed = []
|
||||
cli_path = tmp_path / "logseq-cli.js"
|
||||
cli_path.write_text("// mock cli\n")
|
||||
|
||||
def fake_run_query(cli_path, config_path, root_dir, graph, query):
|
||||
record = {
|
||||
"cli_path": str(cli_path),
|
||||
"config_path": str(config_path),
|
||||
"root_dir": str(root_dir),
|
||||
"graph": graph,
|
||||
"query": query,
|
||||
}
|
||||
if str(config_path).endswith("a.edn"):
|
||||
left_queries.append(record)
|
||||
else:
|
||||
right_queries.append(record)
|
||||
return {
|
||||
"payload": {"status": "ok"},
|
||||
"result": [{"title": query}],
|
||||
}
|
||||
|
||||
compare_graph_queries.run_query = fake_run_query
|
||||
compare_graph_queries.parse_args = lambda: SimpleNamespace(
|
||||
cli=str(cli_path),
|
||||
graph="demo",
|
||||
query=["[:find ?x]", "[:find ?y]"],
|
||||
config_a="/tmp/a.edn",
|
||||
root_dir_a="/tmp/a",
|
||||
config_b="/tmp/b.edn",
|
||||
root_dir_b="/tmp/b",
|
||||
require_result=False,
|
||||
)
|
||||
compare_graph_queries.print = lambda value, **kwargs: printed.append(json.loads(value))
|
||||
|
||||
compare_graph_queries.main()
|
||||
|
||||
assert [item["query"] for item in left_queries] == ["[:find ?x]", "[:find ?y]"]
|
||||
assert [item["query"] for item in right_queries] == ["[:find ?x]", "[:find ?y]"]
|
||||
assert printed == [
|
||||
{
|
||||
"status": "ok",
|
||||
"results": {
|
||||
"[:find ?x]": [{"title": "[:find ?x]"}],
|
||||
"[:find ?y]": [{"title": "[:find ?y]"}],
|
||||
},
|
||||
}
|
||||
]
|
||||
58
cli-e2e/test/logseq/cli/e2e/coverage_test.clj
Normal file
58
cli-e2e/test/logseq/cli/e2e/coverage_test.clj
Normal file
@@ -0,0 +1,58 @@
|
||||
(ns logseq.cli.e2e.coverage-test
|
||||
(:require [clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.coverage :as coverage]
|
||||
[logseq.cli.e2e.manifests :as manifests]))
|
||||
|
||||
(def sample-inventory
|
||||
{:excluded-command-prefixes ["sync" "login" "logout"]
|
||||
:scopes {:global {:options ["--help" "--version"]}
|
||||
:graph {:commands ["graph list" "graph create"]
|
||||
:options ["--file" "--type"]}}})
|
||||
|
||||
(deftest reports-missing-commands-and-options
|
||||
(let [cases [{:id "global-help"
|
||||
:covers {:options {:global ["--help"]}}}
|
||||
{:id "graph-list"
|
||||
:covers {:commands ["graph list"]
|
||||
:options {:graph ["--file"]}}}]
|
||||
report (coverage/coverage-report sample-inventory cases)]
|
||||
(is (= ["graph create"] (:missing-commands report)))
|
||||
(is (= ["--version"] (get-in report [:missing-options :global])))
|
||||
(is (= ["--type"] (get-in report [:missing-options :graph])))))
|
||||
|
||||
(deftest rejects-excluded-commands-in-inventory
|
||||
(let [inventory (assoc-in sample-inventory [:scopes :bad :commands] ["login"])]
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Excluded commands"
|
||||
(coverage/validate-inventory! inventory)))))
|
||||
|
||||
(deftest rejects-excluded-commands-in-case-covers
|
||||
(let [cases [{:id "bad-login"
|
||||
:covers {:commands ["logout"]}}]]
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Excluded commands"
|
||||
(coverage/validate-cases! sample-inventory cases)))))
|
||||
|
||||
(deftest complete-coverage-is-recognized
|
||||
(let [cases [{:id "global"
|
||||
:covers {:options {:global ["--help" "--version"]}}}
|
||||
{:id "graph-create"
|
||||
:covers {:commands ["graph create"]
|
||||
:options {:graph ["--type"]}}}
|
||||
{:id "graph-list"
|
||||
:covers {:commands ["graph list"]
|
||||
:options {:graph ["--file"]}}}]
|
||||
report (coverage/coverage-report sample-inventory cases)]
|
||||
(is (coverage/complete? report))))
|
||||
|
||||
(deftest sync-suite-manifests-cover-mvp-commands
|
||||
(let [inventory (manifests/load-inventory :sync)
|
||||
cases (manifests/load-cases :sync)
|
||||
covered-commands (set (mapcat #(get-in % [:covers :commands]) cases))
|
||||
report (coverage/coverage-report inventory cases)]
|
||||
(is (contains? covered-commands "sync upload"))
|
||||
(is (contains? covered-commands "sync download"))
|
||||
(is (contains? covered-commands "sync status"))
|
||||
(is (coverage/complete? report))))
|
||||
812
cli-e2e/test/logseq/cli/e2e/main_test.clj
Normal file
812
cli-e2e/test/logseq/cli/e2e/main_test.clj
Normal file
@@ -0,0 +1,812 @@
|
||||
(ns logseq.cli.e2e.main-test
|
||||
(:require [babashka.cli :as cli]
|
||||
[clojure.string :as string]
|
||||
[clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.cleanup :as cleanup]
|
||||
[logseq.cli.e2e.main :as main]
|
||||
[logseq.cli.e2e.manifests :as manifests]
|
||||
[logseq.cli.e2e.sync-fixture :as sync-fixture]))
|
||||
|
||||
(def sample-cases
|
||||
[{:id "global-help"
|
||||
:cmds ["node static/logseq-cli.js --help"]
|
||||
:covers {:options {:global ["--help"]}}
|
||||
:tags [:global :smoke]}
|
||||
{:id "graph-create"
|
||||
:cmds ["node static/logseq-cli.js graph create --graph demo"]
|
||||
:covers {:commands ["graph create"]
|
||||
:options {:graph ["--type"]}}
|
||||
:tags [:graph]}
|
||||
{:id "graph-list"
|
||||
:cmds ["node static/logseq-cli.js graph list"]
|
||||
:covers {:commands ["graph list"]
|
||||
:options {:graph ["--file"]}}
|
||||
:tags [:graph :smoke]}])
|
||||
|
||||
(def complete-inventory
|
||||
{:excluded-command-prefixes ["sync" "login" "logout"]
|
||||
:scopes {:global {:options ["--help"]}
|
||||
:graph {:commands ["graph create" "graph list"]
|
||||
:options ["--type" "--file"]}}})
|
||||
|
||||
(def cli-parse-config
|
||||
{:alias {:i :include
|
||||
:h :help}
|
||||
:spec {:jobs {:default 4}}
|
||||
:coerce {:include []
|
||||
:help :boolean
|
||||
:dry-run :boolean
|
||||
:skip-build :boolean
|
||||
:verbose :boolean
|
||||
:timings :boolean
|
||||
:jobs :long}})
|
||||
|
||||
(deftest cli-opts-parses-jobs-as-integer
|
||||
(is (= 3
|
||||
(:jobs (cli/parse-opts ["--jobs" "3"] cli-parse-config)))))
|
||||
|
||||
(deftest cli-opts-defaults-jobs-to-four
|
||||
(is (= 4
|
||||
(:jobs (cli/parse-opts [] cli-parse-config)))))
|
||||
|
||||
(deftest cli-opts-rejects-non-integer-jobs
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Coerce failure"
|
||||
(cli/parse-opts ["--jobs" "nope"] cli-parse-config))))
|
||||
|
||||
(deftest run-rejects-jobs-less-than-one
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"--jobs must be a positive integer"
|
||||
(main/run! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:skip-build true
|
||||
:jobs 0
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})}))))
|
||||
|
||||
(deftest run-non-sync-uses-parallel-runner-when-jobs-greater-than-one
|
||||
(let [parallel-call (atom nil)
|
||||
serial-called? (atom false)]
|
||||
(with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
|
||||
(reset! parallel-call {:case-ids (mapv :id selected-cases)
|
||||
:run-case run-case
|
||||
:run-command run-command
|
||||
:jobs (:jobs opts)})
|
||||
[{:id "global-help" :status :ok}
|
||||
{:id "graph-list" :status :ok}])
|
||||
main/run-selected-cases! (fn [& _]
|
||||
(reset! serial-called? true)
|
||||
(throw (ex-info "serial runner should not be used" {})))]
|
||||
(let [result (main/run! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:jobs 2
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["global-help" "graph-list"] (:case-ids @parallel-call)))
|
||||
(is (= 2 (:jobs @parallel-call)))
|
||||
(is (false? @serial-called?))))))
|
||||
|
||||
(deftest run-sync-suite-uses-parallel-runner-when-jobs-greater-than-one
|
||||
(let [parallel-call (atom nil)
|
||||
serial-called? (atom false)
|
||||
sync-inventory {:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes {:sync {:commands ["sync upload" "sync status"]
|
||||
:options []}}}
|
||||
sync-cases [{:id "sync-upload"
|
||||
:cmds ["node static/logseq-cli.js sync upload"]
|
||||
:covers {:commands ["sync upload"]}}
|
||||
{:id "sync-status"
|
||||
:cmds ["node static/logseq-cli.js sync status"]
|
||||
:covers {:commands ["sync status"]}}]]
|
||||
(with-redefs [main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
|
||||
(reset! parallel-call {:case-ids (mapv :id selected-cases)
|
||||
:run-case run-case
|
||||
:run-command run-command
|
||||
:jobs (:jobs opts)})
|
||||
(mapv (fn [case]
|
||||
{:id (:id case)
|
||||
:status :ok})
|
||||
selected-cases))
|
||||
main/run-selected-cases! (fn [& _]
|
||||
(reset! serial-called? true)
|
||||
(throw (ex-info "serial runner should not be used" {})))]
|
||||
(let [result (main/run! {:suite :sync
|
||||
:inventory sync-inventory
|
||||
:cases sync-cases
|
||||
:skip-build true
|
||||
:jobs 4
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call)))
|
||||
(is (= 4 (:jobs @parallel-call)))
|
||||
(is (false? @serial-called?))))))
|
||||
|
||||
(deftest run-jobs-one-keeps-serial-runner
|
||||
(let [serial-call (atom nil)
|
||||
parallel-called? (atom false)]
|
||||
(with-redefs [main/run-selected-cases! (fn [selected-cases run-case run-command opts]
|
||||
(reset! serial-call {:case-ids (mapv :id selected-cases)
|
||||
:run-case run-case
|
||||
:run-command run-command
|
||||
:jobs (:jobs opts)})
|
||||
(mapv (fn [case]
|
||||
{:id (:id case)
|
||||
:status :ok})
|
||||
selected-cases))
|
||||
main/run-selected-cases-in-parallel! (fn [& _]
|
||||
(reset! parallel-called? true)
|
||||
(throw (ex-info "parallel runner should not be used" {})))]
|
||||
(let [result (main/run! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:jobs 1
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["global-help" "graph-list"] (:case-ids @serial-call)))
|
||||
(is (= 1 (:jobs @serial-call)))
|
||||
(is (false? @parallel-called?))))))
|
||||
|
||||
(deftest parallel-runner-collects-completions-before-rethrowing-failure
|
||||
(let [started (atom [])
|
||||
finished (atom [])
|
||||
started-latch (java.util.concurrent.CountDownLatch. 2)
|
||||
release-success (promise)
|
||||
cases [{:id "global-help"}
|
||||
{:id "graph-list"}]
|
||||
error (try
|
||||
(main/run-selected-cases-in-parallel!
|
||||
cases
|
||||
(fn [case _opts]
|
||||
(swap! started conj (:id case))
|
||||
(.countDown started-latch)
|
||||
(.await started-latch)
|
||||
(if (= "graph-list" (:id case))
|
||||
(do
|
||||
(swap! finished conj [:failed (:id case)])
|
||||
(deliver release-success true)
|
||||
(throw (ex-info "boom" {:id (:id case)})))
|
||||
(do
|
||||
@release-success
|
||||
(swap! finished conj [:ok (:id case)])
|
||||
{:id (:id case)
|
||||
:status :ok})))
|
||||
(fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
{:jobs 2})
|
||||
nil
|
||||
(catch Exception ex
|
||||
ex))]
|
||||
(is (instance? Exception error))
|
||||
(is (= #{"global-help" "graph-list"}
|
||||
(set @started)))
|
||||
(is (= #{[:ok "global-help"]
|
||||
[:failed "graph-list"]}
|
||||
(set @finished)))))
|
||||
|
||||
(deftest parallel-runner-elapsed-ms-starts-when-case-begins-running
|
||||
(let [events (atom [])]
|
||||
(main/run-selected-cases-in-parallel!
|
||||
[{:id "slow-1"}
|
||||
{:id "slow-2"}
|
||||
{:id "fast-after-queue"}]
|
||||
(fn [test-case _opts]
|
||||
(clojure.core/case (:id test-case)
|
||||
"slow-1" (Thread/sleep 180)
|
||||
"slow-2" (Thread/sleep 180)
|
||||
"fast-after-queue" (Thread/sleep 10))
|
||||
{:id (:id test-case)
|
||||
:status :ok})
|
||||
(fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
{:jobs 2
|
||||
:on-case-success (fn [payload]
|
||||
(swap! events conj [(:id (:result payload)) (:elapsed-ms payload)]))})
|
||||
(let [elapsed-map (into {} @events)]
|
||||
(is (< (get elapsed-map "fast-after-queue" 1000) 120)
|
||||
"queued case should measure only its own execution time, not time spent waiting in the pool")
|
||||
(is (>= (get elapsed-map "slow-1" 0) 150))
|
||||
(is (>= (get elapsed-map "slow-2" 0) 150)))))
|
||||
|
||||
(deftest select-cases-supports-case-id
|
||||
(is (= ["graph-create"]
|
||||
(mapv :id (main/select-cases sample-cases {:case "graph-create"})))))
|
||||
|
||||
(deftest select-cases-supports-include-tags
|
||||
(is (= ["global-help" "graph-list"]
|
||||
(mapv :id (main/select-cases sample-cases {:include ["smoke"]})))))
|
||||
|
||||
(deftest run-fails-on-missing-coverage-before-case-execution
|
||||
(let [ran? (atom false)]
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"Missing coverage"
|
||||
(main/run! {:inventory complete-inventory
|
||||
:cases [{:id "global-help"
|
||||
:cmds ["node static/logseq-cli.js --help"]
|
||||
:covers {:options {:global ["--help"]}}}]
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
(reset! ran? true)
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})})))
|
||||
(is (false? @ran?))))
|
||||
|
||||
(deftest run-succeeds-when-coverage-is-complete
|
||||
(let [result (main/run! {:inventory complete-inventory
|
||||
:cases [{:id "global-help"
|
||||
:cmds ["node static/logseq-cli.js --help"]
|
||||
:covers {:options {:global ["--help"]}}}
|
||||
{:id "graph-create"
|
||||
:cmds ["node static/logseq-cli.js graph create --type markdown"]
|
||||
:covers {:commands ["graph create"]
|
||||
:options {:graph ["--type"]}}}
|
||||
{:id "graph-list"
|
||||
:cmds ["node static/logseq-cli.js graph list --file demo.edn"]
|
||||
:covers {:commands ["graph list"]
|
||||
:options {:graph ["--file"]}}}]
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= 3 (count (:cases result))))))
|
||||
|
||||
(deftest targeted-run-skips-full-coverage-check
|
||||
(let [executed (atom [])]
|
||||
(let [result (main/run! {:inventory complete-inventory
|
||||
:cases [{:id "global-help"
|
||||
:cmds ["node static/logseq-cli.js --help"]
|
||||
:covers {:options {:global ["--help"]}}}
|
||||
{:id "graph-create"
|
||||
:cmds ["node static/logseq-cli.js graph create --graph demo"]
|
||||
:covers {:commands ["graph create"]
|
||||
:options {:graph ["--type"]}}}]
|
||||
:case "global-help"
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
(swap! executed conj (:id case))
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["global-help"] @executed)))))
|
||||
|
||||
(deftest run-invokes-case-progress-hooks
|
||||
(let [events (atom [])]
|
||||
(main/run! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})
|
||||
:on-case-start (fn [{:keys [index total case]}]
|
||||
(swap! events conj [:start index total (:id case)]))
|
||||
:on-case-success (fn [{:keys [index total result]}]
|
||||
(swap! events conj [:ok index total (:id result)]))})
|
||||
(is (= [[:start 1 2 "global-help"]
|
||||
[:ok 1 2 "global-help"]
|
||||
[:start 2 2 "graph-list"]
|
||||
[:ok 2 2 "graph-list"]]
|
||||
@events))))
|
||||
|
||||
(deftest run-loads-non-sync-suite-by-default
|
||||
(let [suite-calls (atom [])]
|
||||
(with-redefs [manifests/load-inventory (fn [suite]
|
||||
(swap! suite-calls conj [:inventory suite])
|
||||
complete-inventory)
|
||||
manifests/load-cases (fn [suite]
|
||||
(swap! suite-calls conj [:cases suite])
|
||||
sample-cases)]
|
||||
(let [result (main/run! {:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result))))
|
||||
(is (= [[:inventory :non-sync]
|
||||
[:cases :non-sync]]
|
||||
@suite-calls)))))
|
||||
|
||||
(deftest run-loads-sync-suite-when-explicit
|
||||
(let [suite-calls (atom [])
|
||||
sync-inventory {:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes {:sync {:commands ["sync upload"]
|
||||
:options []}}}
|
||||
sync-cases [{:id "sync-upload"
|
||||
:cmds ["node static/logseq-cli.js sync upload"]
|
||||
:covers {:commands ["sync upload"]}}]]
|
||||
(with-redefs [manifests/load-inventory (fn [suite]
|
||||
(swap! suite-calls conj [:inventory suite])
|
||||
sync-inventory)
|
||||
manifests/load-cases (fn [suite]
|
||||
(swap! suite-calls conj [:cases suite])
|
||||
sync-cases)]
|
||||
(let [result (main/run! {:suite :sync
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result))))
|
||||
(is (= [[:inventory :sync]
|
||||
[:cases :sync]]
|
||||
@suite-calls)))))
|
||||
|
||||
(deftest run-sync-suite-uses-suite-fixture-once-with-parallel-runner
|
||||
(let [before-called (atom 0)
|
||||
after-called (atom 0)
|
||||
prepared-case-ids (atom [])
|
||||
run-case-seen (atom [])
|
||||
parallel-call (atom nil)
|
||||
sync-inventory {:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes {:sync {:commands ["sync upload" "sync status"]
|
||||
:options []}}}
|
||||
sync-cases [{:id "sync-upload"
|
||||
:cmds ["node static/logseq-cli.js sync upload"]
|
||||
:covers {:commands ["sync upload"]}}
|
||||
{:id "sync-status"
|
||||
:cmds ["node static/logseq-cli.js sync status"]
|
||||
:covers {:commands ["sync status"]}}]]
|
||||
(with-redefs [sync-fixture/before-suite! (fn [_]
|
||||
(swap! before-called inc)
|
||||
{:suite :sync})
|
||||
sync-fixture/prepare-case (fn [case _suite-context]
|
||||
(swap! prepared-case-ids conj (:id case))
|
||||
(assoc case :prepared? true))
|
||||
sync-fixture/after-suite! (fn [_ _]
|
||||
(swap! after-called inc))
|
||||
main/run-selected-cases-in-parallel! (fn [selected-cases run-case run-command opts]
|
||||
(reset! parallel-call {:case-ids (mapv :id selected-cases)
|
||||
:jobs (:jobs opts)})
|
||||
(mapv (fn [case]
|
||||
(run-case case {:run-command run-command}))
|
||||
selected-cases))]
|
||||
(let [result (main/run! {:suite :sync
|
||||
:inventory sync-inventory
|
||||
:cases sync-cases
|
||||
:skip-build true
|
||||
:jobs 2
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
(swap! run-case-seen conj [(:id case) (:prepared? case)])
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))))
|
||||
(is (= 1 @before-called))
|
||||
(is (= 1 @after-called))
|
||||
(is (= ["sync-upload" "sync-status"] (:case-ids @parallel-call)))
|
||||
(is (= 2 (:jobs @parallel-call)))
|
||||
(is (= ["sync-upload" "sync-status"] @prepared-case-ids))
|
||||
(is (= [["sync-upload" true]
|
||||
["sync-status" true]]
|
||||
@run-case-seen))))
|
||||
|
||||
(deftest run-sync-suite-forwards-e2ee-password-to-prepare-case
|
||||
(let [seen-passwords (atom [])
|
||||
sync-inventory {:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes {:sync {:commands ["sync status"]
|
||||
:options []}}}
|
||||
sync-cases [{:id "sync-status"
|
||||
:cmds ["node static/logseq-cli.js sync status"]
|
||||
:covers {:commands ["sync status"]}}]]
|
||||
(with-redefs [sync-fixture/before-suite! (fn [_]
|
||||
{:suite :sync})
|
||||
sync-fixture/prepare-case (fn [case suite-context]
|
||||
(swap! seen-passwords conj (:e2ee-password suite-context))
|
||||
case)
|
||||
sync-fixture/after-suite! (fn [_ _] nil)]
|
||||
(main/run! {:suite :sync
|
||||
:inventory sync-inventory
|
||||
:cases sync-cases
|
||||
:skip-build true
|
||||
:e2ee-password "abc 123"
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})}))
|
||||
(is (= ["abc 123"] @seen-passwords))))
|
||||
|
||||
(deftest list-cases-defaults-to-non-sync
|
||||
(let [selected-suite (atom nil)
|
||||
output (with-out-str
|
||||
(with-redefs [manifests/load-cases (fn [suite]
|
||||
(reset! selected-suite suite)
|
||||
[{:id "non-sync-case"}])]
|
||||
(main/list-cases! {})))]
|
||||
(is (= :non-sync @selected-suite))
|
||||
(is (string/includes? output "non-sync-case"))))
|
||||
|
||||
(deftest list-sync-cases-uses-sync-suite
|
||||
(let [selected-suite (atom nil)
|
||||
output (with-out-str
|
||||
(with-redefs [manifests/load-cases (fn [suite]
|
||||
(reset! selected-suite suite)
|
||||
[{:id "sync-case"}])]
|
||||
(main/list-sync-cases! {})))]
|
||||
(is (= :sync @selected-suite))
|
||||
(is (string/includes? output "sync-case"))))
|
||||
|
||||
(deftest test-prints-progress-and-summary
|
||||
(let [output (with-out-str
|
||||
(main/test! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok
|
||||
:cmd (last (:cmds case))})}))]
|
||||
(is (string/includes? output "==> Running cli-e2e cases"))
|
||||
(is (string/includes? output "==> Build preflight: running..."))
|
||||
(is (string/includes? output "==> Build preflight: skipped (--skip-build)"))
|
||||
(is (string/includes? output "==> Prepared 2 case(s), starting execution"))
|
||||
(is (string/includes? output "[1/2] ▶ global-help"))
|
||||
(is (string/includes? output "[1/2] ✓ global-help"))
|
||||
(is (string/includes? output "[2/2] ▶ graph-list"))
|
||||
(is (string/includes? output "[2/2] ✓ graph-list"))
|
||||
(is (string/includes? output "Summary: 2 passed, 0 failed"))))
|
||||
|
||||
(deftest test-parallel-output-omits-meaningless-index-prefixes
|
||||
(let [output (with-out-str
|
||||
(main/test! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:jobs 2
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
(Thread/sleep (if (= "global-help" (:id case)) 25 5))
|
||||
{:id (:id case)
|
||||
:status :ok})}))]
|
||||
(is (string/includes? output "▶ global-help"))
|
||||
(is (string/includes? output "✓ global-help"))
|
||||
(is (string/includes? output "✓ graph-list"))
|
||||
(is (not (string/includes? output "[1/2]")))
|
||||
(is (not (string/includes? output "[2/2]")))))
|
||||
|
||||
(deftest test-timings-prints-step-details-and-slow-summary
|
||||
(let [output (with-out-str
|
||||
(main/test! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:timings true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
(if (= "global-help" (:id case))
|
||||
{:id "global-help"
|
||||
:status :ok
|
||||
:timings [{:phase :setup
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 12
|
||||
:status :ok
|
||||
:cmd "setup-global"}
|
||||
{:phase :main
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 55
|
||||
:status :ok
|
||||
:cmd "main-global"}]}
|
||||
{:id "graph-list"
|
||||
:status :ok
|
||||
:timings [{:phase :main
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 210
|
||||
:status :ok
|
||||
:cmd "main-graph-list"}]}))}))]
|
||||
(is (string/includes? output "==> Step timing enabled (--timings)"))
|
||||
(is (string/includes? output "step timings:"))
|
||||
(is (string/includes? output "Slow steps (top 10):"))
|
||||
(is (string/includes? output "main-graph-list"))))
|
||||
|
||||
(deftest test-without-timings-keeps-output-concise
|
||||
(let [output (with-out-str
|
||||
(main/test! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok
|
||||
:timings [{:phase :main
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 88
|
||||
:status :ok
|
||||
:cmd "hidden-step"}]})}))]
|
||||
(is (not (string/includes? output "==> Step timing enabled (--timings)")))
|
||||
(is (not (string/includes? output "step timings:")))
|
||||
(is (not (string/includes? output "Slow steps (top 10):")))))
|
||||
|
||||
(deftest test-sync-timings-prints-step-details-and-slow-summary
|
||||
(let [sync-inventory {:excluded-command-prefixes ["login" "logout"]
|
||||
:scopes {:sync {:commands ["sync status"]
|
||||
:options []}}}
|
||||
sync-cases [{:id "sync-status-case"
|
||||
:cmds ["node static/logseq-cli.js sync status"]
|
||||
:covers {:commands ["sync status"]}}]
|
||||
output (with-out-str
|
||||
(main/test-sync! {:inventory sync-inventory
|
||||
:cases sync-cases
|
||||
:skip-build true
|
||||
:timings true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok
|
||||
:timings [{:phase :setup
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 10
|
||||
:status :ok
|
||||
:cmd "sync-setup"}
|
||||
{:phase :main
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 150
|
||||
:status :ok
|
||||
:cmd "sync-main"}
|
||||
{:phase :cleanup
|
||||
:step-index 1
|
||||
:step-total 1
|
||||
:elapsed-ms 20
|
||||
:status :ok
|
||||
:cmd "sync-cleanup"}]})}))]
|
||||
(is (string/includes? output "==> Running cli-e2e cases"))
|
||||
(is (string/includes? output "==> Step timing enabled (--timings)"))
|
||||
(is (string/includes? output "step timings:"))
|
||||
(is (string/includes? output "Slow steps (top 10):"))
|
||||
(is (string/includes? output "sync-main"))))
|
||||
|
||||
(deftest test-help-prints-usage-and-skips-execution
|
||||
(let [ran? (atom false)
|
||||
result (atom nil)
|
||||
output (with-out-str
|
||||
(reset! result
|
||||
(main/test! {:help true
|
||||
:run-command (fn [_]
|
||||
(reset! ran? true)
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [_ _]
|
||||
(reset! ran? true)
|
||||
{:id "unexpected"
|
||||
:status :ok})}))) ]
|
||||
(is (= :help (:status @result)))
|
||||
(is (false? @ran?))
|
||||
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test [options]"))
|
||||
(is (string/includes? output "--skip-build"))
|
||||
(is (not (string/includes? output "--force-build")))
|
||||
(is (string/includes? output "--include TAG"))
|
||||
(is (string/includes? output "--case ID"))
|
||||
(is (string/includes? output "--jobs N"))
|
||||
(is (string/includes? output "Run up to N non-sync cases in parallel"))
|
||||
(is (string/includes? output "--skip-build --jobs 4"))
|
||||
(is (string/includes? output "Default: 4"))
|
||||
(is (string/includes? output "--timings"))
|
||||
(is (not (string/includes? output "--e2ee-password")))))
|
||||
|
||||
(deftest test-sync-help-prints-usage-and-skips-execution
|
||||
(let [ran? (atom false)
|
||||
result (atom nil)
|
||||
output (with-out-str
|
||||
(reset! result
|
||||
(main/test-sync! {:help true
|
||||
:run-command (fn [_]
|
||||
(reset! ran? true)
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [_ _]
|
||||
(reset! ran? true)
|
||||
{:id "unexpected"
|
||||
:status :ok})})))]
|
||||
(is (= :help (:status @result)))
|
||||
(is (false? @ran?))
|
||||
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn test-sync [options]"))
|
||||
(is (string/includes? output "--skip-build"))
|
||||
(is (not (string/includes? output "--force-build")))
|
||||
(is (string/includes? output "--include TAG"))
|
||||
(is (string/includes? output "--case ID"))
|
||||
(is (string/includes? output "--jobs N"))
|
||||
(is (string/includes? output "Run up to N sync cases in parallel"))
|
||||
(is (string/includes? output "--jobs 4"))
|
||||
(is (string/includes? output "bb -f cli-e2e/bb.edn test-sync"))
|
||||
(is (string/includes? output "--case sync-upload-download-mvp"))
|
||||
(is (not (string/includes? output "--skip-build --case sync-upload-download-mvp")))
|
||||
(is (string/includes? output "Default: 4"))
|
||||
(is (string/includes? output "--timings"))
|
||||
(is (string/includes? output "--e2ee-password VALUE"))
|
||||
(is (string/includes? output "Default: 11111"))))
|
||||
|
||||
(deftest run-does-not-pass-force-build-to-preflight
|
||||
(let [preflight-call (atom nil)]
|
||||
(with-redefs [logseq.cli.e2e.preflight/run! (fn [opts]
|
||||
(reset! preflight-call opts)
|
||||
{:status :ok
|
||||
:commands []
|
||||
:missing-artifacts []})]
|
||||
(let [result (main/run! {:inventory complete-inventory
|
||||
:cases sample-cases
|
||||
:include ["smoke"]
|
||||
:skip-build false
|
||||
:force-build true
|
||||
:run-command (fn [_]
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:run-case (fn [case _opts]
|
||||
{:id (:id case)
|
||||
:status :ok})})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (not (contains? @preflight-call :force-build)))))))
|
||||
|
||||
(deftest test-single-case-enables-detailed-command-logging
|
||||
(let [command-opts (atom nil)
|
||||
output (with-out-str
|
||||
(main/test! {:inventory complete-inventory
|
||||
:cases [{:id "global-help"
|
||||
:cmds ["node static/logseq-cli.js --help"]
|
||||
:covers {:options {:global ["--help"]}}}]
|
||||
:case "global-help"
|
||||
:skip-build true
|
||||
:run-command (fn [{:keys [cmd] :as opts}]
|
||||
(reset! command-opts opts)
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out "ok"
|
||||
:err ""})
|
||||
:run-case (fn [case {:keys [run-command]}]
|
||||
(let [result (run-command {:cmd (first (:cmds case))
|
||||
:phase :main
|
||||
:step-index 1
|
||||
:step-total 1})]
|
||||
{:id (:id case)
|
||||
:status :ok
|
||||
:cmd (:cmd result)}) )}))]
|
||||
(is (string/includes? output "==> Detailed case logging enabled (--case global-help)"))
|
||||
(is (string/includes? output " [main 1/1] $ node static/logseq-cli.js --help"))
|
||||
(is (true? (:stream-output? @command-opts)))))
|
||||
|
||||
(deftest cleanup-help-prints-usage
|
||||
(let [output (with-out-str (main/cleanup! {:help true}))]
|
||||
(is (string/includes? output "Usage: bb -f cli-e2e/bb.edn cleanup"))
|
||||
(is (string/includes? output "Terminate cli-e2e db-worker-node processes"))
|
||||
(is (string/includes? output "Terminate db-sync server listeners on port 18080"))
|
||||
(is (string/includes? output "Remove cli-e2e temp roots"))
|
||||
(is (string/includes? output "--dry-run"))))
|
||||
|
||||
(deftest cleanup-prints-summary-and-returns-status
|
||||
(with-redefs [cleanup/cleanup-db-worker-processes! (fn [_] {:found-pids [101 202]
|
||||
:killed-pids [101]
|
||||
:failed-pids [202]})
|
||||
cleanup/cleanup-db-sync-port-processes! (fn [_] {:found-pids [303]
|
||||
:killed-pids [303]
|
||||
:failed-pids []})
|
||||
cleanup/cleanup-temp-roots! (fn [_] {:found-dirs ["/tmp/logseq-cli-e2e-a"
|
||||
"/tmp/logseq-cli-e2e-b"]
|
||||
:removed-dirs ["/tmp/logseq-cli-e2e-a"]
|
||||
:failed-dirs ["/tmp/logseq-cli-e2e-b"]})]
|
||||
(let [result (atom nil)
|
||||
output (with-out-str
|
||||
(reset! result (main/cleanup! {})))]
|
||||
(is (= :ok (:status @result)))
|
||||
(is (= [101] (get-in @result [:processes :killed-pids])))
|
||||
(is (= [303] (get-in @result [:db-sync-port-processes :killed-pids])))
|
||||
(is (= ["/tmp/logseq-cli-e2e-a"] (get-in @result [:temp-roots :removed-dirs])))
|
||||
(is (string/includes? output "db-worker-node processes: found 2, killed 1, failed 1"))
|
||||
(is (string/includes? output "db-sync server processes (port 18080): found 1, killed 1, failed 0"))
|
||||
(is (string/includes? output "temp roots: found 2, removed 1, failed 1")))))
|
||||
|
||||
(deftest cleanup-dry-run-prints-summary-and-passes-option
|
||||
(let [process-opts (atom nil)
|
||||
db-sync-opts (atom nil)
|
||||
dir-opts (atom nil)]
|
||||
(with-redefs [cleanup/cleanup-db-worker-processes! (fn [opts]
|
||||
(reset! process-opts opts)
|
||||
{:dry-run? true
|
||||
:found-pids [101 202]
|
||||
:would-kill-pids [101 202]
|
||||
:killed-pids []
|
||||
:failed-pids []})
|
||||
cleanup/cleanup-db-sync-port-processes! (fn [opts]
|
||||
(reset! db-sync-opts opts)
|
||||
{:dry-run? true
|
||||
:found-pids [303]
|
||||
:would-kill-pids [303]
|
||||
:killed-pids []
|
||||
:failed-pids []})
|
||||
cleanup/cleanup-temp-roots! (fn [opts]
|
||||
(reset! dir-opts opts)
|
||||
{:dry-run? true
|
||||
:found-dirs ["/tmp/logseq-cli-e2e-a"]
|
||||
:would-remove-dirs ["/tmp/logseq-cli-e2e-a"]
|
||||
:removed-dirs []
|
||||
:failed-dirs []})]
|
||||
(let [result (atom nil)
|
||||
output (with-out-str
|
||||
(reset! result (main/cleanup! {:dry-run true})))]
|
||||
(is (= {:dry-run true} @process-opts))
|
||||
(is (= {:dry-run true} @db-sync-opts))
|
||||
(is (= {:dry-run true} @dir-opts))
|
||||
(is (= :ok (:status @result)))
|
||||
(is (true? (get-in @result [:processes :dry-run?])))
|
||||
(is (true? (get-in @result [:db-sync-port-processes :dry-run?])))
|
||||
(is (true? (get-in @result [:temp-roots :dry-run?])))
|
||||
(is (string/includes? output "[dry-run] db-worker-node processes: found 2, would kill 2"))
|
||||
(is (string/includes? output "[dry-run] db-sync server processes (port 18080): found 1, would kill 1"))
|
||||
(is (string/includes? output "[dry-run] temp roots: found 1, would remove 1"))))))
|
||||
225
cli-e2e/test/logseq/cli/e2e/manifests_test.clj
Normal file
225
cli-e2e/test/logseq/cli/e2e/manifests_test.clj
Normal file
@@ -0,0 +1,225 @@
|
||||
(ns logseq.cli.e2e.manifests-test
|
||||
(:require [clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.manifests :as manifests]))
|
||||
|
||||
(deftest load-cases-requires-new-map-format
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
[{:id "legacy-a"}
|
||||
{:id "legacy-b"}])]
|
||||
(let [error (try
|
||||
(manifests/load-cases :non-sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))]
|
||||
(is (some? error))
|
||||
(is (re-find #"Invalid cli-e2e manifest format" (.getMessage error)))
|
||||
(is (= "{:templates {...} :cases [...]}"
|
||||
(:expected (ex-data error)))))))
|
||||
|
||||
(deftest load-cases-merges-templates-and-cases-by-rules
|
||||
(with-redefs [manifests/read-edn-file
|
||||
(fn [_]
|
||||
{:templates
|
||||
{:base {:setup ["setup-a"]
|
||||
:cmds ["cmd-a"]
|
||||
:cleanup ["cleanup-a"]
|
||||
:tags [:base]
|
||||
:vars {:nested {:left 1}
|
||||
:only-base true}
|
||||
:covers {:commands ["base-command"]
|
||||
:options {:global ["--base"]}}
|
||||
:expect {:stdout-json-paths {[:status] "ok"
|
||||
[:data :base] 1}}
|
||||
:graph "base-graph"}
|
||||
:addon {:setup ["setup-b"]
|
||||
:cmds ["cmd-b"]
|
||||
:cleanup ["cleanup-b"]
|
||||
:tags [:addon]
|
||||
:vars {:nested {:right 2}}
|
||||
:covers {:options {:graph ["--addon"]}}
|
||||
:expect {:stdout-json-paths {[:data :addon] 2}}
|
||||
:graph "addon-graph"}
|
||||
:shared {:setup ["setup-shared"]
|
||||
:cmds ["cmd-shared"]
|
||||
:cleanup ["cleanup-shared"]
|
||||
:tags [:shared]
|
||||
:vars {:nested {:shared 9}}
|
||||
:graph "shared-graph"}}
|
||||
:cases
|
||||
[{:id "single-parent"
|
||||
:extends :base
|
||||
:cmds ["cmd-case"]
|
||||
:expect {:stdout-json-paths {[:data :case] 3}}
|
||||
:graph "single-parent-graph"}
|
||||
{:id "multi-parent"
|
||||
:extends [:base :addon]
|
||||
:setup ["setup-case"]
|
||||
:cmds ["cmd-case"]
|
||||
:cleanup ["cleanup-case"]
|
||||
:tags [:case]
|
||||
:vars {:nested {:leaf 3}
|
||||
:only-case true}
|
||||
:covers {:commands ["case-command"]
|
||||
:options {:graph ["--case"]}}
|
||||
:expect {:stdout-json-paths {[:data :case] 3}}
|
||||
:graph "case-graph"}
|
||||
{:id "negative-case"
|
||||
:extends :shared
|
||||
:expect {:exit 1
|
||||
:stdout-json-paths {[:error :code] "boom"}}
|
||||
:graph "negative-graph"}]})]
|
||||
(let [[single-parent multi-parent negative-case] (manifests/load-cases :sync)]
|
||||
(testing "supports :extends keyword"
|
||||
(is (= "single-parent" (:id single-parent)))
|
||||
(is (= ["cmd-a" "cmd-case"] (:cmds single-parent)))
|
||||
(is (= "single-parent-graph" (:graph single-parent))))
|
||||
(testing "append merge keys"
|
||||
(is (= ["setup-a" "setup-b" "setup-case"] (:setup multi-parent)))
|
||||
(is (= ["cmd-a" "cmd-b" "cmd-case"] (:cmds multi-parent)))
|
||||
(is (= ["cleanup-a" "cleanup-b" "cleanup-case"] (:cleanup multi-parent)))
|
||||
(is (= [:base :addon :case] (:tags multi-parent))))
|
||||
(testing "deep merge keys"
|
||||
(is (= {:nested {:left 1 :right 2 :leaf 3}
|
||||
:only-base true
|
||||
:only-case true}
|
||||
(:vars multi-parent)))
|
||||
(is (= {:commands ["case-command"]
|
||||
:options {:global ["--base"]
|
||||
:graph ["--case"]}}
|
||||
(:covers multi-parent)))
|
||||
(is (= {[:status] "ok"
|
||||
[:data :base] 1
|
||||
[:data :addon] 2
|
||||
[:data :case] 3}
|
||||
(get-in multi-parent [:expect :stdout-json-paths]))))
|
||||
(testing "child expect overrides inherited maps when parent omits them"
|
||||
(is (= {:exit 1
|
||||
:stdout-json-paths {[:error :code] "boom"}}
|
||||
(:expect negative-case))))
|
||||
(testing "base templates can omit happy-path defaults"
|
||||
(is (= ["setup-shared"] (:setup negative-case)))
|
||||
(is (= ["cmd-shared"] (:cmds negative-case)))
|
||||
(is (= ["cleanup-shared"] (:cleanup negative-case))))
|
||||
(testing "scalar keys are overridden by child"
|
||||
(is (= "case-graph" (:graph multi-parent)))))))
|
||||
|
||||
(deftest load-cases-detects-circular-template-inheritance-with-cycle-path
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
{:templates
|
||||
{:a {:extends :b
|
||||
:setup ["a"]}
|
||||
:b {:extends :a
|
||||
:setup ["b"]}}
|
||||
:cases [{:id "cycle" :extends :a}]})]
|
||||
(let [error (try
|
||||
(manifests/load-cases :sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))]
|
||||
(is (some? error))
|
||||
(is (re-find #"Circular template inheritance" (.getMessage error)))
|
||||
(is (= [:a :b :a] (:cycle (ex-data error)))))))
|
||||
|
||||
(deftest load-cases-validates-extends-entries
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
{:templates {:base {:cmds ["base"]}}
|
||||
:cases [{:id "invalid-extends"
|
||||
:extends [:base "bad"]
|
||||
:cmds ["case"]}]})]
|
||||
(let [error (try
|
||||
(manifests/load-cases :non-sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))]
|
||||
(is (some? error))
|
||||
(is (re-find #"Invalid :extends entries" (.getMessage error)))
|
||||
(is (= ["bad"] (:invalid-entries (ex-data error)))))))
|
||||
|
||||
(deftest load-cases-lint-detects-invalid-extends-references
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
{:templates {:base {:cmds ["base"]}
|
||||
:unused {:extends :missing-template
|
||||
:cmds ["unused"]}}
|
||||
:cases [{:id "valid" :extends :base :cmds ["case"]}
|
||||
{:id "invalid" :extends :also-missing :cmds ["case"]}]})]
|
||||
(let [error (try
|
||||
(manifests/load-cases :sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))
|
||||
issues (:issues (ex-data error))]
|
||||
(is (some? error))
|
||||
(is (re-find #"manifest lint failed" (.getMessage error)))
|
||||
(is (= #{:also-missing :missing-template}
|
||||
(set (map :target (filter #(= :invalid-extends (:type %)) issues))))))))
|
||||
|
||||
(deftest load-cases-lint-detects-duplicate-case-ids
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
{:templates {:base {:cmds ["base"]}}
|
||||
:cases [{:id "dup" :extends :base :cmds ["case-a"]}
|
||||
{:id "dup" :extends :base :cmds ["case-b"]}]})]
|
||||
(let [error (try
|
||||
(manifests/load-cases :non-sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))
|
||||
duplicate-issues (filter #(= :duplicate-case-id (:type %))
|
||||
(:issues (ex-data error)))]
|
||||
(is (some? error))
|
||||
(is (= [{:type :duplicate-case-id :id "dup" :count 2}]
|
||||
(vec duplicate-issues))))))
|
||||
|
||||
(deftest load-cases-lint-detects-unused-templates
|
||||
(with-redefs [manifests/read-edn-file (fn [_]
|
||||
{:templates {:base {:cmds ["base"]}
|
||||
:unused {:cmds ["unused"]}}
|
||||
:cases [{:id "only" :extends :base :cmds ["case"]}]})]
|
||||
(let [error (try
|
||||
(manifests/load-cases :sync)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))
|
||||
unused-issues (filter #(= :unused-template (:type %))
|
||||
(:issues (ex-data error)))]
|
||||
(is (some? error))
|
||||
(is (= [{:type :unused-template :template :unused}]
|
||||
(vec unused-issues))))))
|
||||
|
||||
(deftest sync-multi-batch-operations-uses-state-driven-waits-instead-of-fixed-sleeps
|
||||
(let [cases (manifests/load-cases :sync)
|
||||
multi-batch (some #(when (= "sync-multi-batch-operations" (:id %)) %) cases)
|
||||
commands (:cmds multi-batch)
|
||||
upload-commands (filter #(re-find #"sync upload --graph" %) commands)
|
||||
wait-commands (filter #(re-find #"wait_sync_status\.py" %) commands)]
|
||||
(is (some? multi-batch))
|
||||
(is (not-any? #(= "sleep 1" %) commands))
|
||||
(is (= 1 (count upload-commands)))
|
||||
(is (= 5 (count wait-commands)))
|
||||
(is (= 3 (count (filter #(re-find #"--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) wait-commands))))))
|
||||
|
||||
(deftest sync-manifest-includes-duplicate-upload-negative-case
|
||||
(let [cases (manifests/load-cases :sync)
|
||||
duplicate-upload (some #(when (= "sync-upload-rejects-duplicate-remote-graph" (:id %)) %) cases)
|
||||
json-paths (get-in duplicate-upload [:expect :stdout-json-paths])
|
||||
setup-commands (:setup duplicate-upload)
|
||||
main-commands (:cmds duplicate-upload)]
|
||||
(is (some? duplicate-upload))
|
||||
(is (= 1 (get-in duplicate-upload [:expect :exit])))
|
||||
(is (= "graph-already-exists"
|
||||
(get-in duplicate-upload [:expect :stdout-json-paths [:error :code]])))
|
||||
(is (= ["delete it before uploading again"]
|
||||
(get-in duplicate-upload [:expect :stdout-contains])))
|
||||
(is (= 1 (count (filter #(re-find #"sync upload --graph" %) setup-commands))))
|
||||
(is (= 1 (count (filter #(re-find #"sync upload --graph" %) main-commands))))
|
||||
(is (false? (contains? json-paths [:data :pending-local])))
|
||||
(is (false? (contains? json-paths [:data :pending-asset])))
|
||||
(is (false? (contains? json-paths [:data :pending-server])))
|
||||
(is (false? (contains? json-paths [:data :last-error])))))
|
||||
|
||||
(deftest sync-status-steady-state-does-not-repeat-identical-b-side-steady-state-waits
|
||||
(let [cases (manifests/load-cases :sync)
|
||||
steady-state (some #(when (= "sync-status-steady-state" (:id %)) %) cases)
|
||||
steady-waits (filter #(re-find #"wait_sync_status\.py.+--root-dir '\{\{tmp-dir\}\}/graphs-b'.+--timeout-s 30 --interval-s 1" %) (:cmds steady-state))]
|
||||
(is (some? steady-state))
|
||||
(is (= 1 (count steady-waits)))
|
||||
(is (= 1 (count (filter #(re-find #"sync status --graph" %) (:cmds steady-state)))))))
|
||||
9
cli-e2e/test/logseq/cli/e2e/paths_test.clj
Normal file
9
cli-e2e/test/logseq/cli/e2e/paths_test.clj
Normal file
@@ -0,0 +1,9 @@
|
||||
(ns logseq.cli.e2e.paths-test
|
||||
(:require [clojure.test :refer [deftest is]]
|
||||
[logseq.cli.e2e.paths :as paths]))
|
||||
|
||||
(deftest repo-root-does-not-depend-on-runtime-file-binding
|
||||
(let [expected (paths/repo-root)]
|
||||
(binding [*file* nil]
|
||||
(is (= expected
|
||||
(paths/repo-root))))))
|
||||
95
cli-e2e/test/logseq/cli/e2e/preflight_test.clj
Normal file
95
cli-e2e/test/logseq/cli/e2e/preflight_test.clj
Normal file
@@ -0,0 +1,95 @@
|
||||
(ns logseq.cli.e2e.preflight-test
|
||||
(:require [clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.preflight :as preflight]))
|
||||
|
||||
(def required-artifacts
|
||||
["/repo/static/logseq-cli.js"
|
||||
"/repo/static/db-worker-node.js"
|
||||
"/repo/dist/db-worker-node.js"
|
||||
"/repo/dist/db-worker-node-assets.json"
|
||||
"/repo/deps/db-sync/worker/dist/node-adapter.js"])
|
||||
|
||||
(defn- with-required-artifacts
|
||||
[f]
|
||||
(with-redefs [logseq.cli.e2e.paths/repo-root (constantly "/repo")
|
||||
logseq.cli.e2e.paths/required-artifacts (fn [] required-artifacts)]
|
||||
(f)))
|
||||
|
||||
(deftest build-plan-matches-required-commands
|
||||
(is (= ["clojure -M:cljs compile logseq-cli"
|
||||
"pnpm db-worker-node:compile:bundle"
|
||||
"pnpm --dir deps/db-sync build:node-adapter"]
|
||||
(mapv :cmd preflight/build-plan))))
|
||||
|
||||
(deftest missing-artifacts-returns-unreadable-paths
|
||||
(let [artifacts ["/repo/static/logseq-cli.js"
|
||||
"/repo/static/db-worker-node.js"
|
||||
"/repo/dist/db-worker-node.js"]
|
||||
present? #{"/repo/static/logseq-cli.js"}]
|
||||
(is (= ["/repo/static/db-worker-node.js"
|
||||
"/repo/dist/db-worker-node.js"]
|
||||
(preflight/missing-artifacts artifacts present?)))))
|
||||
|
||||
(deftest skip-build-avoids-running-shell-commands
|
||||
(let [called? (atom false)
|
||||
result (preflight/run! {:skip-build true
|
||||
:run-command (fn [_]
|
||||
(reset! called? true))
|
||||
:file-exists? (constantly false)})]
|
||||
(is (= :skipped (:status result)))
|
||||
(is (false? @called?))))
|
||||
|
||||
(deftest build-runs-commands-even-when-artifacts-are-ready
|
||||
(let [calls (atom [])]
|
||||
(with-required-artifacts
|
||||
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
|
||||
(swap! calls conj cmd)
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:file-exists? (set required-artifacts)})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["clojure -M:cljs compile logseq-cli"
|
||||
"pnpm db-worker-node:compile:bundle"
|
||||
"pnpm --dir deps/db-sync build:node-adapter"]
|
||||
@calls))))))
|
||||
|
||||
(deftest build-runs-commands-when-artifacts-are-partially-present
|
||||
(let [calls (atom [])
|
||||
existing (atom (disj (set required-artifacts)
|
||||
"/repo/dist/db-worker-node-assets.json"))]
|
||||
(with-required-artifacts
|
||||
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
|
||||
(swap! calls conj cmd)
|
||||
(reset! existing (set required-artifacts))
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:file-exists? (fn [path]
|
||||
(contains? @existing path))})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["clojure -M:cljs compile logseq-cli"
|
||||
"pnpm db-worker-node:compile:bundle"
|
||||
"pnpm --dir deps/db-sync build:node-adapter"]
|
||||
@calls))))))
|
||||
|
||||
(deftest build-runs-commands-when-artifacts-are-absent
|
||||
(let [calls (atom [])
|
||||
existing (atom #{})]
|
||||
(with-required-artifacts
|
||||
#(let [result (preflight/run! {:run-command (fn [{:keys [cmd]}]
|
||||
(swap! calls conj cmd)
|
||||
(reset! existing (set required-artifacts))
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
:file-exists? (fn [path]
|
||||
(contains? @existing path))})]
|
||||
(is (= :ok (:status result)))
|
||||
(is (= ["clojure -M:cljs compile logseq-cli"
|
||||
"pnpm db-worker-node:compile:bundle"
|
||||
"pnpm --dir deps/db-sync build:node-adapter"]
|
||||
@calls))))))
|
||||
@@ -0,0 +1,102 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "random_bidirectional_block_ops.py"
|
||||
spec = importlib.util.spec_from_file_location("random_bidirectional_block_ops", MODULE_PATH)
|
||||
random_bidirectional_block_ops = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
sys.modules[spec.name] = random_bidirectional_block_ops
|
||||
spec.loader.exec_module(random_bidirectional_block_ops)
|
||||
|
||||
|
||||
def test_default_profile_is_faster_than_high_stress(monkeypatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
random_bidirectional_block_ops.sys,
|
||||
"argv",
|
||||
[
|
||||
"random_bidirectional_block_ops.py",
|
||||
"--cli",
|
||||
"/tmp/logseq-cli.js",
|
||||
"--graph",
|
||||
"demo",
|
||||
"--config-a",
|
||||
"/tmp/a.edn",
|
||||
"--root-dir-a",
|
||||
"/tmp/a",
|
||||
"--config-b",
|
||||
"/tmp/b.edn",
|
||||
"--root-dir-b",
|
||||
"/tmp/b",
|
||||
"--page",
|
||||
"Home",
|
||||
],
|
||||
)
|
||||
default_args = random_bidirectional_block_ops.parse_args()
|
||||
|
||||
monkeypatch.setattr(
|
||||
random_bidirectional_block_ops.sys,
|
||||
"argv",
|
||||
[
|
||||
"random_bidirectional_block_ops.py",
|
||||
"--cli",
|
||||
"/tmp/logseq-cli.js",
|
||||
"--graph",
|
||||
"demo",
|
||||
"--config-a",
|
||||
"/tmp/a.edn",
|
||||
"--root-dir-a",
|
||||
"/tmp/a",
|
||||
"--config-b",
|
||||
"/tmp/b.edn",
|
||||
"--root-dir-b",
|
||||
"/tmp/b",
|
||||
"--page",
|
||||
"Home",
|
||||
"--profile",
|
||||
"high-stress",
|
||||
],
|
||||
)
|
||||
high_stress_args = random_bidirectional_block_ops.parse_args()
|
||||
|
||||
assert default_args.root_dir_a == "/tmp/a"
|
||||
assert default_args.root_dir_b == "/tmp/b"
|
||||
assert default_args.profile == "default"
|
||||
assert high_stress_args.profile == "high-stress"
|
||||
assert default_args.rounds_per_client < high_stress_args.rounds_per_client
|
||||
|
||||
|
||||
def test_explicit_rounds_override_profile_default(monkeypatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
random_bidirectional_block_ops.sys,
|
||||
"argv",
|
||||
[
|
||||
"random_bidirectional_block_ops.py",
|
||||
"--cli",
|
||||
"/tmp/logseq-cli.js",
|
||||
"--graph",
|
||||
"demo",
|
||||
"--config-a",
|
||||
"/tmp/a.edn",
|
||||
"--root-dir-a",
|
||||
"/tmp/a",
|
||||
"--config-b",
|
||||
"/tmp/b.edn",
|
||||
"--root-dir-b",
|
||||
"/tmp/b",
|
||||
"--page",
|
||||
"Home",
|
||||
"--profile",
|
||||
"high-stress",
|
||||
"--rounds-per-client",
|
||||
"12",
|
||||
],
|
||||
)
|
||||
|
||||
args = random_bidirectional_block_ops.parse_args()
|
||||
|
||||
assert args.profile == "high-stress"
|
||||
assert args.rounds_per_client == 12
|
||||
152
cli-e2e/test/logseq/cli/e2e/runner_test.clj
Normal file
152
cli-e2e/test/logseq/cli/e2e/runner_test.clj
Normal file
@@ -0,0 +1,152 @@
|
||||
(ns logseq.cli.e2e.runner-test
|
||||
(:require [clojure.test :refer [deftest is]]
|
||||
[logseq.cli.e2e.runner :as runner]))
|
||||
|
||||
(deftest render-case-expands-template-values-recursively
|
||||
(let [rendered (runner/render-case
|
||||
{:id "graph-create"
|
||||
:setup ["{{cli}} --graph {{graph-arg}}"]
|
||||
:cmds ["{{cli}} graph info --graph {{graph-arg}}"]
|
||||
:expect {:stdout-json-paths {[:data :graph] "{{graph}}"
|
||||
[:status] "ok"}}}
|
||||
{:cli "node /tmp/logseq-cli.js"
|
||||
:graph "demo"
|
||||
:graph-arg "'demo'"})]
|
||||
(is (= ["node /tmp/logseq-cli.js --graph 'demo'"] (:setup rendered)))
|
||||
(is (= ["node /tmp/logseq-cli.js graph info --graph 'demo'"] (:cmds rendered)))
|
||||
(is (= "demo" (get-in rendered [:expect :stdout-json-paths [:data :graph]])))))
|
||||
|
||||
(deftest run-case-executes-setup-before-main-command
|
||||
(let [calls (atom [])
|
||||
result (runner/run-case!
|
||||
{:id "graph-info"
|
||||
:setup ["setup one" "setup two"]
|
||||
:cmds ["main command one" "main command two"]
|
||||
:expect {:exit 0}}
|
||||
{:context {}
|
||||
:run-command (fn [{:keys [cmd]}]
|
||||
(swap! calls conj cmd)
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})})]
|
||||
(is (= ["setup one" "setup two" "main command one" "main command two"] @calls))
|
||||
(is (= "graph-info" (:id result)))
|
||||
(is (= "main command two" (get-in result [:result :cmd])))))
|
||||
|
||||
(deftest run-case-includes-command-phase-metadata-when-detailed
|
||||
(let [calls (atom [])]
|
||||
(runner/run-case!
|
||||
{:id "graph-info"
|
||||
:setup ["setup one" "setup two"]
|
||||
:cmds ["main command one" "main command two"]
|
||||
:cleanup ["cleanup one"]
|
||||
:expect {:exit 0}}
|
||||
{:context {}
|
||||
:detailed-log? true
|
||||
:run-command (fn [{:keys [cmd phase step-index step-total case-id throw?] :as opts}]
|
||||
(swap! calls conj (select-keys opts [:cmd :phase :step-index :step-total :case-id :throw?]))
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})})
|
||||
(is (= [{:cmd "setup one" :phase :setup :step-index 1 :step-total 2 :case-id "graph-info" :throw? true}
|
||||
{:cmd "setup two" :phase :setup :step-index 2 :step-total 2 :case-id "graph-info" :throw? true}
|
||||
{:cmd "main command one" :phase :main :step-index 1 :step-total 2 :case-id "graph-info" :throw? true}
|
||||
{:cmd "main command two" :phase :main :step-index 2 :step-total 2 :case-id "graph-info" :throw? false}
|
||||
{:cmd "cleanup one" :phase :cleanup :step-index 1 :step-total 1 :case-id "graph-info" :throw? false}]
|
||||
@calls))))
|
||||
|
||||
(deftest run-case-collects-step-timings-when-enabled
|
||||
(let [result (runner/run-case!
|
||||
{:id "graph-info"
|
||||
:setup ["setup one"]
|
||||
:cmds ["main command"]
|
||||
:cleanup ["cleanup one"]
|
||||
:expect {:exit 0}}
|
||||
{:context {}
|
||||
:timings? true
|
||||
:run-command (fn [{:keys [cmd]}]
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})})]
|
||||
(is (= 3 (count (:timings result))))
|
||||
(is (= [:setup :main :cleanup]
|
||||
(mapv :phase (:timings result))))))
|
||||
|
||||
(deftest run-case-attaches-timings-to-error-when-enabled
|
||||
(let [error (try
|
||||
(runner/run-case!
|
||||
{:id "graph-info"
|
||||
:setup ["setup one"]
|
||||
:cmds ["main command"]
|
||||
:cleanup ["cleanup one"]
|
||||
:expect {:exit 0}}
|
||||
{:context {}
|
||||
:timings? true
|
||||
:run-command (fn [{:keys [cmd]}]
|
||||
(when (= cmd "main command")
|
||||
(throw (ex-info "boom" {:cmd cmd})))
|
||||
{:cmd cmd
|
||||
:exit 0
|
||||
:out ""
|
||||
:err ""})})
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo error
|
||||
error))
|
||||
timings (:timings (ex-data error))]
|
||||
(is (= "graph-info" (:case-id (ex-data error))))
|
||||
(is (= [:setup :main :cleanup]
|
||||
(mapv :phase timings)))
|
||||
(is (= :failed (:status (second timings))))))
|
||||
|
||||
(deftest run-case-validates-json-paths-and-nonzero-exit
|
||||
(let [result (runner/run-case!
|
||||
{:id "invalid-shell"
|
||||
:cmds ["node static/logseq-cli.js completion fish"]
|
||||
:expect {:exit 1
|
||||
:stdout-json-paths {[:status] "error"
|
||||
[:error :code] "invalid-options"}}}
|
||||
{:context {}
|
||||
:run-command (fn [{:keys [cmd]}]
|
||||
{:cmd cmd
|
||||
:exit 1
|
||||
:out "{\"status\":\"error\",\"error\":{\"code\":\"invalid-options\"}}"
|
||||
:err ""})})]
|
||||
(is (= 1 (get-in result [:result :exit])))))
|
||||
|
||||
(deftest assert-result-validates-edn-paths
|
||||
(is (nil? (runner/assert-result!
|
||||
{:id "graph-list-edn"
|
||||
:expect {:exit 0
|
||||
:stdout-edn-paths {[:status] :ok
|
||||
[:data :graphs] ["demo"]}}}
|
||||
{:cmd "node cli.js graph list"
|
||||
:exit 0
|
||||
:out "{:status :ok, :data {:graphs [\"demo\"]}}"
|
||||
:err ""}))))
|
||||
|
||||
(deftest assert-result-validates-stdout-not-contains
|
||||
(is (thrown-with-msg?
|
||||
clojure.lang.ExceptionInfo
|
||||
#"stdout contained forbidden text"
|
||||
(runner/assert-result!
|
||||
{:id "remove-block"
|
||||
:expect {:exit 0
|
||||
:stdout-not-contains ["Alpha block"]}}
|
||||
{:cmd "node cli.js show --page Home"
|
||||
:exit 0
|
||||
:out "Home\n- Alpha block"
|
||||
:err ""}))))
|
||||
|
||||
(deftest assert-result-normalizes-json-escaped-windows-paths-for-contains
|
||||
(is (nil? (runner/assert-result!
|
||||
{:id "doctor-dev-script-json"
|
||||
:expect {:exit 0
|
||||
:stdout-contains ["static/db-worker-node.js"
|
||||
"C:\\Users\\demo\\tmp\\graph-export.edn"]}}
|
||||
{:cmd "node cli.js doctor --dev-script"
|
||||
:exit 0
|
||||
:out "{\"status\":\"ok\",\"data\":{\"path\":\"C:\\\\Users\\\\demo\\\\tmp\\\\graph-export.edn\",\"message\":\"Found readable file: C:\\\\Users\\\\demo\\\\project\\\\static\\\\db-worker-node.js\"}}"
|
||||
:err ""}))))
|
||||
54
cli-e2e/test/logseq/cli/e2e/shell_test.clj
Normal file
54
cli-e2e/test/logseq/cli/e2e/shell_test.clj
Normal file
@@ -0,0 +1,54 @@
|
||||
(ns logseq.cli.e2e.shell-test
|
||||
(:require [babashka.process :as process]
|
||||
[clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.shell :as shell]))
|
||||
|
||||
(deftest preserves-the-exact-command-string
|
||||
(let [captured (atom nil)
|
||||
command "node static/logseq-cli.js --graph 'Graph With Space' --help"
|
||||
result (shell/run! {:cmd command
|
||||
:dir "/repo"
|
||||
:executor (fn [cmd opts]
|
||||
(reset! captured {:cmd cmd
|
||||
:opts opts})
|
||||
{:exit 0
|
||||
:out "ok"
|
||||
:err ""})})]
|
||||
(is (= command (:cmd result)))
|
||||
(is (= command (:cmd @captured)))
|
||||
(is (= "/repo" (get-in @captured [:opts :dir])))))
|
||||
|
||||
(deftest shell-failures-include-command-context
|
||||
(let [command "node static/logseq-cli.js completion fish"]
|
||||
(try
|
||||
(shell/run! {:cmd command
|
||||
:executor (fn [_ _]
|
||||
{:exit 2
|
||||
:out ""
|
||||
:err "unsupported shell"})})
|
||||
(is false "Expected shell/run! to throw")
|
||||
(catch clojure.lang.ExceptionInfo ex
|
||||
(is (= command (:cmd (ex-data ex))))
|
||||
(is (= 2 (:exit (ex-data ex))))
|
||||
(is (= "unsupported shell" (:err (ex-data ex))))))))
|
||||
|
||||
(deftest allow-failure-returns-result-without-throwing
|
||||
(let [command "node static/logseq-cli.js graph info --graph missing"
|
||||
result (shell/run! {:cmd command
|
||||
:throw? false
|
||||
:executor (fn [_ _]
|
||||
{:exit 1
|
||||
:out "failed"
|
||||
:err "missing"})})]
|
||||
(is (= command (:cmd result)))
|
||||
(is (= 1 (:exit result)))
|
||||
(is (= "failed" (:out result)))
|
||||
(is (= "missing" (:err result)))))
|
||||
|
||||
(deftest default-executor-uses-resolvable-shell-command
|
||||
(let [captured (atom nil)]
|
||||
(with-redefs [process/process (fn [_opts & args]
|
||||
(reset! captured args)
|
||||
(future {:exit 0}))]
|
||||
(shell/run! {:cmd "echo ok"}))
|
||||
(is (= ["bash" "-c" "echo ok"] @captured))))
|
||||
114
cli-e2e/test/logseq/cli/e2e/sync_fixture_test.clj
Normal file
114
cli-e2e/test/logseq/cli/e2e/sync_fixture_test.clj
Normal file
@@ -0,0 +1,114 @@
|
||||
(ns logseq.cli.e2e.sync-fixture-test
|
||||
(:require [clojure.string :as string]
|
||||
[clojure.test :refer [deftest is testing]]
|
||||
[logseq.cli.e2e.sync-fixture :as sync-fixture]))
|
||||
|
||||
(deftest prepare-case-injects-case-local-sync-resources
|
||||
(let [suite-context {:sync-port "18080"
|
||||
:sync-http-base "http://127.0.0.1:18080"
|
||||
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"}
|
||||
input-case {:id "sync-case"
|
||||
:vars {:existing true}
|
||||
:setup ["mkdir -p '{{tmp-dir}}/graphs-b'"
|
||||
"mkdir -p '{{tmp-dir}}/home/logseq'"
|
||||
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' start --port 18080"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"]
|
||||
:cleanup ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/db_sync_server.py' stop --pid-file '{{tmp-dir}}/db-sync-server.pid'"]}
|
||||
prepared (sync-fixture/prepare-case input-case suite-context)]
|
||||
(is (= "sync-case" (:id prepared)))
|
||||
(is (= ["mkdir -p '{{tmp-dir}}/graphs-b'"
|
||||
"mkdir -p '{{tmp-dir}}/home/logseq'"
|
||||
"cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
"python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"]
|
||||
(:setup prepared)))
|
||||
(is (= ["{{cli}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json server stop --graph {{graph-arg}}"]
|
||||
(:cleanup prepared)))
|
||||
(is (= "http://127.0.0.1:18080" (get-in prepared [:vars :sync-http-base])))
|
||||
(is (= "ws://127.0.0.1:18080/sync/%s" (get-in prepared [:vars :sync-ws-url])))
|
||||
(is (= "18080" (get-in prepared [:vars :sync-port])))
|
||||
(is (= "11111" (get-in prepared [:vars :e2ee-password])))
|
||||
(is (= "11111" (get-in prepared [:vars :e2ee-password-arg])))
|
||||
(is (not (contains? (:vars prepared) :suite-auth-path)))
|
||||
(is (not (contains? (:vars prepared) :suite-config-path)))
|
||||
(is (not-any? #(string/includes? % "suite-auth-path") (:setup prepared)))
|
||||
(is (not-any? #(string/includes? % "suite-config-path") (:setup prepared)))
|
||||
(is (not-any? #(string/includes? % "db_sync_server.py' start") (:setup prepared)))
|
||||
(is (not-any? #(string/includes? % "db_sync_server.py' stop") (:cleanup prepared)))))
|
||||
|
||||
(deftest prepare-case-places-case-local-auth-before-cli-sync-commands
|
||||
(let [suite-context {:suite-tmp-dir "/tmp/sync-suite"
|
||||
:sync-port "18080"
|
||||
:sync-http-base "http://127.0.0.1:18080"
|
||||
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"}
|
||||
prepared (sync-fixture/prepare-case
|
||||
{:id "sync-case"
|
||||
:setup ["mkdir -p '{{tmp-dir}}/graphs-b'"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
|
||||
"{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"]
|
||||
:cleanup []}
|
||||
suite-context)
|
||||
setup (:setup prepared)
|
||||
bootstrap-cmd (nth setup 5)]
|
||||
(is (= "mkdir -p '{{tmp-dir}}/graphs-b'" (first setup)))
|
||||
(is (= "mkdir -p '{{tmp-dir}}/home/logseq'" (second setup)))
|
||||
(is (= "cp ~/logseq/auth.json '{{tmp-dir}}/home/logseq/auth.json'" (nth setup 2)))
|
||||
(is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{config-path}}' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
(nth setup 3)))
|
||||
(is (= "python3 '{{repo-root}}/cli-e2e/scripts/prepare_sync_config.py' --output '{{tmp-dir}}/cli-b.edn' --auth-path '{{tmp-dir}}/home/logseq/auth.json' --http-base '{{sync-http-base}}' --ws-url '{{sync-ws-url}}'"
|
||||
(nth setup 4)))
|
||||
(is (string/includes? bootstrap-cmd "sync ensure-keys"))
|
||||
(is (string/includes? bootstrap-cmd "--upload-keys"))
|
||||
(is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.lock"))
|
||||
(is (string/includes? bootstrap-cmd "/tmp/sync-suite/user-rsa-keys.ready"))
|
||||
(is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json graph create --graph {{graph-arg}} >/dev/null"
|
||||
(nth setup 6)))
|
||||
(is (= "{{cli-home}} --root-dir {{root-dir-arg}} --config {{config-path-arg}} --output json sync ensure-keys --graph {{graph-arg}} --e2ee-password {{e2ee-password-arg}}"
|
||||
(nth setup 7)))
|
||||
(is (not-any? #(and (string/includes? % "sync ensure-keys")
|
||||
(not= % bootstrap-cmd)
|
||||
(string/includes? % "--upload-keys"))
|
||||
setup))))
|
||||
|
||||
(deftest prepare-case-accepts-custom-e2ee-password
|
||||
(let [suite-context {:sync-port "18080"
|
||||
:sync-http-base "http://127.0.0.1:18080"
|
||||
:sync-ws-url "ws://127.0.0.1:18080/sync/%s"
|
||||
:e2ee-password "pass word"}
|
||||
prepared (sync-fixture/prepare-case {:id "sync-case"
|
||||
:setup []
|
||||
:cleanup []}
|
||||
suite-context)]
|
||||
(is (= "pass word" (get-in prepared [:vars :e2ee-password])))
|
||||
(is (= "'pass word'" (get-in prepared [:vars :e2ee-password-arg])))))
|
||||
|
||||
(deftest before-and-after-suite-only-manage-shared-sync-server-lifecycle
|
||||
(let [calls (atom [])
|
||||
run-command (fn [opts]
|
||||
(swap! calls conj opts)
|
||||
{:exit 0
|
||||
:out ""
|
||||
:err ""})
|
||||
suite-context (sync-fixture/before-suite! {:run-command run-command})]
|
||||
(testing "before-suite only starts the shared server"
|
||||
(is (= 1 (count @calls)))
|
||||
(is (string/includes? (:cmd (first @calls)) "db_sync_server.py"))
|
||||
(is (string/includes? (:cmd (first @calls)) " start "))
|
||||
(is (string/includes? (:cmd (first @calls)) "--data-dir"))
|
||||
(is (not (string/includes? (:cmd (first @calls)) "--root-dir")))
|
||||
(is (string/includes? (:cmd (first @calls)) "--port 18080"))
|
||||
(is (string/includes? (:cmd (first @calls)) "--startup-timeout-s 60"))
|
||||
(is (not (string/includes? (:cmd (first @calls)) "prepare_sync_config.py")))
|
||||
(is (not (contains? suite-context :suite-auth-path)))
|
||||
(is (not (contains? suite-context :suite-config-path))))
|
||||
(sync-fixture/after-suite! suite-context {:run-command run-command})
|
||||
(testing "after-suite only stops the shared server once"
|
||||
(is (= 2 (count @calls)))
|
||||
(is (string/includes? (:cmd (last @calls)) "db_sync_server.py"))
|
||||
(is (string/includes? (:cmd (last @calls)) " stop "))
|
||||
(is (false? (:throw? (last @calls)))))))
|
||||
55
cli-e2e/test/logseq/cli/e2e/wait_sync_status_test.py
Normal file
55
cli-e2e/test/logseq/cli/e2e/wait_sync_status_test.py
Normal file
@@ -0,0 +1,55 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).resolve().parents[4] / "scripts" / "wait_sync_status.py"
|
||||
spec = importlib.util.spec_from_file_location("wait_sync_status", MODULE_PATH)
|
||||
wait_sync_status = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(wait_sync_status)
|
||||
|
||||
|
||||
def test_resolved_status_command_is_built_once_and_reused() -> None:
|
||||
args = SimpleNamespace(
|
||||
cli="~/repo/static/logseq-cli.js",
|
||||
root_dir="~/tmp/root",
|
||||
config="~/tmp/cli.edn",
|
||||
graph="demo",
|
||||
)
|
||||
|
||||
command = wait_sync_status.status_command(args)
|
||||
|
||||
assert command[0] == "node"
|
||||
assert command[2] == "--root-dir"
|
||||
assert command[-2:] == ["--graph", "demo"]
|
||||
assert Path(command[1]).is_absolute()
|
||||
assert Path(command[3]).is_absolute()
|
||||
assert Path(command[5]).is_absolute()
|
||||
assert wait_sync_status.status_command(args) is command
|
||||
|
||||
|
||||
def test_run_status_uses_precomputed_command() -> None:
|
||||
command = ["node", "/abs/cli.js", "--graph", "demo"]
|
||||
args = SimpleNamespace(status_command=command)
|
||||
calls = []
|
||||
|
||||
def fake_run(cmd, capture_output, text):
|
||||
calls.append(cmd)
|
||||
return SimpleNamespace(
|
||||
returncode=0,
|
||||
stdout='{"status":"ok","data":{"pending-local":0,"pending-asset":0,"pending-server":0}}',
|
||||
stderr="",
|
||||
)
|
||||
|
||||
original_run = wait_sync_status.subprocess.run
|
||||
wait_sync_status.subprocess.run = fake_run
|
||||
try:
|
||||
payload = wait_sync_status.run_status(args)
|
||||
finally:
|
||||
wait_sync_status.subprocess.run = original_run
|
||||
|
||||
assert payload["status"] == "ok"
|
||||
assert calls == [command]
|
||||
2
deps.edn
2
deps.edn
@@ -24,6 +24,8 @@
|
||||
cljs-http/cljs-http {:mvn/version "0.1.49"}
|
||||
org.babashka/sci {:mvn/version "0.12.51"}
|
||||
org.clj-commons/hickory {:mvn/version "0.7.7"}
|
||||
org.clj-commons/humanize {:mvn/version "1.2"}
|
||||
org.babashka/cli {:mvn/version "0.8.67"}
|
||||
hiccups/hiccups {:mvn/version "0.3.0"}
|
||||
tongue/tongue {:mvn/version "0.4.4"}
|
||||
org.clojure/core.async {:mvn/version "1.8.741"}
|
||||
|
||||
48
deps/cli/src/logseq/cli.cljs
vendored
48
deps/cli/src/logseq/cli.cljs
vendored
@@ -2,6 +2,7 @@
|
||||
"Main ns for Logseq CLI"
|
||||
(:require ["fs" :as fs]
|
||||
["path" :as node-path]
|
||||
["child_process" :as child-process]
|
||||
[babashka.cli :as cli]
|
||||
[clojure.string :as string]
|
||||
[logseq.cli.common.graph :as cli-common-graph]
|
||||
@@ -58,6 +59,29 @@
|
||||
(print-command-help "help" cmd-map)
|
||||
(println "Command" (pr-str command) "does not exist"))))
|
||||
|
||||
(defn- extract-graph-names
|
||||
"Extract graph names from parsed opts for unlock-graph."
|
||||
[{:keys [graph graphs]}]
|
||||
(cond-> []
|
||||
(string? graph) (conj graph)
|
||||
(sequential? graphs) (into graphs)))
|
||||
|
||||
(defn- unlock-graphs!
|
||||
"Stop the db-worker-node server(s) for graph-names so this CLI can open the
|
||||
SQLite database directly"
|
||||
[graph-names]
|
||||
(doseq [graph graph-names]
|
||||
(let [result (.spawnSync child-process
|
||||
"logseq"
|
||||
#js ["server" "stop" "-g" graph]
|
||||
#js {:timeout 5000
|
||||
:stdio "pipe"})]
|
||||
(if (= 0 (.-status result))
|
||||
(println "Unlocked graph:" graph)
|
||||
;; Ignore server not started
|
||||
(when-not (string/includes? (str (.-stdout result)) "server-not-found")
|
||||
(println "Failed to unlock graph:" (pr-str result)))))))
|
||||
|
||||
(defn- lazy-load-fn
|
||||
"Lazy load fn to speed up start time. After nbb requires ~30 namespaces, start time gets close to 1s.
|
||||
Also handles --help on all commands"
|
||||
@@ -65,13 +89,17 @@
|
||||
(fn [& args]
|
||||
(if (get-in (first args) [:opts :help])
|
||||
(help-command {:opts {:command (-> args first :dispatch first)}})
|
||||
(-> (p/let [_ (require (symbol (namespace fn-sym)))]
|
||||
(apply (resolve fn-sym) args))
|
||||
(p/catch (fn [err]
|
||||
(if (= :sci/error (:type (ex-data err)))
|
||||
(nbb.error/print-error-report err)
|
||||
(js/console.error "Error:" err))
|
||||
(js/process.exit 1)))))))
|
||||
(do
|
||||
(when (get-in (first args) [:opts :unlock-graph])
|
||||
(let [graphs (extract-graph-names (get (first args) :opts))]
|
||||
(unlock-graphs! graphs)))
|
||||
(-> (p/let [_ (require (symbol (namespace fn-sym)))]
|
||||
(apply (resolve fn-sym) args))
|
||||
(p/catch (fn [err]
|
||||
(if (= :sci/error (:type (ex-data err)))
|
||||
(nbb.error/print-error-report err)
|
||||
(js/console.error "Error:" err))
|
||||
(js/process.exit 1))))))))
|
||||
|
||||
(def ^:private table*
|
||||
[{:cmds ["list"] :desc "List local graphs"
|
||||
@@ -122,10 +150,14 @@
|
||||
:spec default-spec
|
||||
:fn default-command}])
|
||||
|
||||
|
||||
;; Spec shared with all commands
|
||||
(def ^:private shared-spec
|
||||
{:help {:alias :h
|
||||
:desc "Print help"}})
|
||||
:desc "Print help"}
|
||||
:unlock-graph {:alias :u
|
||||
:coerce :boolean
|
||||
:desc "Unlock graph(s) before running command"}})
|
||||
|
||||
(def ^:private table
|
||||
(mapv (fn [m] (update m :spec #(merge % shared-spec))) table*))
|
||||
|
||||
4
deps/cli/src/logseq/cli/commands/graph.cljs
vendored
4
deps/cli/src/logseq/cli/commands/graph.cljs
vendored
@@ -38,6 +38,6 @@
|
||||
(defn list-graphs
|
||||
[]
|
||||
(let [db-graphs (->> (cli-common-graph/get-db-based-graphs)
|
||||
(map #(string/replace-first % common-config/db-version-prefix ""))
|
||||
(map common-config/strip-leading-db-version-prefix)
|
||||
sort)]
|
||||
(println (string/join "\n" db-graphs))))
|
||||
(println (string/join "\n" db-graphs))))
|
||||
|
||||
18
deps/cli/src/logseq/cli/common/graph.cljs
vendored
18
deps/cli/src/logseq/cli/common/graph.cljs
vendored
@@ -1,23 +1,19 @@
|
||||
(ns ^:node-only logseq.cli.common.graph
|
||||
"Graph related fns shared between CLI and electron"
|
||||
(:require ["fs-extra" :as fs-extra]
|
||||
["os" :as os]
|
||||
["path" :as node-path]
|
||||
[clojure.string :as string]
|
||||
[logseq.common.config :as common-config]
|
||||
[logseq.common.graph :as common-graph]))
|
||||
[logseq.common.graph :as common-graph]
|
||||
[logseq.common.graph-dir :as graph-dir]))
|
||||
|
||||
(defn ^:api graph-name->path
|
||||
[graph-name]
|
||||
(when graph-name
|
||||
(-> graph-name
|
||||
(string/replace "+3A+" ":")
|
||||
(string/replace "++" "/"))))
|
||||
(graph-dir/decode-graph-dir-name graph-name))
|
||||
|
||||
(defn get-db-graphs-dir
|
||||
"Directory where DB graphs are stored"
|
||||
[]
|
||||
(node-path/join (os/homedir) "logseq" "graphs"))
|
||||
(common-graph/expand-home (common-graph/get-default-graphs-dir)))
|
||||
|
||||
(defn get-db-based-graphs
|
||||
[]
|
||||
@@ -27,5 +23,7 @@
|
||||
(remove (fn [s] (= s common-config/unlinked-graphs-dir)))
|
||||
(map graph-name->path)
|
||||
(keep (fn [s]
|
||||
(when-not (string/starts-with? s common-config/file-version-prefix)
|
||||
(str common-config/db-version-prefix s)))))))
|
||||
(when (and (string? s)
|
||||
(not (string/starts-with? s common-config/file-version-prefix)))
|
||||
(common-config/canonicalize-db-version-repo s))))
|
||||
distinct)))
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
(if expand
|
||||
(cond-> (into {} e)
|
||||
true
|
||||
(dissoc e :block/tags :block/order :block/refs :block/name :db/index
|
||||
(dissoc :block/tags :block/order :block/refs :block/name :db/index
|
||||
:logseq.property/default-value)
|
||||
true
|
||||
(update :block/uuid str)
|
||||
@@ -46,7 +46,7 @@
|
||||
(if expand
|
||||
(cond-> (into {} e)
|
||||
true
|
||||
(dissoc e :block/tags :block/order :block/refs :block/name)
|
||||
(dissoc :block/tags :block/order :block/refs :block/name)
|
||||
true
|
||||
(update :block/uuid str)
|
||||
(:logseq.property.class/extends e)
|
||||
|
||||
4
deps/cli/src/logseq/cli/util.cljs
vendored
4
deps/cli/src/logseq/cli/util.cljs
vendored
@@ -4,7 +4,7 @@
|
||||
["path" :as node-path]
|
||||
[clojure.string :as string]
|
||||
[logseq.cli.common.graph :as cli-common-graph]
|
||||
[logseq.db.common.sqlite :as common-sqlite]
|
||||
[logseq.common.graph-dir :as graph-dir]
|
||||
[nbb.error]
|
||||
[promesa.core :as p]))
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
(string/includes? graph "/")
|
||||
((juxt node-path/dirname node-path/basename) graph)
|
||||
:else
|
||||
[(cli-common-graph/get-db-graphs-dir) (common-sqlite/sanitize-db-name graph)]))
|
||||
[(cli-common-graph/get-db-graphs-dir) (graph-dir/repo->encoded-graph-dir-name graph)]))
|
||||
|
||||
(defn get-graph-path
|
||||
"If graph is a file, return its path. Otherwise returns the graph's dir"
|
||||
|
||||
4
deps/common/.carve/config.edn
vendored
4
deps/common/.carve/config.edn
vendored
@@ -8,6 +8,8 @@
|
||||
logseq.common.util.date-time
|
||||
logseq.common.date
|
||||
logseq.common.util.macro
|
||||
logseq.common.cognito-config
|
||||
logseq.common.config
|
||||
logseq.common.defkeywords]
|
||||
logseq.common.defkeywords
|
||||
logseq.common.graph-dir]
|
||||
:report {:format :ignore}}
|
||||
|
||||
4
deps/common/.carve/ignore
vendored
4
deps/common/.carve/ignore
vendored
@@ -3,6 +3,10 @@ logseq.common.graph/get-files
|
||||
;; API fn
|
||||
logseq.common.graph/read-directories
|
||||
;; API fn
|
||||
logseq.common.graph/get-default-graphs-dir
|
||||
;; API fn
|
||||
logseq.common.graph/expand-home
|
||||
;; API fn
|
||||
logseq.common.authorization/verify-jwt
|
||||
|
||||
;; Profile utils
|
||||
|
||||
13
deps/common/src/logseq/common/cognito_config.cljs
vendored
Normal file
13
deps/common/src/logseq/common/cognito_config.cljs
vendored
Normal file
@@ -0,0 +1,13 @@
|
||||
(ns logseq.common.cognito-config
|
||||
"Shared Cognito configuration for frontend and CLI-safe consumers.")
|
||||
|
||||
(def COGNITO-CLIENT-ID
|
||||
"69cs1lgme7p8kbgld8n5kseii6")
|
||||
|
||||
(def CLI-COGNITO-CLIENT-ID
|
||||
"69cs1lgme7p8kbgld8n5kseii6")
|
||||
|
||||
(def OAUTH-DOMAIN
|
||||
"logseq-prod.auth.us-east-1.amazoncognito.com")
|
||||
|
||||
(def OAUTH-SCOPE "email openid phone")
|
||||
20
deps/common/src/logseq/common/config.cljs
vendored
20
deps/common/src/logseq/common/config.cljs
vendored
@@ -33,6 +33,26 @@
|
||||
(defonce db-version-prefix "logseq_db_")
|
||||
(defonce file-version-prefix "logseq_local_")
|
||||
|
||||
(defn strip-leading-db-version-prefix
|
||||
"Strip exactly one leading db prefix for user-facing display values."
|
||||
[s]
|
||||
(if (and (string? s)
|
||||
(string/starts-with? s db-version-prefix))
|
||||
(subs s (count db-version-prefix))
|
||||
s))
|
||||
|
||||
(defn canonicalize-db-version-repo
|
||||
"Normalize any repo/graph name to exactly one leading db prefix."
|
||||
[s]
|
||||
(when (seq s)
|
||||
(let [s (str s)
|
||||
stripped (loop [name' s]
|
||||
(if (string/starts-with? name' db-version-prefix)
|
||||
(recur (subs name' (count db-version-prefix)))
|
||||
name'))]
|
||||
(str db-version-prefix stripped))))
|
||||
|
||||
(defonce default-graphs-dir "~/logseq/graphs")
|
||||
(defonce local-assets-dir "assets")
|
||||
(defonce unlinked-graphs-dir "Unlinked graphs")
|
||||
|
||||
|
||||
14
deps/common/src/logseq/common/graph.cljs
vendored
14
deps/common/src/logseq/common/graph.cljs
vendored
@@ -1,8 +1,10 @@
|
||||
(ns ^:node-only logseq.common.graph
|
||||
"This ns provides common fns for a graph directory and only runs in a node environment"
|
||||
(:require ["fs" :as fs]
|
||||
["os" :as os]
|
||||
["path" :as node-path]
|
||||
[clojure.string :as string]
|
||||
[logseq.common.config :as common-config]
|
||||
[logseq.common.path :as path]))
|
||||
|
||||
(def ^:private win32?
|
||||
@@ -97,3 +99,15 @@ Rules:
|
||||
(->> (readdir graph-dir)
|
||||
(remove (partial ignored-path? graph-dir))
|
||||
(filter #(contains? allowed-formats (get-ext %)))))
|
||||
|
||||
(defn get-default-graphs-dir
|
||||
"Get default dir for storing graphs by first looking in env var."
|
||||
[]
|
||||
(or js/process.env.LOGSEQ_GRAPHS_DIR common-config/default-graphs-dir))
|
||||
|
||||
(defn expand-home
|
||||
"Expands path if it starts with '~'"
|
||||
[path]
|
||||
(if (and (seq path) (string/starts-with? path "~"))
|
||||
(node-path/join (os/homedir) (subs path 1))
|
||||
path))
|
||||
|
||||
57
deps/common/src/logseq/common/graph_dir.cljs
vendored
Normal file
57
deps/common/src/logseq/common/graph_dir.cljs
vendored
Normal file
@@ -0,0 +1,57 @@
|
||||
(ns logseq.common.graph-dir
|
||||
"Platform-agnostic graph directory naming helpers."
|
||||
(:require [clojure.string :as string]
|
||||
[logseq.common.config :as common-config]))
|
||||
|
||||
(defn encode-graph-dir-name
|
||||
[graph-name]
|
||||
(let [encoded (js/encodeURIComponent (or graph-name ""))]
|
||||
(-> encoded
|
||||
(string/replace "%20" " ")
|
||||
(string/replace "~" "%7E")
|
||||
(string/replace "%" "~"))))
|
||||
|
||||
(defn decode-graph-dir-name
|
||||
[dir-name]
|
||||
(when-not (and (string? dir-name)
|
||||
(or (string/includes? dir-name "++")
|
||||
(string/includes? dir-name "+3A+")))
|
||||
(when (some? dir-name)
|
||||
(try
|
||||
(js/decodeURIComponent (string/replace dir-name "~" "%"))
|
||||
(catch :default _
|
||||
nil)))))
|
||||
|
||||
(def ^:private legacy-dir-pattern #"(?:\+\+|\+3A\+|%)")
|
||||
|
||||
(defn decode-legacy-graph-dir-name
|
||||
[dir-name]
|
||||
(when (and (string? dir-name)
|
||||
(re-find legacy-dir-pattern dir-name))
|
||||
(let [compat-name (-> dir-name
|
||||
(string/replace "+3A+" ":")
|
||||
(string/replace "++" "/"))]
|
||||
(try
|
||||
(let [decoded (js/decodeURIComponent compat-name)]
|
||||
(when (seq decoded)
|
||||
decoded))
|
||||
(catch :default _
|
||||
nil)))))
|
||||
|
||||
(defn repo->graph-dir-key
|
||||
[repo]
|
||||
(when (seq repo)
|
||||
(if (string/starts-with? repo common-config/db-version-prefix)
|
||||
(subs repo (count common-config/db-version-prefix))
|
||||
repo)))
|
||||
|
||||
(defn graph-dir-key->encoded-dir-name
|
||||
[graph-dir-key]
|
||||
(when (some? graph-dir-key)
|
||||
(encode-graph-dir-name graph-dir-key)))
|
||||
|
||||
(defn repo->encoded-graph-dir-name
|
||||
[repo]
|
||||
(some-> repo
|
||||
repo->graph-dir-key
|
||||
graph-dir-key->encoded-dir-name))
|
||||
2
deps/db-sync/.carve/ignore
vendored
2
deps/db-sync/.carve/ignore
vendored
@@ -26,6 +26,8 @@ logseq.db-sync.worker/worker
|
||||
;; debugging
|
||||
logseq.db-sync.worker.timing/summary
|
||||
;; API
|
||||
logseq.db-sync.snapshot/finalize-datoms-jsonl-buffer
|
||||
;; API
|
||||
logseq.db-sync.checksum/recompute-checksum-diagnostics
|
||||
;; API (test runner entrypoint)
|
||||
logseq.db-sync.test-runner/main
|
||||
|
||||
@@ -148,7 +148,7 @@
|
||||
(def graphs-list-response-schema
|
||||
[:map
|
||||
[:graphs [:sequential graph-info-schema]]
|
||||
[:user-rsa-keys-exists? :boolean]])
|
||||
[:user-rsa-keys-exists? {:optional true} :boolean]])
|
||||
|
||||
(def graph-create-request-schema
|
||||
[:map
|
||||
|
||||
20
deps/db-sync/src/logseq/db_sync/node/graph.cljs
vendored
20
deps/db-sync/src/logseq/db_sync/node/graph.cljs
vendored
@@ -8,14 +8,18 @@
|
||||
:removeWebSocket (fn [ws] (swap! sockets disj ws))}))
|
||||
|
||||
(defn- env-object [cfg index-db assets-bucket]
|
||||
(doto (js-obj)
|
||||
(aset "DB" index-db)
|
||||
(aset "LOGSEQ_SYNC_ASSETS" assets-bucket)
|
||||
;; Keep node-adapter snapshot stream uncompressed.
|
||||
(aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false")
|
||||
(aset "COGNITO_ISSUER" (:cognito-issuer cfg))
|
||||
(aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg))
|
||||
(aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg))))
|
||||
(let [allow-unverified-jwt-claims (some-> js/process .-env (aget "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS"))
|
||||
env (doto (js-obj)
|
||||
(aset "DB" index-db)
|
||||
(aset "LOGSEQ_SYNC_ASSETS" assets-bucket)
|
||||
;; Keep node-adapter snapshot stream uncompressed.
|
||||
(aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false")
|
||||
(aset "COGNITO_ISSUER" (:cognito-issuer cfg))
|
||||
(aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg))
|
||||
(aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))]
|
||||
(when (some? allow-unverified-jwt-claims)
|
||||
(aset env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" allow-unverified-jwt-claims))
|
||||
env))
|
||||
|
||||
(defn graph-context
|
||||
[{:keys [config index-db assets-bucket]} graph-id]
|
||||
|
||||
22
deps/db-sync/src/logseq/db_sync/node/server.cljs
vendored
22
deps/db-sync/src/logseq/db_sync/node/server.cljs
vendored
@@ -24,15 +24,19 @@
|
||||
(logging/install!)
|
||||
|
||||
(defn- make-env [cfg index-db assets-bucket]
|
||||
(doto (js-obj)
|
||||
(aset "DB" index-db)
|
||||
(aset "LOGSEQ_SYNC_ASSETS" assets-bucket)
|
||||
;; Node adapter serves snapshot transit stream without gzip to avoid
|
||||
;; browser/adapter content-encoding mismatches during graph download.
|
||||
(aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false")
|
||||
(aset "COGNITO_ISSUER" (:cognito-issuer cfg))
|
||||
(aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg))
|
||||
(aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg))))
|
||||
(let [allow-unverified-jwt-claims (some-> js/process .-env (aget "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS"))
|
||||
env (doto (js-obj)
|
||||
(aset "DB" index-db)
|
||||
(aset "LOGSEQ_SYNC_ASSETS" assets-bucket)
|
||||
;; Node adapter serves snapshot transit stream without gzip to avoid
|
||||
;; browser/adapter content-encoding mismatches during graph download.
|
||||
(aset "DB_SYNC_SNAPSHOT_STREAM_GZIP" "false")
|
||||
(aset "COGNITO_ISSUER" (:cognito-issuer cfg))
|
||||
(aset "COGNITO_CLIENT_ID" (:cognito-client-id cfg))
|
||||
(aset "COGNITO_JWKS_URL" (:cognito-jwks-url cfg)))]
|
||||
(when (some? allow-unverified-jwt-claims)
|
||||
(aset env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" allow-unverified-jwt-claims))
|
||||
env))
|
||||
|
||||
(defn- access-allowed?
|
||||
[env graph-id request]
|
||||
|
||||
24
deps/db-sync/src/logseq/db_sync/worker/auth.cljs
vendored
24
deps/db-sync/src/logseq/db_sync/worker/auth.cljs
vendored
@@ -34,12 +34,28 @@
|
||||
(def ^:private recoverable-auth-errors
|
||||
#{"invalid" "iss not found" "aud not found" "exp" "kid"})
|
||||
|
||||
(def ^:private truthy-env-values
|
||||
#{"1" "true" "yes" "on"})
|
||||
|
||||
(defn- recoverable-auth-error?
|
||||
[error]
|
||||
(when error
|
||||
(let [message (or (ex-message error) (some-> error .-message))]
|
||||
(contains? recoverable-auth-errors message))))
|
||||
|
||||
(defn- env-flag-enabled?
|
||||
[env k]
|
||||
(let [v (some-> env (aget k))]
|
||||
(cond
|
||||
(true? v) true
|
||||
(false? v) false
|
||||
(string? v) (contains? truthy-env-values (string/lower-case v))
|
||||
:else false)))
|
||||
|
||||
(defn- allow-unverified-jwt-claims?
|
||||
[env]
|
||||
(env-flag-enabled? env "DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS"))
|
||||
|
||||
(defn- expired-token?
|
||||
[token]
|
||||
(when-let [claims (unsafe-jwt-claims token)]
|
||||
@@ -55,7 +71,13 @@
|
||||
(p/resolved nil)
|
||||
(-> (authorization/verify-jwt token env)
|
||||
(p/catch (fn [error]
|
||||
(if (recoverable-auth-error? error)
|
||||
(cond
|
||||
(recoverable-auth-error? error)
|
||||
nil
|
||||
|
||||
(allow-unverified-jwt-claims? env)
|
||||
(unsafe-jwt-claims token)
|
||||
|
||||
:else
|
||||
(p/rejected error))))))
|
||||
(p/resolved nil))))
|
||||
|
||||
@@ -274,9 +274,9 @@
|
||||
(let [value (.-value chunk)
|
||||
{:keys [rows buffer]} (snapshot/parse-framed-chunk buffer value)
|
||||
rows-count (count rows)
|
||||
reset? (and @reset-pending? (seq rows))]
|
||||
reset? (boolean (and @reset-pending? (seq rows)))]
|
||||
(when (seq rows)
|
||||
(import-snapshot! self rows (true? reset?))
|
||||
(import-snapshot! self rows reset?)
|
||||
(vreset! reset-pending? false))
|
||||
(vswap! total-count + rows-count)
|
||||
(p/recur buffer)))))
|
||||
|
||||
10
deps/db-sync/src/logseq/db_sync/worker/ws.cljs
vendored
10
deps/db-sync/src/logseq/db_sync/worker/ws.cljs
vendored
@@ -28,7 +28,9 @@
|
||||
(.send ws (protocol/encode-message {:type "error" :message "server error"}))))))
|
||||
|
||||
(defn broadcast! [^js self sender msg]
|
||||
(let [clients (.getWebSockets (.-state self))]
|
||||
(doseq [ws clients]
|
||||
(when (and (not= ws sender) (ws-open? ws))
|
||||
(send! ws msg)))))
|
||||
(when-let [state (some-> self .-state)]
|
||||
(when (fn? (.-getWebSockets state))
|
||||
(let [clients (.getWebSockets state)]
|
||||
(doseq [ws clients]
|
||||
(when (and (not= ws sender) (ws-open? ws))
|
||||
(send! ws msg)))))))
|
||||
|
||||
@@ -57,6 +57,22 @@
|
||||
(is (= "jwks" (ex-message error)))
|
||||
(done)))))))
|
||||
|
||||
(deftest auth-claims-jwks-error-falls-back-to-unsafe-claims-when-enabled-test
|
||||
(async done
|
||||
(let [token "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJ1MSJ9.signature"
|
||||
request (js/Request. "http://localhost/graphs"
|
||||
#js {:headers #js {"authorization" (str "Bearer " token)}})
|
||||
env #js {"DB_SYNC_ALLOW_UNVERIFIED_JWT_CLAIMS" "true"}]
|
||||
(-> (p/with-redefs [authorization/verify-jwt
|
||||
(fn [_token _env]
|
||||
(p/rejected (ex-info "jwks" {})))]
|
||||
(p/let [claims (auth/auth-claims request env)]
|
||||
(is (= "u1" (aget claims "sub")))))
|
||||
(p/then (fn [] (done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest auth-claims-expired-jwt-short-circuits-verification-test
|
||||
(async done
|
||||
(let [expired-token "eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjEsInN1YiI6InUxIn0.signature"
|
||||
|
||||
@@ -561,6 +561,56 @@
|
||||
(is (= "pull/ok" (:type pull-response)))
|
||||
(is (empty? (:txs pull-response)))))))
|
||||
|
||||
(deftest tx-batch-keeps-created-by-ref-lookup-payload-test
|
||||
(testing "created-by lookup payload is preserved for save-block tx"
|
||||
(let [sql (test-sql/make-sql)
|
||||
conn (storage/open-conn sql)
|
||||
self #js {:sql sql
|
||||
:conn conn
|
||||
:schema-ready true}
|
||||
page-uuid (random-uuid)
|
||||
missing-user-uuid (random-uuid)
|
||||
missing-user-ref [:block/uuid missing-user-uuid]
|
||||
tx-entry {:tx (protocol/tx->transit [[:db/add -1 :block/uuid page-uuid]
|
||||
[:db/add -1 :block/name "created-by-sanitize-page"]
|
||||
[:db/add -1 :block/title "created-by-sanitize-page"]
|
||||
[:db/add -1 :logseq.property/created-by-ref missing-user-ref]])
|
||||
:outliner-op :save-block}
|
||||
response (with-redefs [ws/broadcast! (fn [& _] nil)]
|
||||
(sync-handler/handle-tx-batch! self nil [tx-entry] 0))
|
||||
page (d/entity @conn [:block/uuid page-uuid])]
|
||||
(is (= "tx/batch/ok" (:type response)))
|
||||
(is (= 1 (:t response)))
|
||||
(is (some? page))
|
||||
(is (= "created-by-sanitize-page" (:block/title page)))
|
||||
(is (= missing-user-ref (:logseq.property/created-by-ref page))))))
|
||||
|
||||
(deftest tx-batch-rejects-missing-page-ref-lookups-test
|
||||
(testing "missing page refs/tags lookup refs reject create-page tx"
|
||||
(let [sql (test-sql/make-sql)
|
||||
conn (storage/open-conn sql)
|
||||
self #js {:sql sql
|
||||
:conn conn
|
||||
:schema-ready true}
|
||||
page-uuid (random-uuid)
|
||||
missing-ref-uuid (random-uuid)
|
||||
missing-tag-uuid (random-uuid)
|
||||
missing-user-uuid (random-uuid)
|
||||
tx-entry {:tx (protocol/tx->transit [[:db/add -1 :block/uuid page-uuid]
|
||||
[:db/add -1 :block/name "optional-ref-sanitize-page"]
|
||||
[:db/add -1 :block/title "optional-ref-sanitize-page"]
|
||||
[:db/add -1 :block/refs [:block/uuid missing-ref-uuid]]
|
||||
[:db/add -1 :block/tags [:block/uuid missing-tag-uuid]]
|
||||
[:db/add -1 :logseq.property/created-by-ref [:block/uuid missing-user-uuid]]])
|
||||
:outliner-op :create-page}
|
||||
response (with-redefs [ws/broadcast! (fn [& _] nil)]
|
||||
(sync-handler/handle-tx-batch! self nil [tx-entry] 0))
|
||||
page (d/entity @conn [:block/uuid page-uuid])]
|
||||
(is (= "tx/reject" (:type response)))
|
||||
(is (= "db transact failed" (:reason response)))
|
||||
(is (= 0 (:t response)))
|
||||
(is (nil? page)))))
|
||||
|
||||
(deftest tx-batch-rejects-while-snapshot-upload-is-in-progress-test
|
||||
(let [sql (test-sql/make-sql)
|
||||
conn (d/create-conn db-schema/schema)
|
||||
@@ -640,6 +690,33 @@
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest import-snapshot-stream-first-non-empty-chunk-applies-reset-test
|
||||
(async done
|
||||
(let [rows [[42 "payload" nil]]
|
||||
frame (#'sync-handler/frame-bytes (snapshot/encode-rows rows))
|
||||
stream (js/ReadableStream.
|
||||
#js {:start (fn [controller]
|
||||
(.enqueue controller frame)
|
||||
(.close controller))})
|
||||
applied (atom [])
|
||||
self #js {:sql (test-sql/make-sql)
|
||||
:conn (d/create-conn db-schema/schema)
|
||||
:schema-ready true}]
|
||||
(-> (p/with-redefs [sync-handler/import-snapshot!
|
||||
(fn [_self rows* reset?]
|
||||
(swap! applied conj {:rows rows*
|
||||
:reset? reset?}))]
|
||||
(p/let [count (#'sync-handler/import-snapshot-stream! self stream true)]
|
||||
(is (= 1 count))
|
||||
(is (= [{:rows rows
|
||||
:reset? true}]
|
||||
@applied))))
|
||||
(p/then (fn []
|
||||
(done)))
|
||||
(p/catch (fn [error]
|
||||
(is false (str error))
|
||||
(done)))))))
|
||||
|
||||
(deftest tx-batch-rejects-when-a-tx-entry-fails-test
|
||||
(testing "db transact failure rejects the batch"
|
||||
(let [sql (test-sql/make-sql)
|
||||
|
||||
2
deps/db/.carve/config.edn
vendored
2
deps/db/.carve/config.edn
vendored
@@ -9,6 +9,8 @@
|
||||
logseq.db.sqlite.create-graph
|
||||
logseq.db.frontend.malli-schema
|
||||
logseq.db.frontend.asset
|
||||
;; Used outside deps/db by frontend worker and electron modules.
|
||||
logseq.db.sqlite.backup
|
||||
;; Some fns are used by frontend but not worth moving over yet
|
||||
logseq.db.frontend.schema
|
||||
logseq.db.frontend.validate
|
||||
|
||||
12
deps/db/src/logseq/db/common/sqlite.cljs
vendored
12
deps/db/src/logseq/db/common/sqlite.cljs
vendored
@@ -3,6 +3,8 @@
|
||||
(:require ["path" :as node-path]
|
||||
[clojure.string :as string]
|
||||
[datascript.core :as d]
|
||||
[logseq.common.graph-dir :as graph-dir]
|
||||
[logseq.common.path :as path]
|
||||
[logseq.db.sqlite.util :as sqlite-util]))
|
||||
|
||||
(defn create-kvs-table!
|
||||
@@ -26,11 +28,11 @@
|
||||
|
||||
(defn get-db-full-path
|
||||
[graphs-dir db-name]
|
||||
(let [db-name' (sanitize-db-name db-name)
|
||||
graph-dir (node-path/join graphs-dir db-name')]
|
||||
[db-name' (node-path/join graph-dir "db.sqlite")]))
|
||||
(let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name)
|
||||
graph-dir (node-path/join graphs-dir graph-dir-name)]
|
||||
[graph-dir-name (path/path-join graph-dir "db.sqlite")]))
|
||||
|
||||
(defn get-db-backups-path
|
||||
[graphs-dir db-name]
|
||||
(let [db-name' (sanitize-db-name db-name)]
|
||||
(node-path/join graphs-dir db-name' "backups")))
|
||||
(let [graph-dir-name (graph-dir/repo->encoded-graph-dir-name db-name)]
|
||||
(path/path-join graphs-dir graph-dir-name "backups")))
|
||||
|
||||
6
deps/db/src/logseq/db/common/sqlite_cli.cljs
vendored
6
deps/db/src/logseq/db/common/sqlite_cli.cljs
vendored
@@ -1,11 +1,11 @@
|
||||
(ns ^:node-only logseq.db.common.sqlite-cli
|
||||
"Primary ns to interact with DB files with node.js based CLIs"
|
||||
(:require ["better-sqlite3" :as sqlite3]
|
||||
["os" :as os]
|
||||
["path" :as node-path]
|
||||
[cljs-bean.core :as bean]
|
||||
[clojure.string :as string]
|
||||
[datascript.storage :refer [IStorage]]
|
||||
[logseq.common.graph :as common-graph]
|
||||
[logseq.db.common.sqlite :as common-sqlite]
|
||||
[logseq.db.frontend.schema :as db-schema]
|
||||
[logseq.db.sqlite.util :as sqlite-util]))
|
||||
@@ -22,6 +22,7 @@
|
||||
(defn- upsert-addr-content!
|
||||
"Upsert addr+data-seq. Should be functionally equivalent to db-worker/upsert-addr-content!"
|
||||
[db data]
|
||||
(assert db ::upsert-addr-content!)
|
||||
(let [insert (.prepare db "INSERT INTO kvs (addr, content, addresses) values ($addr, $content, $addresses) on conflict(addr) do update set content = $content, addresses = $addresses")
|
||||
insert-many (.transaction ^object db
|
||||
(fn [data]
|
||||
@@ -99,5 +100,4 @@
|
||||
;; $ORIGINAL_PWD used by bb tasks to correct current dir
|
||||
(node-path/join (or js/process.env.ORIGINAL_PWD ".") %))]
|
||||
((juxt node-path/dirname node-path/basename) (resolve-path' graph-dir-or-path)))
|
||||
;; TODO: Reuse with get-db-graphs-dir when there is a db ns that is usable by electron i.e. no better-sqlite3
|
||||
[(node-path/join (os/homedir) "logseq" "graphs") graph-dir-or-path])))
|
||||
[(common-graph/expand-home (common-graph/get-default-graphs-dir)) graph-dir-or-path])))
|
||||
|
||||
2
deps/db/src/logseq/db/frontend/db.cljs
vendored
2
deps/db/src/logseq/db/frontend/db.cljs
vendored
@@ -21,7 +21,7 @@
|
||||
(:db/ident property-entity))))
|
||||
|
||||
(defn private-built-in-page?
|
||||
"Private built-in pages should not be navigable or searchable by users. Later it
|
||||
"Private built-in pages should not be navigable, searchable or editable by users. Later it
|
||||
could be useful to use this for the All Pages view"
|
||||
[page]
|
||||
(cond (entity-util/property? page)
|
||||
|
||||
4
deps/db/src/logseq/db/frontend/db_ident.cljc
vendored
4
deps/db/src/logseq/db/frontend/db_ident.cljc
vendored
@@ -75,9 +75,9 @@
|
||||
(assert (not (re-find #"^(logseq|block)(\.|$)" (name user-namespace)))
|
||||
"New ident is not allowed to use an internal namespace")
|
||||
(if #?(:org.babashka/nbb true
|
||||
:cljs (exists? js/process)
|
||||
;; Use $LOGSEQ_STABLE_IDENTS when we want stable idents e.g. tests
|
||||
:cljs (and (exists? js/process) js/process.env.LOGSEQ_STABLE_IDENTS)
|
||||
:default false)
|
||||
;; Used for contexts where we want repeatable idents e.g. tests and CLIs
|
||||
(keyword user-namespace (normalize-ident-name-part name-string))
|
||||
(let [plugin? (string/starts-with? user-namespace "plugin.class.")
|
||||
suffix (str "-"
|
||||
|
||||
12
deps/db/src/logseq/db/frontend/property.cljs
vendored
12
deps/db/src/logseq/db/frontend/property.cljs
vendored
@@ -26,7 +26,7 @@
|
||||
* :cardinality - property cardinality. Default to one/single cardinality if not set
|
||||
* :hide? - Boolean which hides property when set on a block or exported e.g. slides
|
||||
* :public? - Boolean which allows property to be used by user: add and remove property to blocks/pages
|
||||
and queryable via property and has-property rules
|
||||
and queryable via property and has-property rules. When it's not set, it's the same as false
|
||||
* :view-context - Keyword to indicate which view contexts a property can be
|
||||
seen in when :public? is set. Valid values are :page, :block and :never. Property can
|
||||
be viewed in any context if not set
|
||||
@@ -635,6 +635,11 @@
|
||||
:public? false
|
||||
:hide? true}})))
|
||||
|
||||
(def public-built-in-properties
|
||||
(->> built-in-properties
|
||||
(keep (fn [[k v]] (when (get-in v [:schema :public?]) k)))
|
||||
set))
|
||||
|
||||
(def db-attribute-properties
|
||||
"Internal properties that are also db schema attributes"
|
||||
#{:block/alias :block/tags :block/parent
|
||||
@@ -741,6 +746,11 @@
|
||||
;; Disallow tags or page refs as they would create unreferenceable page names
|
||||
(not (re-find #"^(#|\[\[)" s)))
|
||||
|
||||
(defn built-in-closed-values
|
||||
"Gets :closed-values for given built-in property ident"
|
||||
[ident]
|
||||
(get-in built-in-properties [ident :closed-values]))
|
||||
|
||||
(defn get-closed-property-values
|
||||
[db property-id]
|
||||
(when db
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
(def user-allowed-internal-property-types
|
||||
"Internal property types that users are allowed to store. These aren't available in the UI
|
||||
so these would normally be created via EDN or the API."
|
||||
#{:map})
|
||||
#{:map :json :string})
|
||||
|
||||
(assert (set/subset? user-allowed-internal-property-types internal-built-in-property-types))
|
||||
|
||||
|
||||
46
deps/db/src/logseq/db/sqlite/backup.cljs
vendored
Normal file
46
deps/db/src/logseq/db/sqlite/backup.cljs
vendored
Normal file
@@ -0,0 +1,46 @@
|
||||
(ns logseq.db.sqlite.backup
|
||||
"Shared SQLite backup utilities for Node runtimes."
|
||||
(:require ["node:sqlite" :as node-sqlite]
|
||||
[clojure.string :as string]
|
||||
[goog.object :as gobj]
|
||||
[promesa.core :as p]))
|
||||
|
||||
(defn- resolve-database-sync-ctor
|
||||
[]
|
||||
(or (gobj/get node-sqlite "DatabaseSync")
|
||||
(some-> (gobj/get node-sqlite "default")
|
||||
(gobj/get "DatabaseSync"))
|
||||
(let [default-export (gobj/get node-sqlite "default")]
|
||||
(when (fn? default-export)
|
||||
default-export))
|
||||
(throw (ex-info "node:sqlite DatabaseSync constructor missing"
|
||||
{:module-keys (js->clj (js/Object.keys node-sqlite))}))))
|
||||
|
||||
(def ^:private DatabaseSync
|
||||
(resolve-database-sync-ctor))
|
||||
|
||||
(defn- resolve-sqlite-backup-fn
|
||||
[]
|
||||
(or (gobj/get node-sqlite "backup")
|
||||
(some-> (gobj/get node-sqlite "default")
|
||||
(gobj/get "backup"))
|
||||
(throw (ex-info "node:sqlite backup function missing"
|
||||
{:module-keys (js->clj (js/Object.keys node-sqlite))}))))
|
||||
|
||||
(def ^:private sqlite-backup-fn
|
||||
(resolve-sqlite-backup-fn))
|
||||
|
||||
(defn backup-connection!
|
||||
[^js db path]
|
||||
(sqlite-backup-fn db path))
|
||||
|
||||
(defn backup-db-file!
|
||||
[src-path dst-path]
|
||||
(let [db (new DatabaseSync src-path)]
|
||||
(-> (backup-connection! db dst-path)
|
||||
(p/finally (fn []
|
||||
(try
|
||||
(.close db)
|
||||
(catch :default error
|
||||
(when-not (string/includes? (str error) "database is not open")
|
||||
(throw error)))))))))
|
||||
13
deps/graph-parser/README.md
vendored
13
deps/graph-parser/README.md
vendored
@@ -36,10 +36,8 @@ This step is not needed if you're just running the frontend application.
|
||||
|
||||
### Testing
|
||||
|
||||
Since this library is compatible with cljs and nbb-logseq, tests are run against both languages.
|
||||
|
||||
Nbb tests use [nbb-test-runner](https://github.com/nextjournal/nbb-test-runner).
|
||||
Some basic usage:
|
||||
Testing is done with nbb-logseq and
|
||||
[nbb-test-runner](https://github.com/nextjournal/nbb-test-runner). Some basic usage:
|
||||
|
||||
```
|
||||
# Run all tests
|
||||
@@ -50,13 +48,6 @@ $ pnpm test -H
|
||||
$ pnpm test -i focus
|
||||
```
|
||||
|
||||
ClojureScript tests use https://github.com/Olical/cljs-test-runner. To run tests:
|
||||
```
|
||||
clojure -M:test
|
||||
```
|
||||
|
||||
To see available options that can run specific tests or namespaces: `clojure -M:test --help`
|
||||
|
||||
### Managing dependencies
|
||||
|
||||
The package.json dependencies are just for testing and should be updated if there is
|
||||
|
||||
18
deps/outliner/src/logseq/outliner/core.cljs
vendored
18
deps/outliner/src/logseq/outliner/core.cljs
vendored
@@ -461,6 +461,11 @@
|
||||
(throw (ex-info "Block eid doesn't exist"
|
||||
{:block block})))
|
||||
(when-let [entity (d/entity db eid)]
|
||||
(when (outliner-validate/built-in-entity? entity)
|
||||
(throw (ex-info "Built-in nodes can't be modified"
|
||||
{:type :notification
|
||||
:payload {:message "Built-in nodes can't be modified"
|
||||
:type :error}})))
|
||||
(let [*txs-state (atom [])
|
||||
block' (if (de/entity? block)
|
||||
block
|
||||
@@ -838,15 +843,15 @@
|
||||
(let [top-level-blocks (filter-top-level-blocks db blocks)
|
||||
non-consecutive? (and (> (count top-level-blocks) 1) (seq (ldb/get-non-consecutive-blocks db top-level-blocks)))
|
||||
top-level-blocks* (get-top-level-blocks top-level-blocks non-consecutive?)
|
||||
top-level-blocks (remove :logseq.property/built-in? top-level-blocks*)
|
||||
top-level-blocks (remove outliner-validate/built-in-entity? top-level-blocks*)
|
||||
txs-state (ds/new-outliner-txs-state)
|
||||
block-ids (map (fn [b] [:block/uuid (:block/uuid b)]) top-level-blocks)
|
||||
start-block (first top-level-blocks)
|
||||
end-block (last top-level-blocks)
|
||||
delete-one-block? (or (= 1 (count top-level-blocks)) (= start-block end-block))]
|
||||
|
||||
;; Validate before `when` since top-level-blocks will be empty when deleting one built-in block
|
||||
(when (seq (filter :logseq.property/built-in? top-level-blocks*))
|
||||
;; Validate before `when` since top-level-blocks will be empty when deleting one built-in/internal block
|
||||
(when (seq (filter outliner-validate/built-in-entity? top-level-blocks*))
|
||||
(throw (ex-info "Built-in nodes can't be deleted"
|
||||
{:type :notification
|
||||
:payload {:message "Built-in nodes can't be deleted."
|
||||
@@ -934,6 +939,13 @@
|
||||
:as opts}]
|
||||
{:pre [(seq blocks)]}
|
||||
(when (m/validate block-map-or-entity target-block)
|
||||
(doseq [b blocks]
|
||||
(let [entity (d/entity @conn (:db/id b))]
|
||||
(when (outliner-validate/built-in-entity? entity)
|
||||
(throw (ex-info "Built-in nodes can't be modified"
|
||||
{:type :notification
|
||||
:payload {:message "Built-in nodes can't be modified"
|
||||
:type :error}})))))
|
||||
(let [db @conn
|
||||
top-level-blocks (filter-top-level-blocks db blocks)
|
||||
[target-block sibling?] (get-target-block db top-level-blocks target-block opts)
|
||||
|
||||
9
deps/outliner/src/logseq/outliner/page.cljs
vendored
9
deps/outliner/src/logseq/outliner/page.cljs
vendored
@@ -195,11 +195,9 @@
|
||||
;; TODO: Revisit title cleanup as this was copied from file implementation
|
||||
(defn ^:api sanitize-title
|
||||
[title]
|
||||
(let [title (-> (string/trim title)
|
||||
(text/page-ref-un-brackets!)
|
||||
;; remove `#` from tags
|
||||
(string/replace #"^#+" ""))
|
||||
title (common-util/remove-boundary-slashes title)]
|
||||
(let [title (-> (string/trim title)
|
||||
(text/page-ref-un-brackets!))
|
||||
title (common-util/remove-boundary-slashes title)]
|
||||
title))
|
||||
|
||||
(defn- get-page-by-parent-name
|
||||
@@ -310,6 +308,7 @@
|
||||
class? (or class? (some (fn [t] (= :logseq.class/Tag (:db/ident t))) tags))
|
||||
class-ident-namespace? (and class? class-ident-namespace (string? class-ident-namespace))
|
||||
title (sanitize-title title*)
|
||||
_ (outliner-validate/validate-page-title-no-hashtag title {:node {:block/title title}})
|
||||
types (cond class?
|
||||
#{:logseq.class/Tag}
|
||||
today-journal?
|
||||
|
||||
100
deps/outliner/src/logseq/outliner/property.cljs
vendored
100
deps/outliner/src/logseq/outliner/property.cljs
vendored
@@ -94,6 +94,7 @@
|
||||
[entities property-ident]
|
||||
(throw-error-if-deleting-protected-property (map :db/ident entities) property-ident)
|
||||
(when (= :block/tags property-ident) (throw-error-if-removing-private-tag entities))
|
||||
(outliner-validate/disallow-editing-private-built-in-nodes entities)
|
||||
(throw-error-if-deleting-required-property property-ident))
|
||||
|
||||
(defn- build-property-value-tx-data
|
||||
@@ -184,14 +185,18 @@
|
||||
(and new-type old-ref-type? (not ref-type?))
|
||||
(conj [:db/retract (:db/id property) :db/valueType]))))
|
||||
|
||||
(defn- update-property
|
||||
[conn db-ident property schema {:keys [property-name properties]}]
|
||||
(defn- validate-property-name-update
|
||||
[conn property property-name]
|
||||
(when (and (some? property-name) (not= property-name (:block/title property)))
|
||||
(outliner-validate/validate-page-title property-name {:node property})
|
||||
(outliner-validate/validate-page-title-characters property-name {:node property})
|
||||
(outliner-validate/validate-block-title @conn property-name property)
|
||||
(outliner-validate/validate-property-title property-name))
|
||||
(outliner-validate/validate-property-title property-name)))
|
||||
|
||||
(defn- update-property
|
||||
[conn db-ident property schema {:keys [property-name properties]}]
|
||||
(validate-property-name-update conn property property-name)
|
||||
(outliner-validate/validate-editing-built-in-property property schema)
|
||||
(let [changed-property-attrs
|
||||
;; Only update property if something has changed as we are updating a timestamp
|
||||
(cond-> (->> (dissoc schema :db/cardinality)
|
||||
@@ -219,13 +224,21 @@
|
||||
(mapcat
|
||||
(fn [[property-id v]]
|
||||
(build-property-value-tx-data conn property property-id v)) properties)))
|
||||
many->one? (and (db-property/many? property) (= :one (:db/cardinality schema)))]
|
||||
many->one? (and (db-property/many? property)
|
||||
;; For UI calls, :db/cardinality can have :one and :many values
|
||||
(contains? #{:one :db.cardinality/one} (:db/cardinality schema)))]
|
||||
(when (and many->one? (seq (d/datoms @conn :avet db-ident)))
|
||||
(throw (ex-info "Disallowed many to one conversion"
|
||||
{:type :notification
|
||||
:payload {:message "This property can't change from multiple values to one value because it has existing data."
|
||||
:i18n-key :property.validation/many-to-one
|
||||
:type :warning}})))
|
||||
(when (and (contains? changed-property-attrs :logseq.property/type)
|
||||
(seq (d/datoms @conn :avet db-ident)))
|
||||
(throw (ex-info "Disallowed type change with existing data"
|
||||
{:type :notification
|
||||
:payload {:message "This property's type can't be changed because it has existing data."
|
||||
:type :error}})))
|
||||
(when (seq tx-data)
|
||||
(ldb/transact! conn tx-data {:outliner-op :update-property
|
||||
:property-id (:db/id property)}))
|
||||
@@ -407,14 +420,37 @@
|
||||
v)]
|
||||
(find-or-create-property-value conn property-id v')))))
|
||||
|
||||
(defn- convert-ref-property-values
|
||||
[conn property-id value property-type {:keys [many?]}]
|
||||
(if-not (and many? (or (sequential? value) (set? value)))
|
||||
(convert-ref-property-value conn property-id value property-type)
|
||||
(try
|
||||
(mapv #(convert-ref-property-value conn property-id % property-type) value)
|
||||
(catch :default e
|
||||
(throw (ex-info "Failed to convert many property values"
|
||||
(merge
|
||||
{:property-id property-id
|
||||
:property-type property-type
|
||||
:value value
|
||||
:many? many?
|
||||
:value-shape (cond
|
||||
(vector? value) :vector
|
||||
(set? value) :set
|
||||
(sequential? value) :seq
|
||||
:else :single)}
|
||||
(ex-data e))
|
||||
e))))))
|
||||
|
||||
(defn- throw-error-if-self-value
|
||||
[block value ref?]
|
||||
(when (and ref? (= value (:db/id block)))
|
||||
(throw (ex-info "Can't set this block itself as own property value"
|
||||
{:type :notification
|
||||
:payload {:message "Can't set this block itself as own property value."
|
||||
:i18n-key :property.validation/cant-set-self-value
|
||||
:type :error}}))))
|
||||
(let [values (if (or (sequential? value) (set? value)) value [value])]
|
||||
(when (and ref?
|
||||
(some #(= % (:db/id block)) values))
|
||||
(throw (ex-info "Can't set this block itself as own property value"
|
||||
{:type :notification
|
||||
:payload {:message "Can't set this block itself as own property value."
|
||||
:i18n-key :property.validation/cant-set-self-value
|
||||
:type :error}})))))
|
||||
|
||||
(defn batch-remove-property!
|
||||
[conn block-ids property-id]
|
||||
@@ -467,9 +503,21 @@
|
||||
(ldb/transact! conn txs
|
||||
{:outliner-op :batch-remove-property})))))))
|
||||
|
||||
(defn- validate-batch-set-property
|
||||
[conn block-eids property-id v]
|
||||
(outliner-validate/disallow-editing-private-built-in-nodes (mapv #(d/entity @conn %) block-eids))
|
||||
(when (= property-id :block/tags)
|
||||
(outliner-validate/validate-tags-property @conn block-eids v))
|
||||
(when (= property-id :logseq.property.class/extends)
|
||||
(outliner-validate/validate-extends-property
|
||||
@conn
|
||||
(if (number? v) (d/entity @conn v) v)
|
||||
(map #(d/entity @conn %) block-eids))))
|
||||
|
||||
(defn batch-set-property!
|
||||
"Sets properties for multiple blocks. Automatically handles property value refs.
|
||||
Does no validation of property values."
|
||||
Does no validation of property values. For :many properties, passing a collection
|
||||
replaces existing values in one call, while passing a scalar preserves add-single-value behavior."
|
||||
([conn block-ids property-id v]
|
||||
(batch-set-property! conn block-ids property-id v {}))
|
||||
([conn block-ids property-id v options]
|
||||
@@ -478,24 +526,18 @@
|
||||
(if (nil? v)
|
||||
(batch-remove-property! conn block-ids property-id)
|
||||
(let [block-eids (map ->eid block-ids)
|
||||
_ (when (= property-id :block/tags)
|
||||
(outliner-validate/validate-tags-property @conn block-eids v))
|
||||
_ (validate-batch-set-property conn block-eids property-id v)
|
||||
property (d/entity @conn property-id)
|
||||
blocks (keep #(d/entity @conn %) block-eids)
|
||||
_ (when (= (:db/ident property) :logseq.property.class/extends)
|
||||
(outliner-validate/validate-extends-property
|
||||
@conn
|
||||
(if (number? v) (d/entity @conn v) v)
|
||||
blocks))
|
||||
_ (when (nil? property)
|
||||
(throw (ex-info (str "Property " property-id " doesn't exist yet") {:property-id property-id})))
|
||||
property-type (get property :logseq.property/type :default)
|
||||
many? (= :db.cardinality/many (:db/cardinality property))
|
||||
entity-id? (and (:entity-id? options) (number? v))
|
||||
ref? (contains? db-property-type/all-ref-property-types property-type)
|
||||
default-url-not-closed? (and (contains? #{:default :url} property-type)
|
||||
(not (seq (entity-plus/lookup-kv-then-entity property :property/closed-values))))
|
||||
v' (if (and ref? (not entity-id?))
|
||||
(convert-ref-property-value conn property-id v property-type)
|
||||
(convert-ref-property-values conn property-id v property-type {:many? many?})
|
||||
v)
|
||||
_ (when (nil? v')
|
||||
(throw (ex-info "Property value must be not nil" {:v v})))
|
||||
@@ -505,11 +547,17 @@
|
||||
(if-let [block (d/entity @conn eid)]
|
||||
(let [v' (if (and default-url-not-closed?
|
||||
(not (and (keyword? v) entity-id?)))
|
||||
(do
|
||||
(when (number? v')
|
||||
(throw-error-if-invalid-property-value @conn property v'))
|
||||
(let [v (if (number? v') (:block/title (d/entity @conn v')) v')]
|
||||
(convert-ref-property-value conn property-id v property-type)))
|
||||
(let [normalize-default-url-value
|
||||
(fn [value]
|
||||
(if (number? value)
|
||||
(do
|
||||
(throw-error-if-invalid-property-value @conn property value)
|
||||
(:block/title (d/entity @conn value)))
|
||||
value))
|
||||
value' (if (and many? (or (sequential? v') (set? v')))
|
||||
(mapv normalize-default-url-value v')
|
||||
(normalize-default-url-value v'))]
|
||||
(convert-ref-property-values conn property-id value' property-type {:many? many?}))
|
||||
v')]
|
||||
(throw-error-if-self-value block v' ref?)
|
||||
(throw-error-if-invalid-property-value @conn property v')
|
||||
@@ -665,6 +713,7 @@
|
||||
(prn "property-id: " property-id ", property-name: " property-name))
|
||||
(outliner-validate/validate-page-title k-name {:node {:db/ident db-ident'}})
|
||||
(outliner-validate/validate-page-title-characters k-name {:node {:db/ident db-ident'}})
|
||||
(outliner-validate/validate-property-title k-name {:node {:db/ident db-ident'}})
|
||||
(let [db-id (:db/id properties)
|
||||
opts' (cond-> {:title k-name
|
||||
:properties properties}
|
||||
@@ -672,6 +721,7 @@
|
||||
(assoc :block-uuid (:block/uuid (d/entity db db-id))))
|
||||
tx-data (concat
|
||||
[(sqlite-util/build-new-property db-ident' schema opts')]
|
||||
;; Convert page to property
|
||||
(when db-id
|
||||
[[:db/retract db-id :block/tags :logseq.class/Page]]))]
|
||||
(ldb/transact! conn tx-data
|
||||
|
||||
56
deps/outliner/src/logseq/outliner/validate.cljs
vendored
56
deps/outliner/src/logseq/outliner/validate.cljs
vendored
@@ -9,10 +9,11 @@
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.frontend.class :as db-class]
|
||||
[logseq.db.frontend.entity-util :as entity-util]
|
||||
[logseq.db.frontend.malli-schema :as db-malli-schema]
|
||||
[logseq.db.frontend.property :as db-property]))
|
||||
|
||||
(defn ^:api validate-page-title-characters
|
||||
"Validates characters that must not be in a page title"
|
||||
(defn ^:api validate-page-title-no-hashtag
|
||||
"Validates a page title doesn't include hashtag character"
|
||||
[page-title meta-m]
|
||||
(when (string/includes? page-title "#")
|
||||
(throw (ex-info "Page name can't include \"#\"."
|
||||
@@ -20,7 +21,12 @@
|
||||
{:type :notification
|
||||
:payload {:message "Page name can't include \"#\"."
|
||||
:i18n-key :page.validation/name-no-hash
|
||||
:type :warning}}))))
|
||||
:type :warning}})))))
|
||||
|
||||
(defn ^:api validate-page-title-characters
|
||||
"Validates characters that must not be in a page title"
|
||||
[page-title meta-m]
|
||||
(validate-page-title-no-hashtag page-title meta-m)
|
||||
(when (and (string/includes? page-title ns-util/parent-char)
|
||||
(not (common-date/normalize-date page-title nil)))
|
||||
(throw (ex-info "Page name can't include \"/\"."
|
||||
@@ -154,6 +160,22 @@
|
||||
:i18n-key :property.validation/invalid-name
|
||||
:type :error}}))))))
|
||||
|
||||
(defn validate-editing-built-in-property
|
||||
"Validates if built-in property entity is editable for the given attributes to be updated"
|
||||
[entity attribute-map-to-update]
|
||||
;; Update allowed as needed. Keep this as an allowed list to default to safe editing for built-in entities
|
||||
(let [allowed-attributes #{:logseq.property/hide-empty-value :logseq.property/description}]
|
||||
(when-let [disallowed (and (:logseq.property/built-in? entity)
|
||||
(not-empty (set/difference (set (keys attribute-map-to-update))
|
||||
allowed-attributes)))]
|
||||
(throw (ex-info "Given built-in property's attributes are not editable"
|
||||
(merge
|
||||
{:type :notification
|
||||
:payload {:message "Can't change the given attributes for a built-in property"
|
||||
:type :error}}
|
||||
{:property (:db/ident entity)
|
||||
:disallowed-attributes disallowed}))))))
|
||||
|
||||
(defn- validate-extends-property-have-correct-type
|
||||
"Validates whether given parent and children are classes"
|
||||
[parent-ent child-ents]
|
||||
@@ -231,9 +253,21 @@
|
||||
:property-id :block/tags
|
||||
:property-value v})))))
|
||||
|
||||
(defn built-in-entity?
|
||||
"Returns true when the entity is a built-in. Ideally checking
|
||||
:logseq.property/built-in? would be enough but not all built-in nodes have
|
||||
that property. Covers:
|
||||
- entities marked with :logseq.property/built-in? (built-in pages, classes, properties)
|
||||
- file entities (logseq/config.edn, custom.css, etc.)
|
||||
- entities whose :db/ident belongs to an internal namespace (KV entries, empty-placeholder)"
|
||||
[ent]
|
||||
(or (:logseq.property/built-in? ent)
|
||||
(:file/path ent)
|
||||
(some-> (:db/ident ent) db-malli-schema/internal-ident?)))
|
||||
|
||||
(defn- disallow-tagging-a-built-in-entity
|
||||
[db block-eids & {:keys [delete?]}]
|
||||
(when-let [built-in-ent (some #(when (:logseq.property/built-in? %) %)
|
||||
(when-let [built-in-ent (some #(when (built-in-entity? %) %)
|
||||
(map #(d/entity db %) block-eids))]
|
||||
(throw (ex-info (str (if delete? "Can't remove tag" "Can't add tag")
|
||||
" on built-in " (pr-str (:block/title built-in-ent)))
|
||||
@@ -328,3 +362,17 @@
|
||||
(disallow-tagging-a-built-in-entity db block-eids {:delete? true})
|
||||
(disallow-node-cant-tag-with-private-tags db block-eids v {:delete? true})
|
||||
(disallow-removing-page-tag db block-eids v))
|
||||
|
||||
(defn disallow-editing-private-built-in-nodes
|
||||
"Disallow editing private :built-in nodes. This explicit validation is needed for contexts
|
||||
like CLI and API which allow users to edit any built-in entity whereas the app guards this
|
||||
by not allowing users to navigate to private built-in nodes"
|
||||
[entities]
|
||||
(doseq [entity entities]
|
||||
(when (and (built-in-entity? entity)
|
||||
;; This also checks private status of non-page ents like ents with :kv/value
|
||||
(ldb/private-built-in-page? entity))
|
||||
(throw (ex-info "Built-in private nodes can't be modified"
|
||||
{:type :notification
|
||||
:payload {:message "Built-in private nodes can't be modified"
|
||||
:type :error}})))))
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
[datascript.core :as d]
|
||||
[logseq.db :as ldb]
|
||||
[logseq.db.test.helper :as db-test]
|
||||
[logseq.outliner.core :as outliner-core]))
|
||||
[logseq.outliner.core :as outliner-core]
|
||||
[logseq.common.config :as common-config]))
|
||||
|
||||
(deftest test-delete-block-with-default-property
|
||||
(testing "Delete block with default property hard retracts the block subtree"
|
||||
@@ -50,3 +51,49 @@
|
||||
child' (db-test/find-block-by-content @conn "child")]
|
||||
(is (nil? parent'))
|
||||
(is (nil? child')))))
|
||||
|
||||
(deftest delete-blocks-rejects-built-in-entities
|
||||
(let [conn (db-test/create-conn)]
|
||||
(testing "built-in page is rejected"
|
||||
(let [recycle-page (ldb/get-page @conn common-config/recycle-page-name)]
|
||||
(is (true? (:logseq.property/built-in? recycle-page)))
|
||||
(is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted"
|
||||
(db-test/silence-stderr (outliner-core/delete-blocks! conn [recycle-page] {}))))))
|
||||
|
||||
(testing "built-in idents that are not a class or property like empty-placeholder are rejected"
|
||||
(let [placeholder (d/entity @conn :logseq.property/empty-placeholder)]
|
||||
(is (some? (:block/uuid placeholder)))
|
||||
(is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted"
|
||||
(db-test/silence-stderr (outliner-core/delete-blocks! conn [placeholder] {}))))))
|
||||
|
||||
(testing "file entity is rejected"
|
||||
(let [file (->> (d/datoms @conn :avet :file/path)
|
||||
first
|
||||
:e
|
||||
(d/entity @conn))]
|
||||
(is (some? (:file/path file)))
|
||||
(is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted"
|
||||
(db-test/silence-stderr (outliner-core/delete-blocks! conn [file] {}))))))
|
||||
|
||||
(testing "KV entity is rejected"
|
||||
(let [kv (d/entity @conn :logseq.kv/db-type)]
|
||||
(is (some? (:db/id kv)))
|
||||
(is (thrown-with-msg? js/Error #"Built-in nodes can't be deleted"
|
||||
(db-test/silence-stderr (outliner-core/delete-blocks! conn [kv] {}))))))))
|
||||
|
||||
(deftest save-block-rejects-built-in-entity
|
||||
(let [conn (db-test/create-conn)
|
||||
placeholder (d/entity @conn :logseq.property/description)]
|
||||
(is (thrown-with-msg? js/Error #"Built-in.*can't be modified"
|
||||
(db-test/silence-stderr
|
||||
(outliner-core/save-block! conn {:db/id (:db/id placeholder) :block/title "hacked"}))))))
|
||||
|
||||
(deftest move-blocks-rejects-built-in-entity
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
[{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "target"}]}])
|
||||
placeholder (d/entity @conn :logseq.property/description)
|
||||
target (db-test/find-block-by-content @conn "target")]
|
||||
(is (thrown-with-msg? js/Error #"Built-in.*can't be modified"
|
||||
(db-test/silence-stderr
|
||||
(outliner-core/move-blocks! conn [placeholder] target {:sibling? true}))))))
|
||||
@@ -97,7 +97,19 @@
|
||||
js/Error
|
||||
#"can't include \"/"
|
||||
(outliner-page/create! conn "foo/bar" {}))
|
||||
"Page can't have '/'n title")))
|
||||
"Page can't have '/'n title")
|
||||
|
||||
(is (thrown-with-msg?
|
||||
js/Error
|
||||
#"can't include \"#\""
|
||||
(outliner-page/create! conn "foo#bar" {}))
|
||||
"Page can't have '#' in title")
|
||||
|
||||
(is (thrown-with-msg?
|
||||
js/Error
|
||||
#"can't include \"#\""
|
||||
(outliner-page/create! conn "#tagstyle" {}))
|
||||
"Page can't have leading '#' in title")))
|
||||
|
||||
(deftest delete-page
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
|
||||
@@ -45,6 +45,20 @@
|
||||
(:block/title (d/entity @conn :user.property/p1-2)))
|
||||
"3rd property gets unique ident"))))
|
||||
|
||||
(deftest upsert-property-rejects-type-change-with-existing-data
|
||||
(testing "Changing type is rejected when property has values"
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
[{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "b1" :build/properties {:status "active"}}]}])]
|
||||
(is (thrown-with-msg?
|
||||
js/Error #"Disallowed type change with existing data"
|
||||
(outliner-property/upsert-property! conn :user.property/status {:logseq.property/type :number} {})))))
|
||||
|
||||
(testing "Changing type is allowed when property has no values"
|
||||
(let [conn (db-test/create-conn-with-blocks {:properties {:empty-prop {:logseq.property/type :default}}})]
|
||||
(outliner-property/upsert-property! conn :user.property/empty-prop {:logseq.property/type :number} {})
|
||||
(is (= :number (:logseq.property/type (d/entity @conn :user.property/empty-prop)))))))
|
||||
|
||||
(deftest convert-property-input-string
|
||||
(testing "Convert property input string according to its schema type"
|
||||
(let [test-uuid (random-uuid)]
|
||||
@@ -185,17 +199,68 @@
|
||||
(is (nil? (:user.property/default updated-block)) "Block property is deleted")))
|
||||
|
||||
(deftest batch-set-property!
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
[{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "item 1"}
|
||||
{:block/title "item 2"}]}])
|
||||
block-ids (map #(-> (db-test/find-block-by-content @conn %) :block/uuid) ["item 1" "item 2"])
|
||||
_ (outliner-property/batch-set-property! conn block-ids :logseq.property/order-list-type "number")
|
||||
updated-blocks (map #(db-test/find-block-by-content @conn %) ["item 1" "item 2"])]
|
||||
(is (= ["number" "number"]
|
||||
(map #(db-property/property-value-content (:logseq.property/order-list-type %))
|
||||
updated-blocks))
|
||||
"Property values are batch set")))
|
||||
(testing "Set built-in property values for multiple blocks"
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
[{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "item 1"}
|
||||
{:block/title "item 2"}]}])
|
||||
block-ids (map #(-> (db-test/find-block-by-content @conn %) :block/uuid) ["item 1" "item 2"])
|
||||
_ (outliner-property/batch-set-property! conn block-ids :logseq.property/order-list-type "number")
|
||||
updated-blocks (map #(db-test/find-block-by-content @conn %) ["item 1" "item 2"])]
|
||||
(is (= ["number" "number"]
|
||||
(map #(db-property/property-value-content (:logseq.property/order-list-type %))
|
||||
updated-blocks))
|
||||
"Property values are batch set")))
|
||||
|
||||
(testing "Set custom default-many property values from string vector"
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
{:properties {:user.property/reproducible-steps {:logseq.property/type :default
|
||||
:db/cardinality :db.cardinality/many
|
||||
:logseq.property/public? true}}
|
||||
:pages-and-blocks [{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "target"}]}]})
|
||||
block-id (:block/uuid (db-test/find-block-by-content @conn "target"))]
|
||||
(outliner-property/batch-set-property! conn [block-id] :user.property/reproducible-steps ["Step 1" "Step 2" "Step 3"])
|
||||
(is (= #{"Step 1" "Step 2" "Step 3"}
|
||||
(->> (:user.property/reproducible-steps (d/entity @conn [:block/uuid block-id]))
|
||||
(map db-property/property-value-content)
|
||||
set))
|
||||
"String vector values are persisted as many property values")))
|
||||
|
||||
(testing "Set custom default-many property values from id vector"
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
{:properties {:user.property/reproducible-steps {:logseq.property/type :default
|
||||
:db/cardinality :db.cardinality/many
|
||||
:logseq.property/public? true}}
|
||||
:pages-and-blocks [{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "source"
|
||||
:build/properties {:user.property/reproducible-steps #{"Step 1" "Step 2" "Step 3"}}}
|
||||
{:block/title "target"}]}]})
|
||||
source-values (->> (:user.property/reproducible-steps (db-test/find-block-by-content @conn "source"))
|
||||
(map :db/id)
|
||||
vec)
|
||||
target-id (:block/uuid (db-test/find-block-by-content @conn "target"))]
|
||||
(outliner-property/batch-set-property! conn [target-id] :user.property/reproducible-steps source-values)
|
||||
(is (= #{"Step 1" "Step 2" "Step 3"}
|
||||
(->> (:user.property/reproducible-steps (d/entity @conn [:block/uuid target-id]))
|
||||
(map db-property/property-value-content)
|
||||
set))
|
||||
"Id vector values are persisted as many property values")))
|
||||
|
||||
(testing "Invalid many values throw and don't partially persist"
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
{:properties {:user.property/reproducible-steps {:logseq.property/type :default
|
||||
:db/cardinality :db.cardinality/many
|
||||
:logseq.property/public? true}}
|
||||
:pages-and-blocks [{:page {:block/title "page1"}
|
||||
:blocks [{:block/title "target"}]}]})
|
||||
block-id (:block/uuid (db-test/find-block-by-content @conn "target"))]
|
||||
(is (thrown-with-msg?
|
||||
js/Error
|
||||
#"Schema validation failed"
|
||||
(outliner-property/batch-set-property! conn [block-id] :user.property/reproducible-steps [999999])))
|
||||
(is (nil? (:user.property/reproducible-steps (d/entity @conn [:block/uuid block-id])))
|
||||
"No partial values are persisted on failure"))))
|
||||
|
||||
(deftest status-property-setting-classes
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
@@ -223,6 +288,14 @@
|
||||
(mapv :db/ident (:block/tags (d/entity @conn [:block/uuid project]))))
|
||||
"Doesn't add Task to block when it is already tagged")))
|
||||
|
||||
(deftest batch-set-property-rejects-private-built-in-entity
|
||||
(let [conn (db-test/create-conn)
|
||||
placeholder (d/entity @conn :logseq.property/empty-placeholder)
|
||||
prop (d/entity @conn :logseq.property/description)]
|
||||
(is (thrown-with-msg? js/Error #"Built-in.*can't be modified"
|
||||
(db-test/silence-stderr
|
||||
(outliner-property/batch-set-property! conn [(:db/id placeholder)] (:db/ident prop) "hacked"))))))
|
||||
|
||||
(deftest batch-remove-property!
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
{:classes {:C1 {}}
|
||||
@@ -247,6 +320,14 @@
|
||||
#"Can't remove required"
|
||||
(outliner-property/batch-remove-property! conn [(:db/id (d/entity @conn :user.class/C1))] :logseq.property.class/extends)))))
|
||||
|
||||
(deftest batch-remove-property-rejects-private-built-in-entity
|
||||
(let [conn (db-test/create-conn)
|
||||
placeholder (d/entity @conn :logseq.property/empty-placeholder)
|
||||
prop (d/entity @conn :logseq.property/description)]
|
||||
(is (thrown-with-msg? js/Error #"Built-in.*can't be modified"
|
||||
(db-test/silence-stderr
|
||||
(outliner-property/batch-remove-property! conn [(:db/id placeholder)] (:db/ident prop)))))))
|
||||
|
||||
(deftest add-existing-values-to-closed-values!
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
[{:page {:block/title "page1"}
|
||||
@@ -379,4 +460,4 @@
|
||||
(:db/id (d/entity @conn :user.class/C1)))
|
||||
(is (= [:logseq.class/Root]
|
||||
(:logseq.property.class/extends (db-test/readable-properties (d/entity @conn :user.class/C3))))
|
||||
"Extends property is restored back to Root")))
|
||||
"Extends property is restored back to Root")))
|
||||
@@ -248,6 +248,27 @@
|
||||
(outliner-validate/validate-tags-property-deletion @conn [(:db/id page)] :logseq.class/Page))
|
||||
"Page without parent can't remove #Page")))
|
||||
|
||||
(deftest validate-editing-built-in-property
|
||||
(let [conn (db-test/create-conn-with-blocks
|
||||
{:properties {:myprop {:logseq.property/type :default}}})
|
||||
user-prop (d/entity @conn :user.property/myprop)
|
||||
built-in-prop (d/entity @conn :block/tags)]
|
||||
|
||||
(testing "user property can edit any attribute"
|
||||
(is (nil? (outliner-validate/validate-editing-built-in-property
|
||||
user-prop {:db/cardinality :db.cardinality/many}))))
|
||||
|
||||
(testing "built-in property cannot edit disallowed attribute"
|
||||
(is (thrown-with-msg?
|
||||
js/Error
|
||||
#"not editable"
|
||||
(outliner-validate/validate-editing-built-in-property
|
||||
built-in-prop {:block/title "renamed"}))))
|
||||
|
||||
(testing "built-in property can edit allowed attribute"
|
||||
(is (nil? (outliner-validate/validate-editing-built-in-property
|
||||
built-in-prop {:logseq.property/hide-empty-value true}))))))
|
||||
|
||||
;; Try as many of the validations against a new graph to confirm
|
||||
;; that validations make sense and are valid for a new graph
|
||||
(deftest new-graph-should-be-valid
|
||||
|
||||
6
dist/logseq.js
vendored
Executable file
6
dist/logseq.js
vendored
Executable file
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env node
|
||||
"use strict";
|
||||
|
||||
const path = require("path");
|
||||
|
||||
require(path.resolve(__dirname, "../static/logseq-cli.js"));
|
||||
166
docs/agent-guide/001-logseq-cli.md
Normal file
166
docs/agent-guide/001-logseq-cli.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Logseq CLI Implementation Plan
|
||||
|
||||
Goal: Build a new Logseq CLI in ClojureScript that runs on Node.js and connects to the db-worker-node server.
|
||||
|
||||
Architecture: The CLI is a Node-targeted ClojureScript program built via shadow-cljs and packaged with a small JavaScript launcher.
|
||||
The CLI speaks a simple request and response protocol to the existing db-worker-node HTTP or WebSocket API and exposes high-level subcommands for users.
|
||||
|
||||
Tech Stack: ClojureScript, shadow-cljs :node-script target, Node.js runtime, existing db-worker-node server.
|
||||
|
||||
Related: Relates to docs/agent-guide/task--basic-logseq-cli.md and docs/agent-guide/task--db-worker-nodejs-compatible.md.
|
||||
|
||||
## Problem statement
|
||||
|
||||
We need a new Logseq CLI that is independent of any existing CLI code in the repo.
|
||||
The CLI must run in Node.js, be written in ClojureScript, and connect to the db-worker-node server started from dist/db-worker-node.js.
|
||||
The CLI should provide a stable interface for scripting and troubleshooting, and it should be easy to extend with new commands.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
I will add an integration test that starts db-worker-node on a test port and verifies the CLI can connect and run a simple graph/content request.
|
||||
I will add unit tests for command parsing, configuration precedence, and error formatting.
|
||||
I will add unit tests for the client transport layer to ensure timeouts behave correctly.
|
||||
I will add unit tests for new graph/content commands (parsing, validation, and request mapping).
|
||||
I will add integration tests for graph lifecycle commands and content commands against a real db-worker-node.
|
||||
I will follow @test-driven-development for all behavior changes.
|
||||
NOTE: I will write *all* tests before I add any implementation behavior.
|
||||
|
||||
## Architecture sketch
|
||||
|
||||
The CLI is a Node program that parses flags, loads config, and sends requests to db-worker-node.
|
||||
The db-worker-node server is already built from the :db-worker-node shadow-cljs target and listens on a random localhost TCP port recorded in the lock file.
|
||||
|
||||
ASCII diagram:
|
||||
|
||||
+--------------+ HTTP or WS +---------------------+
|
||||
| logseq-cli | -----------------------> | db-worker-node |
|
||||
| node script | <----------------------- | server on random port |
|
||||
+--------------+ +---------------------+
|
||||
|
||||
## Assumptions
|
||||
|
||||
The db-worker-node server exposes a stable API for a small set of requests needed by the CLI.
|
||||
The CLI always uses localhost and discovers the server port from the lock file.
|
||||
The CLI will use JSON for request and response bodies for ease of scripting.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps.
|
||||
2. Read @test-driven-development guidelines and confirm the red phase will include all CLI tests first.
|
||||
3. Identify existing db-worker-node request handlers and document their request and response shapes.
|
||||
4. Define the initial CLI command surface as a table that includes command, input, output, and errors.
|
||||
5. Decide on transport protocol based on db-worker-node capabilities and document the selection.
|
||||
6. Add a new shadow-cljs build target named :logseq-cli with :target :node-script and a dedicated output file in static/.
|
||||
7. Create a new namespace for the CLI entrypoint in src/main/cli/main.cljs and wire it as the :main for the build.
|
||||
8. Create src/main/cli/config.cljs with config resolution order of CLI flags, env vars, then config file.
|
||||
9. Create src/main/cli/transport.cljs with a small client that can send requests and parse responses.
|
||||
10. Create src/main/cli/commands.cljs with pure functions that map parsed args to transport requests.
|
||||
11. Create src/main/cli/format.cljs that formats success and error output for human and machine usage.
|
||||
12. Add unit tests in src/test/logseq/cli for config precedence, command parsing, and error formatting behavior.
|
||||
13. Add integration tests in src/test/logseq/cli that start db-worker-node and invoke the CLI entrypoint.
|
||||
14. Run tests in red phase with bb dev:test -v and confirm failures are behavior-related.
|
||||
15. Implement the minimal code to make the tests pass and re-run in green phase.
|
||||
16. Refactor for naming and reuse while keeping tests green.
|
||||
17. Document how to build and run the CLI in a short section in README.md.
|
||||
|
||||
## Current status (2026-01-14)
|
||||
|
||||
Implemented:
|
||||
- CLI build target, entrypoint, config resolution, transport, formatting, and command wiring.
|
||||
- Graph commands: list/create/switch/remove/validate/info.
|
||||
- Content commands: add/remove/search/tree.
|
||||
- Unit tests for config/commands/format/transport and integration tests for graph/content commands.
|
||||
- CLI docs moved to `docs/cli/logseq-cli.md` and linked from README.
|
||||
|
||||
Not fully aligned with plan:
|
||||
- Red-first TDD sequence was not strictly followed (some tests added after initial implementation).
|
||||
- README section was replaced by a link to the dedicated doc.
|
||||
- `search` currently queries `:block/title` only (no page name/content search).
|
||||
|
||||
Open follow-ups (optional):
|
||||
- Expand `search` to include page name/content and update tests.
|
||||
- Add any additional graph metadata to `graph-info` beyond `:logseq.kv/graph-created-at` and `:logseq.kv/schema-version`.
|
||||
|
||||
## Command surface definition
|
||||
|
||||
| Command | Input | Output | Errors |
|
||||
| --- | --- | --- | --- |
|
||||
| graph-list | none | list of graphs | server unavailable, timeout |
|
||||
| graph-create | graph name | created graph + set current graph | invalid name, server unavailable |
|
||||
| graph-switch | graph name | switched graph + set current graph | missing graph, server unavailable |
|
||||
| graph-remove | graph name | removal confirmation | missing graph, server unavailable |
|
||||
| graph-validate | graph name or current graph | validation result | missing graph, server unavailable |
|
||||
| graph-info | graph name or current graph | graph metadata/info | missing graph, server unavailable |
|
||||
| add | block/page payload | created block IDs | invalid input, server unavailable |
|
||||
| remove | block/page id or name | removal confirmation | invalid input, server unavailable |
|
||||
| search | text query | matched blocks/pages | invalid input, server unavailable |
|
||||
| tree | block/page id or name | hierarchical tree output | invalid input, server unavailable |
|
||||
|
||||
## Edge cases
|
||||
|
||||
The db-worker-node server is not running or the lock file points to a stale server.
|
||||
The response payload is invalid JSON or missing fields.
|
||||
The request times out or the server closes the connection early.
|
||||
The user passes incompatible flags or unknown commands.
|
||||
The CLI is run on Windows where path and quoting rules differ.
|
||||
Graph commands are invoked without a current graph configured.
|
||||
Content commands are invoked without specifying a graph and no current graph is set.
|
||||
Content commands refer to missing pages/blocks.
|
||||
Graph removal is attempted while a graph is open.
|
||||
|
||||
## Testing commands and expected output
|
||||
|
||||
Run a single unit test in red phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.config-test/test-config-precedence'
|
||||
```
|
||||
|
||||
Expected output includes a failing assertion and ends with a non-zero exit code.
|
||||
|
||||
Run the full unit test suite in green phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.commands-test'
|
||||
```
|
||||
|
||||
Expected output includes 0 failures and 0 errors.
|
||||
|
||||
Run lint and unit tests when all work is complete.
|
||||
|
||||
```bash
|
||||
bb dev:lint-and-test
|
||||
```
|
||||
|
||||
Expected output includes successful linting and tests with exit code 0.
|
||||
|
||||
## Testing Details
|
||||
|
||||
I will add behavior-driven tests that verify the CLI connects to a real db-worker-node process and that each command returns the expected output for valid input.
|
||||
I will keep unit tests focused on pure functions like parsing, formatting, and config resolution, and avoid mocking internal implementation details.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- Add a new shadow-cljs build target for the CLI with a node-script output in static/.
|
||||
- Create a dedicated CLI entrypoint namespace that handles args, logging, and exit codes.
|
||||
- Implement config resolution for flags, env vars, and optional config file.
|
||||
- Implement a transport client with timeouts and explicit error mapping.
|
||||
- Define a small command map with functions that return request objects and output renderers.
|
||||
- Add structured JSON output mode for scripting alongside human-readable output.
|
||||
- Ensure the CLI exits with non-zero status codes on errors.
|
||||
- Document build and run steps, including starting db-worker-node first.
|
||||
- Add graph management commands that map to db-worker thread-apis.
|
||||
- Add graph content commands (add/remove/search/tree) with clear input formats and output.
|
||||
- Persist/resolve a “current graph” for commands that default to current context.
|
||||
|
||||
## Question
|
||||
|
||||
Which exact db-worker-node endpoints and request schemas should the CLI use for graph/content commands.
|
||||
- Answer: all thread-apis are available in http endpoint, check @src/main/frontend/worker/db_worker_node.cljs
|
||||
|
||||
Do we want WebSocket or HTTP as the default transport for the CLI.
|
||||
- HTTP
|
||||
|
||||
Can I consult the clojure-expert and research-agent agents for architecture and reference implementations as required by the planning guidelines.
|
||||
- yes
|
||||
---
|
||||
165
docs/agent-guide/002-logseq-cli-subcommands.md
Normal file
165
docs/agent-guide/002-logseq-cli-subcommands.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# Logseq CLI Subcommands Implementation Plan
|
||||
|
||||
Goal: Replace the CLI argument parser with babashka/cli and expose every command as a subcommand with consistent help and output formats.
|
||||
|
||||
Architecture: The CLI remains a Node-targeted ClojureScript program built via shadow-cljs, but command parsing moves to babashka/cli with an explicit subcommand map. The CLI entrypoint will delegate to per-subcommand parsers and handlers that return a consistent result envelope that is rendered to human, JSON, or EDN output.
|
||||
|
||||
Tech Stack: ClojureScript, babashka/cli, shadow-cljs, Node.js runtime, db-worker-node HTTP API.
|
||||
|
||||
Related: Builds on docs/agent-guide/001-logseq-cli.md.
|
||||
|
||||
## Problem statement
|
||||
|
||||
The current CLI uses clojure.tools.cli with a flat flag set and manual command detection.
|
||||
This limits help text, makes subcommand-specific options awkward, and complicates output formatting consistency.
|
||||
We need to migrate to babashka/cli so that each command is a first-class subcommand with its own help, and so output formats are consistent across all commands.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
I will add unit tests that validate babashka/cli subcommand parsing for every command and its flags.
|
||||
I will add unit tests that assert each subcommand renders help and that top-level help includes all subcommands.
|
||||
I will add unit tests that verify output formatting for human, JSON, and EDN across success and error paths for each subcommand.
|
||||
I will add integration tests that invoke the Node CLI with subcommands and verify consistent output formats for graph and content commands.
|
||||
NOTE: I will write all tests before I add any implementation behavior.
|
||||
|
||||
## Architecture sketch
|
||||
|
||||
+--------------+ HTTP +---------------------+
|
||||
| logseq-cli | -----------------> | db-worker-node |
|
||||
| node script | <----------------- | server on random port |
|
||||
+--------------+ +---------------------+
|
||||
|
||||
## Command and output surface
|
||||
|
||||
The CLI will expose these subcommands and shared output controls.
|
||||
|
||||
| Subcommand | Purpose | Output formats | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| graph list | List graphs | human, json, edn | Replaces graph-list |
|
||||
| graph create | Create graph | human, json, edn | Replaces graph-create |
|
||||
| graph switch | Switch current graph | human, json, edn | Replaces graph-switch |
|
||||
| graph remove | Remove graph | human, json, edn | Replaces graph-remove |
|
||||
| graph validate | Validate graph | human, json, edn | Replaces graph-validate |
|
||||
| graph info | Graph metadata | human, json, edn | Replaces graph-info |
|
||||
| block add | Add blocks | human, json, edn | Replaces add |
|
||||
| block remove | Remove block or page | human, json, edn | Replaces remove |
|
||||
| search | Search graph | human, json, edn | Replaces search |
|
||||
| block tree | Show tree | human, json, edn | Replaces tree |
|
||||
|
||||
The plan assumes a single global output flag that defaults to human, and each subcommand may also accept it.
|
||||
|
||||
## Subcommand map design
|
||||
|
||||
Global options apply to all subcommands and are parsed before subcommand options.
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| --help | Show help | Available at top level and per subcommand. |
|
||||
| --version | Show version | Prints build time and revision. |
|
||||
| --config PATH | Config file path | Defaults to ~/logseq/cli.edn. |
|
||||
| --graph GRAPH | Graph name | Used as current graph. |
|
||||
| --timeout-ms MS | Request timeout | Integer milliseconds. |
|
||||
| --output FORMAT | Output format | One of human, json, edn. |
|
||||
|
||||
Each subcommand uses a nested path and its own options.
|
||||
|
||||
| Subcommand path | Required args | Options | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| graph list | none | --output | Lists all graphs. |
|
||||
| graph create | none | --graph GRAPH, --output | Creates and switches graph. |
|
||||
| graph switch | none | --graph GRAPH, --output | Switches current graph. |
|
||||
| graph remove | none | --graph GRAPH, --output | Removes graph. |
|
||||
| graph validate | none | --graph GRAPH, --output | Validates graph. |
|
||||
| graph info | none | --graph GRAPH, --output | Shows metadata, defaults to config repo if omitted. |
|
||||
| block add | none | --content TEXT, --blocks EDN, --blocks-file PATH, --page PAGE, --parent UUID, --output | Content source is required, with file and text variants. |
|
||||
| block remove | none | --block UUID, --page PAGE, --output | One of block or page is required. |
|
||||
| search | QUERY | --type page|block|tag|property|all, --tag NAME, --case-sensitive, --sort updated-at|created-at, --order asc|desc, --output | Search text is positional and required. Human output columns: ID (db/id), TITLE. Block reference UUIDs in text are resolved recursively up to 10 levels. |
|
||||
| block tree | none | --block UUID, --page PAGE, --format FORMAT, --output | One of block or page is required, and format controls tree rendering. |
|
||||
|
||||
## Plan
|
||||
|
||||
1. Consult the clojure-expert agent about babashka/cli idioms for nested subcommands and help generation.
|
||||
2. Consult the research-agent for a reference implementation of babashka/cli subcommand parsing in ClojureScript, including Node usage.
|
||||
3. Review current CLI documentation in docs/cli/logseq-cli.md and list all existing flags and examples that must be preserved.
|
||||
4. Review the current parser and action mapping in src/main/logseq/cli/commands.cljs and list which options are command-specific versus global.
|
||||
5. Create a babashka/cli command map design and capture it in this document as a table of subcommands, arguments, and defaults.
|
||||
6. Write new unit tests for top-level help output in src/test/logseq/cli/commands_test.cljs that assert subcommand listing and usage text.
|
||||
7. Write new unit tests for each subcommand parse path in src/test/logseq/cli/commands_test.cljs covering required args, missing args, and unknown flags.
|
||||
8. Write new unit tests in src/test/logseq/cli/format_test.cljs that assert human, json, and edn output for success and error results.
|
||||
9. Write new unit tests in src/test/logseq/cli/config_test.cljs for output format precedence between flags, env, and config file.
|
||||
10. Write new integration tests in src/test/logseq/cli/integration_test.cljs that invoke the built CLI with subcommands and verify outputs for at least one graph and one block command in each format.
|
||||
11. Run the new tests to confirm they fail for the current parser and output handling.
|
||||
12. Replace the parser in src/main/logseq/cli/commands.cljs with babashka/cli, using a subcommand map and per-command option specs.
|
||||
13. Update src/main/logseq/cli/main.cljs to route to babashka/cli and return subcommand-specific help when requested.
|
||||
14. Update src/main/logseq/cli/config.cljs to add a unified output format option and ensure json and edn are both supported.
|
||||
15. Update src/main/logseq/cli/format.cljs so that all commands emit consistent human, json, or edn output using a single option path.
|
||||
16. Update docs/cli/logseq-cli.md to document subcommands, shared output flags, and per-subcommand help examples.
|
||||
17. Run the unit test suite with bb dev:test -v 'logseq.cli.commands-test' and confirm 0 failures and 0 errors.
|
||||
18. Run lint and tests with bb dev:lint-and-test and confirm a zero exit code.
|
||||
19. Refactor for naming clarity, shared helpers, and reduced duplication while keeping tests green.
|
||||
|
||||
## Status
|
||||
|
||||
- Completed: Plan tasks 1-10, 12-19.
|
||||
- Skipped: Plan task 11 (red-phase confirmation no longer applicable after parser swap).
|
||||
|
||||
## Edge cases
|
||||
|
||||
Missing subcommand should show top-level help with a non-zero exit code.
|
||||
Unknown subcommands should show a helpful error that includes the available subcommands.
|
||||
Subcommand-specific help should not require a working db-worker-node server.
|
||||
Output format flags should be accepted both at the top level and at subcommand level without conflict.
|
||||
Existing config keys such as :output-format and the legacy --json flag should either be preserved or mapped with a clear deprecation path.
|
||||
Windows quoting should be covered for block add subcommand with multi-word content arguments.
|
||||
|
||||
## Testing commands and expected output
|
||||
|
||||
Run a single failing unit test in red phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.commands-test/test-help-output'
|
||||
```
|
||||
|
||||
Expected output includes a failing assertion about subcommand help text and ends with a non-zero exit code.
|
||||
|
||||
Run the full unit test suite in green phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -r logseq.cli.*
|
||||
```
|
||||
|
||||
Expected output includes 0 failures and 0 errors.
|
||||
|
||||
Run lint and unit tests when all work is complete.
|
||||
|
||||
```bash
|
||||
bb dev:lint-and-test
|
||||
```
|
||||
|
||||
Expected output includes successful linting and tests with exit code 0.
|
||||
|
||||
## Testing Details
|
||||
|
||||
The unit tests will exercise parsing and output formatting behavior without mocking internal parser details.
|
||||
The integration tests will start db-worker-node on a random port and invoke the CLI entrypoint with subcommands to verify end-to-end behavior.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- Replace clojure.tools.cli usage with babashka/cli and define a nested subcommand map for graph and block groups.
|
||||
- Keep global options for server connection and output format and merge them with per-subcommand options.
|
||||
- Normalize output format selection to :human, :json, or :edn and route it through a single formatting function.
|
||||
- Preserve config precedence across flags, env vars, and config file while adding the output format option.
|
||||
- Ensure each subcommand has a help string and usage text generated by babashka/cli.
|
||||
- Keep error envelopes consistent with current :status and :error keys to avoid breaking existing scripts.
|
||||
- Update CLI docs to show subcommand usage and output format examples.
|
||||
- Add a transition note for legacy command names if backward compatibility is required.
|
||||
|
||||
## Question
|
||||
|
||||
Should we keep backwards compatibility for legacy command names like graph-list and add, or require the new subcommand forms only.
|
||||
- Answer: No need to keep backwards compatibility
|
||||
Should we retain the --json flag as an alias for --output json or remove it after a deprecation period.
|
||||
- Answer: remove --json, only keep --output
|
||||
Do we want --output edn and --output json to be accepted at both the top level and per-subcommand level.
|
||||
- Answer: yes, accept at both levels
|
||||
---
|
||||
115
docs/agent-guide/003-db-worker-node-cli-orchestration.md
Normal file
115
docs/agent-guide/003-db-worker-node-cli-orchestration.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# db-worker-node and logseq-cli Orchestration Plan
|
||||
|
||||
Goal: Based on the current `logseq-cli` and `db-worker-node` implementations, refactor db-worker-node to be repo-bound with locking, make logseq-cli fully manage db-worker-node lifecycle, and add server subcommands for management.
|
||||
|
||||
## Background and Current State (from existing code)
|
||||
|
||||
- `db-worker-node` currently accepts `--repo` but it is optional; it can open/switch graphs via `thread-api/create-or-open-db` at runtime. Entrypoint: `src/main/frontend/worker/db_worker_node.cljs`.
|
||||
- `logseq-cli` connects to an existing db-worker-node on localhost using the port recorded in the lock file; it does not start/stop the server. Entrypoint: `src/main/logseq/cli/main.cljs`, `src/main/logseq/cli/commands.cljs`, `src/main/logseq/cli/transport.cljs`.
|
||||
- Tests explicitly start db-worker-node (`src/test/logseq/cli/integration_test.cljs`, `src/test/frontend/worker/db_worker_node_test.cljs`).
|
||||
|
||||
## Requirements
|
||||
|
||||
1. Refactor `db-worker-node`: startup must require `--repo`; on startup it must open or create that graph; it must not switch graphs at runtime; it must create a lock file so a graph can be served by only one db-worker-node instance; it only needs to bind to localhost.
|
||||
2. In `logseq-cli`, all commands requiring `--graph` or any graph operations must connect to or create the corresponding db-worker-node server.
|
||||
3. db-worker-node server must not be started manually; logseq-cli is fully responsible.
|
||||
4. Add `server` subcommand(s) to logseq-cli for managing db-worker-node servers.
|
||||
|
||||
## Scope / Non-goals
|
||||
|
||||
- In scope: db-worker-node startup and lifecycle, repo binding enforcement, lock files, CLI server orchestration, management commands, tests/docs.
|
||||
- Out of scope: changing db-worker core query/write protocol; changing db-worker-node HTTP API semantics beyond repo binding constraints.
|
||||
|
||||
## Proposed Design
|
||||
|
||||
- **Repo-bound server**: db-worker-node opens a single repo at startup and refuses repo changes for the lifetime of the process. It only listens on localhost.
|
||||
- **Lock file**: each repo directory has a lock file to ensure one server per graph. Lock contains metadata for status/cleanup; db-worker-node handles it by default, and logseq-cli handles cases db-worker-node cannot.
|
||||
- **CLI orchestration**: logseq-cli discovers/starts/reuses db-worker-node servers per repo. It is the only entrypoint for starting servers.
|
||||
- **Server subcommands**: add `server list|status|start|stop|restart` (or similar) to manage servers explicitly.
|
||||
|
||||
## Detailed Design
|
||||
|
||||
### 1) db-worker-node: required repo, repo binding, lock file
|
||||
|
||||
Files:
|
||||
- `src/main/frontend/worker/db_worker_node.cljs`
|
||||
- `src/main/frontend/worker/platform/node.cljs` (for data-dir / repo dir resolution)
|
||||
- Optional new helper: `src/main/frontend/worker/db_worker_node_lock.cljs`
|
||||
|
||||
Key changes:
|
||||
- **Argument parsing**: `--repo` becomes required. If missing, print help and exit non-zero. Host binding is restricted to localhost (e.g., `127.0.0.1`) regardless of input.
|
||||
- **Startup flow**: replace `<maybe-open-repo!` with a required `create-or-open-db` for the repo; store `bound-repo`.
|
||||
- **Reject switching**:
|
||||
- In `/v1/invoke`, for repo-scoped thread APIs, validate `args[0]` matches `bound-repo`.
|
||||
- Reject `thread-api/create-or-open-db`, `thread-api/unsafe-unlink-db`, etc. when repo differs.
|
||||
- Return 409/400 with `:repo-mismatch` error shape.
|
||||
- **Lock file**:
|
||||
- Location: inside repo dir (e.g. `~/logseq/cli-graphs/<graph>/db-worker.lock`).
|
||||
- Content: JSON `{repo, pid, host, port, startedAt}`.
|
||||
- Creation: exclusive create (`fs.open` with `wx`) or atomic temp + rename. If exists, fail with “graph already locked”.
|
||||
- Cleanup: delete lock file on stop (`stop!`) and on SIGINT/SIGTERM.
|
||||
- Stale lock: if lock exists but pid is dead, allow replacement (db-worker-node first; CLI can repair when server cannot).
|
||||
|
||||
### 2) logseq-cli: auto start/reuse db-worker-node per repo
|
||||
|
||||
Files:
|
||||
- `src/main/logseq/cli/commands.cljs`
|
||||
- `src/main/logseq/cli/main.cljs`
|
||||
- `src/main/logseq/cli/config.cljs`
|
||||
- New: `src/main/logseq/cli/server.cljs` (process management + lock handling)
|
||||
|
||||
Key changes:
|
||||
- **Repo resolution**: for all graph/content commands, require `--graph` or resolved repo from config; otherwise error.
|
||||
- **Ensure server** (new helper `ensure-server!`):
|
||||
1. Derive data-dir, repo dir, and lock file path from repo.
|
||||
2. If lock file exists, read port/pid; probe `/healthz` + `/readyz`.
|
||||
3. If healthy, reuse existing server; build the connection URL from localhost and the lock file port.
|
||||
4. If unhealthy or stale, attempt to spawn a new server; if db-worker-node cannot handle the lock situation, CLI repairs the lock then retries.
|
||||
5. Spawn via `child_process.spawn`: `./dist/db-worker-node.js --repo <repo> --data-dir <...>`.
|
||||
6. Resolve actual port from the lock file written by db-worker-node.
|
||||
- **Connection URL**: derived from the repo lock file; host is always localhost and the port is always discovered from the lock file.
|
||||
|
||||
### 3) CLI `server` subcommands
|
||||
|
||||
Suggested command group:
|
||||
- `server list`: list servers from lock files (repo, pid, port, status).
|
||||
- `server start --graph <name>`: start server for repo.
|
||||
- `server stop --graph <name>`: stop server (SIGTERM or `/v1/shutdown`).
|
||||
- `server restart --graph <name>`: stop + start.
|
||||
|
||||
Implementation notes:
|
||||
- `start|stop|restart` require `--graph`.
|
||||
- `list` scans data-dir for repo directories, reads lock files, and verifies status.
|
||||
- Consider adding `/v1/shutdown` in db-worker-node for graceful stop.
|
||||
|
||||
## Compatibility / Migration
|
||||
|
||||
- No need to preserve compatibility for existing env vars, config keys, or flags related to db-worker-node or CLI server connectivity; remove them if they are no longer needed.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- **Unit tests**:
|
||||
- CLI: repo resolution, server orchestration logic, lock parsing, error codes (`src/test/logseq/cli/*`).
|
||||
- db-worker-node: repo required, repo mismatch rejection, lock file create/cleanup (`src/test/frontend/worker/db_worker_node_test.cljs`).
|
||||
- **Integration tests**:
|
||||
- CLI runs graph/content commands without manual server startup (`src/test/logseq/cli/integration_test.cljs`).
|
||||
- Concurrent startup of same repo must fail due to lock.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. **db-worker-node binding & lock file**: repo required + repo enforcement + lock creation/cleanup.
|
||||
2. **CLI server module**: `ensure-server!` with lock/health checks and spawning.
|
||||
3. **CLI command updates**: graph/content commands require repo and auto-start server; add `server` subcommands.
|
||||
4. **Tests + docs**: update/extend tests and adjust CLI docs (`docs/cli/logseq-cli.md`).
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should `graph list` require `--graph`? If not, define a “global” server or out-of-band access to data-dir.
|
||||
- Answer: No --graph needed, using 'out-of-band access to data-dir' way
|
||||
2. Lock file format and location: confirm cross-platform expectations (Windows paths/permissions).
|
||||
- lockfile name:`db-worker.lock`,
|
||||
- Location: inside repo dir (e.g. `~/logseq/cli-graphs/<graph>/db-worker.lock`).
|
||||
- only consider linux/macos for now
|
||||
3. Who owns lock cleanup and stale lock handling: primarily db-worker-node; CLI only steps in for cases db-worker-node cannot handle.
|
||||
4. Add `/v1/shutdown` for graceful stop vs. SIGTERM from CLI?
|
||||
5. db-worker-node servers should keep running unless `logseq-cli server stop` is invoked or the process exits unexpectedly; in the latter case, CLI should handle lockfile cleanup on restart.
|
||||
218
docs/agent-guide/004-logseq-cli-verb-subcommands.md
Normal file
218
docs/agent-guide/004-logseq-cli-verb-subcommands.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# Logseq CLI Verb-First Subcommands Implementation Plan
|
||||
|
||||
Goal: Refactor logseq-cli to remove the block subcommand group and replace it with verb-first subcommands for list, add, remove, search, and show.
|
||||
|
||||
Architecture: Keep the babashka/cli dispatch table but reorganize it into verb-first subcommands, and route each verb to typed actions for pages, blocks, tags, and properties through existing db-worker-node thread-apis.
|
||||
|
||||
Tech Stack: ClojureScript, babashka/cli, db-worker-node thread-apis, Datascript queries.
|
||||
|
||||
Related: Builds on docs/agent-guide/003-db-worker-node-cli-orchestration.md and docs/agent-guide/002-logseq-cli-subcommands.md.
|
||||
|
||||
## Problem statement
|
||||
|
||||
The current CLI uses a block subcommand group, which makes the interface noun-first and inconsistent with graph and server commands.
|
||||
|
||||
We need to make the CLI verb-first, so that content operations are consistent with other tooling and easier to extend for new resource types.
|
||||
|
||||
The refactor must preserve existing behaviors for add, remove, search, and tree while adding list commands for pages, tags, and properties, and renaming tree to show.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
I will add unit tests that cover verb-first parsing, group help output, and option validation for list, add, remove, search, and tree.
|
||||
|
||||
I will add unit tests that assert list option parsing for each list subtype and that invalid flag combinations are rejected.
|
||||
|
||||
I will add integration tests that run list, add, remove, search, and tree against a real db-worker-node with a test graph and assert output shapes.
|
||||
|
||||
I will follow @test-driven-development for all behavior changes.
|
||||
|
||||
NOTE: I will write *all* tests before I add any implementation behavior.
|
||||
|
||||
## Command surface
|
||||
|
||||
The block subcommand group is removed and replaced with verb-first subcommands.
|
||||
|
||||
Help output groups commands into two sections, with Graph Inspect and Edit first, and Graph Management last.
|
||||
|
||||
Group names and order:
|
||||
|
||||
| Group | Commands | Order |
|
||||
| --- | --- | --- |
|
||||
| Graph Inspect and Edit | list, add, remove, search, show | First |
|
||||
| Graph Management | graph, server | Last |
|
||||
|
||||
| Command | Subcommand | Purpose |
|
||||
| --- | --- | --- |
|
||||
| list | page | List pages |
|
||||
| list | tag | List tags |
|
||||
| list | property | List properties |
|
||||
| add | block | Add blocks |
|
||||
| add | page | Create page |
|
||||
| remove | block | Remove block |
|
||||
| remove | page | Remove page |
|
||||
| search | none | Search across resources |
|
||||
| show | none | Show block tree |
|
||||
|
||||
Global options remain unchanged and are shared across all commands.
|
||||
|
||||
## List options detail
|
||||
|
||||
The list command should expose a consistent shape across resource types.
|
||||
|
||||
Common list options:
|
||||
|
||||
| Option | Applies to | Purpose | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| --expand | page, tag, property | Include expanded metadata | Maps to existing api-list-* expand behavior. |
|
||||
| --limit N | page, tag, property | Limit results | Implemented in CLI after fetch unless server supports it. |
|
||||
| --offset N | page, tag, property | Offset results | Implemented in CLI after fetch unless server supports it. |
|
||||
| --sort FIELD | page, tag, property | Sort results | Field whitelist per type. |
|
||||
| --order asc|desc | page, tag, property | Sort direction | Defaults to desc. |
|
||||
| --output FORMAT | all | Output format | Existing output handling. |
|
||||
|
||||
List page options:
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| --include-journal | Include journal pages | Default is include all. |
|
||||
| --journal-only | Only journal pages | Requires journal detection in api-list-pages. |
|
||||
| --include-hidden | Include hidden pages | Requires a flag to bypass entity-util/hidden? filtering. |
|
||||
| --updated-after ISO8601 | Filter by updated-at | Compare to :block/updated-at. |
|
||||
| --created-after ISO8601 | Filter by created-at | Compare to :block/created-at. |
|
||||
| --fields FIELD,FIELD | Select output fields | |
|
||||
|
||||
List tag options:
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| --include-built-in | Include built-in classes | Built-in tags are currently included by default, clarify behavior. |
|
||||
| --with-properties | Include class properties | Uses :logseq.property.class/properties when expanded. |
|
||||
| --with-extends | Include class extends | Uses :logseq.property.class/extends when expanded. |
|
||||
| --fields FIELD,FIELD | Select output fields | |
|
||||
|
||||
List property options:
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| --include-built-in | Include built-in properties | Built-in properties are currently included by default, clarify behavior. |
|
||||
| --with-classes | Include property classes | Uses :logseq.property/classes when expanded. |
|
||||
| --with-type | Include property type | Uses :logseq.property/type. |
|
||||
| --fields FIELD,FIELD | Select output fields | |
|
||||
|
||||
List block is removed to avoid overlap with search.
|
||||
|
||||
## Search options detail
|
||||
|
||||
Search has no subcommands and searches across pages, blocks, tags, and properties by default. Human output columns are `ID` (db/id) and `TITLE`.
|
||||
Search and show outputs resolve block reference UUIDs in text. Nested references are resolved recursively up to 10 levels (e.g., `[[<uuid1>]]` → `[[some text [[<uuid2>]]]]`, then `<uuid2>` is also replaced).
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| QUERY | Search text | Required positional argument. |
|
||||
| --type page|block|tag|property|all | Restrict types | Default is all. |
|
||||
| --tag NAME | Restrict to a specific tag | Tag is a class page, e.g. Page, Asset, Task. |
|
||||
| --case-sensitive | Case sensitive search | Default is case-insensitive. |
|
||||
| --sort updated-at|created-at | Sort results | Default is relevance or stable order. |
|
||||
| --order asc|desc | Sort direction | Defaults to desc for time sorts. |
|
||||
|
||||
## Tree options detail
|
||||
|
||||
Show has no subcommands and returns the block tree for a page or block.
|
||||
|
||||
| Option | Purpose | Notes |
|
||||
| --- | --- | --- |
|
||||
| --id ID | Tree root by :db/id | Mutually exclusive with other identifiers. |
|
||||
| --uuid UUID | Tree root by :block/uuid | Mutually exclusive with other identifiers. |
|
||||
| --page-name NAME | Tree root by :block/title for a #Page block | Must be a page. |
|
||||
| --level N | Limit tree depth | N >= 1, default 10. |
|
||||
| --format text|json|edn | Output format | Existing behavior. |
|
||||
|
||||
## Plan
|
||||
|
||||
1. Review current CLI command parsing and action routing in src/main/logseq/cli/commands.cljs to map block group behavior to verb-first commands.
|
||||
2. Add failing unit tests in src/test/logseq/cli/commands_test.cljs for verb-first help output and parse behavior for list, add, remove, search, and show.
|
||||
3. Add failing unit tests that assert list subtype option parsing and validation for list page, list tag, and list property.
|
||||
4. Add failing unit tests that assert search defaults to all types and respects --type and positional text.
|
||||
5. Add failing unit tests that assert show accepts --page-name, --uuid, or --id and rejects missing targets.
|
||||
6. Run bb dev:test -v 'logseq.cli.commands-test/test-parse-args' and confirm failures are about the new verbs and options.
|
||||
7. Update src/main/logseq/cli/commands.cljs to replace block subcommands with verb-first entries and to add list subcommand group.
|
||||
8. Update summary helpers in src/main/logseq/cli/commands.cljs to show group help for list, add, and remove instead of block, and to render help groups as Graph Inspect and Edit first, Graph Management last.
|
||||
9. Update src/main/logseq/cli/main.cljs usage string to reflect the verb-first command surface.
|
||||
10. Add list option specs in src/main/logseq/cli/commands.cljs and update validation to enforce required args and mutually exclusive flags.
|
||||
11. Implement list actions in src/main/logseq/cli/commands.cljs that call existing thread-apis for pages, tags, and properties.
|
||||
12. Implement add page using thread-api/apply-outliner-ops with :create-page in src/main/logseq/cli/commands.cljs.
|
||||
13. Update search logic in src/main/logseq/cli/commands.cljs to query all resource types and honor --type, --tag, --sort, and --order.
|
||||
14. Update show logic in src/main/logseq/cli/commands.cljs to support --id, --uuid, --page-name, and --level.
|
||||
15. Add failing integration tests in src/test/logseq/cli/integration_test.cljs for list page, list tag, list property, add page, remove page, search all, and show.
|
||||
16. Run bb dev:test -v 'logseq.cli.integration-test/test-cli-list-and-search' and confirm failures before implementation.
|
||||
17. Implement behavior for list, add, remove, search, and show until all tests pass.
|
||||
18. Update docs/cli/logseq-cli.md with new verb-first commands and examples.
|
||||
19. Run bb dev:test -r logseq.cli.* and confirm 0 failures and 0 errors.
|
||||
20. Run bb dev:lint-and-test and confirm a zero exit code.
|
||||
|
||||
## Edge cases
|
||||
|
||||
Missing subcommand should return group help for list, add, or remove, and still exit with a non-zero status.
|
||||
|
||||
Unknown subcommands should return a helpful error message that lists the valid subcommands.
|
||||
|
||||
Add block should still default to today’s journal page when no page is provided and no parent is provided.
|
||||
|
||||
Search across all types should avoid duplicate hits when a tag or property is also a page with the same title.
|
||||
|
||||
Show should return a deterministic order based on :block/order.
|
||||
|
||||
## Testing commands and expected output
|
||||
|
||||
Run a single unit test in red phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.commands-test/test-parse-args'
|
||||
```
|
||||
|
||||
Expected output includes failing assertions about the new verb-first commands and ends with a non-zero exit code.
|
||||
|
||||
Run the integration tests in red phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.integration-test/test-cli-list-add-search-show-remove'
|
||||
```
|
||||
|
||||
Expected output includes failing assertions about list and search output and ends with a non-zero exit code.
|
||||
|
||||
Run the full suite in green phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -r logseq.cli.*
|
||||
```
|
||||
|
||||
Expected output includes 0 failures and 0 errors.
|
||||
|
||||
Run lint and tests after all changes.
|
||||
|
||||
```bash
|
||||
bb dev:lint-and-test
|
||||
```
|
||||
|
||||
Expected output includes successful linting and tests with exit code 0.
|
||||
|
||||
## Testing Details
|
||||
|
||||
The unit tests will validate parsing, help output, and option validation for each new verb-first command.
|
||||
|
||||
The integration tests will create a temporary graph, add pages, tags, and properties, and verify list, search, and tree output against db-worker-node behavior.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- Replace block group entries with list, add, remove, search, and show in src/main/logseq/cli/commands.cljs.
|
||||
- Add list subtype specs and validation, including common list options and per-type field filtering in src/main/logseq/cli/commands.cljs.
|
||||
- Extend search to combine page, block, tag, and property queries and to enforce --type and --tag behavior in src/main/logseq/cli/commands.cljs.
|
||||
- Preserve existing add block and remove block behavior while changing only the command paths and option names.
|
||||
- Rename tree to show and add id, uuid, page-name, and level parsing in src/main/logseq/cli/commands.cljs.
|
||||
- Update docs/cli/logseq-cli.md to show new usage and examples.
|
||||
|
||||
## Question
|
||||
|
||||
None.
|
||||
|
||||
---
|
||||
160
docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md
Normal file
160
docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Logseq CLI Output and db-worker-node Log Implementation Plan
|
||||
|
||||
Goal: Improve all logseq-cli human output for clarity and usability, and write db-worker-node logs to <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log with retention of the most recent 7 logs.
|
||||
|
||||
Architecture: Add a dedicated human output formatter with per-command renderers and consistent error messaging in the CLI formatting layer.
|
||||
Add a file-based glogi appender for db-worker-node that writes logs into the graph-specific data directory using dated filenames and retention while keeping console logging behavior explicit and configurable.
|
||||
|
||||
Tech Stack: ClojureScript, babashka/cli, lambdaisland.glogi, Node.js fs/path.
|
||||
|
||||
Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md.
|
||||
|
||||
## Human Output Specification
|
||||
|
||||
Target: plain text, no ANSI colors. Each command has a stable layout and ordering.
|
||||
|
||||
| Command | OK output (human) | Empty output | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| graph list | Table with header `GRAPH` and rows of graph names, followed by `Count: N` | Header + `Count: 0` | Data from `{:graphs [...]}` |
|
||||
| graph create | `Graph created: <graph>` | n/a | Use graph name from action/options |
|
||||
| graph switch | `Graph switched: <graph>` | n/a | Use graph name from action/options |
|
||||
| graph remove | `Graph removed: <graph>` | n/a | Use graph name from action/options |
|
||||
| graph validate | `Graph validated: <graph>` | n/a | Use graph name from action/options |
|
||||
| graph info | Lines: `Graph: <graph>`, `Created at: <relative time>`, `Schema version: <v>` | n/a | Use `:logseq.kv/*` data; show `-` if missing; `Created at` should use the same human-friendly relative format as list outputs |
|
||||
| server list | Table with header `REPO STATUS HOST PORT PID`, rows for servers, followed by `Count: N` | Header + `Count: 0` | Data from `{:servers [...]}` |
|
||||
| server status/start/stop/restart | `Server <status>: <repo>` + details line `Host: <host> Port: <port>` when available | n/a | Use `:status` keyword where present |
|
||||
| list page/tag/property | Table with header (fields vary by command) and rows, followed by `Count: N` | Header + `Count: 0` | Defaults: page/tag/property `ID TITLE UPDATED-AT CREATED-AT` (ID uses `:db/id`); if `:db/ident` present, include `IDENT` column |
|
||||
| add block | `Added blocks: <count> (repo: <repo>)` | n/a | Count = number of blocks submitted |
|
||||
| add page | `Added page: <page> (repo: <repo>)` | n/a | |
|
||||
| remove block | `Removed block: <block-id> (repo: <repo>)` | n/a | Prefer UUID if available |
|
||||
| remove page | `Removed page: <page> (repo: <repo>)` | n/a | |
|
||||
| search | Table with header `TYPE TITLE/CONTENT UUID UPDATED-AT CREATED-AT`, rows in stable order, followed by `Count: N` | Header + `Count: 0` | For block rows use content snippet; for tag/property rows omit timestamps |
|
||||
| show (text) | Raw tree text with `:db/id` as first column, trimmed | n/a | Tree lines should be prefixed by `:db/id` followed by the tree glyphs. Example:<br/><br/>`id1 block1`<br/>`id2 ├── b2`<br/>`id3 │ └── b3`<br/>`id4 ├── b4`<br/>`id5 │ ├── b5`<br/>`id6 │ │ └── b6`<br/>`id7 │ └── b7`<br/>`id8 └── b8`<br/><br/>If a block title spans multiple lines, show the first line on the tree line and indent the remaining lines under the glyph column. Example:<br/><br/>`168 Jan 18th, 2026`<br/>`169 ├── b1`<br/>`173 ├── aaaxx`<br/>`174 ├── block-line1`<br/>` │ block-line2`<br/>`175 └── cccc`<br/><br/>For `--format json|edn`, keep existing structured output |
|
||||
| errors | `Error (<code>): <message>` + optional `Hint: <hint>` line | n/a | Ensure error codes are stable and consistent |
|
||||
|
||||
## Problem statement
|
||||
|
||||
The current logseq-cli human output is mostly raw pr-str output, which is hard to read and inconsistent across commands.
|
||||
Users need clearer, command-specific summaries, stable table formats, and more helpful error messages for CLI usage.
|
||||
The db-worker-node process currently logs only to console, but operational debugging requires a per-graph log file stored under the data directory with simple retention.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
I will add unit tests for human output formatting functions to ensure stable, readable rendering for each command result shape.
|
||||
I will add unit tests for error formatting to ensure consistent human output for common failure cases.
|
||||
I will add an integration test that starts db-worker-node and verifies that a log file is created at <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log.
|
||||
I will add an integration test that exercises a log-producing db-worker-node action and asserts the log file contains the expected log entries.
|
||||
I will follow @test-driven-development for all behavior changes.
|
||||
NOTE: I will write *all* tests before I add any implementation behavior.
|
||||
|
||||
## Architecture sketch
|
||||
|
||||
The CLI outputs structured results and the formatter converts them into human-friendly text based on the command and payload.
|
||||
The db-worker-node daemon configures glogi to append to a per-graph log file under the repo-specific data directory.
|
||||
|
||||
ASCII diagram:
|
||||
|
||||
+-----------------+ format-result +------------------------+
|
||||
| logseq-cli | --------------------> | human output formatter |
|
||||
| result payloads | <-------------------- | command renderers |
|
||||
+-----------------+ +------------------------+
|
||||
|
||||
+------------------+ glogi appender +-------------------------------------+
|
||||
| db-worker-node | -----------------> | <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log |
|
||||
+------------------+ +-------------------------------------+
|
||||
|
||||
## Implementation plan
|
||||
|
||||
1. Use tool(update_plan) to track the full task list and include the @test-driven-development red-green-refactor steps.
|
||||
2. Read @test-driven-development guidelines and confirm the red phase will include all CLI output and log file tests first.
|
||||
3. Review existing CLI output shapes in src/main/logseq/cli/commands.cljs to catalog the current :data payloads by command.
|
||||
4. Review current formatting in src/main/logseq/cli/format.cljs and identify all human output paths that need command-specific rendering.
|
||||
5. Define a human output specification table in docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md that maps each command to its target human output layout.
|
||||
6. Add unit test scaffolding for CLI formatting in src/test/logseq/cli/format_test.cljs (or a new namespace) using representative :status/:data payloads.
|
||||
7. Write a failing unit test for list commands to ensure human output renders a table with a header and row count.
|
||||
8. Write a failing unit test for add/remove commands to ensure human output renders a succinct success line with key identifiers.
|
||||
9. Write a failing unit test for graph management commands to ensure human output includes graph name and status text.
|
||||
10. Write a failing unit test for server commands to ensure human output includes repo, status, host, and port when available.
|
||||
11. Write a failing unit test for search and show commands to ensure human output includes result counts and stable ordering.
|
||||
12. Write a failing unit test for error formatting to ensure error codes and helpful hints are included in human output.
|
||||
13. Add a failing integration test in src/test/frontend/worker/db_worker_node_test.cljs (or a new namespace) that starts db-worker-node and asserts the log file exists at <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log.
|
||||
14. Add a failing integration test that performs a db-worker-node action and asserts at least one log line is appended to the log file.
|
||||
15. Implement a command-aware human formatter in src/main/logseq/cli/format.cljs, using a dispatch on command or data shape.
|
||||
16. Update src/main/logseq/cli/main.cljs to pass command context into the formatter so it can choose the correct renderer.
|
||||
17. Normalize CLI result payloads in src/main/logseq/cli/commands.cljs to include explicit command identifiers where needed for formatting.
|
||||
18. Ensure human output uses consistent spacing, headers, and ordering for list output, and avoids raw EDN dumps in normal cases.
|
||||
19. Add a utility for table rendering with fixed column widths and truncation behavior in src/main/logseq/cli/format.cljs or a new helper namespace.
|
||||
20. Implement db-worker-node log file setup in src/main/frontend/worker/db_worker_node.cljs using lambdaisland.glogi appenders.
|
||||
21. Compute the log path using frontend.worker.db-worker-node-lock/repo-dir and ensure the directory exists before writing.
|
||||
22. Configure glogi to append to <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log and define whether console logging remains enabled.
|
||||
23. Update help text in src/main/frontend/worker/db_worker_node.cljs to document the log file location and log-level flag behavior.
|
||||
24. Update docs/cli/logseq-cli.md with the new human output expectations and any new formatting options.
|
||||
25. Run unit tests in the red phase to confirm failures, then implement minimal changes to make them pass.
|
||||
26. Run bb dev:test -v 'logseq.cli.commands-test' and bb dev:test -v 'frontend.worker.db-worker-node-test' in the green phase.
|
||||
27. Run bb dev:lint-and-test after all changes to validate lint and unit tests.
|
||||
|
||||
## Edge cases
|
||||
|
||||
The repo name contains characters that change the pool directory name, so the log file path must use worker-util/get-pool-name consistently.
|
||||
The data directory is on a filesystem without write permissions, which should surface a clear error message and non-zero exit code.
|
||||
Multiple db-worker-node instances for different repos should not overwrite each other’s log files.
|
||||
The log file should be created even if no requests are served yet and only startup logs are emitted.
|
||||
Human output should remain stable when list results are empty or fields are missing.
|
||||
The human formatter should avoid printing large nested maps by default for search or show results.
|
||||
|
||||
## Testing commands and expected output
|
||||
|
||||
Run a focused unit test during the red phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'logseq.cli.format-test/test-human-output-list'
|
||||
```
|
||||
|
||||
Expected output includes a failing assertion and exits with a non-zero status code.
|
||||
|
||||
Run the db-worker-node log integration test in the green phase.
|
||||
|
||||
```bash
|
||||
bb dev:test -v 'frontend.worker.db-worker-node-test/test-log-file-created'
|
||||
```
|
||||
|
||||
Expected output includes 0 failures and 0 errors.
|
||||
|
||||
Run the full lint and unit test suite when all changes are complete.
|
||||
|
||||
```bash
|
||||
bb dev:lint-and-test
|
||||
```
|
||||
|
||||
Expected output includes successful linting and tests with exit code 0.
|
||||
|
||||
## Testing Details
|
||||
|
||||
I will validate human output formatting by asserting on complete rendered strings for representative payloads instead of inspecting internal formatting helpers.
|
||||
I will validate db-worker-node logging by checking file existence, dated filename format, and that only the most recent 7 log files remain after multiple startups.
|
||||
I will assert that a known log event is present after a startup or invoke action.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- Add a command-aware human output renderer that produces tables, summaries, and success lines based on command and result payloads.
|
||||
- Standardize human error output to include error codes, messages, and actionable hints when possible.
|
||||
- Ensure human output defaults to stable ordering and includes a count line for list and search commands.
|
||||
- Add a table rendering helper with column width limits and truncation rules.
|
||||
- Pass command context through CLI result objects so the formatter can select the correct renderer.
|
||||
- Configure db-worker-node glogi to append logs to <data-dir>/<graph-dir>/db-worker-node-YYYYMMDD.log.
|
||||
- Enforce log retention by keeping only the most recent 7 dated log files per graph directory.
|
||||
- Ensure the log directory exists before log initialization and keep the log file path deterministic.
|
||||
- Document log file location and new human output behavior in CLI documentation.
|
||||
- Keep JSON and EDN outputs unchanged for scripting compatibility.
|
||||
- Preserve existing exit codes and error handling semantics in the CLI.
|
||||
|
||||
## Question
|
||||
|
||||
Should the human output include color or ANSI styling, or should it remain plain text for maximal portability.
|
||||
Answer: Remain plain text for maximal portability.
|
||||
Should db-worker-node log to both console and file, or file-only to avoid duplicate logs in CLI output.
|
||||
Answer: File-only to avoid duplicate logs in CLI output.
|
||||
Is log rotation or size management required for db-worker-node.log, or is simple append-only acceptable.
|
||||
Answer: Use dated log filenames and keep only the most recent 7 log files.
|
||||
|
||||
---
|
||||
83
docs/agent-guide/006-logseq-cli-import-export.md
Normal file
83
docs/agent-guide/006-logseq-cli-import-export.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Logseq CLI Import/Export Plan
|
||||
|
||||
Goal: Add logseq-cli support for import/export with EDN and SQLite formats using the existing db-worker-node server.
|
||||
|
||||
Architecture: Extend logseq-cli command parsing and execution to invoke db-worker-node thread APIs for export and import, with minimal new APIs to handle EDN import and SQLite binary payloads over HTTP.
|
||||
|
||||
Tech Stack: ClojureScript, babashka/cli, db-worker-node HTTP /v1/invoke, datascript, sqlite-export helpers, Node fs/path.
|
||||
|
||||
Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md and docs/agent-guide/003-db-worker-node-cli-orchestration.md.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Import types: edn, sqlite.
|
||||
- Export types: edn, sqlite.
|
||||
- CLI must work against db-worker-node with repo binding and lock file behavior.
|
||||
- Output files must be written to user-specified paths with clear success/error messages.
|
||||
|
||||
## Proposed CLI UX
|
||||
|
||||
Prefer graph-scoped subcommands to keep import/export with graph management:
|
||||
|
||||
- `logseq graph export --type edn --file <path> [--graph <graph>]`
|
||||
- `logseq graph export --type sqlite --file <path> [--graph <graph>]`
|
||||
- `logseq graph import --type edn --input <path> --graph <graph>`
|
||||
- `logseq graph import --type sqlite --input <path> --graph <graph>`
|
||||
|
||||
Notes:
|
||||
- `graph import` only supports importing into a new graph name; it must not overwrite an existing graph.
|
||||
- `--graph` is required for import, and required unless the current graph is set in config for export.
|
||||
|
||||
## Current Capabilities (Baseline)
|
||||
|
||||
- db-worker-node supports `:thread-api/export-edn`, `:thread-api/export-db`, and `:thread-api/import-db`.
|
||||
- logseq-cli can invoke db-worker-node via `src/main/logseq/cli/transport.cljs` and write files with `transport/write-output`.
|
||||
- EDN import exists in the app layer (`frontend.handler.db-based.import`) but not in db-worker-node.
|
||||
- SQLite import in db-worker-node writes the sqlite file, but CLI needs a full reload step to reflect new data.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
1. Review existing CLI command table and action pipeline in `src/main/logseq/cli/commands.cljs` and `src/main/logseq/cli/main.cljs` to locate insertion points for new graph import/export actions.
|
||||
2. Add new CLI specs for import/export flags (type, input, output, export-type, mode) in `src/main/logseq/cli/commands.cljs`.
|
||||
3. Extend the command table with `graph import` and `graph export` entries and ensure help output includes them.
|
||||
4. Add action builders for import/export that validate repo presence, file paths, and allowed types (edn, sqlite).
|
||||
5. Add CLI helpers for reading input files and writing output files in `src/main/logseq/cli/transport.cljs` (or a new helper namespace), keeping the existing `write-output` behavior for EDN and DB files.
|
||||
6. Implement export execution:
|
||||
- EDN: call `thread-api/export-edn` (graph-only) and write EDN file.
|
||||
- SQLite: call a new db-worker-node export API that returns a base64 string or transit-safe binary; decode to Buffer and write `.sqlite` file.
|
||||
7. Implement import execution:
|
||||
- EDN: read EDN file, pass data to a new db-worker-node `thread-api/import-edn` (see below), and return a summary message.
|
||||
- SQLite: read file as Buffer, pass to a new db-worker-node `thread-api/import-sqlite` (or reuse `import-db` with a wrapper that closes/reopens the repo).
|
||||
- Always stop and restart the db-worker-node server from the CLI around import to ensure a clean reload.
|
||||
8. Add db-worker-node thread APIs in `src/main/frontend/worker/db_core.cljs`:
|
||||
- `:thread-api/import-edn` to convert export EDN into tx data via `logseq.db.sqlite.export/build-import` and transact with `:tx-meta` including `::sqlite-export/imported-data? true` so the pipeline rebuilds refs.
|
||||
- `:thread-api/export-db-base64` (or similar) to return a base64 string for SQLite export over HTTP.
|
||||
- `:thread-api/import-db-base64` (or similar) to accept base64 input, close existing sqlite connections, import db data, and reopen the repo (or invoke `:thread-api/create-or-open-db` with `:import-type :sqlite-db`).
|
||||
9. Update db-worker-node server validation (repo binding) if the new thread APIs need special argument shapes.
|
||||
10. Update CLI output formatting in `src/main/logseq/cli/format.cljs` to print concise success lines like `Exported <type> to <path>` and `Imported <type> from <path>`.
|
||||
11. Update documentation in `docs/cli/logseq-cli.md` with new commands, examples, and file format notes.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
- Add CLI parsing tests for `graph import` and `graph export` options in `src/test/logseq/cli/commands_test.cljs` (or a new namespace).
|
||||
- Add integration tests in `src/test/logseq/cli/integration_test.cljs` to:
|
||||
- export EDN and SQLite from a test graph and assert output files exist and are non-empty.
|
||||
- import EDN into a new graph and verify a known page/block exists via CLI `show` or `list`.
|
||||
- import SQLite into a new graph and verify graph metadata or page count.
|
||||
- Add db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` for the new import/export thread APIs (EDN build-import path and base64 DB export/import).
|
||||
- Follow @test-driven-development: write failing tests before implementation.
|
||||
|
||||
## Edge Cases
|
||||
|
||||
- Large SQLite exports may exceed JSON limits if not base64/transit encoded; ensure streaming-safe or chunked base64 handling.
|
||||
- Import should fail fast if the repo is missing and `--graph` is not provided, or if input file does not exist.
|
||||
- SQLite import while the repo is open must close/reopen connections to avoid stale datascript state.
|
||||
- EDN import should validate the export shape and surface readable errors when EDN is invalid or incompatible.
|
||||
- Overwrite behavior should be explicit for SQLite imports to prevent accidental data loss.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. `graph import` only imports into a new graph; it must not overwrite an existing graph.
|
||||
2. No `--mode` flag; both EDN and SQLite imports are replace-style imports.
|
||||
3. CLI always stops and restarts db-worker-node around imports.
|
||||
4. `graph export --type edn` is graph-only for now (no page/view/blocks).
|
||||
@@ -0,0 +1,75 @@
|
||||
# Logseq CLI Thread API Keywords And Command Split Implementation Plan
|
||||
|
||||
Goal: Replace thread-api string usage with keywords, standardize CLI repo/graph option to --graph, and split logseq.cli.commands into per-subcommand namespaces.
|
||||
|
||||
Architecture: Update transport and db-worker-node boundaries to accept keyword methods while still serializing over HTTP. Refactor CLI command parsing into a shared dispatcher plus per-subcommand namespaces under a new command directory. Keep existing CLI behavior and output stable while updating option naming and error hints.
|
||||
|
||||
Tech Stack: ClojureScript, babashka.cli, promesa, Logseq db-worker-node.
|
||||
|
||||
Related: Builds on docs/agent-guide/004-logseq-cli-verb-subcommands.md, docs/agent-guide/005-logseq-cli-output-and-db-worker-node-log.md, docs/agent-guide/006-logseq-cli-import-export.md.
|
||||
|
||||
## Problem statement
|
||||
|
||||
The current CLI and db-worker-node codebase mixes thread-api method strings with keyword-based APIs, which makes it easy to introduce mismatches and reduces consistency with the thread-api macro design. The logseq.cli.commands namespace has grown large and mixes parsing, validation, and execution for multiple command groups, which makes maintenance and ownership difficult.
|
||||
|
||||
## Testing Plan
|
||||
|
||||
I will add unit tests to ensure all CLI thread-api method invocations use keywords and still serialize correctly through transport when invoking db-worker-node. I will add unit tests for command parsing and action building to cover the new per-subcommand namespaces and ensure summaries and help still match expected output. I will update db-worker-node tests to assert keyword method handling for repo validation and allowed non-repo methods. NOTE: I will write *all* tests before I add any implementation behavior.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Review @prompts/review.md to align plan with review expectations.
|
||||
|
||||
2. Enumerate all thread-api string usages in CLI, db-worker-node, and tests using `rg -n -- "\"thread-api/" src/main src/test`.
|
||||
|
||||
3. Enumerate all --graph references in CLI code and tests using `rg -n -- "--graph" src/main src/test` and confirm there are no docs references outside CLI.
|
||||
|
||||
4. Define a small utility in `src/main/logseq/cli/transport.cljs` to normalize thread-api method arguments to keywords for callers while serializing over HTTP as string names.
|
||||
|
||||
5. Add tests in `src/test/logseq/cli/transport_test.cljs` that pass a keyword method to transport invoke and assert the outgoing payload uses the string name and the response handling is unchanged.
|
||||
|
||||
6. Update all CLI calls in `src/main/logseq/cli/commands.cljs` to pass keyword thread-api names to transport invoke and to any action maps that store method identifiers.
|
||||
|
||||
7. Update CLI tests in `src/test/logseq/cli/commands_test.cljs` and `src/test/logseq/cli/integration_test.cljs` to expect keyword thread-api methods where they assert method calls.
|
||||
|
||||
8. Update db-worker-node internal invocation and repo validation in `src/main/frontend/worker/db_worker_node.cljs` to coerce incoming method values to keywords for comparisons and logging, while still accepting string input from JSON payloads.
|
||||
|
||||
9. Update any db-worker-node tests in `src/test/frontend/worker/db_worker_node_test.cljs` that assert method strings to use keyword expectations and verify non-repo method handling.
|
||||
|
||||
10. ~~Replace any --graph option references in CLI formatting and tests by updating `src/main/logseq/cli/format.cljs` and `src/test/logseq/cli/format_test.cljs` to use --repo.~~
|
||||
|
||||
11. Search for any `:graph` option wiring in `src/main/logseq/cli/commands.cljs` and remove CLI option parsing for --graph, including any help or usage text, while preserving graph-specific subcommands like `graph create`.
|
||||
|
||||
12. Introduce a new directory `src/main/logseq/cli/command/` and split `logseq.cli.commands` into per-subcommand namespaces such as `logseq.cli.command.graph`, `logseq.cli.command.server`, `logseq.cli.command.list`, `logseq.cli.command.add`, `logseq.cli.command.remove`, `logseq.cli.command.search`, and `logseq.cli.command.show`.
|
||||
|
||||
13. Create a small shared namespace like `src/main/logseq/cli/command/core.cljs` for global option spec, parsing helpers, summary formatting, and shared validation utilities, ensuring no behavior changes in parsing or summaries.
|
||||
|
||||
14. Update `src/main/logseq/cli/commands.cljs` to become a thin facade that assembles tables from per-subcommand modules, delegates parse-args and build-action to those modules, and exposes execute dispatching without changing public API.
|
||||
|
||||
15. Update `src/main/logseq/cli/main.cljs` requires to match any namespace changes and confirm the CLI usage summary still renders the same command list.
|
||||
|
||||
16. Update tests in `src/test/logseq/cli/commands_test.cljs` to import any moved namespaces or use the facade namespace, and ensure all help summary snapshots still pass.
|
||||
|
||||
17. Run unit tests for CLI and db-worker-node with `bb dev:test -v 'logseq.cli.commands-test'`, `bb dev:test -v 'logseq.cli.transport-test'`, and `bb dev:test -v 'frontend.worker.db-worker-node-test'` and fix failures.
|
||||
|
||||
18. Run the full lint and unit test suite with `bb dev:lint-and-test` after all changes are complete.
|
||||
|
||||
## Testing Details
|
||||
|
||||
The tests will focus on behavior by asserting that CLI invocations still produce the same actions and outputs while enforcing keyword-based thread-api methods and updated --repo hints. The db-worker-node tests will assert that repo validation and non-repo method bypass logic behave correctly when methods are provided as keywords or strings. The command split will be validated by reusing existing parse-args and summary tests to ensure no behavioral regression in user-facing help or dispatch.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
- Normalize thread-api method values at transport and db-worker-node boundaries to accept keywords and serialize as strings over HTTP.
|
||||
- Replace all explicit "thread-api/..." literals in CLI and db-worker-node call sites with :thread-api/... keywords.
|
||||
- Preserve graph subcommands and graph naming semantics while standardizing on :graph options.
|
||||
- Move per-subcommand parsing and execution helpers into `src/main/logseq/cli/command/` namespaces and keep `logseq.cli.commands` as a facade.
|
||||
- Keep action map shapes stable to avoid downstream changes in format or execution.
|
||||
- Update tests to match keyword method expectations and new module layout.
|
||||
- Ensure public CLI output and behavior remain unchanged
|
||||
|
||||
## Question
|
||||
|
||||
~~Resolved: Remove --graph entirely and fail fast on any --graph usage.~~
|
||||
|
||||
---
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user