mirror of
https://github.com/anomalyco/opencode.git
synced 2026-02-01 22:48:16 +00:00
test(app): smoke tests spec
This commit is contained in:
Adam
253
specs/08-app-e2e-smoke-suite.md
Normal file
253
specs/08-app-e2e-smoke-suite.md
Normal file
@@ -0,0 +1,253 @@
|
||||
## App E2E Smoke Suite (CI)
|
||||
|
||||
Implement a small set of high-signal, low-flake Playwright tests to run in CI.
|
||||
|
||||
These tests are intended to catch regressions in the “core shell” of the app (navigation, dialogs, prompt UX, file viewer, terminal), without relying on model output.
|
||||
|
||||
---
|
||||
|
||||
### Summary
|
||||
|
||||
Add 6 smoke tests to `packages/app/e2e/`:
|
||||
|
||||
- Settings dialog: open, switch tabs, close
|
||||
- Prompt slash command: `/open` opens the file picker dialog
|
||||
- Prompt @mention: `@<file>` inserts a file pill token
|
||||
- Model picker: open model selection and choose a model
|
||||
- File viewer: open a known file and assert contents render
|
||||
- Terminal: open terminal, verify Ghostty mounts, create a second terminal
|
||||
|
||||
---
|
||||
|
||||
### Progress
|
||||
|
||||
- [x] 1. Settings dialog open / switch / close (`packages/app/e2e/settings.spec.ts`)
|
||||
- [ ] 2. Prompt slash command path: `/open` opens file picker
|
||||
- [ ] 3. Prompt @mention inserts a file pill token
|
||||
- [ ] 4. Model selection UI works end-to-end
|
||||
- [ ] 5. File viewer renders real file content
|
||||
- [ ] 8. Terminal init + create new terminal
|
||||
|
||||
---
|
||||
|
||||
### Goals
|
||||
|
||||
- Tests run reliably in CI using the existing local runner (`packages/app/script/e2e-local.ts`).
|
||||
- Cover “wiring” regressions across UI + backend APIs:
|
||||
- dialogs + command routing
|
||||
- prompt contenteditable parsing
|
||||
- file search + file read + code viewer render
|
||||
- terminal open + pty creation + Ghostty mount
|
||||
- Avoid assertions that depend on LLM output.
|
||||
- Keep runtime low (these should be “smoke”, not full workflows).
|
||||
|
||||
---
|
||||
|
||||
### Non-goals
|
||||
|
||||
- Verifying complex model behavior, streaming correctness, or tool call semantics.
|
||||
- Testing provider auth flows (CI has no secrets).
|
||||
- Testing share, MCP, or LSP download flows (disabled in the e2e runner).
|
||||
|
||||
---
|
||||
|
||||
### Current State
|
||||
|
||||
Existing tests in `packages/app/e2e/` already cover:
|
||||
|
||||
- Home renders + server picker opens
|
||||
- Directory route redirects to `/session`
|
||||
- Sidebar collapse/expand
|
||||
- Command palette opens/closes
|
||||
- Basic session open + prompt input + (optional) prompt/reply flow
|
||||
- File open via palette (but shallow assertion: tab exists)
|
||||
- Terminal panel toggles (but doesn’t assert Ghostty mounted)
|
||||
- Context panel open
|
||||
|
||||
We want to add a focused smoke layer that increases coverage of the most regression-prone UI paths.
|
||||
|
||||
---
|
||||
|
||||
### Proposed Tests
|
||||
|
||||
All tests should use the shared fixtures in:
|
||||
|
||||
- `packages/app/e2e/fixtures.ts` (for `sdk`, `directory`, `gotoSession`)
|
||||
- `packages/app/e2e/utils.ts` (for `modKey`, `promptSelector`, `terminalToggleKey`)
|
||||
|
||||
Prefer creating new spec files rather than overloading existing ones, so it’s easy to run these tests as a group via grep.
|
||||
|
||||
Suggested file layout:
|
||||
|
||||
- `packages/app/e2e/settings.spec.ts`
|
||||
- `packages/app/e2e/prompt-slash-open.spec.ts`
|
||||
- `packages/app/e2e/prompt-mention.spec.ts`
|
||||
- `packages/app/e2e/model-picker.spec.ts`
|
||||
- `packages/app/e2e/file-viewer.spec.ts`
|
||||
- `packages/app/e2e/terminal-init.spec.ts`
|
||||
|
||||
Name each test with a “smoke” prefix so CI can run only this suite if needed.
|
||||
|
||||
#### 1) Settings dialog open / switch / close
|
||||
|
||||
Purpose: catch regressions in dialog infra, settings rendering, tabs.
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Open settings via keybind (preferred for stability): `await page.keyboard.press(`${modKey}+Comma`)`.
|
||||
3. Assert dialog visible (`page.getByRole('dialog')`).
|
||||
4. Click the "Shortcuts" tab (role `tab`, name "Shortcuts").
|
||||
5. Assert shortcuts view renders (e.g. the search field placeholder or reset button exists).
|
||||
6. Close with `Escape` and assert dialog removed.
|
||||
|
||||
Notes:
|
||||
|
||||
- If `Meta+Comma` / `Control+Comma` key name is flaky, fall back to clicking the sidebar settings icon.
|
||||
- Favor role-based selectors over brittle class selectors.
|
||||
- If `Escape` doesn’t dismiss reliably (tooltips can intercept), fall back to clicking the dialog overlay.
|
||||
|
||||
Implementation: `packages/app/e2e/settings.spec.ts`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Settings dialog opens reliably.
|
||||
- Switching to Shortcuts tab works.
|
||||
- Escape closes the dialog.
|
||||
|
||||
#### 2) Prompt slash command path: `/open` opens file picker
|
||||
|
||||
Purpose: validate contenteditable parsing + slash popover + builtin command dispatch (distinct from `mod+p`).
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Click prompt (`promptSelector`).
|
||||
3. Type `/open`.
|
||||
4. Press `Enter` (while slash popover is active).
|
||||
5. Assert a dialog appears and contains a textbox (the file picker search input).
|
||||
6. Close dialog with `Escape`.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- `/open` triggers `file.open` and opens `DialogSelectFile`.
|
||||
|
||||
#### 3) Prompt @mention inserts a file pill token
|
||||
|
||||
Purpose: validate the most fragile prompt behavior: structured tokens inside contenteditable.
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Focus the prompt.
|
||||
3. Type `@packages/app/package.json`.
|
||||
4. Press `Tab` to accept the active @mention suggestion.
|
||||
5. Assert a pill element is inserted:
|
||||
- `page.locator('[data-component="prompt-input"] [data-type="file"][data-path="packages/app/package.json"]')` exists.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- A file pill is inserted and has the expected `data-*` attributes.
|
||||
- Prompt editor remains interactable (e.g. typing a trailing space works).
|
||||
|
||||
#### 4) Model selection UI works end-to-end
|
||||
|
||||
Purpose: validate model list rendering, selection wiring, and prompt footer updating.
|
||||
|
||||
Implementation approach:
|
||||
|
||||
- Use `/model` to open the model selection dialog (builtin command).
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Focus prompt, type `/model`, press `Enter`.
|
||||
3. In the model dialog, use the search field to filter to a deterministic model, e.g. `qwen3-coder`.
|
||||
4. Select the first matching model.
|
||||
5. Assert dialog closed.
|
||||
6. Assert the prompt footer now shows the chosen model name.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- A model can be selected without requiring provider auth.
|
||||
- The prompt footer reflects the new selection.
|
||||
|
||||
#### 5) File viewer renders real file content
|
||||
|
||||
Purpose: ensure file search + open + file.read + code viewer render all work.
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Open file picker (either `mod+p` or `/open`).
|
||||
3. Search for `packages/app/package.json`.
|
||||
4. Click the matching file result.
|
||||
5. Assert the code viewer contains a known substring:
|
||||
- `"name": "@opencode-ai/app"`.
|
||||
6. Optionally assert the file tab is active and visible.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Code view shows expected content (not just “tab exists”).
|
||||
|
||||
#### 8) Terminal init + create new terminal
|
||||
|
||||
Purpose: ensure terminal isn’t only “visible”, but actually mounted and functional.
|
||||
|
||||
Steps:
|
||||
|
||||
1. `await gotoSession()`.
|
||||
2. Open terminal with `terminalToggleKey` (currently `Control+Backquote`).
|
||||
3. Assert terminal container exists and is visible: `[data-component="terminal"]`.
|
||||
4. Assert Ghostty textarea exists: `[data-component="terminal"] textarea`.
|
||||
5. Create a new terminal via keybind (`terminal.new` is `ctrl+alt+t`).
|
||||
6. Assert terminal tab count increases to 2.
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Ghostty mounts (textarea present).
|
||||
- Creating a new terminal results in a second tab.
|
||||
|
||||
---
|
||||
|
||||
### CI Stability + Flake Avoidance
|
||||
|
||||
These tests run with `fullyParallel: true` in `packages/app/playwright.config.ts`. Keep them isolated and deterministic.
|
||||
|
||||
- Avoid ordering-based assertions: never assume a “first” session/project/file is stable unless you filtered by unique text.
|
||||
- Prefer deterministic targets:
|
||||
- use `packages/app/package.json` rather than bare `package.json` (multiple hits possible)
|
||||
- filter model search to a specific id/name (e.g. `qwen3-coder`)
|
||||
- Prefer robust selectors:
|
||||
- role selectors: `getByRole('dialog')`, `getByRole('textbox')`, `getByRole('tab')`
|
||||
- stable data attributes already present: `promptSelector`, `[data-component="terminal"]`
|
||||
- Keep tests local and fast:
|
||||
- do not submit prompts that require real model replies
|
||||
- avoid `page.waitForTimeout`; use `expect(...).toBeVisible()` and `expect.poll` when needed
|
||||
- Watch for silent UI failures:
|
||||
- capture `page.on('pageerror')` and fail test if any are emitted
|
||||
- optionally capture console errors (`page.on('console', ...)`) and fail on `type==='error'`
|
||||
- Cleanup:
|
||||
- these tests should not need to create sessions
|
||||
- if a test ever creates sessions or PTYs directly, clean up with SDK calls in `finally`
|
||||
|
||||
---
|
||||
|
||||
### Validation Plan
|
||||
|
||||
Run locally:
|
||||
|
||||
- `cd packages/app`
|
||||
- `bun run test:e2e:local -- --grep smoke`
|
||||
|
||||
Verify:
|
||||
|
||||
- all new tests pass consistently across multiple runs
|
||||
- overall e2e suite time does not increase significantly
|
||||
|
||||
---
|
||||
|
||||
### Open Questions
|
||||
|
||||
- Should we add a small helper in `packages/app/e2e/utils.ts` for “type into prompt contenteditable” to reduce duplication?
|
||||
- Do we want to gate these smoke tests with a dedicated `@smoke` naming convention (or `test.describe('smoke', ...)`) so CI can target them explicitly?
|
||||
Reference in New Issue
Block a user