Skip to content

test(e2e): add Phase C plans — project create, export jar, delete permanently#1029

Merged
wenytang-ms merged 4 commits into
mainfrom
e2e/phase-c-ui-coverage
Jun 9, 2026
Merged

test(e2e): add Phase C plans — project create, export jar, delete permanently#1029
wenytang-ms merged 4 commits into
mainfrom
e2e/phase-c-ui-coverage

Conversation

@wenytang-ms

@wenytang-ms wenytang-ms commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Phase C of the E2E test-plan coverage push, building on PRs #1027 and #1028.

What this PR adds

Three new YAML plans under test/e2e-plans/ that cover the remaining high-value Java Manager commands not yet exercised by Phase A (lifecycle / new types / file ops / view modes / project explorer) or Phase B (referenced libraries):

Plan Command(s) Steps
java-dep-project-creation.yaml java.project.create 15
java-dep-export-jar.yaml java.view.package.exportJar 14
java-dep-delete-permanent.yaml java.view.package.deleteFilePermanently 16

After this PR, most of the extension's user-facing commands are covered by deterministic E2E plans. The remaining uncovered surface is either trivial (copyFilePath / copyRelativeFilePath — clipboard ops needing an autotest helper) or out of scope (revealFileInOS — no in-VS-Code state to assert).

Design notes

  • project.create uses the No build tools branch — the only project type whose scaffolding lives entirely inside this repo at templates/invisible-project/. Maven / Gradle / Spring / Quarkus all delegate to third-party extensions and are therefore intentionally out of scope. The plan asserts on-disk presence of the scaffolded files (src/App.java, .vscode/settings.json); the vscode.openFolder second-window spawn is intentionally not interacted with, since Playwright is attached to the original window's renderer and the new window is a separate Electron process.
  • exportJar pins java.project.exportJar.targetPath via a user-level settings: block so the wizard's showSaveDialog branch is skipped (getDefaultTask short-circuits to the configured path when it is non-empty). User-level was chosen over workspace-level to side-step an autotest limitation: its workspaceSettings: merge uses JSON.parse on the existing .vscode/settings.json, and the maven fixture's settings file contains JSONC comments. Worth filing as an autotest follow-up.
  • deleteFilePermanently has no plain-click UI affordance for normal local files — the context-menu entry and delete keybinding are both gated on !explorerResourceMoveableToTrash (see package.json menus/keybindings contributions). Invocation is therefore by command id with a tree-selection fallback, the same pattern removeLibrary uses in the classpath plan.

Flake fixes folded in

Two follow-up commits remove verify: LLM screenshot assertions on three expandTreeItem steps in java-dep-refresh-generated-files.yaml and on select-interface in java-dep-new-types.yaml. The LLM verifier is unreliable on already-expanded trees and on quick-picks that advance immediately after selection — it occasionally downgrades a mechanically-successful step to fail, then contradicts itself in the post-mortem analysis. The deterministic verifyTreeItem / fillQuickInput / file-creation steps that follow each affected action provide authoritative ground truth, so the LLM assertion was pure redundancy with a non-trivial false-negative rate.

Verification

Locally green:

Auto-refresh targeted (#914):     16/16
Build Lifecycle:                  26/26
Classpath / Referenced Libraries: 33/33
Permanent Delete (NEW):           16/16
Export Jar (NEW):                 14/14
File Operations:                  41/41
New File Types:                   75/75
Create Project (NEW):             15/15
Project Explorer:                 21/21
Refresh Generated Files (#914):   21/21
View Modes & Refresh:             30/30

Total: 11 plans / 308 steps — 0 failures, 0 crashes

CI auto-discovers plans via the discover-plans matrix in .github/workflows/e2eUI.yml, so no workflow changes are required.

…manently

Adds three new E2E test plans that cover the remaining high-value
Java Manager commands not yet exercised by Phase A or Phase B:

  - test/e2e-plans/java-dep-project-creation.yaml
      Covers java.project.create — the flagship "Java: Create Java
      Project..." wizard, exercised on the "No build tools" branch
      (the only project type whose scaffolding lives in this repo;
      every other type delegates to a third-party extension).
      Asserts on-disk presence of the scaffolded template files.

  - test/e2e-plans/java-dep-export-jar.yaml
      Covers java.view.package.exportJar — the multi-step Export Jar
      wizard. Pins java.project.exportJar.targetPath via user settings
      so the showSaveDialog branch is skipped, then asserts the jar
      file exists at the configured target path after the Jdtls
      generation step finishes.

  - test/e2e-plans/java-dep-delete-permanent.yaml
      Covers java.view.package.deleteFilePermanently — the non-trash
      branch of file removal. Has no plain-click UI affordance for
      regular local files (the context menu and delete keybinding
      are both gated on !explorerResourceMoveableToTrash), so the
      command is invoked directly by id with tree-selection fallback,
      same pattern as removeLibrary in the classpath plan.

All three plans pass locally (45/45 new steps). Full suite remains
green across 11 plans / 308 steps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
wenytang-ms and others added 2 commits June 9, 2026 10:24
The LLM screenshot check on expand-existing-pkg / expand-source-root /
expand-project is inherently flaky on already-expanded trees — the
recent CI run downgraded a mechanically-successful expand from pass to
fail, then immediately contradicted itself in the post-mortem analysis
('was already expanded before the action and remains expanded after').

The deterministic verifyTreeItem steps that follow each expand provide
authoritative ground truth: if the expand didn't happen, the child item
won't be visible. So the LLM verify lines are pure redundancy with no
upside and a non-trivial false-negative rate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Same flake pattern as e633cd0: the LLM screenshot check is unreliable
when a quick-pick advances immediately after a selection. On Windows CI
the LLM saw the second wizard page (source-folder prompt) and concluded
'Interface not selected', when in fact the wizard had advanced exactly
because Interface was selected.

The downstream fillQuickInput steps and final file-creation assertion
provide deterministic verification of the whole flow — if Interface
hadn't been picked, the file would have been created with the wrong
extension or template.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Phase C E2E YAML plans to expand deterministic coverage of Java Dependency’s high-value commands (project creation, export jar, and permanent delete), while also reducing flakiness in a couple of existing plans by removing unreliable verify: screenshot assertions.

Changes:

  • Add new E2E plan for java.project.create using the “No build tools” (invisible project) path and on-disk assertions.
  • Add new E2E plan for java.view.package.exportJar by pinning java.project.exportJar.targetPath and verifying the jar is produced.
  • Add new E2E plan for java.view.package.deleteFilePermanently using command invocation + selection fallback and verifying deletion on disk and in-tree.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test/e2e-plans/java-dep-refresh-generated-files.yaml Removes flaky verify: on expand steps; relies on deterministic verifyTreeItem checks.
test/e2e-plans/java-dep-new-types.yaml Removes a flaky verify: on quick-pick selection; relies on downstream deterministic assertions.
test/e2e-plans/java-dep-project-creation.yaml New plan covering java.project.create (No build tools) with filesystem-based verification.
test/e2e-plans/java-dep-export-jar.yaml New plan covering export-jar wizard end-to-end with pinned target path and jar existence verification.
test/e2e-plans/java-dep-delete-permanent.yaml New plan covering permanent delete via command id + selection fallback, verifying disk + tree removal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/e2e-plans/java-dep-export-jar.yaml Outdated
Comment thread test/e2e-plans/java-dep-project-creation.yaml Outdated
Two header comments were internally inconsistent with the plan body:

1. java-dep-export-jar.yaml line 21-22 said 'we set it via
   workspaceSettings' but the setup block actually uses user-level
   \settings:\. The detailed rationale further down already explains
   the user-level choice; the upstream summary now matches.

2. java-dep-project-creation.yaml line 35-36 said the template ships
   'plus an empty lib/ ensure-dir'. The template directory in the repo
   contains only README.md, .vscode/settings.json and src/App.java —
   the empty lib/ is created at runtime by scaffoldSimpleProject via
   \se.ensureDir(path.join(projectRoot, 'lib'))\ (projectController.ts:185),
   not by the template copy. Reworded to make the distinction explicit.

Doc-only changes; YAML behaviour and assertions unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@wenytang-ms wenytang-ms merged commit 3858fdd into main Jun 9, 2026
56 of 58 checks passed
@wenytang-ms wenytang-ms deleted the e2e/phase-c-ui-coverage branch June 9, 2026 05:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants