Skip to content

Commit ae0dfd8

Browse files
elenatanasoiuelenatanasoiu
authored
Merge pull request #1901 from github/elena/move-test-scenarios-into-markdown
Move test plan into markdown file
2 parents 9e1da8f + 1a6f532 commit ae0dfd8

4 files changed

Lines changed: 281 additions & 0 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,7 @@ Pre-recorded scenarios are stored in `./src/mocks/scenarios`. However, it's poss
221221

222222
## Releasing (write access required)
223223

224+
1. Go through [our test plan](/extensions/ql-vscode/docs/test-plan.md) to ensure that the extension is working as expected.
224225
1. Double-check the `CHANGELOG.md` contains all desired change comments and has the version to be released with date at the top.
225226
* Go through all recent PRs and make sure they are properly accounted for.
226227
* Make sure all changelog entries have links back to their PR(s) if appropriate.

‎extensions/ql-vscode/docs/images/highlighted-code-snippet.png‎

41.7 KB
Loading

‎extensions/ql-vscode/docs/images/results-table.png‎

24.2 KB
Loading

‎extensions/ql-vscode/docs/test-plan.md‎

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# Test Plan
2+
3+
This document describes the manual test plan for the QL extension for Visual Studio Code.
4+
5+
The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
6+
effort vs value basis).
7+
8+
#### What this doesn't cover
9+
We don't need to test features (and permutations of features) that are covered by automated tests.
10+
11+
### Before releasing the VS Code extension
12+
- Go through the required test cases listed below
13+
- Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
14+
choose to go through some of the Optional Test Cases.
15+
- Run a query using the existing version of the extension (to generate an "old" query history item)
16+
17+
## Required Test Cases
18+
19+
### Pre-requisites
20+
21+
- Flip the `codeQL.canary` flag. This will enable MRVA in the extension.
22+
23+
### Test Case 1: MRVA - Running a problem path query and viewing results
24+
25+
1. Open the [UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
26+
2. Run a MRVA against the following repo list:
27+
```
28+
"test-repo-list": [
29+
"angular-cn/ng-nice",
30+
"apache/hadoop",
31+
"apache/hive"
32+
]
33+
```
34+
3. Check that a notification message pops up and the results view is opened.
35+
4. Check the query history. It should:
36+
- Show that an item has been added to the query history
37+
- The item should be marked as "in progress".
38+
5. Once the query starts:
39+
- Check the results view
40+
- Check the code paths view, including the code paths drop down menu.
41+
- Check that the repository filter box works
42+
- Click links to files/locations on GitHub
43+
- Check that the query history item is updated to show the number of results
44+
6. Once the query completes:
45+
- Check that the query history item is updated to show the query status as "complete"
46+
47+
### Test Case 2: MRVA - Running a problem query and viewing results
48+
49+
1. Open the [ReDoS query](https://github.com/github/codeql/blob/main/javascript/ql/src/Performance/ReDoS.ql).
50+
2. Run a MRVA against the "Top 10" repositories.
51+
3. Check the notification message. It should:
52+
- Show the number of repos that are going to be queried
53+
- Provide a link to the actions workflow
54+
4. Check the query history. It should:
55+
- Show that an item has been added to the query history
56+
- The item should be marked as "in progress".
57+
5. Once the query starts:
58+
- Check that a notification is shown with a link to the results view
59+
- Check that the results are rendered with an alert message and a highlighted code snippet:
60+
![highlighted-code-snippet](images/highlighted-code-snippet.png)
61+
62+
### Test Case 3: MRVA - Running a non-problem query and viewing results
63+
64+
1. Open the [FunLinesOfCode query](https://github.com/github/codeql/blob/main/cpp/ql/src/Metrics/Functions/FunLinesOfCode.ql).
65+
2. Run a MRVA against a single repository (e.g. `google/brotli`).
66+
3. Once the query starts:
67+
- Open the query results
68+
- Check that the results show up in a table:
69+
![results-table](images/results-table.png)
70+
71+
### Test Case 4: MRVA - Interacting with query history
72+
73+
1. Click a history item (for MRVA):
74+
- Check that exporting results works
75+
- Check that sorting results works
76+
- Check that copying repo lists works
77+
2. Open the query directory (containing results):
78+
- Check that the correct directory is opened and there are results in it
79+
3. Open variant analysis on GitHub
80+
- Check that the correct workflow is opened
81+
82+
### Test Case 5: MRVA - Canceling a variant analysis run
83+
84+
Run one of the above MRVAs, but cancel it from within VS Code:
85+
- Check that the query is canceled and the query history item is updated.
86+
- Check that the workflow run is also canceled.
87+
- Check that any available results are visible in VS Code.
88+
89+
### Test Case 6: MRVA - Change to a different colour theme
90+
91+
Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
92+
Are there any components that are not showing up?
93+
94+
## Optional Test Cases
95+
96+
These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA queries.
97+
98+
### Selecting repositories to run on
99+
100+
#### Test case 1: Running a query on a single repository
101+
1. When the repository exists and is public
102+
1. Has a CodeQL database for the correct language
103+
2. Has a CodeQL database for another language
104+
3. Does not have any CodeQL databases
105+
2. When the repository exists and is private
106+
1. Is accessible and has a CodeQL database
107+
2. Is not accessible
108+
3. When the repository does not exist
109+
110+
#### Test case 2: Running a query on a custom repository list
111+
1. The repository list is non-empty
112+
1. All repositories in the list have a CodeQL database
113+
2. Some but not all repositories in the list have a CodeQL database
114+
3. No repositories in the list have a CodeQL database
115+
2. The repository list is empty
116+
117+
#### Test case 3: Running a query on all repositories in an organization
118+
1. The org exists
119+
1. The org contains repositories that have CodeQL databases
120+
2. The org contains repositories of the right language but without CodeQL databases
121+
3. The org contains repositories not of the right language
122+
4. The org contains private repositories that are inaccessible
123+
2. The org does not exist
124+
125+
### Using different types of controller repos
126+
127+
#### Test case 1: Running a query when the controller repository is public
128+
1. Can run queries on public repositories
129+
2. Can not run queries on private repositories
130+
131+
#### Test case 2: Running a query when the controller repository is private
132+
1. Can run queries on public repositories
133+
2. Can run queries on private repositories
134+
135+
#### Test case 3: Running a query when the controller repo exists but you do not have write access
136+
1. Cannot run queries
137+
138+
#### Test case 4: Running a query when the controller repo doesn’t exist
139+
1. Cannot run queries
140+
141+
#### Test case 5: Running a query when the "config field" for the controller repo is not set
142+
1. Cannot run queries
143+
144+
### Query History
145+
146+
This requires running a MRVA query and viewing the query history.
147+
148+
The first test case specifies actions that you can do when the query is first run and is in "pending" state. We start
149+
with this since it has quite a limited number of actions you can do.
150+
151+
#### Test case 1: When variant analysis state is "pending"
152+
1. Starts monitoring variant analysis
153+
2. Cannot open query history item
154+
3. Can delete a query history item
155+
1. Item is removed from list in UI
156+
2. Files on dist are deleted (can get to files using "open query directory")
157+
4. Can sort query history items
158+
1. By name
159+
2. By query date
160+
3. By result count
161+
5. Cannot open query directory
162+
6. Can open query that produced these results
163+
1. When the file still exists and has not moved
164+
2. When the file does not exist
165+
7. Cannot open variant analysis on github
166+
8. Cannot copy repository list
167+
9. Cannot export results
168+
10. Cannot select to create a gist
169+
11. Cannot select to save as markdown
170+
12. Cannot cancel analysis
171+
172+
#### Test case 2: When the variant analysis state is not "pending"
173+
1. Query history is loaded when VSCode starts
174+
2. Handles when action workflow was canceled while VSCode was closed
175+
3. Can open query history item
176+
1. Manually by clicking on them
177+
2. Automatically when VSCode starts (if they were open when VSCode was last used)
178+
4. Can delete a query history item
179+
1. Item is removed from list in UI
180+
2. Files on dist are deleted (can get to files using "open query directory")
181+
5. Can sort query history items
182+
1. By name
183+
2. By query date
184+
3. By result count
185+
6. Can open query directory
186+
7. Can open query that produced these results
187+
1. When the file still exists and has not moved
188+
2. When the file does not exist
189+
8. Can open variant analysis on github
190+
9. Can copy repository list
191+
1. Text is copied to clipboard
192+
2. Text is a valid repository list
193+
10. Can export results
194+
11. Can select to create gist
195+
1. A gist is created
196+
2. The first thing in the gist is a summary
197+
3. Contains a file for each repository with results
198+
4. A popup links you to the gist
199+
12. Can select to save as markdown
200+
1. A directory is created on disk
201+
2. Contains a summary file
202+
3. Contains a file for each repository with results
203+
4. A popup allows you to open the directory
204+
205+
#### Test case 3: When variant analysis state is "in_progress"
206+
1. Starts monitoring variant analysis
207+
1. Ready results are downloaded
208+
2. Can cancel analysis
209+
1. Causes the actions run to be canceled
210+
211+
#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
212+
1. Stops monitoring variant analysis
213+
1. All results are downloaded if state is succeeded
214+
2. Otherwise, ready results are downloaded, if any are available
215+
2. Cannot cancel analysis
216+
217+
### MRVA results view
218+
219+
This requires running a MRVA query and seeing the results view.
220+
221+
#### Test case 1: When variant analysis state is "pending"
222+
1. Can open a results view
223+
2. Results view opens automatically
224+
- When starting variant analysis run
225+
- When VSCode opens (if view was open when VSCode was closed)
226+
3. Results view is empty
227+
228+
#### Test case 2: When variant analysis state is not "pending"
229+
1. Can open a results view
230+
2. Results view opens automatically
231+
1. When starting variant analysis run
232+
2. When VSCode opens (if view was open when VSCode was closed)
233+
3. Can copy repository list
234+
1. Text is copied to clipboard
235+
2. Text is a valid repository list
236+
4. Can export results
237+
1. Only includes repos that you have selected (also see section from query history)
238+
5. Can cancel analysis
239+
6. Can open query file
240+
1. When the file still exists and has not moved
241+
2. When the file does not exist
242+
7. Can open query text
243+
8. Can sort repos
244+
1. By name
245+
2. By results
246+
3. By stars
247+
4. By last commit
248+
9. Can filter repos
249+
10. Shows correct statistics
250+
1. Total number of results
251+
2. Total number of repositories
252+
3. Duration
253+
11. Can see live results
254+
1. Results appear in extension as soon as each query is completed
255+
12. Can view interpreted results (i.e. for a "problem" query)
256+
1. Can view non-path results
257+
2. Can view code paths for "path-problem" queries
258+
13. Can view raw results (i.e. for a non "problem" query)
259+
1. Renders a table
260+
14. Can see skipped repositories
261+
1. Can see repos with no db in a tab
262+
1. Shown warning that explains the tab
263+
2. Can see repos with no access in a tab
264+
1. Shown warning that explains the tab
265+
3. Only shows tab when there are skipped repos
266+
15. Result downloads
267+
1. All results are downloaded automatically
268+
2. Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
269+
3. Only 3 items are downloaded at a time
270+
4. Results for completed queries are still downloaded when
271+
1. Some but not all queries failed
272+
2. The variant analysis was canceled after some queries completed
273+
274+
#### Test case 3: When variant analysis state is in "succeeded" state
275+
1. Can view logs
276+
2. All results are downloaded
277+
278+
#### Test case 4: When variant analysis is in "failed" or "canceled" state
279+
1. Can view logs
280+
1. Results for finished queries are still downloaded.

0 commit comments

Comments
 (0)