You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/test-plan.md
+78-21Lines changed: 78 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,16 +11,67 @@ We don't need to test features (and permutations of features) that are covered b
11
11
12
12
## Before releasing the VS Code extension
13
13
14
-
- Go through the required test cases listed below
14
+
- Run at least one local query and MRVA using the existing version of the extension (to generate "old" query history items).
15
+
- Go through the required test cases listed below.
15
16
- Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
16
17
choose to go through some of the Optional Test Cases.
17
-
- Run a query using the existing version of the extension (to generate an "old" query history item)
18
18
19
19
## Required Test Cases
20
20
21
-
### Test Case 1: MRVA - Running a problem path query and viewing results
21
+
### Local databases
22
22
23
-
1. Open the [UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
23
+
#### Test case 1: Download a database from GitHub
24
+
25
+
1. Click "Download Database from GitHub" and enter `angular-cn/ng-nice` and select the javascript language if prompted
26
+
27
+
#### Test case 2: Import a database from an archive
28
+
29
+
1. Obtain a javascript database for `babel/babel`
30
+
- You can do `gh api "/repos/babel/babel/code-scanning/codeql/databases/javascript" -H "Accept: application/zip" > babel.zip` to fetch a database from GitHub.
31
+
2. Click "Choose Database from Archive" and select the file you just downloaded above.
32
+
33
+
### Local queries
34
+
35
+
#### Test case 1: Running a path problem query and viewing results
36
+
37
+
1. Open the [javascript UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
38
+
2. Select the `angular-cn/ng-nice` database (or download it if you don't have one already)
39
+
3. Run a local query.
40
+
4. Once the query completes:
41
+
- Check that the result messages are rendered
42
+
- Check that the paths can be opened and paths are rendered correctly
43
+
- Check that alert locations can be clicked on
44
+
45
+
#### Test case 2: Running a problem query and viewing results
46
+
47
+
1. Open the [javascript UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
48
+
2. Select the `babel/babel` database (or download it if you don't have one already)
49
+
3. Run a local query.
50
+
4. Once the query completes:
51
+
- Check that the result messages are rendered
52
+
- Check that alert locations can be clicked on
53
+
54
+
#### Test case 3: Running a non-probem query and viewing results
55
+
56
+
1. Open the [cpp FunLinesOfCode query](https://github.com/github/codeql/blob/main/cpp/ql/src/Metrics/Functions/FunLinesOfCode.ql).
57
+
2. Select the `google/brotli` database (or download it if you don't have one already)
58
+
3. Run a local query.
59
+
4. Once the query completes:
60
+
- Check that the results table is rendered
61
+
- Check that alert locations can be clicked on
62
+
63
+
#### Test case 3: Can use AST viewer
64
+
65
+
1. Click on any code location from a previous query to open a source file from a database
66
+
2. Open the AST viewing panel and click "View AST"
67
+
3. Once the AST is computed:
68
+
- Check that it can be navigated
69
+
70
+
### MRVA
71
+
72
+
#### Test Case 1: Running a path problem query and viewing results
73
+
74
+
1. Open the [javascript UnsafeJQueryPlugin query](https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql).
24
75
2. Run a MRVA against the following repo list:
25
76
26
77
```json
@@ -47,50 +98,56 @@ choose to go through some of the Optional Test Cases.
47
98
6. Once the query completes:
48
99
- Check that the query history item is updated to show the query status as "complete"
49
100
50
-
### Test Case 2: MRVA - Running a problem query and viewing results
101
+
####Test Case 2: Running a problem query and viewing results
51
102
52
-
1. Open the [ReDoS query](https://github.com/github/codeql/blob/main/javascript/ql/src/Performance/ReDoS.ql).
103
+
1. Open the [javascript ReDoS query](https://github.com/github/codeql/blob/main/javascript/ql/src/Performance/ReDoS.ql).
53
104
2. Run a MRVA against the "Top 10" repositories.
54
-
3. Check that there is a notification message.
105
+
3. Check that a notification message pops up and the results view is opened.
55
106
4. Check the query history. It should:
56
107
- Show that an item has been added to the query history
57
108
- The item should be marked as "in progress".
58
-
5. Once the query starts:
59
-
- Check that a notification is shown with a link to the results view
109
+
5. Once the query completes:
60
110
- Check that the results are rendered with an alert message and a highlighted code snippet:
#### Test Case 3: Running a non-problem query and viewing results
64
115
65
-
1. Open the [FunLinesOfCode query](https://github.com/github/codeql/blob/main/cpp/ql/src/Metrics/Functions/FunLinesOfCode.ql).
116
+
1. Open the [cpp FunLinesOfCode query](https://github.com/github/codeql/blob/main/cpp/ql/src/Metrics/Functions/FunLinesOfCode.ql).
66
117
2. Run a MRVA against a single repository (e.g. `google/brotli`).
67
-
3. Once the query starts:
68
-
- Open the query results
118
+
3. Check that a notification message pops up and the results view is opened.
119
+
4. Check the query history. It should:
120
+
- Show that an item has been added to the query history
121
+
- The item should be marked as "in progress".
122
+
5. Once the query completes:
69
123
- Check that the results show up in a table:
70
-

71
124
72
-
### Test Case 4: MRVA - Interacting with query history
125
+

126
+
127
+
#### Test Case 4: Interacting with query history
73
128
74
129
1. Click a history item (for MRVA):
75
130
- Check that exporting results works
76
131
- Check that sorting results works
77
132
- Check that copying repo lists works
78
-
2. Open the query results directory:
133
+
2.Click "Open Results Directory":
79
134
- Check that the correct directory is opened and there are results in it
80
-
3. View logs
135
+
3.Click "View Logs":
81
136
- Check that the correct workflow is opened
82
137
83
-
### Test Case 5: MRVA - Canceling a variant analysis run
138
+
####Test Case 5: Canceling a variant analysis run
84
139
85
140
Run one of the above MRVAs, but cancel it from within VS Code:
86
141
87
142
- Check that the query is canceled and the query history item is updated.
88
143
- Check that the workflow run is also canceled.
89
144
- Check that any available results are visible in VS Code.
90
145
91
-
### Test Case 6: MRVA - Change to a different colour theme
146
+
### General
147
+
148
+
#### Test case 1: Change to a different colour theme
92
149
93
-
Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
150
+
Open at least one of the above MRVAs and at least one local query, then try changing to a different colour theme and check that everything looks sensible.
0 commit comments