Skip to content

Commit 3ea7fdc

Browse files
ci: wire framework-checks, generation-retention and napi-fuzz into the witness matrix
All three were local-only, so nothing caught their regressions. framework-checks had silently stopped testing astro: its installs used --legacy-peer-deps (the fork's 6.0.3-bridge.N prerelease satisfies no ^6 peer range), which also dropped @astrojs/check's own peers, so astro check stopped at its install prompt for both TNB and stock and the parity compare passed on two identical prompts. Installs now resolve against stock typescript (STOCK_TSSERVER_PATH's package, which replaces the private npm pack) and node_modules/typescript is relinked per run; framework versions are pinned and a changed pin reinstalls; the astro cases fail when no Result line was printed; the TNB side links the real package root, so the isolated CI tools copy works. TNB_FW_CACHE had no setter and is gone. generation-retention defaults --tsserver to lib/tsserver.js and the stock calibration to STOCK_TSSERVER_PATH, the matrix's no-argument shape. framework-checks joins wg3, the other two wg4 (the ~23s groups). Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
1 parent 2f43435 commit 3ea7fdc

5 files changed

Lines changed: 65 additions & 50 deletions

File tree

‎AGENTS.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,8 @@ TNB is a tsgo-backed TypeScript fork: upstream `microsoft/TypeScript` and `micro
3232

3333
- `npm run check:lib` / `check:enums` / `check:sourcefile-guard`
3434
- `npm run check:go-as-guards` — Go-side Type-cast guard: every `Type.As*()` chain deref and unguarded nil-family assignment in the tsgo patches/overlay is fixed or carries an inline `// asguard:exempt` reason (issues #69/#70 bug class)
35-
- Witnesses: 86 across wg0–wg6, all wired in CI (`.github/workflows/ci.yml`), single source of truth `tools/ci-witness-groups.mjs` — `node tools/ci-witness-groups.mjs all` validates dup/missing/orphan/local-only/baseline wiring and emits the matrix (`matrix` mode feeds the ci.yml prepare job).
36-
- Local-only — run on demand, not in the matrix (reasons in the matrix header comment): framework-checks, generation-retention, napi-fuzz, completion-latency, postedit-latency, perf-edit-rpc, perf-qi-rpc, typing-cpuprof.
35+
- Witnesses: 89 across wg0–wg6, all wired in CI (`.github/workflows/ci.yml`), single source of truth `tools/ci-witness-groups.mjs` — `node tools/ci-witness-groups.mjs all` validates dup/missing/orphan/local-only/baseline wiring and emits the matrix (`matrix` mode feeds the ci.yml prepare job).
36+
- Local-only — run on demand, not in the matrix (measurement, not pass/fail): completion-latency, postedit-latency, perf-edit-rpc, perf-qi-rpc, typing-cpuprof.
3737
- Locally hostable gates: `check:sim-nav` and `check:sourcefile-guard` need a volar checkout. ci.yml's recipe, which is the one to copy (`VOLAR_SHA` is the ci.yml env pin — set it in your shell, a shallow clone does not carry the tag):
3838
```sh
3939
export VOLAR_SHA=<.github/workflows/ci.yml value>
@@ -69,9 +69,9 @@ TNB is a tsgo-backed TypeScript fork: upstream `microsoft/TypeScript` and `micro
6969
- `triage-cli-only-flags` — CLI-only options (`--strict`, `--target`, …) ride updateSnapshot to Go; diagnostics must move like stock
7070
- `triage-electron-abi` — issue #44: win32 bridge.node loads under Electron-as-node (wired out-of-matrix: ci.yml build job + nightly test-bridge-win32)
7171
- `triage-external-edits` — issue #49: disk rewrites reach tsgo across estree/tsserver/tsc-watch host classes (tsc-watch: the thin program registers a watcher per program file — it never passes files through getSourceFile, where stock creates them)
72-
- `triage-generation-retention` (local-only) — 100 edit+F12 rounds hold a ≈0 heapUsed slope (no closure pins a generation island)
73-
- `triage-framework-checks` (local-only) — svelte-check/astro-check/glint vs stock parity (fixtures under /tmp/tnb-fw-fixtures)
74-
- `triage-napi-fuzz` (local-only) — NAPI payload fuzz probe
72+
- `triage-generation-retention` — 100 edit+F12 rounds hold a ≈0 heapUsed slope (no closure pins a generation island)
73+
- `triage-framework-checks` — svelte-check/astro-check/glint vs stock parity (pinned framework versions installed under /tmp/tnb-fw-fixtures against stock typescript, then relinked to TNB/stock per run)
74+
- `triage-napi-fuzz` — NAPI payload fuzz: echo round-trips byte-identical on the string and binary paths
7575
- `triage-display-tokens` — issue #4: decoded RemoteNode display tokens (optionality token, template-head rawText)
7676
- `triage-computed-literal` — binder-literal computed property names print as-written
7777
- `triage-refs-exportspec` / `triage-quickinfo-emptyparity` — wg0 sweep pair: refs on re-exported classes + quickinfo empty-result parity (sweep-ls-throws regenerates /tmp/tnb-sweep-fixtures first)

‎tools/ci-witness-groups.mjs‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,13 @@
2828
// 2026-09-26: triage-external-edits joined wg5 (~8s) once its estree fixture
2929
// self-provisions — local-only let plain `tsc -w` lose every source-file
3030
// watcher unnoticed.
31+
// 2026-09-26: framework-checks joined wg3 (pinned framework versions, peer
32+
// tree resolved against stock typescript — it had silently stopped running
33+
// astro check), generation-retention (defaults to lib/tsserver.js +
34+
// STOCK_TSSERVER_PATH calibration) and napi-fuzz joined wg4; wg3/wg4 were
35+
// the ~23s groups.
3136
// Witnesses intentionally NOT in the matrix are local-only — the machine-
32-
// readable list lives in LOCAL_ONLY below (reasons: framework-checks needs
33-
// the /tmp/tnb-fw-fixtures framework installs (network, heavy);
34-
// generation-retention needs in-child GC marks + --stock-tsserver slope
35-
// calibration; napi-fuzz is a NAPI payload fuzz probe; the perf series is
37+
// readable list lives in LOCAL_ONLY below (reason: the perf series is
3638
// measurement, not pass/fail). triage-electron-abi / triage-sim-nav-shard
3739
// are wired outside the matrix — OWNED_ELSEWHERE below (electron-abi:
3840
// direct ci.yml build-job step + nightly test-bridge-win32; sim-nav-shard:
@@ -46,13 +48,10 @@ import * as fs from 'node:fs';
4648
import * as path from 'node:path';
4749
import { fileURLToPath } from 'node:url';
4850

49-
const TOTAL = 86;
51+
const TOTAL = 89;
5052

5153
// Witnesses intentionally NOT in the matrix — run on demand (reasons above).
5254
const LOCAL_ONLY = [
53-
'triage-framework-checks',
54-
'triage-generation-retention',
55-
'triage-napi-fuzz',
5655
'triage-completion-latency',
5756
'triage-postedit-latency',
5857
'triage-perf-edit-rpc',
@@ -92,12 +91,15 @@ const groups = [
9291
witnesses: [
9392
'triage-f2r5-defcases', // ~67s
9493
'triage-tostring-gtd-vue', // ~61s
94+
'triage-framework-checks', // ~28s cold (npm installs), ~12s warm
9595
],
9696
},
9797
{
9898
witnesses: [
9999
'triage-qi-delete-min', // ~65s
100100
'triage-f2r6-xvue', // ~62s
101+
'triage-generation-retention', // ~4s (stock calibration + 100 tnb rounds)
102+
'triage-napi-fuzz', // <1s
101103
],
102104
},
103105
{

‎tools/triage-external-edits.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'
3535
// The package the estree fixture links as `typescript`: lib's realpath parent,
3636
// since an isolated tools copy (the CI witness job) has lib/ but no package.json.
3737
const tnbPackageRoot = path.dirname(fs.realpathSync(path.join(repoRoot, 'lib')));
38-
const cacheRoot = process.env.TNB_FW_CACHE ?? '/tmp/tnb-fw-fixtures';
38+
const cacheRoot = '/tmp/tnb-fw-fixtures';
3939
const stockDir = path.join(cacheRoot, 'stock-ts');
4040
const estreeDir = path.join(cacheRoot, 'estree');
4141
const scratchRoot = '/tmp/tnb-49-repro';

‎tools/triage-framework-checks.mjs‎

Lines changed: 42 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
* Framework checker compatibility witness: svelte-check / astro-check / glint
44
* against the TNB fork vs stock typescript. Each case builds a minimal project
55
* in a cache dir (deps installed once, then reused), runs the tool with TNB
6-
* and with stock typescript@6.0.3 (packed on demand), and asserts the
6+
* and with stock typescript (STOCK_TSSERVER_PATH's package), and asserts the
77
* invariants for the failure modes that were fixed:
88
* - glint: must NOT be fail-silent (virtual App.gts→App.ts via readFile
99
* override reaches tsgo; was: 0 errors where stock reports them)
@@ -16,21 +16,31 @@
1616
* Usage: node tools/triage-framework-checks.mjs [svelte|astro|astro7|glint...]
1717
* Exit: 0 = PASS, 1 = FAIL. Network required on first run (npm installs).
1818
*
19+
* Installs resolve against stock typescript: the fork's prerelease version
20+
* (6.0.3-bridge.N) satisfies no `^6` peer range, and --legacy-peer-deps
21+
* would drop every other peer too (@astrojs/check lost @emnapi/runtime and
22+
* astro check never ran). node_modules/typescript is relinked per run.
23+
*
1924
* v5 classification: the glint/svelte invariants are diagnostic-SET
2025
* semantics (bridge-contract surface, stock-gated). The astro full-text
2126
* parity covers diagnostic content (codes/spans — contract) but would also
2227
* trip on engine-owned presentation (tsgo's own diagnostic text/code
2328
* choices, e.g. the TS6196 class); such a failure is a KNOWN registration
2429
* (pristine tsgo reference), not a revert-to-stock patch.
2530
*/
26-
import { execFileSync, execSync } from 'node:child_process';
31+
import { execFileSync } from 'node:child_process';
2732
import * as fs from 'node:fs';
2833
import * as path from 'node:path';
2934
import { fileURLToPath } from 'node:url';
3035

3136
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
32-
const cacheRoot = process.env.TNB_FW_CACHE ?? '/tmp/tnb-fw-fixtures';
33-
const stockDir = path.join(cacheRoot, 'stock-ts');
37+
// The CI witness job runs from an isolated tools copy whose lib/ symlinks
38+
// into the checkout; the package root is wherever lib/ really lives.
39+
const tnbPackageRoot = path.dirname(fs.realpathSync(path.join(repoRoot, 'lib')));
40+
const cacheRoot = '/tmp/tnb-fw-fixtures';
41+
if (!process.env.STOCK_TSSERVER_PATH) throw new Error('STOCK_TSSERVER_PATH is required (stock typescript package)');
42+
const stockPackage = path.resolve(path.dirname(process.env.STOCK_TSSERVER_PATH), '..');
43+
const stockVersion = JSON.parse(fs.readFileSync(path.join(stockPackage, 'package.json'), 'utf8')).version;
3444
const stripAnsi = (s) => s.replace(/\x1b\[[0-9;]*m/g, '');
3545

3646
const TSCONFIG_BASE = {
@@ -40,13 +50,17 @@ const TSCONFIG_BASE = {
4050
},
4151
};
4252

53+
// The first astro invocation on a machine prints a one-time telemetry notice,
54+
// which would land on whichever of TNB/stock ran first.
55+
const astroEnv = { ...process.env, ASTRO_TELEMETRY_DISABLED: '1' };
56+
4357
const CASES = {
4458
glint: {
4559
deps: {
46-
'@glint/core': '^1.5.0',
47-
'@glint/environment-ember-loose': '^1.5.0',
48-
'@glint/environment-ember-template-imports': '^1.5.0',
49-
'@glint/template': '^1.5.0',
60+
'@glint/core': '1.5.2',
61+
'@glint/environment-ember-loose': '1.5.2',
62+
'@glint/environment-ember-template-imports': '1.5.2',
63+
'@glint/template': '1.9.0',
5064
},
5165
files: {
5266
'tsconfig.json': JSON.stringify({ ...TSCONFIG_BASE, include: ['src/**/*.ts', 'src/**/*.gts'], glint: { environment: ['ember-loose', 'ember-template-imports'] } }, null, 2),
@@ -70,7 +84,7 @@ export default class App extends Component<{ Args: Args }> {
7084
},
7185
},
7286
svelte: {
73-
deps: { svelte: '^5.0.0', 'svelte-check': '^4.0.0', tslib: '^2.6.0' },
87+
deps: { svelte: '5.57.1', 'svelte-check': '4.7.6', tslib: '2.8.1' },
7488
files: {
7589
'tsconfig.json': JSON.stringify({ ...TSCONFIG_BASE, compilerOptions: { ...TSCONFIG_BASE.compilerOptions, allowJs: true, checkJs: true }, include: ['src/**/*.svelte', 'src/**/*.ts'] }, null, 2),
7690
'src/util.ts': 'export function double(n: number): number { return n * 2; }\n',
@@ -118,7 +132,7 @@ export default class App extends Component<{ Args: Args }> {
118132
},
119133
},
120134
astro: {
121-
deps: { astro: '^5.0.0', '@astrojs/check': '^0.9.0' },
135+
deps: { astro: '5.18.2', '@astrojs/check': '0.9.10' },
122136
files: {
123137
'tsconfig.json': JSON.stringify({ extends: 'astro/tsconfigs/strict', compilerOptions: { noEmit: true }, include: ['.astro/types.d.ts', 'src/**/*'] }, null, 2),
124138
'src/util.ts': 'export function double(n: number): number { return n * 2; }\n',
@@ -131,13 +145,14 @@ const wrong: string = double(count);
131145
<p>{double(count)}</p>
132146
`,
133147
},
134-
run: (dir) => execFileSync('npx', ['astro', 'check'], { cwd: dir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }),
148+
run: (dir) => execFileSync('npx', ['astro', 'check'], { cwd: dir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], env: astroEnv }),
135149
check(out, outStock) {
136150
const norm = (s) => stripAnsi(s)
137151
.replace(/[^\n]*TNB ACTIVE[^\n]*\n?/g, '') // TNB banner line
138152
.replace(/\d{2}:\d{2}:\d{2}/g, 'TT:TT:TT')
139153
.replace(/\d+(?:\.\d+)?m?s\b/g, 'Xms');
140154
const a = norm(out).trim(), b = norm(outStock).trim();
155+
if (!/Result \(\d+ files?\)/.test(a)) return `astro check did not run:\n${a}`;
141156
if (a !== b) return `output mismatch vs stock:\n--- tnb ---\n${a}\n--- stock ---\n${b}`;
142157
return undefined;
143158
},
@@ -167,7 +182,7 @@ const { title } = Astro.props;
167182
<div>{title}</div>
168183
`,
169184
},
170-
run: (dir) => execFileSync('npx', ['astro', 'check'], { cwd: dir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }),
185+
run: (dir) => execFileSync('npx', ['astro', 'check'], { cwd: dir, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], env: astroEnv }),
171186
check(out, outStock) {
172187
const t = stripAnsi(out);
173188
if (/ts\(2304\):\s*Cannot find name 'Fragment'/.test(t)) return "false positive: global Fragment unresolved (#38 — plugin ambient roots dropped on snapshot rebuild)";
@@ -177,34 +192,32 @@ const { title } = Astro.props;
177192
.replace(/\d{2}:\d{2}:\d{2}/g, 'TT:TT:TT')
178193
.replace(/\d+(?:\.\d+)?m?s\b/g, 'Xms');
179194
const a = norm(out).trim(), b = norm(outStock).trim();
195+
if (!/Result \(\d+ files?\)/.test(a)) return `astro check did not run:\n${a}`;
180196
if (a !== b) return `output mismatch vs stock:\n--- tnb ---\n${a}\n--- stock ---\n${b}`;
181197
return undefined;
182198
},
183199
},
184200
};
185201

186-
function ensureStock() {
187-
if (fs.existsSync(path.join(stockDir, 'package', 'lib', 'typescript.js'))) return;
188-
fs.mkdirSync(stockDir, { recursive: true });
189-
execSync('npm pack typescript@6.0.3 --silent', { cwd: stockDir, stdio: 'ignore' });
190-
execSync('tar -xzf typescript-6.0.3.tgz', { cwd: stockDir, stdio: 'ignore' });
191-
}
192-
193202
function ensureCase(name, def) {
194203
const dir = path.join(cacheRoot, name);
195204
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
196205
for (const [rel, content] of Object.entries(def.files)) {
197206
fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true });
198207
fs.writeFileSync(path.join(dir, rel), content);
199208
}
200-
const pkg = {
209+
const pkg = JSON.stringify({
201210
name: `tnb-fw-${name}`, private: true, type: 'module',
202-
dependencies: { ...def.deps, typescript: `file:${repoRoot}` },
203-
};
204-
fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify(pkg, null, 2));
205-
if (!fs.existsSync(path.join(dir, 'node_modules'))) {
206-
console.log(`[${name}] installing deps (first run)…`);
207-
execFileSync('npm', ['install', '--no-audit', '--no-fund', '--legacy-peer-deps'], { cwd: dir, stdio: 'inherit' });
211+
dependencies: { ...def.deps, typescript: stockVersion },
212+
}, null, 2);
213+
const pkgPath = path.join(dir, 'package.json');
214+
const nodeModules = path.join(dir, 'node_modules');
215+
if (!fs.existsSync(nodeModules) || !fs.existsSync(pkgPath) || fs.readFileSync(pkgPath, 'utf8') !== pkg) {
216+
console.log(`[${name}] installing deps…`);
217+
fs.rmSync(nodeModules, { recursive: true, force: true });
218+
fs.rmSync(path.join(dir, 'package-lock.json'), { force: true });
219+
fs.writeFileSync(pkgPath, pkg);
220+
execFileSync('npm', ['install', '--no-audit', '--no-fund'], { cwd: dir, stdio: 'inherit' });
208221
}
209222
return dir;
210223
}
@@ -213,13 +226,13 @@ function runCase(name, def) {
213226
const dir = ensureCase(name, def);
214227
const tsLink = path.join(dir, 'node_modules', 'typescript');
215228
const linkTo = (target) => { fs.rmSync(tsLink, { force: true, recursive: true }); fs.symlinkSync(target, tsLink); };
216-
linkTo(repoRoot);
229+
linkTo(tnbPackageRoot);
217230
let outTnb;
218231
try { outTnb = def.run(dir); } catch (e) { outTnb = (e.stdout ?? '') + (e.stderr ?? ''); }
219-
linkTo(path.join(stockDir, 'package'));
232+
linkTo(stockPackage);
220233
let outStock;
221234
try { outStock = def.run(dir); } catch (e) { outStock = (e.stdout ?? '') + (e.stderr ?? ''); }
222-
linkTo(repoRoot);
235+
linkTo(tnbPackageRoot);
223236
const problem = def.check(outTnb, outStock);
224237
if (problem) {
225238
console.error(`[${name}] FAIL: ${problem}`);
@@ -229,7 +242,6 @@ function runCase(name, def) {
229242
return true;
230243
}
231244

232-
ensureStock();
233245
const wanted = process.argv.slice(2);
234246
const names = wanted.length ? wanted : Object.keys(CASES);
235247
let ok = true;

‎tools/triage-generation-retention.mjs‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,12 @@
1818
* flat on this protocol, so the absolute floor dominates in practice.
1919
*
2020
* Usage:
21-
* node tools/triage-generation-retention.mjs --tsserver=<path>
21+
* node tools/triage-generation-retention.mjs [--tsserver=<path>]
2222
* [--label=tnb] [--rounds=100] [--marks=2,25,50,75,100]
2323
* [--max-slope-mb=0.05] [--stock-tsserver=<path>]
2424
* [--keep-fixture=1] [--out=<json>]
25+
* --tsserver defaults to this repo's lib/tsserver.js, --stock-tsserver to
26+
* STOCK_TSSERVER_PATH (the witness-matrix shape: no arguments).
2527
*
2628
* Exit 0 = within threshold, 1 = regression, 2 = setup/usage error.
2729
*/
@@ -40,9 +42,9 @@ function argVal(name, dflt) {
4042
return m ? m.split('=')[1] : dflt;
4143
}
4244

43-
const TSSERVER = argVal('tsserver', '');
44-
const STOCK_TSSERVER = argVal('stock-tsserver', '');
45-
const LABEL = argVal('label', 'target');
45+
const TSSERVER = argVal('tsserver', path.join(repoRoot, 'lib', 'tsserver.js'));
46+
const STOCK_TSSERVER = argVal('stock-tsserver', process.env.STOCK_TSSERVER_PATH ?? '');
47+
const LABEL = argVal('label', 'tnb');
4648
const ROUNDS = Number(argVal('rounds', '100'));
4749
const MARKS = argVal('marks', '2,25,50,75,100').split(',').map(Number).filter(n => n > 0).sort((a, b) => a - b);
4850
const MAX_SLOPE_MB = Number(argVal('max-slope-mb', '0.05'));
@@ -51,11 +53,10 @@ const OUT = argVal('out', '');
5153

5254
function usage(msg) {
5355
if (msg) console.error(msg);
54-
console.error('usage: node tools/triage-generation-retention.mjs --tsserver=<path> [--rounds=100] [--marks=2,25,50,75,100] [--max-slope-mb=0.05] [--stock-tsserver=<path>]');
56+
console.error('usage: node tools/triage-generation-retention.mjs [--tsserver=<path>] [--rounds=100] [--marks=2,25,50,75,100] [--max-slope-mb=0.05] [--stock-tsserver=<path>]');
5557
process.exit(2);
5658
}
5759

58-
if (!TSSERVER) usage('missing --tsserver');
5960
if (!fs.existsSync(TSSERVER)) usage(`missing tsserver: ${TSSERVER}`);
6061
if (STOCK_TSSERVER && !fs.existsSync(STOCK_TSSERVER)) usage(`missing stock tsserver: ${STOCK_TSSERVER}`);
6162
if (!fs.existsSync(gcMarksPath)) usage(`missing helper: ${gcMarksPath}`);

0 commit comments

Comments
 (0)