Skip to content

fix(appkit): non-blocking typegen on Analytics#406

Merged
atilafassina merged 13 commits into
mainfrom
typegen-uncaught
Jun 5, 2026
Merged

fix(appkit): non-blocking typegen on Analytics#406
atilafassina merged 13 commits into
mainfrom
typegen-uncaught

Conversation

@atilafassina
Copy link
Copy Markdown
Contributor

@atilafassina atilafassina commented Jun 2, 2026

Summary

Makes type generation warehouse-aware and non-blocking by default.

Previously typegen ran DESCRIBE QUERY blind to the warehouse's lifecycle state and to the SDK's real error shapes, producing two wrong outcomes — a STOPPED warehouse was misread as "empty" (every query emitted result: unknown and the known-good types were discarded), and an unreachable host was treated as a fatal build failure. On top of that it blocked npm install / npm dev on a cold warehouse and could crash uncaught.

After this PR:

  • Non-blocking is the default (appkit generate-types, the dev server, postinstall / predev). Typegen writes the best types it can immediately — the last-known-good cached type where the SQL is unchanged, otherwise result: unknown — and never blocks on, or fails because of, the warehouse. Real types then refresh in the background.
  • --block (CI / prebuild) waits for readiness and produces accurate types, failing fast only when the warehouse genuinely can't serve them (deleted) or the SQL is wrong.

Behavior

A pure policy function maps (warehouse state × mode) to an action; the orchestration executes it.

Warehouse state non-blocking (default) blocking (--block)
RUNNING degrade now; describe in background describe
STARTING degrade now; wait → describe in background wait → describe
STOPPED / STOPPING degrade now; start → wait → describe in background start → wait → describe
DELETED / DELETING degrade, leave as-is (no error) fatal (exit 1)
Unreachable (connectivity) degrade (reuse cache or unknown) degrade — never fatal
SQL error (DESCRIBE … FAILED) unknown; reported throws → fails build
  • Degrading reuses the last-known-good cached type when the SQL hash is unchanged, else emits result: unknown. Degraded types are never persisted, so a transient outage can't poison the cache and a fixed query recovers on the next run.
  • The generated .d.ts is always written before any throw.
  • The background refresh mechanism differs by host: the dev Vite plugin runs an in-process, single-flight, abortable watch (covering a RUNNING warehouse too); the CLI spawns a detached --block worker behind a single-flight lock (stale-stealable after 6 min), so postinstall returns instantly and types refresh once the worker finishes.

Key pieces

  • preflight.ts — pure decidePreflight(state, mode) policy; fully table-tested, no I/O.
  • warehouse-status.tsgetWarehouseState, startWarehouse, and a bounded, abortable waitUntilRunning (exp backoff; treatStoppedAsTransient so it polls through the STOPPED → STARTING transition right after a start).
  • query-registry.ts — pre-flight orchestration + per-query classification (ok / syntax / connectivity / non-terminal → degrade), with a per-query backstop so a mid-run state change can't resurface the "empty" misclassification.
  • vite-plugin.ts — dev: instant degrade in the foreground, background lifecycle (including a RUNNING warehouse) with single-flight coalescing and abort-on-shutdown.
  • cli/commands/generate-types.ts + spawn-lock.ts — non-blocking CLI degrades then spawns the detached worker behind the lock; positive --block flag (replaces --no-block); hidden internal --worker-lock.
  • Connectivity classifier now recognizes the SDK's DNS wrapper ("Can't connect to <url>" carrying numeric code 500) → degrade, not fatal.
  • template/package.jsonpostinstall / predev → non-blocking default; prebuild--block; dropped the redundant typegen:no-block script.
  • Docstype-generation.md documents the non-blocking default and --block.

Testing

  • Exhaustive table-driven tests for the pure policy (every state × mode).
  • Warehouse-status: state mapping, connectivity-vs-fatal classification, bounded-wait backoff / timeout via fake timers.
  • Orchestration: degrade-all skips describes; wait-then-proceed; start → wait → describe; fatal throws after the .d.ts is written; the per-query non-terminal backstop.
  • Dev plugin: instant degrade, background-describes RUNNING / STARTING / STOPPED, swallows DELETED, aborts on shutdown.
  • CLI: spawns exactly one detached worker; the single-flight lock prevents stacking; a stale lock is stolen; spawn failure is non-fatal; the worker releases the lock.
  • Full suite green (appkit + shared); build:package + publint clean.

This pull request and its description were written by Isaac.

Type generation threw an uncaught error whenever the SQL warehouse was
down. Every DESCRIBE QUERY failed, all queries degraded to an unknown
result, and generateFromEntryPoint unconditionally threw an aggregate
"Type generation failed" error that escaped uncaught at the Vite plugin
(un-awaited generate()) and the CLI (sync cmd.parse()) call sites.

Distinguish connectivity failures from genuine SQL errors:

- Connectivity (executeStatement rejects): degrade silently. Reuse the
  last-known-good cached type if present, otherwise emit an unknown
  result. Never fatal, so a transient outage no longer fails a build.
- SQL error (reachable warehouse, DESCRIBE FAILED): surface via a typed
  TypegenSyntaxError so the existing prod-throws / dev-warns gate
  applies. Eligible to fail prod builds only.

Also stop caching unknown results: only successful describes with a
result schema are persisted, so a transient outage never poisons the
cache and a fixed query recovers on the next run.

PR1 of 2 (user-visible behavior). PR2 will await the Vite
buildStart/watcher, use parseAsync().catch() in the CLI, and add
degrade/throw regression tests.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
Add the regression coverage the warehouse-down crash slipped through: the
prior suite tested rejection->unknown as graceful but never connected it to
the aggregate throw in generateFromEntryPoint.

query-registry (generate-queries.test.ts):
- connectivity reuses the last-known-good cached type
- empty result (described, no columns) -> unknown, not syntax, not cached
- syntax (FAILED) -> recorded in syntaxErrors, not cached
- cache HIT serves the stored type without a warehouse call
- legacy retry-flagged entry is re-described, not reused
- mixed run records only the syntax failure; failures are not persisted

generateFromEntryPoint (index.test.ts):
- syntax errors throw TypegenSyntaxError
- connectivity-only failures do NOT throw (the warehouse-down regression)
- the .d.ts is written before the throw

Layers 1+2 of the test plan; Layer 3 (analytics vite-plugin) and Layer 4
(CLI exit codes) land in PR2 with their await/parseAsync refactors.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
Signed-off-by: Atila Fassina <atila@fassina.eu>
@atilafassina atilafassina marked this pull request as ready for review June 3, 2026 12:39
@atilafassina atilafassina requested a review from a team as a code owner June 3, 2026 12:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR prevents analytics type generation from crashing when the SQL warehouse is unreachable by distinguishing connectivity failures (non-fatal; reuse cached types or emit unknown) from genuine SQL errors (warehouse reachable but DESCRIBE QUERY returns FAILED, which can be treated as fatal by callers).

Changes:

  • Classifies per-query DESCRIBE QUERY outcomes into success, syntax error (FAILED), connectivity error (request rejected), and empty result; only successful results are cached.
  • Updates the typegen entrypoint to throw a typed TypegenSyntaxError only when genuine SQL errors occur, and only after writing the .d.ts output.
  • Adds/updates tests to cover “warehouse down doesn’t crash” and caching behavior during connectivity vs syntax failures.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
packages/appkit/src/type-generator/types.ts Adds QuerySyntaxError and QueryGenerationResult types to represent non-fatal vs fatal describe outcomes.
packages/appkit/src/type-generator/query-registry.ts Implements failure classification, prevents caching unknown, reuses prior cached types on connectivity failures, and returns {schemas, syntaxErrors}.
packages/appkit/src/type-generator/index.ts Introduces TypegenSyntaxError and gates throwing on syntaxErrors after emitting .d.ts.
packages/appkit/src/type-generator/tests/index.test.ts Adds tests asserting syntax errors throw (after writing output) while connectivity failures do not throw.
packages/appkit/src/type-generator/tests/generate-queries.test.ts Expands coverage for caching rules, offline behavior, empty result handling, and syntax error reporting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/appkit/src/type-generator/query-registry.ts
Comment thread packages/appkit/src/type-generator/query-registry.ts
Comment thread packages/appkit/src/type-generator/tests/index.test.ts
Signed-off-by: Atila Fassina <atila@fassina.eu>
Add a warehouse-status pre-flight to typegen and rework error handling so a
stopped, starting, or unreachable warehouse degrades gracefully instead of
crashing or emitting EMPTY types.

- Classify connectivity (incl. the SDK's "Can't connect to <url>"/code-500 DNS
  wrapper) as OFFLINE, and non-terminal describe states (PENDING/RUNNING) as
  degraded rather than EMPTY.
- Surface typegen errors as their actionable message (no internal stack trace);
  format and de-duplicate the failure output.
- Add a warehouse status probe + pure pre-flight policy; block in the CLI/build,
  roll forward (degrade) in dev.
- Dev: regenerate types in the background once the warehouse reaches RUNNING
  (single-flight guarded); auto-start a stopped warehouse in dev.
- CLI: --no-block degrades instead of describing so postinstall never blocks.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
…locking

Replace the dev/blocking/degrade modes with two: non-blocking (default) and
blocking. non-blocking always degrades (skip probe + DESCRIBE, write cache-or-
unknown instantly); blocking keeps the current probe+wait flow. The CLI default
flips to non-blocking and --no-block becomes a positive --block flag. The dev
plugin runs the foreground in non-blocking (instant degrade) while its
background warehouse-watch regenerates in blocking so real types still land.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
In blocking mode a STOPPED/STOPPING warehouse is now started and waited on
(startWaitProceed: startWarehouse -> waitUntilRunning -> describe) instead of
failing fast. Only DELETED/DELETING is fatal. STARTING still waits; RUNNING
describes. The write-the-.d.ts-then-throw invariant is preserved for the fatal
and wait-timeout paths.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
The dev background lifecycle returned early for RUNNING, a leftover from when
the foreground described it synchronously. Now that the foreground always
degrades instantly (non-blocking), RUNNING must also background-describe or a
running warehouse never gets real types in dev. Only DELETED/DELETING leaves
degraded; single-flight coalescing and abort-on-shutdown are preserved.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
The default (non-blocking) generate-types now writes degraded types, then spawns
a detached `generate-types --block` worker behind a single-flight lock and exits
0 -- so postinstall/predev never block on warehouse state. The worker does the
full blocking lifecycle in the background, refreshes real types, and releases the
lock on exit (process-exit guard covers a hard fail). A stale lock from a crashed
worker is stolen after 6 min; spawn failure is non-fatal to the foreground.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
…block

Template postinstall/predev now run the non-blocking default `appkit
generate-types` (instant degrade + background refresh) instead of a dedicated
no-block script; prebuild keeps `--block` for accurate CI types. Removed the now
redundant `typegen:no-block` script. Documented the non-blocking default and the
`--block` flag in the type-generation guide.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
A failed DESCRIBE logged the raw SQL error message per query during the describe
loop, and the aggregated TypegenSyntaxError (printed by the Vite plugin / CLI)
carries the same message in its formatted block -- so every SQL syntax error
showed up twice in dev. Drop the per-query warn; the formatted block and the
summary table already surface it.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
@atilafassina atilafassina changed the title fix(appkit): don't crash typegen when the warehouse is unreachable fix(appkit): non-blocking typegen on Analytics Jun 5, 2026
Comment thread template/package.json Outdated
…er tsx

The non-blocking CLI spawned its background worker as `node <argv[1]>` without
the parent's node/loader flags. Run from source via tsx, argv[1] is a .ts file
that plain node can't parse, so the worker died silently (detached + stdio
ignore) and the degraded types never refreshed -- the queries appeared to never
run. Forward process.execArgv, which carries tsx's --require/--import loader
flags (and is empty for the built bin, so production is unaffected), so the
worker runs under the same runtime as the parent.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
Rename the user-facing CLI flag --block to --wait (commander option property
block -> wait), including the detached worker's self-spawn arg, the --help
example, the template prebuild script, and the type-generation guide. The
internal "blocking"/"non-blocking" PreflightMode names are unchanged -- they
describe runtime behaviour and aren't user-facing. The flag only ever existed
on this branch (unreleased), so no deprecation alias is needed.

Co-authored-by: Isaac
Signed-off-by: Atila Fassina <atila@fassina.eu>
@atilafassina atilafassina merged commit 31ab42e into main Jun 5, 2026
9 checks passed
@atilafassina atilafassina deleted the typegen-uncaught branch June 5, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants