Skip to content

feat(telemetry): record sanitised detail for unrecognised commands - #342

Open
MattieTK wants to merge 3 commits into
mainfrom
feat/telemetry-unknown-command-detail
Open

MattieTK wants to merge 3 commits into
mainfrom
feat/telemetry-unknown-command-detail

Conversation

@MattieTK

@MattieTK MattieTK commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Unknown-command telemetry only reported command: "unknown command" with empty args, so we could not see what was attempted. That bucket also mixed unrelated failures: the most common case is a real leaf given an extra positional (cf zones get <id>, cf dns records list example.com), because yargs' strict-commands check runs at leaf level too. Several failures were not reported at all: multiple unknown words at the root, root-level typos with a "Did you mean" suggestion, and flag-only invocations such as cf --bogus.

.fail now captures where resolution stopped while yargs still holds that level's context, and the parse-error reporter adds:

  • resolutionFailure: unknown-subcommand | surplus-positional | suggested-command | missing-command
  • knownCommand and knownCommandIsLeaf
  • unknownTokens (at most three) and unknownTokenCount
  • suggestedCommand
  • unknownFlagCount, with known cf flag names in argsUsed

A token is sent only when it is the first unresolved token, sits where a command name was expected, and is a word from cf's command catalogue. Everything else becomes a shape label (<hex>, <uuid>, <domain>, <path>, <email>, <word> …): any later token (cf workers deploy dev records ["deploy", "<word>"], since dev may be a value for the intended command) and any surplus positional after a complete command. IDs and resource names are not sent. Flag values are never sent.

Short flags are named using the global aliases plus the single-character option aliases from the catalogue, so a command's -f is recorded as force. The aliases come from the whole catalogue rather than the resolved command, matching how option names are checked; the catalogue does not list the generated --force on delete commands, so a per-command map would miss it. Unrecognised short flags such as -x are counted in unknownFlagCount but never named. Reading a short-flag group stops at the first unrecognised letter, as the rest may be its value.

Existing command values are kept for dashboard continuity: in-group typos and plural "Unknown commands:" failures still report the group or leaf name, now with the extra fields. Near-miss typos of cli telemetry are resolved before the exclusion check, so they stay unreported.

The catalogue is loaded lazily on this error path only, and not at all when telemetry is disabled. telemetry.md is updated.

Example (cf zones get 023e…353 --zone-id abc):

{"command":"unknown command","resolutionFailure":"surplus-positional","knownCommand":"zones get","knownCommandIsLeaf":true,"unknownTokens":["<hex>"],"unknownTokenCount":1,"unknownFlagCount":0,"argsUsed":["zone-id"]}

Expect a small rise in unknown command volume from the root-level cases that were previously dropped.

Validation:

  • pnpm check
  • pnpm test src/__tests__/telemetry-errors.test.ts src/__tests__/lib/telemetry/command-resolution.test.ts
  • Built bin/cf run with DEBUG=1 and no Sparrow key against adversarial argv (IDs, domains, paths, emails, --flag=value, tokens after --)

@pkg-pr-new

pkg-pr-new Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
npm i https://pkg.pr.new/cf@342

commit: 4c4a480

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

cf --help startup

mean stddev min max
PR 272.7 ms 5.3 ms 265.1 ms 285.9 ms
base 274.0 ms 5.2 ms 266.7 ms 289.6 ms

Δ (PR vs base): ↓ -1.3 ms (-0.5%)

Measured with hyperfine on ubuntu-latest (3 warmup runs, 20 timed runs). CI numbers carry ±a few ms of runner jitter; treat small deltas as noise.

@clanky-the-clanker

clanky-the-clanker Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

✅ Approved — No issues found. Unrecognised tokens are reduced to catalogue command words or shape labels, flag values are never read, and the catalogue loads only on the error path when telemetry is enabled.

The decision is advisory: this comment never approves, blocks or requests changes on the pull request.


✅ All checks passed

🔒 Security: ✅ LGTM

I found no privacy or security issues.

  • Command words: Raw tokens are sent only when they match a closed vocabulary from the command catalogue.
  • Other tokens: IDs, domains, paths, emails, URLs and resource names become shape labels.
  • Suggestions: suggestedCommand is checked against a strict name pattern before it is sent.
  • Flags: Unrecognised flags and short flags are counted, never named.

📖 AGENTS.md: ✅ LGTM

This pull request does not change any agent guidance files, and I found nothing in the change that calls for an update to them.


📋 Code Quality: ✅ LGTM

I found no correctness, privacy or test-coverage problems in the changed code.

  • Token sanitisation: Only the first unresolved token can be sent as a word, and only when it is a word in the cf command catalogue (sanitizeCommandToken, command-resolution.ts:143-185). Surplus positionals at a leaf and every later token get shape labels. Flag values are never read.
  • yargs internals: The reads of yargs internal state sit inside a try/catch with type guards, so a change in yargs falls back to the old behaviour.
  • Stale state: The captured failure is used only when it is attached to the exact error that was thrown, so state from an earlier failure cannot leak into a later report.
  • cli telemetry exclusion: It is checked against the intended command, so near-miss typos of cli telemetry stay unreported.
  • Tests: They cover each failure kind, short and long flags, the three-token cap, -- handling, a catalogue that fails to load, and the disabled-telemetry path.

⚡ Performance: ✅ LGTM

I found no performance issues.

  • When the catalogue loads: It is read only on the parse-error path, after the dispatcher.enabled check, so nothing loads when telemetry is off.
  • Caching: The catalogue is cached for the life of the process.
  • Work per token: At most three tokens are processed, each cut to 64 characters, and the regexes run in linear time.

📝 Documentation: ✅ LGTM

telemetry.md and the changeset describe the new fields and how they are sanitised. I found no issues.


Reviewed commit: 4c4a4802

Comment thread packages/cli/src/lib/telemetry/command-resolution.ts Outdated
Comment thread packages/cli/src/lib/telemetry/command-resolution.ts Outdated
@ask-bonk

ask-bonk Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

🤖 I've done a thorough review of your PR.

Adds sanitized telemetry context for command-resolution failures.

Posted 2 actionable inline findings.

github run

@MattieTK
MattieTK marked this pull request as draft October 17, 2026 10:20
Unknown-command telemetry only said "unknown command", so we could not
tell an agent typing a Wrangler habit apart from a real leaf given an
extra positional (for example `cf zones get <id>`). Some failures were
not reported at all: several unknown words, root-level typos with a
suggestion, and flag-only invocations.

`.fail` now captures where resolution stopped while yargs still holds
that level's context, and the parse-error reporter adds
resolutionFailure, knownCommand, knownCommandIsLeaf, unknownTokens
(at most three), unknownTokenCount, suggestedCommand and
unknownFlagCount. A token is sent only when it is a word from cf's
command catalogue; anything else becomes a shape label such as <hex>,
<domain> or <path>, so IDs and resource names never leave the machine.
Flag names are kept only when they are cf options.

The `command` value is unchanged for continuity, except that the plural
"Unknown commands:" message is now treated like the singular one. The
catalogue is loaded lazily on this error path, and only when telemetry
is enabled.
Restore the existing command values so dashboards stay continuous: only yargs' singular "Unknown command:" message, or a failure at the top level, reports command "unknown command". Plural and in-group typo failures keep the group or leaf name and carry the new resolution fields instead.

Near-miss typos of cli telemetry are now resolved to the intended command before the telemetry exclusion check, so they are not reported. Surplus positionals after a complete command are always sent as a shape label, never as a matching catalogue word, because they are almost always user values.
Only the first unresolved token is known to sit where a command name was
expected. After an unknown word such as `deploy` in `cf workers deploy dev`,
later tokens may be values for the intended command, so they are now always
reduced to a shape label even when they match a cf command word.

Short flags were only read through the global aliases, so a command's own
`-f` was neither named nor counted, and unknown short flags such as `-x`
were skipped although the docs say unrecognised flags are counted. The
vocabulary now carries the catalogue's single-character option aliases, and
unrecognised short-flag letters are counted (never sent). The aliases are
taken from the whole catalogue, matching the scope of the option names: the
catalogue does not list the generated `--force` of each delete command, so
a map for the resolved command alone would miss it. A letter that names
different options in different commands is left out and counted instead.
@MattieTK
MattieTK force-pushed the feat/telemetry-unknown-command-detail branch from b12c6ff to 4c4a480 Compare October 21, 2026 11:38
@MattieTK

Copy link
Copy Markdown
Member Author

Unknown short flags such as -x are now counted in unknownFlagCount (never named), and command-specific short aliases come from the catalogue; telemetry.md and the changeset are updated to match.

@MattieTK
MattieTK marked this pull request as ready for review October 21, 2026 11:38
@ask-bonk

ask-bonk Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

LGTM

github run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant