diff --git a/CHANGELOG.md b/CHANGELOG.md index 81467354..c49f3f67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ - A shebang that runs `env` says what runs: `#!/usr/bin/env -S bash -e` runs `bash -e` with the script, and `#!/usr/bin/env bash -e` warns that Linux passes `bash -e` to env as one word, which fails. - The shell state follows static `eval` strings and trap actions: `eval 'x=1'` sets `x` where `eval` runs, and what a trap's action sets "may" be set after the `trap`. Before, any name might have been set after an `eval` or a trap's action. - A string that a command runs as a script is explained as one: `bash -c '…'`, `sh -c`, `eval`, a trap's action, `su -c`, `ssh host '…'`, `watch '…'`, also under `sudo`, `xargs`, `find -exec`, `docker exec` and `kubectl exec`. Its commands and expansions appear under the string's token at their own characters (`$HOME` in `bash -c "echo \$HOME"` highlights `\$HOME`), its notes join the line's in the shell that runs it (an `sh -c` string gets the POSIX checks), and the Problems list shows its dialect issues. A string with an expansion (`eval "$cmd"`, the `"$args"` of `eval set -- "$args"`) says its text is known only when it runs. Where something may not be there, the string says why: `sh` may be dash, a trap's action is read when its signal comes, and a new shell starts with `extglob` off. Before, such a string was one opaque argument. -- New notes on conditions that the values the script set decide, and on the commands that never run: "`$mode` is `fast` here, so this condition is always false: the commands after its `then` never run", "… so the loop stops at its first test: its body never runs", "… so the left side of `&&` always fails", "… so only `fast)` of this `case` runs". A function defined in such a branch says that it is not the function there. Conditions of literals only (`while true`, `if false`) say nothing more. A condition's words are read as bash expands them, so `[[ $f == "$p"* ]]`, `${x:-default}`, `${x#prefix}`, `~` and `$(( ))` are decided, and so is `[ $x = '*' ]` after `set -f`. +- New notes on conditions that the values the script set decide, and on the commands that never run: "`$mode` is `fast` here, so this condition is always false: the commands after its `then` never run", "… so the loop stops at its first test: its body never runs", "… so the left side of `&&` always fails", "… so only `fast)` of this `case` runs". A function defined in such a branch says that it is not the function there. Conditions of literals only (`while true`, `if false`) say nothing more. A condition's words are read as bash expands them, so `[[ $f == "$p"* ]]`, `${x:-default}`, `${x#prefix}`, `~` and `$(( ))` (with the values the script set, such as `$?`) are decided, and so is `[ $x = '*' ]` after `set -f`; the right of `=~` too, once expanded, when it matches plain text (`[[ $x =~ ^$p ]]`, a quoted part as plain text). - bats test files (`.bats`, or a `bats` shebang) are explained: `@test "…" {` as the test bats runs in a process of its own with `set -e`, and `run` (with `!`, `-N`, `--separate-stderr`), `load`, `skip`, `bats_require_minimum_version` and `bats_pipe`, with a pitfall for a `! cmd` that cannot fail a test and for `run` in a pipeline, whose `$status` and `$output` are lost. VS Code, the Language Server and the command-line tool (`--kind bats`) recognise them, and the web page has a bats sample. Conditions in a bats file are decided as in a script: `run` sets `$status`, `$output` and `$lines`, and only a file that calls `load` or `bats_load_library` (a file not read) or `bats_pipe` (a pipeline run in the test's shell) gets no decisions at all. Before, `@test` was a syntax error and `run` an unknown command. - The command-line tool takes `--sarif ` with `--check`: it also writes the problems to that file as SARIF, whatever `--format` prints. The GitHub Action uses it, so it checks the scripts once for both the annotations and the SARIF file. @@ -15,7 +15,7 @@ - A VS Code setting set below its minimum (`bashExplainer.inline.maxLength` under 20, `bashExplainer.hover.maxDepth` or `hover.followDelay` under 0) now uses its default, as the command-line tool already did. Before, VS Code used the number as given. - On the web page, changing the file kind changes only how the script is explained: the text stays, also an untouched sample (an empty editor still gets the new kind's sample). "Load the sample" loads the current kind's sample, and when it replaces your own text, a notice with an "Undo" button appears for a few seconds. Before, choosing a kind replaced an untouched sample with the new kind's sample, and "Load the sample" could be undone only with Ctrl+Z. -- A branch that cannot run, because its condition is decided by values the script itself set, no longer changes how the lines after it are explained: after `x=1; if [ "$x" = 2 ]; then set -f; fi`, a glob is a glob again, and after `if false; then f() { …; }; fi`, `f` is not the function. Code after `break`, `continue`, `return` or `exit` no longer counts either. Before, both sides of every branch counted, so such lines said "may" or "on some paths". A condition is decided only from variables that nothing but plain assignments (`x=1`, `export x=1`) writes anywhere in the script, and not at all in a script that runs what cannot be read (`source`, `eval "$s"`, `$cmd`, an indirection such as `${!v}`, `${!1}` or `${!v@Q}`, whose subscript may assign any variable, a trap whose action may `return` from where a signal comes): there, both sides still count. Arithmetic that reads one of bash's own numbers (`shift $((OPTIND-1))`, `(( EUID == 0 ))`) keeps the decisions (before, it turned off every decision in the script), and text assigned to `OPTIND`, `RANDOM`, `SRANDOM` or `HISTCMD`, which bash runs as arithmetic (`OPTIND='x=5'` sets `x`), counts as setting what it sets. Conditions in a trap's action are never decided. The text of an alias is read with the command it is expanded at, and with what the script declares: `m[k]=1` in an alias after `declare -A m` keeps the decisions (before, it turned off every decision in the script). +- A branch that cannot run, because its condition is decided by values the script itself set, no longer changes how the lines after it are explained: after `x=1; if [ "$x" = 2 ]; then set -f; fi`, a glob is a glob again, and after `if false; then f() { …; }; fi`, `f` is not the function. Code after `break`, `continue`, `return` or `exit` no longer counts either. Before, both sides of every branch counted, so such lines said "may" or "on some paths". A condition is decided only from variables that nothing but plain assignments (`x=1`, `export x=1`) writes anywhere in the script, and not at all in a script that runs what cannot be read (`source`, `eval "$s"`, `$cmd`, an indirection such as `${!v}`, `${!1}` or `${!v@Q}`, whose subscript may assign any variable, a trap whose action may `return` from where a signal comes): there, both sides still count. Arithmetic that reads one of bash's own numbers (`shift $((OPTIND-1))`, `(( EUID == 0 ))`) keeps the decisions (before, it turned off every decision in the script). Conditions in a trap's action are never decided. The text of an alias is read with the command it is expanded at, and with what the script declares: `m[k]=1` in an alias after `declare -A m` keeps the decisions (before, it turned off every decision in the script). ### Fixed @@ -37,11 +37,12 @@ - Quick fixes (the rewrites offered for dialect warnings) are offered in every language of the `bashExplainer.languages` setting, and follow a change of that setting, as the hover does. Before, they were offered in `shellscript` documents only. - `"$@"` inside single quotes is text, so the words after `bash -c 'f "$@"'` are no longer said to depend on how many words it gives. - Examples and dialect checks read a command's options as the command reads them. `grep -e -v foo` searches for `-v`; its example said `-v` inverts the match. `sed -ie 's/a/b/' -e 's/c/d/' f` runs only `s/c/d/`, since `e` is the backup suffix of `-i`; the example ran both scripts. `sed -i.bak` now shows the file rewritten. The input of `grep -efoo data.txt | sort` and `awk '{ print }' v=1 | sort` is labelled as it is read (a file, then stdin), and a pipeline whose later stage reads a file instead of the pipe (`printf … | grep -e -v foo`) gets no whole-pipeline example. `tr --delete` is `-d`, and `find … -exec cmd {} '+'` runs the command once, quoted or not. The dialect check no longer takes `read -p -a v` for `read -a` (`-a` is the prompt), and finds `trap ERR`. The examples of sort, uniq, cut, tr, head, tail, seq, basename, dirname, wc, xargs, sed, grep, awk, perl and `read` read the options as the explanation does, so a long prefix (`sort --rev`, `sed --expr`), `tail +2` and `tr -C` now get an example. `xargs --replace echo {}` and `xargs --max-lines echo` run `echo` (before, `echo` was taken for the value), and `-i`, `-l` are followed. `seq 1 -w 3`, which seq refuses, gets no example; before, it showed output. `cat -n`, `tac -s x` and `rev FILE` in a pipeline, and `cat -n f | sort`, get no example; before, the example passed the input on as if they had no option. The `git log` example shows as many commits as `--max-count 2` asks (before, 3). -- Explanations match bash in more places: `x=1 ''` (a command name that is not found, so `x` is not set), the variables that `exec {fd}>file`, `wait -p var` and `coproc name` set, and an error in `$(( ))` (a division by 0) or an assignment to a readonly variable, which discards the rest of the line, also when it is in a function the line calls (before, only the rest of the command, or nothing). A name written in the option's own word (`printf -vx 2`, `read -rax`, `wait -n -px`) is the variable set (before, `x` kept its old value, and a condition on it could be called decided), and `wait -p x` unsets `x` before it sets it (before, `x` kept its old value when no job ended). `exec cmd` in `( )` or `$( )` says that it ends only the subshell (before, it said nothing of the subshell). After a `trap` whose action calls a function, what the function sets may have changed, also when the script called it before from the same state (`f() { x=2; }; x=1; f; x=1; trap f USR1`: `x` may be 2 after it, as bash has once the signal comes; before, it was 1). `'exit' 3` with quotes has its exit status known, 3, as `exit 3` has (before, it was the status of an unknown command). `$(true)`, `$(:)` and `$(false)` count as empty only where the name runs the builtin, not after `true() { echo a; }`, an alias of it or `enable -n true` (before, `x=$(true)` was taken to set `x` to nothing there). Functions defined after `set -a +a` or `set foo -a` (where `-a` is an argument) are not exported, so conditions in them are decided (before, none were). In a function whose every call is in a subshell (`m() { cd /tmp; }; echo | m`), `cd` says "It may run in a subshell (where its function is called; what the script does not show may call it too)" when a sourced file, a trap's action or an alias may also call the function (before, "It runs in a subshell", although `. ./lib.sh` may call it in the shell itself); a word that names the function without calling it (`command m`, `declare -f m`, `type m`) no longer turns this into "may" (before, it did). `cmd &> >(tee log)` and `&>> >(…)` send both outputs to the process substitution, also after a block or a loop (`{ …; } &> >(tee log)`, `done &> >(tee log)`) and before the command name (`&> >(tee log) cmd`) (before, it was read as an argument of the command, and the redirection as missing its target; after a block or a loop it was a syntax error, and before the name it was the command). +- Explanations match bash in more places: `x=1 ''` (a command name that is not found, so `x` is not set), the variables that `exec {fd}>file`, `wait -p var` and `coproc name` set, and an error in `$(( ))` (a division by 0) or an assignment to a readonly variable, which discards the rest of the line, also when it is in a function the line calls (before, only the rest of the command, or nothing). A name written in the option's own word (`printf -vx 2`, `read -rax`, `wait -n -px`) is the variable set (before, `x` kept its old value, and a condition on it could be called decided), and `wait -p x` unsets `x` before it sets it (before, `x` kept its old value when no job ended). `exec cmd` in `( )` or `$( )` says that it ends only the subshell (before, it said nothing of the subshell). After a `trap` whose action calls a function, what the function sets may have changed, also when the script called it before from the same state (`f() { x=2; }; x=1; f; x=1; trap f USR1`: `x` may be 2 after it, as bash has once the signal comes; before, it was 1). Text given to an integer runs as arithmetic, as bash runs it: after `OPTIND='x=5'` (bash's `OPTIND`, `RANDOM`, `SRANDOM` and `HISTCMD` are integers from the start) or `declare -i n; n='x=5'`, `x` is 5, also for the decisions of conditions; before a command, `n+='x=5' cmd` sets `x` too, as bash does (only the first assignment of `n` there), but `n='x=5' cmd` does not; and `declare -i n; n=x` makes `n` 0 when `x` is not set (before, `x` kept its value, and `n` was not known). `'exit' 3` with quotes has its exit status known, 3, as `exit 3` has (before, it was the status of an unknown command). `$(true)`, `$(:)` and `$(false)` count as empty only where the name runs the builtin, not after `true() { echo a; }`, an alias of it or `enable -n true` (before, `x=$(true)` was taken to set `x` to nothing there). Functions defined after `set -a +a` or `set foo -a` (where `-a` is an argument) are not exported, so conditions in them are decided (before, none were). In a function whose every call is in a subshell (`m() { cd /tmp; }; echo | m`), `cd` says "It may run in a subshell (where its function is called; what the script does not show may call it too)" when a sourced file, a trap's action or an alias may also call the function (before, "It runs in a subshell", although `. ./lib.sh` may call it in the shell itself); a word that names the function without calling it (`command m`, `declare -f m`, `type m`) no longer turns this into "may" (before, it did). `cmd &> >(tee log)` and `&>> >(…)` send both outputs to the process substitution, also after a block or a loop (`{ …; } &> >(tee log)`, `done &> >(tee log)`) and before the command name (`&> >(tee log) cmd`) (before, it was read as an argument of the command, and the redirection as missing its target; after a block or a loop it was a syntax error, and before the name it was the command). - `unset f` says it removes the function when there is no variable `f`, as bash does. A pipeline whose stage is a function or an alias of the script (`sort() { tac; }; … | sort`) no longer shows the output of the program of that name as an example. Before, the first said "Unset the variable" and the second showed the output of the program, which the function does not give. - The ⚠ notes on words and pipelines follow what bash does here. After `IFS=; set -f`, `cp $a b` no longer says that `$a` is split (the token already said it is not); `X=1 echo "\$X"` no longer warns that `$X` is expanded first (`\$X` is text); a variable set by `((count++))`, `printf -v` or `mapfile` at the end of a pipeline, or by a function called there, is now reported as lost after it, and `echo '$n'` no longer counts as a later read. An alias of `rm` is taken for `rm`. - The "Idiom" notes are read from the commands, not from the text. They are found with other spacing, order and quoting: `while true ; do` is an endless loop, `done /dev/null >/dev/null`, `exec >> >(tee -a log)`, `$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )`, `while [ ${#} -gt 0 ]`, `while (( $# != 0 ))`, `trap cleanup 0` and `[[ $BASH_SOURCE != $0 ]] && return` are known. They are no longer given where bash does something else: `trap - EXIT`, `trap '' EXIT` and `trap -p EXIT` set no clean-up; `read -rd'' f` reads up to an `f`, not a NUL byte; `while [ $# > 0 ]` writes to a file named `0` and does not end; and text in quotes (`echo '$(mktemp)'`) is not run. Before, each of these got the idiom's note. - A script is explained in POSIX mode only when its shebang starts bash in it (`#!/bin/bash --posix`, or `#!/usr/bin/env -S` with `bash --posix`, `bash -o posix` or `POSIXLY_CORRECT=1 bash`), and `POSIXLY_CORRECT` is then set, as bash sets it. Before, `--posix` anywhere on the shebang line counted, also in `#!/bin/bash --posix -e` and `#!/usr/bin/env bash --posix`, where Linux passes the rest of the line as one word and bash does not start. +- A script that assigns an element before a function or `eval` (`n[0]=5 f`) is explained: bash refuses the assignment ("not a valid identifier") and runs the command. The same assignment before a special builtin in POSIX mode (`set -o posix; n[0]=5 :`) no longer fails either. Before, explaining such a script failed. [All changes](https://github.com/sf624/bash-explainer/compare/v0.7.0...main) diff --git a/README.ja.md b/README.ja.md index 20c73c95..12ab5a3d 100644 --- a/README.ja.md +++ b/README.ja.md @@ -287,6 +287,7 @@ VS Code 1.85 以降が必要です。ネイティブの依存はなく（構文 - bash と同じには読めない書き方があります。zsh 独自の書き方（`*(.)` や `${(f)var}`。シェルを zsh にすると、その行は「解析できない」と表示されます）、`echo (`（bash では構文エラーですが、エラーなしで受け付けます）、算術式の中で他の項と並んだ 64 進数の桁 `@` です（`$((64#@))` だけなら 62 と読めますが、`$((64#@ + 1))` では式全体が1つの語になります）。`[ ]` では、構文解析の結果によらず、bash の `test` と同じ規則で引数をまとめます（`-o` は `-a` より、`-a` は `!` より弱く結びつく。オペランドを取る位置の `-a` と `-o` は単項演算子）。 - 改行が Windows 形式（CRLF）のスクリプトは、LF のものとして解説します。また、bash は各行の最後の語に CR を含めて読むので、そのことを警告（`crlf`。最初の該当行の最後の語に付きます）で伝えます。このままにしたいときは、独立した行に `# bash-explainer disable-file=crlf` と書きます（1行目はシェバンなので、その前に `disable=` は置けません）。そのようなスクリプトのヒアドキュメントには、bash と同じく終わりの行がありません。 - コマンドがスクリプトとして実行する文字列（`bash -c '…'`・`eval`・trap の動作・`ssh host '…'`）をスクリプトとして解説するのは、中身がそのまま書かれているときだけです。展開を含む文字列（`eval "$cmd"`・`bash -c "cd $dir"`）の中身は実行するときに決まります。`bash file` や `source file` で実行するファイルは読みません。 +- POSIX モードでは、特殊組み込みコマンドの前に置いた配列要素への代入（`set -o posix; n[0]=5 :`）で、bash は「not a valid identifier」と表示してスクリプトを終了します。解説ではその行が実行されたものとして続きます。 ## プライバシー diff --git a/README.md b/README.md index b931ff56..7d1c365a 100644 --- a/README.md +++ b/README.md @@ -287,6 +287,7 @@ The command-line tool, the Language Server and the GitHub Action are one npm pac - A few constructs are not read as bash reads them: zsh-only syntax such as `*(.)` and `${(f)var}` (with the shell set to zsh, such a line is marked as not parsed), `echo (` (bash: syntax error; it is accepted without an error), and the base-64 digit `@` in arithmetic next to other terms (`$((64#@))` alone is read as 62, but in `$((64#@ + 1))` the whole expression becomes one word). In a `[ ]`, the arguments are grouped by bash's own rules for `test` (`-o` below `-a` below `!`; `-a` and `-o` as unary operators where an operand is expected), whatever the parser says. - A script with Windows line endings (CRLF) is explained as if it had LF, and a warning (`crlf`, on the last word of the first such line) says that bash reads the CR as part of the last word of each line. To keep the file as it is, write `# bash-explainer disable-file=crlf` on a line of its own (the first line is the shebang, so `disable=` before it is not possible). A here-document in such a script has no closing line, as in bash. - A string that a command runs as a script (`bash -c '…'`, `eval`, a trap's action, `ssh host '…'`) is explained as a script only when its text is written out: one with an expansion in it (`eval "$cmd"`, `bash -c "cd $dir"`) is known only when it runs. A file run with `bash file` or `source file` is not read. +- In POSIX mode, an assignment to an array element before a special builtin (`set -o posix; n[0]=5 :`) ends the script in bash, with "not a valid identifier"; the explanation goes on as if the line had run. ## Privacy diff --git a/docs/design/refactor.md b/docs/design/refactor.md index 56e43358..4c06ed14 100644 --- a/docs/design/refactor.md +++ b/docs/design/refactor.md @@ -210,12 +210,12 @@ Verdicts are from the verification at 1f9b651. Line numbers are those of that co | A9 | Analyzer god class | confirmed (facts) | 3.1 | PR #237 (part); rest in 3.1 (next PR) | Moved out of src/analyzer.ts (PR #237, every dump unchanged and the same output as main on every line of snippets.json, sample.sh and coverage.sh): the public types (src/analyzer-types.ts; analyzer.ts re-exports them, and src/ui, hosts, settings, service, document import them from there), `nestedScripts` (src/semantics/nested.ts), the example builders of a command's run, `ioExample`, `interactive`, the date and git log examples (src/examples/commands.ts, chosen by `CommandSpec.example`), `visible` (src/ctlchars.ts), the explainers of the embedded languages (src/embedded/registry.ts, see D11). The per-call mutable fields (line, shell, opts, state, status, context, notes, facts, examples, argCtx, slotEnvs) are one `LineContext` made new by each `explainLine`. Left for the next 3.1 PR: the expansion renderers (`expansion`, `expansionPart`, `expansionSummary`… about 300 lines, which read the analyzer's state and token helpers), and the glob, test and pattern examples, which read the state | | A10 | idioms match statement text by regex | confirmed (idioms.ts:164, 174, 229) | 2.3 (while idioms), 3.1 (rest) | while idioms: PR #219; rest: PR #231 | The `while` idioms (endless loop, `read` over a file or `< <(…)`, `getopts`, `getopt`) match the tree (`Shape` in src/idioms.ts); test/facts-from-model.test.ts "idioms follow the tree, not the spelling". 3.1 (PR #231): every idiom is a shape of the tree; no statement text is read. Words are their static values (`staticWordValue`), options come from the argv parse (`has`, `values`; wrapped commands through the parse's command roles: `sudo xargs -0`), parameters from the expansion nodes (`$#`, `${BASH_SOURCE[0]}`), redirections from the redirect nodes. `isStrictMode(text)` became a shape of the `set` command (test/idioms.test.ts "set option bundles, compared with bash" now goes through `findIdioms`). Fixes of contradictions, each run in bash ("says nothing where bash does something else"): `trap - EXIT`, `trap '' EXIT`, `trap -p EXIT`; `read -rd'' f` (the delimiter is `f`); `while [ $# > 0 ]` (a redirection: the loop does not end); `echo '$(mktemp)' IFS= read -r x` (text). New spellings ("reads the words as bash makes them"): `$(cd …; pwd)`, `IFS= read -d '' -r`, `2>/dev/null >/dev/null`, `[ "$0" = "$BASH_SOURCE" ]`, `exec 1> >(tee)`. Found on the way: the unbash adapter read `&> >(tee log)` as a redirection with no target and an argument; it now joins them (test/syntax-adapter-unbash.test.ts "sends `&>` and `&>>` to a process substitution, as bash does"). A corpus run of the old and new `findIdioms` over snippets.json and the fixtures, line by line, found no other difference; every dump unchanged. Review of #231: `exec >> >(tee)` gets the idiom as `>` does (bash sends stdout through it). Follow-up (PR #245): the adapter join covered only a simple command's suffix; now the `&` of a `&>`/`&>>` that unbash leaves without a process-substitution target is read as a blank and put back (every one in the script: unbash tells one error at a time, so it reads again until no new one appears), so `{ …; } &> >(tee log)`, `done &> >(tee log)`, `f() { …; } &> >(tee log)` and `&> >(tee log) cmd` parse as bash does (test/syntax-adapter-unbash.test.ts "sends `&>` to a process substitution after a compound command or before the command name", with bash -n, a run in bash, and bash-ast through the snippets); and `cd -- "$(dirname -- …)" &> /dev/null && pwd`, `${#}`, `(( $# != 0 ))`, `trap … 0`/`exit`, `[[ $BASH_SOURCE != $0 ]]` get their idioms (test/facts-from-model.test.ts "knows the common spellings of each idiom, as bash runs them") | | S1 | Builtin effects hand-coded in Walker.builtin and Scan.command; `-v` regex ×3; duplicates | confirmed (Walker.builtin 3048–3192, Scan.command 472–588, set -a 582–584; DECLARATIONS in trust.ts once) | 2.1 | PR #221 | src/semantics/effects.ts reads the spec: `ValueSpec`/`OperandSpec` `type: 'variable'` with `sets` (`read`, `read -a`, `printf -v`, `getopts`, `mapfile`, `wait -p`), `type: 'function'` (`mapfile -C`, `complete -F/-C`), `shellEffect.byDefault` (REPLY, MAPFILE) and `shellEffect.alters` (options, aliases, builtins: trust's `CHANGES_SHELL` is gone); the walk and the trust scan read `variableWrites`/`callbacks`. `command`/`builtin` are unwrapped by the spec's command slot (`wrapped`), `set -a` by `setOptionWrites` (last one wins, also used by the walk's `set`). One `DECLARATIONS` (also for the trust scan's list of builtins that only name functions; the parser keeps its own in unbash.ts), `SET_LETTERS` and `commandName` (effects.ts), one `BASH_ONLY_BUILTINS` (dict/builtins.ts). test/effects.test.ts (every case against bash; fixes `printf -vx`, `read -rax`, `wait -n -px`, `wait -p x` unsetting first, `set foo -a`, `set -a +a`, `'exit' 3`); probe tests and test/decisions.test.ts unchanged; sample.sh dump unchanged. Left: the walk keeps its handlers for what is not a variable a word names (`eval`/`trap`, `cd`'s PWD, getopts' OPTIND/OPTARG, `set`/`shopt`/`alias`/`enable`/`hash` values, `unset`); the variables bash sets itself go to 2.6's catalog | -| S2 | Second word evaluator in the walk; four variable catalogs | confirmed | 2.6 | catalog: PR #225; evaluator: PR #226; rest in 3.2 (bash's integers in the walk; `=~` and `$(( ))` in conditions; the cost of the walk's evaluation) | `BASH_VARIABLES` in src/semantics/vars.ts (every variable of the manual's 5.1 and 5.2 lists, and TERM, COPROC_PID): set at start with its attributes, dynamic, numeric, read as arithmetic when assigned, set by a builtin or a construct; `DYNAMIC`, `NUMERIC_DYNAMIC`, `SPECIAL` and `SHELL_SET` are gone, their readers use `changedByBash`, `alwaysNumeric`, `changedUnseen`, `setAtStart`, `assignsArithmetic`. test/bash-variables.test.ts (against `env -i bash`: `compgen -v` and `declare -p` at start, assignments to each kind, the builtins that set them, every name of bashref.html when bash-doc is installed); test/trust.test.ts "reads bash's own variables from the catalog" (fixes: `shift $((OPTIND-1))`, `(( EUID == 0 ))` no longer turn off every decision; `OPTIND='x=5'` sets x). An unset gives no text (`unset OPTIND` keeps the decisions; test/trust.test.ts, same case). Probe tests unchanged; sample.sh dump unchanged. Left, for 3.2: the walk does not give OPTIND, RANDOM, SRANDOM and HISTCMD the integer attribute at start, so after `OPTIND='x=5'` its state keeps `x=1` (decisions stay right: the trust scan does not trust `x`) Evaluator (PR #226): `condPieces`/`condValue`/`condPattern`/`condFields` are gone (no wrappers left); the walk calls `wordIn` (src/semantics/words.ts), the evaluator of `wordFields`, for the words of `[ ]`, `[[ ]]`, `case` and a command name, through a `ValueSource` (the walk's: one object reading `variableText` with its trust and reads, `$?`, `~` from HOME; the model's: `ModelValues`). Its `builtinRuns` decides whether `$(true)`, `$(:)`, `$(false)` print nothing (the walk: functions, aliases, `enable -n` of its state; the model: `resolveName`); the walk's assignments (`staticText`) and the trust scan's values read it too. test/conditions.test.ts "the status after a statement against bash" (rows against bash: `[[ abc == $p* ]]`, which the walk decided false, `${x:-w}`, `${x#p}`, `~`, `$(( ))`, braces, `set -f`, `${x:?}` failing, `$(true)` after a function, an alias or `enable -n` of `true`, also through `x=$(true)` and `declare`); test/words.test.ts "takes `$(true)` for empty only where `true` runs the builtin" (with bash). Explain time (esbuild bundle, main → PR): a 1800-line script of 1500 conditions, walk 100 → 145–170 ms, all lines 375 → 420 ms; bash_completion (2290 lines), walk 58 → 78 ms, all lines 415 → 445 ms. Left, for 3.2: `=~` decides only a plain literal on its right (`p=b; [[ abc =~ $p ]]` is not decided); `$(( ))` in a condition's word uses the constant-only `evalArith`, not the walk's arithmetic; and the walk's evaluation costs about 1.5× on condition-heavy scripts (a meaning and an evaluator per read) | +| S2 | Second word evaluator in the walk; four variable catalogs | confirmed | 2.6 | catalog: PR #225; evaluator: PR #226; the rest (3.2): PR #242 | `BASH_VARIABLES` in src/semantics/vars.ts (every variable of the manual's 5.1 and 5.2 lists, and TERM, COPROC_PID): set at start with its attributes, dynamic, numeric, read as arithmetic when assigned, set by a builtin or a construct; `DYNAMIC`, `NUMERIC_DYNAMIC`, `SPECIAL` and `SHELL_SET` are gone, their readers use `changedByBash`, `alwaysNumeric`, `changedUnseen`, `setAtStart`, `assignsArithmetic`. test/bash-variables.test.ts (against `env -i bash`: `compgen -v` and `declare -p` at start, assignments to each kind, the builtins that set them, every name of bashref.html when bash-doc is installed); test/trust.test.ts "reads bash's own variables from the catalog" (fixes: `shift $((OPTIND-1))`, `(( EUID == 0 ))` no longer turn off every decision; `OPTIND='x=5'` sets x). An unset gives no text (`unset OPTIND` keeps the decisions; test/trust.test.ts, same case). Probe tests unchanged; sample.sh dump unchanged. Left, for 3.2: the walk does not give OPTIND, RANDOM, SRANDOM and HISTCMD the integer attribute at start, so after `OPTIND='x=5'` its state keeps `x=1` (decisions stay right: the trust scan does not trust `x`) Evaluator (PR #226): `condPieces`/`condValue`/`condPattern`/`condFields` are gone (no wrappers left); the walk calls `wordIn` (src/semantics/words.ts), the evaluator of `wordFields`, for the words of `[ ]`, `[[ ]]`, `case` and a command name, through a `ValueSource` (the walk's: one object reading `variableText` with its trust and reads, `$?`, `~` from HOME; the model's: `ModelValues`). Its `builtinRuns` decides whether `$(true)`, `$(:)`, `$(false)` print nothing (the walk: functions, aliases, `enable -n` of its state; the model: `resolveName`); the walk's assignments (`staticText`) and the trust scan's values read it too. test/conditions.test.ts "the status after a statement against bash" (rows against bash: `[[ abc == $p* ]]`, which the walk decided false, `${x:-w}`, `${x#p}`, `~`, `$(( ))`, braces, `set -f`, `${x:?}` failing, `$(true)` after a function, an alias or `enable -n` of `true`, also through `x=$(true)` and `declare`); test/words.test.ts "takes `$(true)` for empty only where `true` runs the builtin" (with bash). Explain time (esbuild bundle, main → PR): a 1800-line script of 1500 conditions, walk 100 → 145–170 ms, all lines 375 → 420 ms; bash_completion (2290 lines), walk 58 → 78 ms, all lines 415 → 445 ms. Left, for 3.2: `=~` decides only a plain literal on its right (`p=b; [[ abc =~ $p ]]` is not decided); `$(( ))` in a condition's word uses the constant-only `evalArith`, not the walk's arithmetic; and the walk's evaluation costs about 1.5× on condition-heavy scripts (a meaning and an evaluator per read) 3.2 part (PR #242): the start state gives OPTIND, RANDOM, SRANDOM and HISTCMD the integer attribute (`INTEGERS_AT_START` in vars.ts, also the baseline of test/state-probes.test.ts), and text given to an integer runs through the walk's arithmetic with what it assigns (`WalkArith.integer`): test/state.test.ts "runs the text given to an integer as arithmetic…" (bash: `OPTIND='x=5'`, `declare -i n; n='x=5'`, `n+=`, 64-bit wrap, `unset OPTIND`, `local OPTIND`, an error; before a function or `eval`, `n='x=5' f` sets no x, as bash makes the temporary without the integer attribute, but `n+='x=7' f` does, also before a builtin or a program, `n+='x=7' true`; an element, `n[0]=5 f`, is refused as bash does: before, the model failed on it); `OPTIND=1` keeps status 0 (test/conditions.test.ts rows). The right of `=~` is read with `wordIn(…, 'regex')` (quoted characters escaped as bash does) and decided when it matches plain text (`plainRegex`); `$(( ))` in a condition's word runs with the walk's arithmetic (`ValueSource.arith`): test/conditions.test.ts rows (`p=b; [[ abc =~ $p ]]`, `"$p"`, `^a$p`, `a\.c`, `\w`, `false; [[ $(( $? + 1 )) == 2 ]]`; a name arithmetic reads stays untrusted, so `n=2; [[ $(( n + 1 )) == 3 ]]` is not decided). Cost: a plain word is its own value at once, `commandName` is kept per node, the walk reads a tree's definitions in one pass; esbuild bundle, main → PR, a 1,801-line script of conditions (`[[ ]]`, `[ ]`, `case`, `=~`, `(( ))`) 145 → 112 ms, bash_completion 95 → 84 ms, the fixtures 15 → 11 ms (medians) | | S3 | errexit done twice | corrected: the code agrees; status.ts:259–260 and semantics.md:809 are stale (CLAUDE.md is right). `errexitAt` is redundant for the value but supplies `errexitWhy` | 1.5 (docs), 2.5 (one source) | PR #208 (docs); PR #220 | docs only (status.ts comment, semantics.md 4.11). 2.5: `errexitAt`/`fromCalls` are gone; `statusModel(...).errexit` reads errexit and its reason from the run state (`ShellState.errexitWhy`, set by `substitutionState`, cleared by `set ±e`, carried into functions called in `$( )`; a recursive call in `$( )`, also through another function, walks the body again with errexit as on both paths). test/status-probes.test.ts and test/state-probes.test.ts unchanged (plus the new posix rows); test/status.test.ts "says why errexit is what it is" and "turns errexit off in `$( )` in bash, unless inherit_errexit or POSIX mode" (recursive and mutually recursive calls, against bash -ec) (new rows: functions called in `$( )`, nested `$( )`, `#!/bin/bash --posix`); sample.sh dump unchanged | | S4 | bats: any host function call blinds the trust; host threaded; run re-parsed | confirmed (one `skip` blinds the whole file) | 2.1 | PR #222 | bats' functions carry effects in src/dict/bats.ts (`run`: `shellEffect.always` status, output, lines, BATS_RUN_COMMAND, maybe stderr/stderr_lines; `load`, `bats_load_library`: `shellEffect.runsUnread: 'file'`; `bats_pipe`, which evals its pipeline here: `'command-line'`, which makes the whole file trust nothing); the walk and the trust scan read a host function by its spec as a builtin (not behind `command`/`builtin`, not when the file defines its own). `host` is the spec map (`HostCommands`): `scriptTrust` caches per host and tree (no bypass), `shellState`'s key names it. test/bats-real.test.ts rows "skip in a test does not keep a condition…", "run's command runs in a subshell: a condition after it is decided", "bats_require_minimum_version, and bats_pipe run by run…", "bats_pipe called by the file…" (`lastpipe`; against bats); test/hosts.test.ts "reads bats' functions by their specs…". The walk does not run `run`'s command (`run eval S`: docs/design/bats.md); `Walker.facts` passes the host for strings this shell runs; a child shell's model (`bash -c`) gets none, since bats does not export its functions | -| S5 | Walker god class; hand-listed reuse key; `opaque` | corrected: about 50 fields, 19 mutable (not ~40 mutable) | 3.2 | planned | | +| S5 | Walker god class; hand-listed reuse key; `opaque` | corrected: about 50 fields, 19 mutable (not ~40 mutable) | 3.2 | PR #240 (split), PR #241 (reuse key, `opaque`) | The Walker is split (PR #240): src/semantics/walk/walker.ts (the walk: statements, jumps, loops, calls), arith.ts, conditions.ts, builtins.ts (the builtin effects and assignments), aliases.ts, shared.ts; the state and its merge in src/semantics/shellstate.ts; state.ts keeps the model. Every test, probe and fixture dump unchanged. PR #241: the modes are one record (`Walker.mode`: dead, quiet) and the reuse key is derived from it (`reuseKey`: each mode, and whether an error ends a line); a reused walk makes its changes again for what collects them, which fixes a trap's action that reuses a walk: test/state.test.ts "a walk reused makes its changes again…" (bash: `f() { x=2; }; x=1; f; x=1; trap f USR1; kill -USR1 $$` gives 2; before, the state said 1), and the dead-code case. `opaque` is `ShellState.opaque`, merged at joins, and a change other code may see (`widen`); the arithmetic memos depend on it: test/state.test.ts "says so on the paths that ran what the walk does not read…" (bash; a branch that sources nothing reads an unset name as unset). Probe tests and test/decisions.test.ts unchanged | | S6 | Unseen calls decided twice (env.ts counts names) | confirmed | 2.2 | PR #224 | `EnvModel` reads `WalkFacts.closed`, which is `Trust.closed` (false after a walk that found a reason to trust nothing); the count of name words is gone. A call in an `eval`'s string is now a call the walk sees for both (it runs the string there), one in a trap's action or an alias's text is not. test/env.test.ts "runs a function body where it is called" (new rows: `. ./lib.sh` makes it unsure, `declare -f m`, `type m`, `command -v m`, `unset -f` do not; the alias row is now unsure: the scan reads an alias's text where it is defined and does not expand aliases inside other strings, so it cannot tell which uses the walk sees, `trap g EXIT` calling `f` through `g`); the note on a body that may run in a subshell says so when every seen call is in one ("what the script does not show may call it too"), test/env.test.ts "says that a builtin which acts on its shell acts on the subshell only"; test/trust.test.ts "calls a function in an eval's string where the walk runs it" (bash); test/env-probes.test.ts and the other probes unchanged | -| S7 | Trust scan re-derives structure from text; two parsers per string | confirmed | 2.2 (eval, trap, indirection, slices), 3.2 (alias texts) | 2.2 part: PR #224; alias part planned, 3.2 | Indirection is read from the adapter's `!` token and name node (`indirect` in trust.ts), slices from the `:` operator token. This fixes indirections the regex missed, each of which may assign through a subscript: `${!1}`, `${!@}`, `${!*}`, `${!v@Q}`, `${!a[@]:-x}` (before: conditions after them decided). test/trust.test.ts "reads an indirection from the tree" (19 forms against bash; `${!#}`, `${!-}`, `${!a[@]}`, `${!v@}` stay trusted). The strings of `eval` and traps are parsed once, by the model's `Scripts` (`StringTrees`), and the scan reads those trees; `foreignParser` and the per-text dedupe are gone; the scan is made once per model (the trees are the model's), and its facts cover the strings' statements (`Trust.reads`), with what the whole script declares: test/trust.test.ts "reads the strings the walk runs from the walk's own trees" (`declare -A m; eval 'm[k]=1'`, bash). Left, taken over by 3.2: an alias's text is still parsed twice (the scan parses the value once per word through `StringTrees`; the walk parses it with the rest of each command it starts), and `Walker.facts` keeps a scan of that tree (`treeFacts`). sample.sh dump unchanged; explain time of a 400-line script of evals, traps and aliases: 1.4 s vs 1.5 s on main | +| S7 | Trust scan re-derives structure from text; two parsers per string | confirmed | 2.2 (eval, trap, indirection, slices), 3.2 (alias texts) | 2.2 part: PR #224; alias part: PR #241 | Indirection is read from the adapter's `!` token and name node (`indirect` in trust.ts), slices from the `:` operator token. This fixes indirections the regex missed, each of which may assign through a subscript: `${!1}`, `${!@}`, `${!*}`, `${!v@Q}`, `${!a[@]:-x}` (before: conditions after them decided). test/trust.test.ts "reads an indirection from the tree" (19 forms against bash; `${!#}`, `${!-}`, `${!a[@]}`, `${!v@}` stay trusted). The strings of `eval` and traps are parsed once, by the model's `Scripts` (`StringTrees`), and the scan reads those trees; `foreignParser` and the per-text dedupe are gone; the scan is made once per model (the trees are the model's), and its facts cover the strings' statements (`Trust.reads`), with what the whole script declares: test/trust.test.ts "reads the strings the walk runs from the walk's own trees" (`declare -A m; eval 'm[k]=1'`, bash). Left, taken over by 3.2: an alias's text is still parsed twice (the scan parses the value once per word through `StringTrees`; the walk parses it with the rest of each command it starts), and `Walker.facts` keeps a scan of that tree (`treeFacts`). sample.sh dump unchanged; explain time of a 400-line script of evals, traps and aliases: 1.4 s vs 1.5 s on main Alias part (PR #241): the trees of the aliases the walk expands (an alias's text with the rest of the command) are the model's (`Scripts.expansion`), and the scan reads them at each command named like an alias the script defines (the text foreign, the rest where the command is; not an alias in itself; at most 1,000 from each command of the script, so a chain of aliases that doubles them trusts nothing, but many plain uses do not; and 20,000 in all, past which the walk runs the aliases in trees shared by text and nothing is trusted); `treeFacts`, `Trust.reads` and the global cache of alias texts are gone; an alias no command names is read alone, as before. test/trust.test.ts "reads the aliases the walk expands from the walk's own trees…" (bash: `m[k]=1` in an alias after `declare -A m` keeps the decisions; before, nothing was trusted; the bound, and 1,200 plain uses of aliases that keep the decisions, with and without expand_aliases) and the `alias 'g,h=f'` row (bash expands it) | | S8 | `aliasTrees` cache never evicts | confirmed (grows with distinct alias texts) | 1.5 | PR #208 | test/state.test.ts "the cache of alias texts" (300 alias texts, then a repeat: the values are right; the bound itself is not observable, so not tested) | | S9 | Stale doc (model.ts, layers diagram), c8 ignores, stray comment | corrected: words.ts:846–847 is a rethrow guard (needs a reason, stays); status.ts:363 is the unreachable one | 1.5 | PR #208 | docs and comments only; `enclosingBlock` has no unreachable return; coverage check | | D1 | Embedded languages read options from raw words; `grep --invert` | confirmed; `grep --invert` part is D3's | 1.1 (readers), 3.5 (prefixes) | 1.1 part: PR #212; prefixes: PR #232 (grep, sed, tr); the rest: PR #248 | cmdFlags, regexFlavor, sedScripts, jqVars, trContext, the perl letters and `-i`, the yq flags read the parse. Since PR #248, awkInvocation and the yq `-o` format read the parse too (awk reads options only before the program, as gawk: examples-awk, options-fixes "awk" rows); long prefixes (`grep --invert`, `sed --expr`, `tr --del`) are read since PR #232 (test/prefixes-real.test.ts) | diff --git a/src/semantics/conditions.ts b/src/semantics/conditions.ts index 583603f7..b2b56a02 100644 --- a/src/semantics/conditions.ts +++ b/src/semantics/conditions.ts @@ -136,3 +136,13 @@ export function patternMatches(subject: string, pattern: string): boolean | unde if (!re) return undefined; return re.test(subject); } + +/** + * A regex of `[[ =~ ]]` (its quoted characters escaped with a backslash: src/semantics/words.ts `wordIn`) that matches + * plain text, when it is one: letters, digits, ` _,:/-` and characters escaped (a backslash before a letter or a digit + * is an operator: `\w`, `\1`), maybe anchored at the start (`^`) and at the end (`$`). + */ +export function plainRegex(re: string | undefined): { start: boolean; text: string; end: boolean } | undefined { + const m = re === undefined ? null : /^(\^?)((?:[A-Za-z0-9_ ,:/-]|\\[^A-Za-z0-9])*)(\$?)$/.exec(re); + return m ? { start: m[1] !== '', text: m[2].replace(/\\([^])/g, '$1'), end: m[3] !== '' } : undefined; +} diff --git a/src/semantics/effects.ts b/src/semantics/effects.ts index 59d53c0a..e15fdb84 100644 --- a/src/semantics/effects.ts +++ b/src/semantics/effects.ts @@ -18,9 +18,12 @@ export const SET_LETTERS: Readonly> = { t: 'onecmd', u: 'nounset', v: 'verbose', x: 'xtrace', B: 'braceexpand', C: 'noclobber', E: 'errtrace', H: 'histexpand', P: 'physical', T: 'functrace', }; +const names = new WeakMap(); /** The static name of the simple command `c` (`'exit'` is `exit`), or undefined when it is not known here (`$cmd`). */ export function commandName(c: SyntaxNode): string | undefined { - return staticWordValue(field(c, 'name')!.namedChildren[0]); + let name = names.get(c); + if (name === undefined) names.set(c, (name = staticWordValue(field(c, 'name')!.namedChildren[0]) ?? null)); + return name ?? undefined; } /** The name of the simple command `c` when bash may read it as an alias: a plain word, with no quote or expansion. */ diff --git a/src/semantics/vars.ts b/src/semantics/vars.ts index 4746353f..c6775289 100644 --- a/src/semantics/vars.ts +++ b/src/semantics/vars.ts @@ -114,6 +114,9 @@ export const setAtStart = (name: string): boolean => known(name)?.start !== unde /** Whether the text assigned to `name` is read as arithmetic (`OPTIND`, `RANDOM`: integers from the start). */ export const assignsArithmetic = (name: string): boolean => !!known(name)?.arithmetic; +/** The variables bash makes integers when it starts, whose text it reads as arithmetic (`OPTIND`, `RANDOM`…). */ +export const INTEGERS_AT_START: readonly string[] = Object.keys(BASH_VARIABLES).filter(assignsArithmetic); + /** Whether `name` is readonly from the start (`UID`): what the script assigns to it fails. */ export const readonlyAtStart = (name: string): boolean => known(name)?.start?.includes('r') ?? false; diff --git a/src/semantics/walk/aliases.ts b/src/semantics/walk/aliases.ts index f5fd3983..e9fddaf6 100644 --- a/src/semantics/walk/aliases.ts +++ b/src/semantics/walk/aliases.ts @@ -1,7 +1,7 @@ // The aliases of the walk of the shell state (src/semantics/walk/walker.ts): the `alias` commands of the script, and // a command whose name is an alias, which runs the alias's text with the rest of the command, read again. The trees of // those texts are the model's (`Scripts.expansion`): the trust scan reads the same ones (src/semantics/trust.ts). -import { descendants, fields, type SyntaxNode } from '../../syntax/tree'; +import { fields, type SyntaxNode } from '../../syntax/tree'; import { aliasName, aliasOf, commandName } from '../effects'; import { type AliasState, merge, option, type ShellState, withStatus } from '../shellstate'; import type { Walker } from './walker'; @@ -15,10 +15,10 @@ export class WalkAliases { /** The aliases being expanded (an alias is not expanded again inside itself). */ private readonly expanding = new Set(); - /** Records the `alias` commands under `root`. */ - collect(root: SyntaxNode): void { - for (const c of descendants(root, ['command'])) { - if (commandName(c) !== 'alias') continue; + /** Records the `alias` commands among `nodes`. */ + collect(nodes: readonly SyntaxNode[]): void { + for (const c of nodes) { + if (c.type !== 'command' || commandName(c) !== 'alias') continue; for (const a of fields(c, 'argument')) { const name = aliasOf(a)?.name; const defs = name ? this.defs.get(name) ?? [] : []; diff --git a/src/semantics/walk/arith.ts b/src/semantics/walk/arith.ts index b14ba4ae..c3ac8e51 100644 --- a/src/semantics/walk/arith.ts +++ b/src/semantics/walk/arith.ts @@ -178,6 +178,27 @@ export class WalkArith { } } + /** + * The text given to an integer variable (`declare -i n; n='x = 5'`), run as arithmetic at `at`, as bash does: the + * state after it (what it assigns), and its value when known (not where bash fails, nor from a value it is unsure of). + * A constant is read at once (an assignment that arithmetic makes, `(( n = 3 ))`, gives one). + */ + integer(text: string, st: ShellState, at: SyntaxNode): { st: ShellState; value?: bigint } { + const constant = evalArithWrapping(text); + if (constant) return { st, value: constant.value }; + const { unsure } = this; + try { + const r = this.arithGuard(text, st, at, () => this.arithText(text, st, at, true)); + return this.unsure ? { st: r.st } : r; + } catch (e) { + if (e instanceof ArithError) return { st: e.st }; + /* c8 ignore next -- only an arithmetic error is thrown on purpose */ + throw e; + } finally { + this.unsure = unsure; + } + } + /** `arithScript` with no guard: `ArithUnknown` goes up to the expression of the script that holds it. */ private arithEval(nodes: readonly SyntaxNode[], st: ShellState): { st: ShellState; value?: bigint } { if (!nodes.length) return { st, value: 0n }; diff --git a/src/semantics/walk/builtins.ts b/src/semantics/walk/builtins.ts index 3932b7f3..6ba643af 100644 --- a/src/semantics/walk/builtins.ts +++ b/src/semantics/walk/builtins.ts @@ -1,13 +1,13 @@ // The builtin effects of the walk of the shell state (src/semantics/walk/walker.ts): what the builtins and the // assignments change in the state (`set`, `shopt`, `alias`, `enable`, `hash`, the declarations, `unset`, `read`…), // read with their specs (src/semantics/effects.ts). -import { evalArith } from '../../arith'; +import { evalArithWrapping } from '../../arith'; import { interpret, type ArgRole, type ArgvParse } from '../../argv/interpret'; import type { CommandSpec } from '../../argv/spec'; import { argWord, prefixOf } from '../../argv/words'; import { BASH_ONLY_BUILTINS, builtins } from '../../dict/builtins'; import { staticWordValue } from '../../staticword'; -import { descendants, field, type SyntaxNode } from '../../syntax/tree'; +import { descendants, field, fields, type SyntaxNode } from '../../syntax/tree'; import { aliasOf, builtinParse, DECLARATIONS, setOptionWrites, variableWrites, wrapped } from '../effects'; import { attribute, type Attribute, merge, option, POSIX_ON, type ShellState, type Tri, withoutErrexitWhy } from '../shellstate'; import { optionRoles, readDeclarationWords, type Declaration } from '../vars'; @@ -319,7 +319,7 @@ export class WalkBuiltins { // (The state of an assignment in a declaration has the attributes the declaration gives. A value that may not be a // valid expression, given to an integer, may discard the rest of the line: see `assignmentError`.) if (o.node.type === 'variable_assignment') this.w.record(o.node, st); - if ((attrs.i?.state ?? 'off') !== 'off' && !o.subscript && (o.assign.text === undefined || evalArith(o.assign.text) === undefined)) this.w.abort(st, false); + if ((attrs.i?.state ?? 'off') !== 'off' && !o.subscript && this.mayFail(o.assign.text, st, o.node)) this.w.abort(st, false); return this.assign(st, name, false, o.assign, o.node, nameref, true); } @@ -384,15 +384,18 @@ export class WalkBuiltins { return name; } - /** The assignment `a`; `statement`: one with no command (`x=1`, not `x=1 cmd`), where an error discards the rest of the line. */ - assignNode(a: SyntaxNode, st: ShellState, statement = false): ShellState { + /** + * The assignment `a`; `statement`: one with no command (`x=1`, not `x=1 cmd`), where an error discards the rest of + * the line; `temporary`: one before a function call (`x=1 f`), see `assign`. + */ + assignNode(a: SyntaxNode, st: ShellState, statement = false, temporary = false): ShellState { const nameNode = field(a, 'name')!; const value = field(a, 'value'); const subscript = nameNode.type === 'subscript'; const name = subscript ? field(nameNode, 'name')!.text : nameNode.text; const text = value ? this.w.conditions.staticText(value, st, true) : ''; - if (statement) this.assignmentError(st, name, value?.type === 'array' ? '' : text); - return this.assign(st, name, subscript, { append: a.children.some((c) => c.type === '+='), text, array: value?.type === 'array', source: value }, a, false); + if (statement) this.assignmentError(st, name, value?.type === 'array' ? '' : text, a); + return this.assign(st, name, subscript, { append: a.children.some((c) => c.type === '+='), text, array: value?.type === 'array', source: value }, a, false, false, temporary); } /** @@ -400,20 +403,64 @@ export class WalkBuiltins { * variable (`declare -i n; n=1/0`), is an error that discards the rest of the line, as an expansion error does * (see `abort`): surely when the variable surely is readonly, else it may. */ - private assignmentError(st: ShellState, name: string, text: string | undefined): void { + private assignmentError(st: ShellState, name: string, text: string | undefined, at: SyntaxNode): void { const target = this.resolve(st, name); const v = target === undefined ? undefined : st.variables.get(target); const readonly = v?.attrs.r?.state ?? 'off'; if (readonly === 'on') throw new ExpansionError(st, true); - if (readonly === 'unknown' || ((v?.attrs.i?.state ?? 'off') !== 'off' && (text === undefined || evalArith(text) === undefined))) this.w.abort(st, false); + if (readonly === 'unknown' || ((v?.attrs.i?.state ?? 'off') !== 'off' && this.mayFail(text, st, at))) this.w.abort(st, false); } - /** `name=value`, `name+=value`, `name[i]=value`, `name=(…)`; `nameref`: the assignment of a `declare -n`. */ - assign(st: ShellState, name: string, subscript: boolean, how: NonNullable, at: SyntaxNode, nameref: boolean, declaring = false): ShellState { + /** Whether `text`, given to an integer at `at`, may fail: it is not known, or the walk's arithmetic gives it no value. */ + private mayFail(text: string | undefined, st: ShellState, at: SyntaxNode): boolean { + return text === undefined || this.w.arith.integer(text, st, at).value === undefined; + } + + /** + * The assignments before the builtin or program `n` runs (`n+='x=7' true`), which are the command's alone: only + * `+=` to an integer leaves something, the arithmetic its text runs (bash adds it to the value, running it), and + * only the first assignment of a name: the variable it makes has no attribute (`n+='x=7' n+='x=9' true` sets x to + * 7). Not before `eval`, which makes them in the shell itself (`Walker.evalScript`). + */ + appendArith(assignments: SyntaxNode[], n: SyntaxNode, st: ShellState): ShellState { + let name = field(n, 'name')!.text; + let args = fields(n, 'argument'); + while (name === 'command' || name === 'builtin') { + const inner = wrapped(name, args); + if ('lookup' in inner || inner.name === undefined) break; + ({ name, args } = inner); + } + if (name === 'eval') return st; + const seen = new Set(); + for (const a of assignments) { + const nameNode = field(a, 'name')!; + const value = field(a, 'value'); + const first = !seen.has(nameNode.text); + seen.add(nameNode.text); + if (!first || nameNode.type === 'subscript' || !value || !a.children.some((c) => c.type === '+=')) continue; + const target = this.resolve(st, nameNode.text); + const text = this.w.conditions.staticText(value, st, true); + if (target === undefined || attribute(st, target, 'i') !== 'on' || text === undefined) continue; + st = this.w.arith.integer(text, st, a).st; + } + return st; + } + + /** + * `name=value`, `name+=value`, `name[i]=value`, `name=(…)`; `nameref`: the assignment of a `declare -n`; + * `temporary`: one before a function call (`n=1+1 f`), whose variable bash makes without the integer and case + * attributes: `=` gives it the text as it is (`n='x=5' f` sets no x), but `+=` still runs an integer's arithmetic + * (then the variable made has no attribute either: a second `+=` of it appends text). + */ + assign(st: ShellState, name: string, subscript: boolean, how: NonNullable, at: SyntaxNode, nameref: boolean, declaring = false, temporary = false): ShellState { // Assigning to a nameref assigns to the variable it names. const target = nameref ? name : this.resolve(st, name); if (!target) return st; - const prev = st.variables.get(target); + let prev = st.variables.get(target); + if (temporary && !how.append && prev) { + st = this.withoutCase(st, target); + prev = st.variables.get(target); + } // A readonly variable keeps its value. if (prev?.attrs.r?.state === 'on' && !declaring) return st; if (subscript || how.array) { @@ -424,9 +471,11 @@ export class WalkBuiltins { const is = (l: string): boolean => prev?.attrs[l]?.state === 'on'; if (is('a') || is('A')) text = undefined; else if (is('i') && text !== undefined) { - const v = evalArith(text); - const old = prev?.value !== undefined ? evalArith(prev.value) : how.append ? undefined : 0; - text = v === undefined || old === undefined ? undefined : String(how.append ? old + v : v); + // (The text is run as arithmetic, with what it assigns: `n='x = 5'` sets x.) + const r = this.w.arith.integer(text, st, at); + st = r.st; + const old = prev?.value !== undefined ? evalArithWrapping(prev.value)?.value : how.append ? undefined : 0n; + text = r.value === undefined || old === undefined ? undefined : String(BigInt.asIntN(64, how.append ? old + r.value : r.value)); } else if (how.append) text = prev?.set === 'on' && prev.value !== undefined && text !== undefined ? prev.value + text : prev?.set === 'off' ? text : undefined; if (text !== undefined && is('l')) text = text.toLowerCase(); if (text !== undefined && is('u')) text = text.toUpperCase(); @@ -434,7 +483,15 @@ export class WalkBuiltins { if (is('n') && !nameref && (text === undefined || !NAME.test(text))) return st; // (An integer or a case attribute changes the value: then the word alone does not give it.) const source = text === undefined && !how.append && !['i', 'l', 'u'].some(is) ? how.source : undefined; - return this.setVariable(st, target, at, text, false, is('a') || is('A'), source); + st = this.setVariable(st, target, at, text, false, is('a') || is('A'), source); + return temporary ? this.withoutCase(st, target) : st; + } + + /** `name` without the integer and case attributes (a variable made before a function call, see `assign`). */ + private withoutCase(st: ShellState, name: string): ShellState { + const v = st.variables.get(name)!; + const { i: _i, l: _l, u: _u, ...attrs } = v.attrs; + return Object.keys(attrs).length === Object.keys(v.attrs).length ? st : this.w.put(st, 'variables', name, { ...v, attrs }); } /** Sets `name` (to `value`, when known), exporting it under `set -a` (not an array: bash cannot export one). */ diff --git a/src/semantics/walk/conditions.ts b/src/semantics/walk/conditions.ts index 41c34c55..b284744e 100644 --- a/src/semantics/walk/conditions.ts +++ b/src/semantics/walk/conditions.ts @@ -1,8 +1,9 @@ // The conditions of the walk of the shell state (src/semantics/walk/walker.ts): the status of a statement when it is // known (`[ ]`, `[[ ]]`, `true`, `(( ))`, assignments), the lists of a `case` that run, and what each condition decided. +import { evalArith } from '../../arith'; import { staticWordValue, substitutionNames } from '../../staticword'; import { descendants, field, fields, type SyntaxNode } from '../../syntax/tree'; -import { compareIntegers, patternMatches, testStatus, type TestEnv, unaryTest } from '../conditions'; +import { compareIntegers, patternMatches, plainRegex, testStatus, type TestEnv, unaryTest } from '../conditions'; import { attribute, EMPTY, option, type ShellState, type Tri } from '../shellstate'; import type { Decision } from '../state'; import { type ValueSource, wordIn } from '../words'; @@ -89,7 +90,10 @@ export class WalkConditions { for (const a of assignments) { const name = field(a, 'name')!.text; const target = NAME.test(name) ? this.w.builtins.resolve(st, name) : undefined; - if (!target || attribute(st, target, 'r') !== 'off' || attribute(st, target, 'i') !== 'off') return undefined; + if (!target || attribute(st, target, 'r') !== 'off') return undefined; + // (An integer runs its text: only a constant surely gives a number, `OPTIND=1`.) + const value = field(a, 'value'); + if (attribute(st, target, 'i') !== 'off' && (!value || evalArith(this.staticText(value, st, true) ?? '') === undefined)) return undefined; } return 0; } @@ -191,11 +195,12 @@ export class WalkConditions { return m === undefined ? undefined : m === (op !== '!='); } if (op === '=~') { - // Only a regex of plain characters, maybe anchored: `^abc`, `abc$`, `abc`; not under nocasematch, which it follows. + // A regex that matches plain text, maybe anchored (`^abc`, `abc$`), once expanded (`$p`; quoted characters + // are plain); not under nocasematch, which it follows. const nocase = option(st, 'shopt', 'nocasematch'); - const m = right && (nocase === 'off' || nocase === 'default') && ['regex', 'word', 'number'].includes(right.type) ? /^(\^?)([A-Za-z0-9_ ,:/-]*)(\$?)$/.exec(right.text) : null; + const m = right && (nocase === 'off' || nocase === 'default') ? plainRegex(wordIn(right, st, this.values(st), 'regex')) : undefined; if (!m) return undefined; - return m[1] && m[3] ? value === m[2] : m[1] ? value.startsWith(m[2]) : m[3] ? value.endsWith(m[2]) : value.includes(m[2]); + return m.start && m.end ? value === m.text : m.start ? value.startsWith(m.text) : m.end ? value.endsWith(m.text) : value.includes(m.text); } if (INTEGER_TESTS.has(op)) { // Arithmetic: decimal integers only (`010` is octal there, `x` a variable), with spaces, tabs and newlines around. @@ -235,8 +240,33 @@ export class WalkConditions { }, tilde: (user) => (user === '' ? this.source.variable('HOME', this.w.node!) ?? undefined : undefined), builtinRuns: (name) => this.builtinRuns(name, this.valuesState), + arith: (n) => this.arithValue(n, this.valuesState), }; + /** + * The value of the arithmetic expansion `n` in a word of a condition, run from `st` with the walk's arithmetic: when + * it assigns nothing, and reads only values decisions trust (see `variableText`). Undefined where bash fails on it. + */ + private arithValue(n: SyntaxNode, st: ShellState): bigint | undefined { + const { assumed } = this.w; + const { unsure } = this.w.arith; + this.w.assumed = false; + try { + const r = this.w.arith.arithScript(n.namedChildren, st); + return r.st === st && !this.w.arith.unsure && !this.w.assumed ? r.value : undefined; + // A guard: the walk ran the expansion with the words of the statement (`defaults`), so an error in it ended the + // statement before its condition is read. + /* c8 ignore start */ + } catch (e) { + if (e instanceof ArithError) return undefined; + throw e; + /* c8 ignore stop */ + } finally { + this.w.assumed = assumed; + this.w.arith.unsure = unsure; + } + } + /** The walk's values (`source`), read from `st`. */ private values(st: ShellState): ValueSource { this.valuesState = st; diff --git a/src/semantics/walk/walker.ts b/src/semantics/walk/walker.ts index 76d441bc..ca0e87ed 100644 --- a/src/semantics/walk/walker.ts +++ b/src/semantics/walk/walker.ts @@ -16,7 +16,7 @@ import { } from '../shellstate'; import type { Scripts } from '../state'; import { type HostCommands, NO_HOST, type Trust } from '../trust'; -import { alwaysNumeric, changedByBash, readDeclaration } from '../vars'; +import { alwaysNumeric, changedByBash, INTEGERS_AT_START, readDeclaration } from '../vars'; import { WalkAliases } from './aliases'; import { WalkArith } from './arith'; import { WalkBuiltins } from './builtins'; @@ -91,6 +91,12 @@ function passed(name: string, v: VariableState): VariableState | undefined { : { declared: 'unknown', set: 'unknown', attrs: { x: { state: 'unknown', at: v.attrs.x!.at } }, at: v.at }; } +/** + * Whether the assignment before a command name makes a variable for it: not one to an element (`n[0]=5 f`), which + * bash refuses ("not a valid identifier") before it runs the command, assigning nothing. + */ +const temporary = (a: SyntaxNode): boolean => field(a, 'name')!.type !== 'subscript'; + /** The walk of a tree (`program`), with its parts. */ export class Walker { readonly arith = new WalkArith(this); @@ -219,12 +225,20 @@ export class Walker { return states?.[0]; } - /** The state the script starts with; `root`: the node that sets POSIXLY_CORRECT in POSIX mode (`bash --posix` sets it to `y`). */ - initial(root?: SyntaxNode): ShellState { + /** + * The state a shell starts with, as set at `at` (the root): bash's own integers (`OPTIND`, `RANDOM`…: what the script + * assigns them is arithmetic, `OPTIND='x=5'` sets x); in POSIX mode, its options and POSIXLY_CORRECT (`bash --posix` + * sets it to `y`). + */ + initial(at: SyntaxNode): ShellState { let st = EMPTY; + if (this.bash) { + const attrs = { i: { state: 'on' as const, at } }; + st = { ...st, variables: st.variables.withAll(new Map(INTEGERS_AT_START.map((name) => [name, { declared: 'on', set: 'on', attrs, at }]))) }; + } if (this.posix) { for (const kind of ['set', 'shopt'] as const) for (const name of POSIX_ON[kind]) st = { ...st, [kind]: st[kind].with(name, 'on') }; - if (root) st = { ...st, variables: st.variables.with('POSIXLY_CORRECT', { declared: 'on', set: 'on', attrs: {}, value: 'y', at: root }) }; + st = { ...st, variables: st.variables.with('POSIXLY_CORRECT', { declared: 'on', set: 'on', attrs: {}, value: 'y', at }) }; } else if (this.shell && POSIX_SHELLS.has(this.shell)) { st = { ...st, set: st.set.with('posix', 'on') }; } @@ -239,7 +253,7 @@ export class Walker { * shell that reads startup files (`-l`, `-i`, an exported BASH_ENV) may have any name the script has. */ childStart(outer: ShellState, env: ExecEnv, pass: ChildPass, at: SyntaxNode): ShellState { - let st = this.initial(); + let st = this.initial(at); const exported = (name: string): boolean => (outer.variables.get(name)?.attrs.x?.state ?? 'off') === 'on'; const reach = reachOf(env, exported('SHELLOPTS')); // (`sh` may be bash, which reads exported functions, SHELLOPTS and BASHOPTS, or dash, which reads none.) @@ -247,7 +261,8 @@ export class Walker { if (reach.variables !== 'none') { let vars = outer.variables.mapped(passed, passedMemo); for (const name of pass.removed) vars = vars.with(name, undefined); - st = { ...st, variables: vars }; + // (bash sets its own at start, whatever the environment gives.) + st = { ...st, variables: vars.withAll(new Map(st.variables)) }; } // (`env -i x=1 bash -c …` passes `x`.) for (const name of pass.given) if (NAME.test(name)) st = { ...st, variables: st.variables.with(name, { declared: 'on', set: 'on', attrs: { x: { state: 'on', at } }, at }) }; @@ -300,21 +315,25 @@ export class Walker { return st; } - /** Records the function definitions and the `alias` commands under `root`. */ - private collectDefinitions(root: SyntaxNode): void { - for (const f of descendants(root, ['function_definition'])) { + /** Records the function definitions and the `alias` commands under `root`; gives the nodes of `types` there. */ + private collectDefinitions(root: SyntaxNode, types = ['function_definition', 'command']): SyntaxNode[] { + const nodes = descendants(root, types); + for (const f of nodes) { + if (f.type !== 'function_definition') continue; const name = field(f, 'name')!.text; const defs = this.functionDefs.get(name) ?? []; if (!defs.includes(f)) this.functionDefs.set(name, [...defs, f]); } - this.aliases.collect(root); + this.aliases.collect(nodes); + return nodes; } program(root: SyntaxNode, start: ShellState): ShellState { - this.collectDefinitions(root); - const unset = new Set([...descendants(root, ['unset_command']).flatMap((u) => u.namedChildren), - ...descendants(root, ['command']).filter((c) => commandName(c) === 'unset').flatMap((c) => fields(c, 'argument'))].map((w) => staticWordValue(w) ?? w.text)); - for (const def of descendants(root, ['function_definition'])) { + // (One pass over the tree: the definitions, and the names `unset` may remove.) + const nodes = this.collectDefinitions(root, ['function_definition', 'command', 'unset_command']); + const unset = new Set([...nodes.filter((u) => u.type === 'unset_command').flatMap((u) => u.namedChildren), + ...nodes.filter((c) => c.type === 'command' && commandName(c) === 'unset').flatMap((c) => fields(c, 'argument'))].map((w) => staticWordValue(w) ?? w.text)); + for (const def of nodes.filter((d) => d.type === 'function_definition')) { // (At the top level, or first in a list there: `f() { …; } && complete -F f x` always defines it.) let x = def; while (x.parent?.type === 'list' && x.parent.namedChildren[0] === x) x = x.parent; @@ -1252,11 +1271,11 @@ export class Walker { } else if (temporaries.length) { const frame: Frame = { def, saved: new Map() }; let entry = st; - for (const a of temporaries) { + for (const a of temporaries.filter(temporary)) { const name = field(a, 'name')!.text; if (!frame.saved.has(name)) frame.saved.set(name, entry.variables.get(name)); // (bash exports them to the commands the body runs.) - entry = this.builtins.giveAttribute(this.builtins.assignNode(a, entry), name, 'x', a); + entry = this.builtins.giveAttribute(this.builtins.assignNode(a, entry, false, true), name, 'x', a); } exit = this.walkBody(def, entry, frame); } else { @@ -1410,14 +1429,16 @@ export class Walker { if (expanded) return expanded; // In POSIX mode, the assignments before a special builtin stay after it, exported. const posix = option(before, 'set', 'posix'); - if (assignments.length && name && builtins[name]?.builtin === 'special' && (posix === 'on' || posix === 'unknown')) { + const special = assignments.length > 0 && !!name && builtins[name]?.builtin === 'special' && (posix === 'on' || posix === 'unknown'); + if (special) { let kept = st; - for (const a of assignments) kept = this.builtins.giveAttribute(this.builtins.assignNode(a, kept), field(a, 'name')!.text, 'x', a); - st = posix === 'on' ? kept : merge([st, kept], this.posix); + for (const a of assignments.filter(temporary)) kept = this.builtins.giveAttribute(this.builtins.assignNode(a, kept), field(a, 'name')!.text, 'x', a); + st = posix === 'on' ? kept : merge([this.builtins.appendArith(assignments, n, st), kept], this.posix); } if (name === undefined) return withStatus(st, this.conditions.simpleStatus(n, before)); const fn = this.functionCalled(name, st); if (fn) return withStatus(this.call(fn, st, assignments), undefined); + if (!special) st = this.builtins.appendArith(assignments, n, st); return withStatus(this.builtins.builtin(name, fields(n, 'argument'), n, st, true), this.conditions.simpleStatus(n, before)); } @@ -1488,10 +1509,10 @@ export class Walker { if (!temps.length || option(st, 'set', 'posix') === 'on') return this.runScript(script, n, st); const saved = new Map(); let entry = st; - for (const a of temps) { + for (const a of temps.filter(temporary)) { const v = field(a, 'name')!.text; if (!saved.has(v)) saved.set(v, entry.variables.get(v)); - entry = this.builtins.giveAttribute(this.builtins.assignNode(a, entry), v, 'x', a); + entry = this.builtins.giveAttribute(this.builtins.assignNode(a, entry, false, true), v, 'x', a); } let after = this.runScript(script, n, entry); for (const [v, was] of saved) after = this.put(after, 'variables', v, was); diff --git a/src/semantics/words.ts b/src/semantics/words.ts index f3a2b65e..7489c495 100644 --- a/src/semantics/words.ts +++ b/src/semantics/words.ts @@ -507,6 +507,8 @@ export interface ValueSource { tilde(user: string, w: WordMeaning): string | undefined; /** Whether the command `name` of the substitution `at` (`$(true)`) runs the builtin: not a function, an alias, nor a program after `enable -n`. */ builtinRuns(name: string, at: SyntaxNode): boolean; + /** The value of the arithmetic expansion `n`, when it is known (the walk's arithmetic); without it, of a constant expression only. */ + arith?(n: SyntaxNode): bigint | undefined; } /** The values of the model where the word runs, with the environment `env` for what the script does not set. */ @@ -576,7 +578,7 @@ class Evaluator { /** The pieces of a node of the word, or undefined when they are not known here. */ pieces(n: SyntaxNode, quoted: boolean): Piece[] | undefined { switch (n.type) { - case 'word': case 'number': case 'extglob_pattern': case 'variable_name': case 'brace_expression': + case 'word': case 'number': case 'extglob_pattern': case 'variable_name': case 'brace_expression': case 'regex': return plain(decodedText(n)); case 'raw_string': return [{ text: n.text.slice(1, -1), quoted: true, expanded: false }]; case 'ansi_c_string': { @@ -598,7 +600,7 @@ class Evaluator { } case 'arithmetic_expansion': { const inner = n.text.replace(/^\$(?:\(\(|\[)/, '').replace(/(?:\)\)|\])$/, ''); - const v = evalArith(inner); + const v = this.source.arith ? this.source.arith(n) : evalArith(inner); return v === undefined ? undefined : [{ text: String(v), quoted, expanded: true }]; } } @@ -963,12 +965,16 @@ function splitFields(w: WordMeaning, ev: Evaluator): Char[][] | undefined { /** * The word that holds `n` as the walk of the state reads it in a condition (src/semantics/walk/conditions.ts), from the state `st` * and the values of `source`: its fields (`[ ]`, `test`, a command name; with no file to match a glob against, and - * `ifs: false` when the IFS is not to be followed), its one field (`[[ ]]`, `case`), or its pattern (`[[ == ]]`, a - * `case` pattern: quoted characters escaped). Undefined when it is not known, or bash fails on it. + * `ifs: false` when the IFS is not to be followed), its one field (`[[ ]]`, `case`), its pattern (`[[ == ]]`, a + * `case` pattern: quoted characters escaped), or its regex (`[[ =~ ]]`: quoted characters other than letters, digits + * and `_` escaped with a backslash, as bash does). Undefined when it is not known, or bash fails on it. */ export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: 'fields', ifs: boolean): string[] | undefined; -export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: 'value' | 'pattern'): string | undefined; -export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: 'fields' | 'value' | 'pattern', ifs = true): string[] | string | undefined { +export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: 'value' | 'pattern' | 'regex'): string | undefined; +export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: 'fields' | 'value' | 'pattern' | 'regex', ifs = true): string[] | string | undefined { + // (Most words of conditions are plain text, which is their field, value, pattern and regex as it is: read at once. + // `n` is the whole word: the right side of `[[ == ]]` comes in parts only when it starts with an expansion.) + if ((n.type === 'word' || n.type === 'number') && PLAIN_TEXT.test(n.text)) return as === 'fields' ? [n.text] : n.text; const found = wordOf(n); /* c8 ignore next -- a guard: the walk reads only words of a command, of `[[ ]]` and of `case` */ if (!found) return undefined; @@ -985,7 +991,7 @@ export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: ' if (as !== 'fields') { // (One field: `[[ ]]` and `case` neither split nor drop an empty word.) const f = split[0]; - return as === 'value' ? f.map((c) => c.c).join('') : patternOfChars(f); + return as === 'value' ? f.map((c) => c.c).join('') : as === 'regex' ? f.map((c) => (c.quoted && /\W/.test(c.c) ? `\\${c.c}` : c.c)).join('') : patternOfChars(f); } const fields: string[] = []; for (const f of split) { @@ -1001,6 +1007,9 @@ export function wordIn(n: SyntaxNode, st: ShellState, source: ValueSource, as: ' } } +/** A word that bash takes as it is written: no expansion, quote, escape, tilde, brace, glob or extended glob. */ +const PLAIN_TEXT = /^[\w.,:/@%+=-]+$/; + /** A field as a pattern for globToRegex: quoted characters escaped. */ function patternOfChars(f: readonly Char[]): string { return f.map((c) => (c.quoted && /[\\*?[\]()|!@+]/.test(c.c) ? `\\${c.c}` : c.c)).join(''); diff --git a/test/conditions.test.ts b/test/conditions.test.ts index ec7a29fb..745ba480 100644 --- a/test/conditions.test.ts +++ b/test/conditions.test.ts @@ -117,7 +117,7 @@ describe('the status after a statement against bash (src/semantics/state.ts)', ( const rows: [string, boolean][] = [ ['true', true], ['false', true], [':', true], ['! true', true], ['! false', true], ['true | false', true], ['false | true', true], ['set -o pipefail; false | true', true], ['set -o pipefail; false | true | true', true], ['[ -f x ] && set -o pipefail; false | true', false], - ['x=1', true], ['x=1 y=2', true], ['x=1 >/dev/null', true], ['x=$(false)', false], ['readonly r=1; r=2', false], ['declare -i n; n=x', false], + ['x=1', true], ['x=1 y=2', true], ['x=1 >/dev/null', true], ['x=$(false)', false], ['readonly r=1; r=2', false], ['declare -i n; n=x', false], ['declare -i n; n=3', true], ['OPTIND=1', true], ["OPTIND='1/0'", false], ['set -u; x=$nope', false], ['x=${nope?}', false], ['[ ]', true], ['[ a ]', true], ['[ -n "" ]', true], ['[ a = b ]', true], ['[ 1 -lt 2 ]', true], ['[ a -lt 2 ]', true], ['test a', true], ['test', true], ['x=; [ -n $x ]', true], ['x="a b"; [ -n $x ]', true], ['x="a b"; [ -n "$x" ]', true], ['x="*"; [ $x = a ]', false], ['IFS=:; x=a; [ $x = a ]', false], @@ -126,7 +126,7 @@ describe('the status after a statement against bash (src/semantics/state.ts)', ( ['[ a ] 2>/dev/null', true], ['[ a ] 2>&1', true], ['[ $nope = a ]', false], ['NOPE=1; [ "$NOPE" = 1 ]', true], ['[ "$HOME" = 1 ]', false], ['[[ a ]]', true], ['[[ -z "" ]]', true], ['[[ a == a* ]]', true], ['[[ ab != a? ]]', true], ['[[ a == "a*" ]]', true], ['[[ a* == "a*" ]]', true], ['p="a*"; [[ ab == $p ]]', true], ['p="a*"; [[ ab == "$p" ]]', true], ['[[ 1 -eq 01 ]]', false], ['[[ 2 -gt 1 ]]', true], ['[[ " 2" -ge 3 ]]', true], - ['[[ a < b ]]', false], ['[[ abc =~ ^ab ]]', true], ['[[ abc =~ c$ ]]', true], ['[[ abc =~ bd ]]', true], ['[[ abc =~ ^abc$ ]]', true], ['[[ abc =~ a.c ]]', false], ['[[ 12 =~ 1 ]]', true], ['[[ abc =~ "b" ]]', false], + ['[[ a < b ]]', false], ['[[ abc =~ ^ab ]]', true], ['[[ abc =~ c$ ]]', true], ['[[ abc =~ bd ]]', true], ['[[ abc =~ ^abc$ ]]', true], ['[[ abc =~ a.c ]]', false], ['[[ 12 =~ 1 ]]', true], ['[[ abc =~ "b" ]]', true], ['[[ -o errexit ]]', false], ['set -e; [[ -o errexit ]]', true], ['set -o noglob; [ -o noglob ]', true], ['[[ ! ( a == b ) ]]', true], ['[[ -f /nonexistent || a ]]', true], ['[[ -f /nonexistent && a ]]', false], ['[[ 1 == 2 && $nope == 2 ]]', true], ['x=1; [[ $x == 1 && $nope == 2 ]]', false], ['[[ -f /nonexistent || "" ]]', false], ['[[ -f /nonexistent && "" ]]', true], ['[[ a = ~ ]]', false], ['[[ a=~/x == a ]]', false], @@ -154,6 +154,14 @@ describe('the status after a statement against bash (src/semantics/state.ts)', ( ['shopt -s expand_aliases\nalias true="echo a"\n[ "$(true)" = "" ]', false], ['true() { echo a; }; x=$(true); [ "$x" = "" ]', false], ['true() { echo a; }; declare x=$(true); [ "$x" = "" ]', false], ['IFS=:; x="a:b"; [ $x = a ]', false], ['x=; [[ a == b || ${x:?empty} == a ]]', false], + // The right of `=~` once expanded (quoted characters are plain), and `$(( ))` with the walk's arithmetic. + ['p=b; [[ abc =~ $p ]]', true], ['p=b; [[ abc =~ "$p" ]]', true], ["p='^a'; [[ abc =~ $p ]]", true], ["p='^a'; [[ abc =~ \"$p\" ]]", true], + ['[[ a.c =~ a\\.c ]]', true], ['[[ abc =~ a\\.c ]]', true], ["p='a.c'; [[ abc =~ $p ]]", false], ["p='\\w'; [[ a =~ $p ]]", false], ['[[ a =~ \\w ]]', true], + ['[[ "a b" =~ "a "b$ ]]', true], ['p=b; [[ abc =~ ^a$p ]]', true], ['[[ abc =~ $nope ]]', false], ['x=c; [[ bc == b$x ]]', true], + ['false; [[ $(( $? + 1 )) == 2 ]]', true], ['true; [ $(( $? * 2 )) -eq 0 ]', true], ['false; case $(( $? + 1 )) in 2) false;; esac', true], + // (A name arithmetic reads is never trusted: src/semantics/trust.ts.) + ['n=2; [[ $(( n + 1 )) == 3 ]]', false], + ['[[ $(( nope + 1 )) == 1 ]]', false], ['[[ $(( 1 / 0 )) == 1 ]]', false], ['a=(1); [[ $(( a[0] )) == 1 ]]', false], // What the model does not follow, or must not decide. ['set -o pipefail; [ -f /nonexistent ] | true', false], ['set -o pipefail; true | true', true], ['if true; then false; elif true; then true; fi', true], ['[() { return 3; }; [ a ]', false], ['set +e; [[ -o errexit ]]', true], ['set -o posix; [ a \\< b ]', false], ['[ -f /nonexistent ] && set -o posix; [ a \\< b ]', false], diff --git a/test/state-probes.test.ts b/test/state-probes.test.ts index a63412b2..f4782b02 100644 --- a/test/state-probes.test.ts +++ b/test/state-probes.test.ts @@ -11,10 +11,16 @@ import * as assert from 'assert'; import { createUnbashParser } from '../src/syntax/adapters/unbash'; import { descendants, field, fields } from '../src/syntax/tree'; import { defaultOn, option, shellState, ShellState } from '../src/semantics/state'; +import { INTEGERS_AT_START } from '../src/semantics/vars'; import { instrument, readProbeTable, StateDiff } from './support/probe'; -/** The attributes bash gives its own variables (as in the baseline of the probes), which the model does not follow. */ -const BASELINE_ATTRS: Record = { OPTIND: 'i', PWD: 'x', OLDPWD: 'x', HOME: 'x', PATH: 'x', LC_ALL: 'x', SHLVL: 'x' }; +/** + * The attributes bash gives its own variables (as in the baseline of the probes): the integers, which the model gives + * them at start too (src/semantics/vars.ts), and the exported ones, which it does not follow. + */ +const BASELINE_ATTRS: Record = { + ...Object.fromEntries(INTEGERS_AT_START.map((name) => [name, 'i'])), PWD: 'x', OLDPWD: 'x', HOME: 'x', PATH: 'x', LC_ALL: 'x', SHLVL: 'x', +}; /** How bash's `declare -p` writes a scalar value. */ const ESCAPES: Record = { '\x07': 'a', '\b': 'b', '\x1b': 'E', '\f': 'f', '\n': 'n', '\r': 'r', '\t': 't', '\v': 'v', "'": "'", '\\': '\\' }; diff --git a/test/state.test.ts b/test/state.test.ts index 4463af93..fbe13e9b 100644 --- a/test/state.test.ts +++ b/test/state.test.ts @@ -151,8 +151,9 @@ describe('shell state: paths', () => { describe('shell state: variables', () => { it('follows assignments, attributes and unset', () => { + // (`n=x` runs `x` as arithmetic: x, not set, is 0, as in bash run with no such variable.) const st = end('a=1\na+=2\nb+=x\nc=$x\nc+=1\nd=(1 2)\nd[3]=4\ne[1]=x\ndeclare -A m\nm[k]=v\nm=()\ndeclare -i n=2+3\nn+=1\nn=x\ndeclare -l lo=AB\ndeclare -u up=ab\ni=\nunset i\nunset "d[1]"'); - assert.deepStrictEqual([variable(st, 'a')?.value, variable(st, 'b')?.value, variable(st, 'c')?.value, variable(st, 'n')?.value, variable(st, 'lo')?.value, variable(st, 'up')?.value], ['12', undefined, undefined, undefined, 'ab', 'AB']); + assert.deepStrictEqual([variable(st, 'a')?.value, variable(st, 'b')?.value, variable(st, 'c')?.value, variable(st, 'n')?.value, variable(st, 'lo')?.value, variable(st, 'up')?.value], ['12', undefined, undefined, '0', 'ab', 'AB']); assert.deepStrictEqual(['d', 'e', 'm'].map((v) => [attribute(st, v, 'a'), attribute(st, v, 'A')]), [['on', 'off'], ['on', 'off'], ['off', 'on']]); assert.deepStrictEqual([variable(st, 'i')?.set, variable(st, 'd')?.set], ['off', 'on']); assert.strictEqual(variable(end('declare -i n; n+=1'), 'n')?.value, undefined); @@ -192,7 +193,8 @@ describe('shell state: variables', () => { assert.deepStrictEqual(['a', 'b', 'pv', 'o', 'OPTIND', 'OPTARG', 'd', 'e'].map((v) => variable(st, v)?.set), ['on', 'on', 'on', 'on', 'on', 'unknown', 'on', 'on']); assert.strictEqual(variable(end('read'), 'REPLY')?.set, 'on'); assert.strictEqual(attribute(end("printf -v 'a[1]' x"), 'a', 'a'), 'on'); - assert.strictEqual(end('printf -v $v x; mapfile $m').variables.size, 0); + // (Nothing but the integers bash has at start.) + assert.deepStrictEqual([...end('printf -v $v x; mapfile $m').variables.keys()], [...end('').variables.keys()]); assert.deepStrictEqual(['arr', 'MAPFILE', 'lines'].map((v) => attribute(st, v, 'a')), ['on', 'on', 'on']); assert.deepStrictEqual([variable(st, 'f'), variable(st, 'g')], [undefined, undefined]); assert.strictEqual(variable(end('x=${y:=1}'), 'y')?.set, 'on'); @@ -513,6 +515,63 @@ describe('shell state: arithmetic', () => { // A value read before is not taken again for one that may have changed. assert.deepStrictEqual(after('x=\'foo + 1\'; (( a = x )); source ./c.sh; (( b = x ))', 'a', 'b'), ['set=1', 'unknown']); }); + it("runs the text given to an integer as arithmetic, with what it assigns; bash's own integers are so from the start", () => { + // (A script file, where an error discards the rest of its line only: `bash -c` discards the rest of the string.) + const bash = (src: string): string => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'integers-')); + try { + fs.writeFileSync(path.join(dir, 's.sh'), src); + return spawnSync('bash', ['s.sh'], { cwd: dir, encoding: 'utf8', env: { PATH: '/usr/bin:/bin' }, timeout: 5000 }).stdout; + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }; + // (bash makes OPTIND, RANDOM, SRANDOM and HISTCMD integers when it starts.) + assert.deepStrictEqual(bash('declare -p OPTIND RANDOM SRANDOM HISTCMD').split('\n').filter(Boolean).map((l) => l.split(' ')[1]), ['-i', '-i', '-i', '-i']); + assert.deepStrictEqual(['OPTIND', 'RANDOM', 'SRANDOM', 'HISTCMD'].map((v) => attribute(end(''), v, 'i')), ['on', 'on', 'on', 'on']); + // Each script, the names to look at, what bash has (`$name`), and what the model says (see `after`). + const rows: [string, string[], string, string][] = [ + ["x=1; OPTIND='x=5'", ['x', 'OPTIND'], '5 5', 'set=5 set=5'], + ["x=1; RANDOM='x=6'", ['x'], '6', 'set=6'], + ["declare -i n; n='x=5'", ['x', 'n'], '5 5', 'set=5 set=5'], + ["declare -i n=3; n+='x=5'", ['x', 'n'], '5 8', 'set=5 set=8'], + ["declare -i n; n='x=5, 2 ** 63'", ['x', 'n'], `5 ${-(2n ** 63n)}`, `set=5 set=${-(2n ** 63n)}`], + // (No longer an integer: unset, or a local of the same name.) + ["x=1; unset OPTIND; OPTIND='x=5'", ['x'], '1', 'set=1'], + ["x=1; f() { local OPTIND; OPTIND='x=5'; }; f", ['x'], '1', 'set=1'], + // (bash fails on it, and leaves n as it was, but what it assigned before the error stays; the model takes the + // line as one that may end at the assignment.) + ["declare -i n; n='x=5, 1/0'", ['x', 'n'], '5', 'unknown unknown'], + // (Before a function or `eval`, the variable is made without the integer attribute: `=` gives it the text as it + // is, and sets no x; `+=` still runs the arithmetic. Before, `n='x=5' f` set x.) + ["declare -i n; f() { :; }; x=1; n='x=5' f", ['x'], '1', 'set=1'], + ["declare -i n; x=1; n='x=5' eval :", ['x'], '1', 'set=1'], + ["declare -i n=4; f() { :; }; x=1; n+='x=7' f", ['x', 'n'], '7 4', 'set=7 set=4'], + // (Before a builtin or a program too: only `+=` to an integer runs its text. Before, the model kept x=1.) + ...[['true', '7'], ['/bin/true', '7'], ['builtin eval :', '7'], ['command -v true >/dev/null', '7'], ['command', '7']].map(([cmd, x]): [string, string[], string, string] => + [`declare -i n=4; x=1; n+='x=7' ${cmd}`, ['x', 'n'], `${x} 4`, `set=${x} set=4`]), + ["declare -i n=4; x=1; n+='x++' eval :", ['x', 'n'], '2 4', 'set=2 set=4'], + // (The variable made has no attribute: a second assignment of it appends text. Before, the model ran x=9.) + ...['true', 'f', 'eval :'].map((cmd): [string, string[], string, string] => [`declare -i n=4; f() { :; }; x=1; n+='x=7' n+='x=9' ${cmd}`, ['x', 'n'], '7 4', 'set=7 set=4']), + ["declare -i n=4; f() { :; }; x=1; n='x=5' n+='x=9' f", ['x', 'n'], '1 4', 'set=1 set=4'], + ["declare -i n=4; x=1; n+= true", ['x', 'n'], '1 4', 'set=1 set=4'], + ["declare -i n=4; x=1; n[0]+='x=7' true", ['x', 'n'], '1 4', 'set=1 set=4'], + ["m=4; x=1; m+='x=7' true", ['x', 'm'], '1 4', 'set=1 set=4'], + ["declare -i n=4; x=1; declare -n q=n; q+='x=7' true", ['x', 'n'], '7 4', 'set=7 set=4'], + ["declare -i n=4; x=1; declare -n z=\"$1\"; z+='x=7' true", ['x', 'n'], '1 4', 'set=1 set=4'], + // (An element before a function or `eval`: bash refuses it, "not a valid identifier", and runs the command. + // Before, the model failed on it.) + ["n=(1); f() { :; }; n[0]=5 f", ['n'], '1', 'set'], + ["n=(1); n[0]=5 eval :", ['n'], '1', 'set'], + ]; + // (In POSIX mode, before a special builtin, bash's error ends the script; the model keeps n.) + assert.strictEqual(bash('set -o posix; n=(1); n[0]=5 :\necho $n'), ''); + assert.deepStrictEqual(after('set -o posix; n=(1); n[0]=5 :', 'n'), ['set']); + for (const [src, names, theirs, ours] of rows) { + assert.strictEqual(bash(`${src}\necho ${names.map((v) => `$${v}`).join(' ')}`), `${theirs}\n`, src); + assert.strictEqual(after(src, ...names).join(' '), ours, src); + } + }); it('says so on the paths that ran what the walk does not read, and after they join', () => { // (bash: c.sh sets foo; the branch that does not source it reads foo unset; after the `if`, either may have run.) const src = 'if [ "$1" ]; then source ./c.sh; else (( a = foo + 1 )); : "$a"; fi; (( b = foo + 1 )); echo "$a $b"';