diff --git a/CHANGELOG.md b/CHANGELOG.md index caa8679e..81467354 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ - A VS Code setting set below its minimum (`bashExplainer.inline.maxLength` under 20, `bashExplainer.hover.maxDepth` or `hover.followDelay` under 0) now uses its default, as the command-line tool already did. Before, VS Code used the number as given. - On the web page, changing the file kind changes only how the script is explained: the text stays, also an untouched sample (an empty editor still gets the new kind's sample). "Load the sample" loads the current kind's sample, and when it replaces your own text, a notice with an "Undo" button appears for a few seconds. Before, choosing a kind replaced an untouched sample with the new kind's sample, and "Load the sample" could be undone only with Ctrl+Z. -- A branch that cannot run, because its condition is decided by values the script itself set, no longer changes how the lines after it are explained: after `x=1; if [ "$x" = 2 ]; then set -f; fi`, a glob is a glob again, and after `if false; then f() { …; }; fi`, `f` is not the function. Code after `break`, `continue`, `return` or `exit` no longer counts either. Before, both sides of every branch counted, so such lines said "may" or "on some paths". A condition is decided only from variables that nothing but plain assignments (`x=1`, `export x=1`) writes anywhere in the script, and not at all in a script that runs what cannot be read (`source`, `eval "$s"`, `$cmd`, an indirection such as `${!v}`, `${!1}` or `${!v@Q}`, whose subscript may assign any variable, a trap whose action may `return` from where a signal comes): there, both sides still count. Arithmetic that reads one of bash's own numbers (`shift $((OPTIND-1))`, `(( EUID == 0 ))`) keeps the decisions (before, it turned off every decision in the script), and text assigned to `OPTIND`, `RANDOM`, `SRANDOM` or `HISTCMD`, which bash runs as arithmetic (`OPTIND='x=5'` sets `x`), counts as setting what it sets. Conditions in a trap's action are never decided. +- A branch that cannot run, because its condition is decided by values the script itself set, no longer changes how the lines after it are explained: after `x=1; if [ "$x" = 2 ]; then set -f; fi`, a glob is a glob again, and after `if false; then f() { …; }; fi`, `f` is not the function. Code after `break`, `continue`, `return` or `exit` no longer counts either. Before, both sides of every branch counted, so such lines said "may" or "on some paths". A condition is decided only from variables that nothing but plain assignments (`x=1`, `export x=1`) writes anywhere in the script, and not at all in a script that runs what cannot be read (`source`, `eval "$s"`, `$cmd`, an indirection such as `${!v}`, `${!1}` or `${!v@Q}`, whose subscript may assign any variable, a trap whose action may `return` from where a signal comes): there, both sides still count. Arithmetic that reads one of bash's own numbers (`shift $((OPTIND-1))`, `(( EUID == 0 ))`) keeps the decisions (before, it turned off every decision in the script), and text assigned to `OPTIND`, `RANDOM`, `SRANDOM` or `HISTCMD`, which bash runs as arithmetic (`OPTIND='x=5'` sets `x`), counts as setting what it sets. Conditions in a trap's action are never decided. The text of an alias is read with the command it is expanded at, and with what the script declares: `m[k]=1` in an alias after `declare -A m` keeps the decisions (before, it turned off every decision in the script). ### Fixed @@ -37,7 +37,7 @@ - Quick fixes (the rewrites offered for dialect warnings) are offered in every language of the `bashExplainer.languages` setting, and follow a change of that setting, as the hover does. Before, they were offered in `shellscript` documents only. - `"$@"` inside single quotes is text, so the words after `bash -c 'f "$@"'` are no longer said to depend on how many words it gives. - Examples and dialect checks read a command's options as the command reads them. `grep -e -v foo` searches for `-v`; its example said `-v` inverts the match. `sed -ie 's/a/b/' -e 's/c/d/' f` runs only `s/c/d/`, since `e` is the backup suffix of `-i`; the example ran both scripts. `sed -i.bak` now shows the file rewritten. The input of `grep -efoo data.txt | sort` and `awk '{ print }' v=1 | sort` is labelled as it is read (a file, then stdin), and a pipeline whose later stage reads a file instead of the pipe (`printf … | grep -e -v foo`) gets no whole-pipeline example. `tr --delete` is `-d`, and `find … -exec cmd {} '+'` runs the command once, quoted or not. The dialect check no longer takes `read -p -a v` for `read -a` (`-a` is the prompt), and finds `trap ERR`. The examples of sort, uniq, cut, tr, head, tail, seq, basename, dirname, wc, xargs, sed, grep, awk, perl and `read` read the options as the explanation does, so a long prefix (`sort --rev`, `sed --expr`), `tail +2` and `tr -C` now get an example. `xargs --replace echo {}` and `xargs --max-lines echo` run `echo` (before, `echo` was taken for the value), and `-i`, `-l` are followed. `seq 1 -w 3`, which seq refuses, gets no example; before, it showed output. `cat -n`, `tac -s x` and `rev FILE` in a pipeline, and `cat -n f | sort`, get no example; before, the example passed the input on as if they had no option. The `git log` example shows as many commits as `--max-count 2` asks (before, 3). -- Explanations match bash in more places: `x=1 ''` (a command name that is not found, so `x` is not set), the variables that `exec {fd}>file`, `wait -p var` and `coproc name` set, and an error in `$(( ))` (a division by 0) or an assignment to a readonly variable, which discards the rest of the line, also when it is in a function the line calls (before, only the rest of the command, or nothing). A name written in the option's own word (`printf -vx 2`, `read -rax`, `wait -n -px`) is the variable set (before, `x` kept its old value, and a condition on it could be called decided), and `wait -p x` unsets `x` before it sets it (before, `x` kept its old value when no job ended). `exec cmd` in `( )` or `$( )` says that it ends only the subshell (before, it said nothing of the subshell). `'exit' 3` with quotes has its exit status known, 3, as `exit 3` has (before, it was the status of an unknown command). `$(true)`, `$(:)` and `$(false)` count as empty only where the name runs the builtin, not after `true() { echo a; }`, an alias of it or `enable -n true` (before, `x=$(true)` was taken to set `x` to nothing there). Functions defined after `set -a +a` or `set foo -a` (where `-a` is an argument) are not exported, so conditions in them are decided (before, none were). In a function whose every call is in a subshell (`m() { cd /tmp; }; echo | m`), `cd` says "It may run in a subshell (where its function is called; what the script does not show may call it too)" when a sourced file, a trap's action or an alias may also call the function (before, "It runs in a subshell", although `. ./lib.sh` may call it in the shell itself); a word that names the function without calling it (`command m`, `declare -f m`, `type m`) no longer turns this into "may" (before, it did). `cmd &> >(tee log)` and `&>> >(…)` send both outputs to the process substitution, also after a block or a loop (`{ …; } &> >(tee log)`, `done &> >(tee log)`) and before the command name (`&> >(tee log) cmd`) (before, it was read as an argument of the command, and the redirection as missing its target; after a block or a loop it was a syntax error, and before the name it was the command). +- Explanations match bash in more places: `x=1 ''` (a command name that is not found, so `x` is not set), the variables that `exec {fd}>file`, `wait -p var` and `coproc name` set, and an error in `$(( ))` (a division by 0) or an assignment to a readonly variable, which discards the rest of the line, also when it is in a function the line calls (before, only the rest of the command, or nothing). A name written in the option's own word (`printf -vx 2`, `read -rax`, `wait -n -px`) is the variable set (before, `x` kept its old value, and a condition on it could be called decided), and `wait -p x` unsets `x` before it sets it (before, `x` kept its old value when no job ended). `exec cmd` in `( )` or `$( )` says that it ends only the subshell (before, it said nothing of the subshell). After a `trap` whose action calls a function, what the function sets may have changed, also when the script called it before from the same state (`f() { x=2; }; x=1; f; x=1; trap f USR1`: `x` may be 2 after it, as bash has once the signal comes; before, it was 1). `'exit' 3` with quotes has its exit status known, 3, as `exit 3` has (before, it was the status of an unknown command). `$(true)`, `$(:)` and `$(false)` count as empty only where the name runs the builtin, not after `true() { echo a; }`, an alias of it or `enable -n true` (before, `x=$(true)` was taken to set `x` to nothing there). Functions defined after `set -a +a` or `set foo -a` (where `-a` is an argument) are not exported, so conditions in them are decided (before, none were). In a function whose every call is in a subshell (`m() { cd /tmp; }; echo | m`), `cd` says "It may run in a subshell (where its function is called; what the script does not show may call it too)" when a sourced file, a trap's action or an alias may also call the function (before, "It runs in a subshell", although `. ./lib.sh` may call it in the shell itself); a word that names the function without calling it (`command m`, `declare -f m`, `type m`) no longer turns this into "may" (before, it did). `cmd &> >(tee log)` and `&>> >(…)` send both outputs to the process substitution, also after a block or a loop (`{ …; } &> >(tee log)`, `done &> >(tee log)`) and before the command name (`&> >(tee log) cmd`) (before, it was read as an argument of the command, and the redirection as missing its target; after a block or a loop it was a syntax error, and before the name it was the command). - `unset f` says it removes the function when there is no variable `f`, as bash does. A pipeline whose stage is a function or an alias of the script (`sort() { tac; }; … | sort`) no longer shows the output of the program of that name as an example. Before, the first said "Unset the variable" and the second showed the output of the program, which the function does not give. - The ⚠ notes on words and pipelines follow what bash does here. After `IFS=; set -f`, `cp $a b` no longer says that `$a` is split (the token already said it is not); `X=1 echo "\$X"` no longer warns that `$X` is expanded first (`\$X` is text); a variable set by `((count++))`, `printf -v` or `mapfile` at the end of a pipeline, or by a function called there, is now reported as lost after it, and `echo '$n'` no longer counts as a later read. An alias of `rm` is taken for `rm`. - The "Idiom" notes are read from the commands, not from the text. They are found with other spacing, order and quoting: `while true ; do` is an endless loop, `done /dev/null >/dev/null`, `exec >> >(tee -a log)`, `$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )`, `while [ ${#} -gt 0 ]`, `while (( $# != 0 ))`, `trap cleanup 0` and `[[ $BASH_SOURCE != $0 ]] && return` are known. They are no longer given where bash does something else: `trap - EXIT`, `trap '' EXIT` and `trap -p EXIT` set no clean-up; `read -rd'' f` reads up to an `f`, not a NUL byte; `while [ $# > 0 ]` writes to a file named `0` and does not end; and text in quotes (`echo '$(mktemp)'`) is not run. Before, each of these got the idiom's note. diff --git a/docs/design/semantics.md b/docs/design/semantics.md index d1c36028..3a41d711 100644 --- a/docs/design/semantics.md +++ b/docs/design/semantics.md @@ -979,7 +979,7 @@ An expansion error ends a path too. An error of arithmetic outside `(( ))`, `let These replace the earlier deny-lists (`eval` or `source` before a condition, a builtin given a name not known, an exported function, a loop in which such a thing happens), which three reviews each found one more way around. -**Strings run as scripts (4.13).** The walk runs a static `eval` string where `eval` runs, so its conditions are decided and its dead code said as elsewhere. It runs a trap's action where `trap` sets it (only for what it changes) and after the top level, but decides nothing in it and says none of it is dead: a signal may run it at any point, and an EXIT trap runs not only at the end or at `exit`, but also at `${u:?}`, at a `set -u` error and at a signal, with values the walk does not see there. The trust scan reads the same trees as the walk (the model's `Scripts`, `StringTrees`: each string is parsed once), as foreign text (a trap's action may run at any time); so what a statement of a string has that the walk does not run (subscripts, offsets that may fail, names read as arithmetic) is found as for the script's own statements, with what the whole script declares (`Trust.reads`). Only the tree of an alias's text with the rest of its command, which the walk parses as it expands it, has a scan of its own (`treeFacts`). A trap's action that may leave what a signal interrupts, otherwise than by `exit`, makes the script trust nothing: an error that may end the line around it (bash: `declare -i q; trap "q='1+'" USR1` discards the rest of the line `kill -USR1 $$` is on), or a `return`, `break` or `continue` that its own functions and loops do not hold (`trap return USR1` returns from the function the signal interrupts). An EXIT trap's action ends with the shell, so it may. A string the walk does not read (an expansion or a glob in it, or more than `NESTED_DEPTH` levels deep) may do anything, so the script is walked again with nothing trusted (`Walker.found`), as for arithmetic that reads text not known. Each line of a string is a unit, as a line of the script: an expansion error discards the rest of the string's line, and the shell goes on after `eval` (bash 5.2: `eval 'echo $((1/0)); echo a'; echo b` prints `b`). An error that may happen there may also end the line of the `eval` (bash does so for an integer given text that is no expression, `declare -i n; eval 'n=1/0'`, and for a subscript that fails). The model of another shell (`bash -c S`) decides from the values that reach it, so it trusts a variable only when the outer model does too (`withinTrust`), and nothing when the outer model trusts nothing or the shell reads startup files (`-l`, `-i`, an exported `BASH_ENV`). +**Strings run as scripts (4.13).** The walk runs a static `eval` string where `eval` runs, so its conditions are decided and its dead code said as elsewhere. It runs a trap's action where `trap` sets it (only for what it changes) and after the top level, but decides nothing in it and says none of it is dead: a signal may run it at any point, and an EXIT trap runs not only at the end or at `exit`, but also at `${u:?}`, at a `set -u` error and at a signal, with values the walk does not see there. The trust scan reads the same trees as the walk (the model's `Scripts`, `StringTrees`: each string is parsed once), as foreign text (a trap's action may run at any time); so what a statement of a string has that the walk does not run (subscripts, offsets that may fail, names read as arithmetic) is found as for the script's own statements, with what the whole script declares. So are the trees of the aliases the walk expands (an alias's text with the rest of the command, `StringTrees.expansion`), read at each command named like an alias the script defines: the text as foreign text, the rest of the command where the command is. A trap's action that may leave what a signal interrupts, otherwise than by `exit`, makes the script trust nothing: an error that may end the line around it (bash: `declare -i q; trap "q='1+'" USR1` discards the rest of the line `kill -USR1 $$` is on), or a `return`, `break` or `continue` that its own functions and loops do not hold (`trap return USR1` returns from the function the signal interrupts). An EXIT trap's action ends with the shell, so it may. A string the walk does not read (an expansion or a glob in it, or more than `NESTED_DEPTH` levels deep) may do anything, so the script is walked again with nothing trusted (`Walker.found`), as for arithmetic that reads text not known. Each line of a string is a unit, as a line of the script: an expansion error discards the rest of the string's line, and the shell goes on after `eval` (bash 5.2: `eval 'echo $((1/0)); echo a'; echo b` prints `b`). An error that may happen there may also end the line of the `eval` (bash does so for an integer given text that is no expression, `declare -i n; eval 'n=1/0'`, and for a subscript that fails). The model of another shell (`bash -c S`) decides from the values that reach it, so it trusts a variable only when the outer model does too (`withinTrust`), and nothing when the outer model trusts nothing or the shell reads startup files (`-l`, `-i`, an exported `BASH_ENV`). **Visible effects.** The state after a decided branch (variables, options, functions, aliases: what B2 to B5 read). `StateModel.condition(n)` gives, for an `if`, `elif`, loop, `&&`/`||` list or `case`, the decision every walk made and the variables it read (with their values). The analyzer adds a note (not a pitfall: a value set at the top of a script, `DEBUG=0`, is a common way to switch code off) when the decision reads a variable or `$?`; a condition of literals only (`while true`, `if false`) says nothing, as its meaning is plain: diff --git a/src/semantics/effects.ts b/src/semantics/effects.ts index 13e05cb0..59d53c0a 100644 --- a/src/semantics/effects.ts +++ b/src/semantics/effects.ts @@ -23,6 +23,22 @@ export function commandName(c: SyntaxNode): string | undefined { return staticWordValue(field(c, 'name')!.namedChildren[0]); } +/** The name of the simple command `c` when bash may read it as an alias: a plain word, with no quote or expansion. */ +export function aliasName(c: SyntaxNode): string | undefined { + const word = field(c, 'name')!.namedChildren[0]; + const name = commandName(c); + return word.type === 'word' && word.text === name ? name : undefined; +} + +/** `alias name=value` (or `'name=value'`): the name and what it stands for (static, or the text without its outer quotes). */ +export function aliasOf(a: SyntaxNode): { name: string; value: string } | undefined { + const v = staticWordValue(a); + const text = v ?? a.text; + const m = /^([\w.:@%+-]+)=(['"]?)([\s\S]*)\2$/.exec(text); + if (!m) return undefined; + return { name: m[1], value: v !== undefined ? v.slice(v.indexOf('=') + 1) : m[3] }; +} + /** The words of a builtin read with its spec (or `spec`: a function of the host, bats' `run`). */ export function builtinParse(name: string, args: readonly SyntaxNode[], spec = builtins[name]): ArgvParse { return interpret(spec, args.map(argWord), { name }); diff --git a/src/semantics/shellstate.ts b/src/semantics/shellstate.ts index 595878dd..91f4a67f 100644 --- a/src/semantics/shellstate.ts +++ b/src/semantics/shellstate.ts @@ -238,6 +238,12 @@ export interface ShellState { * under `set -e`). What runs from it is dead code; merging drops it. */ readonly unreachable?: true; + /** + * Set on the state of a path that ran what the walk does not read (a sourced file, a string of `eval` or a trap's + * action that is not static text, the startup files of another shell): there a name the script has not set may be + * set. Merging keeps it: such a path may have been taken. + */ + readonly opaque?: true; } /** Why errexit has its value in a `$( )` (`ShellState.errexitWhy`). */ @@ -266,7 +272,7 @@ export function attribute(st: ShellState, name: string, letter: string): Tri { return st.variables.get(name)?.attrs[letter]?.state ?? 'off'; } -export type Kind = Exclude; +export type Kind = Exclude; export type Entry = ShellState[K] extends NameMap ? V : never; export const EMPTY: ShellState = { @@ -349,6 +355,7 @@ export function merge(states: ShellState[], posix = false): ShellState { hashed: NameMap.merge(a.hashed, b.hashed, (_k, x, y) => mergeAlias(x, y)), ...(a.status === b.status && a.status !== undefined ? { status: a.status } : {}), ...mergeErrexitWhy(a.errexitWhy, b.errexitWhy), + ...(a.opaque || b.opaque ? { opaque: true as const } : {}), })); } @@ -374,7 +381,7 @@ export function sameState(a: ShellState, b: ShellState): boolean { const k = Object.keys(x.attrs); return k.length === Object.keys(y.attrs).length && k.every((l) => x.attrs[l].state === y.attrs[l]?.state); }; - return a === b || (a.status === b.status && !a.unreachable === !b.unreachable && a.errexitWhy === b.errexitWhy && NameMap.same(a.set, b.set, (x, y) => x === y) && NameMap.same(a.shopt, b.shopt, (x, y) => x === y) + return a === b || (a.status === b.status && !a.unreachable === !b.unreachable && !a.opaque === !b.opaque && a.errexitWhy === b.errexitWhy && NameMap.same(a.set, b.set, (x, y) => x === y) && NameMap.same(a.shopt, b.shopt, (x, y) => x === y) && NameMap.same(a.variables, b.variables, (x, y) => x === y || (x.declared === y.declared && x.set === y.set && x.value === y.value && attrs(x, y))) && NameMap.same(a.functions, b.functions, (x, y) => x === y || (x.state === y.state && x.def === y.def && x.exported === y.exported && x.readonly === y.readonly)) && NameMap.same(a.aliases, b.aliases, (x, y) => x === y || (x.state === y.state && x.value === y.value)) diff --git a/src/semantics/state.ts b/src/semantics/state.ts index a34902e4..bb024a13 100644 --- a/src/semantics/state.ts +++ b/src/semantics/state.ts @@ -136,10 +136,17 @@ export function docLine(n: SyntaxNode): number { return lo; } +/** + * How many trees of aliases expanded at commands a model makes, one per command (`Scripts.expansion`): each alias + * whose text names another doubles them at each command that names it, and each tree is kept. + */ +export const ALIAS_TREES = 20_000; + /** * The strings a model reads as scripts (`StateModel.script`), by their first word, and the trees of those that run in - * its own shell (`eval`, a trap's action), by root: the walk runs them, so their nodes are in this model. The trust - * scan reads the same trees (`StringTrees`): each string is parsed once. + * its own shell (`eval`, a trap's action), by root: the walk runs them, so their nodes are in this model. Also the trees + * of the aliases the walk expands (an alias's text with the rest of a command). The trust scan reads the same trees + * (`StringTrees`): each string is parsed once. */ export class Scripts implements StringTrees { readonly byWord = new Map(); @@ -147,6 +154,12 @@ export class Scripts implements StringTrees { /** The text and tree of each string, by its first word (none: not static text); the trees of the texts of `alias`, by word. */ private readonly parsed = new Map(); private readonly aliases = new Map(); + /** The trees of the aliases expanded at each command, by the command and the alias's text. */ + private readonly expansions = new Map>(); + private expansionCount = 0; + /** Past `ALIAS_TREES`, one tree per text (the alias's with the rest of the command), whatever the command. */ + private readonly shared = new Map(); + capped = false; model!: StateModel; constructor(readonly depth: number) {} @@ -165,8 +178,27 @@ export class Scripts implements StringTrees { alias(word: SyntaxNode, text: string): SyntaxNode { let root = this.aliases.get(word); + if (!root) this.aliases.set(word, (root = parser().parse(text).root)); + return root; + } + + expansion(command: SyntaxNode, text: string): SyntaxNode { + let byText = this.expansions.get(command); + if (!byText) this.expansions.set(command, (byText = new Map())); + let root = byText.get(text); if (!root) { - this.aliases.set(word, (root = parser().parse(text).root)); + // (The rest of the command: what follows its name, read with the alias's text, as bash reads it.) + const full = text + command.text.slice(field(command, 'name')!.endIndex - command.startIndex); + if (this.expansionCount >= ALIAS_TREES) { + // (The walk still runs what the alias does, in a tree it shares with the other commands of the same text; + // the trust scan, which tells the trees apart, trusts nothing then.) + this.capped = true; + root = this.shared.get(full); + if (!root) this.shared.set(full, (root = parser().parse(full).root)); + return root; + } + this.expansionCount++; + byText.set(text, (root = parser().parse(full).root)); } return root; } diff --git a/src/semantics/trust.ts b/src/semantics/trust.ts index b20c938c..f21efd8f 100644 --- a/src/semantics/trust.ts +++ b/src/semantics/trust.ts @@ -19,24 +19,23 @@ // - A function is closed (its decisions and dead code may be said, and src/semantics/env.ts says for sure where its // body runs) when every call of it is one the walk sees, in the script or in an `eval`'s string: not exported // (`export -f`, `set -a`), not named as an argument (`complete -F f`, `trap f INT`), not called from the text of a -// trap (which runs at any time) or an alias (the scan reads it where it is defined, and does not expand aliases -// inside other strings, so it cannot tell which uses the walk sees: `trap g EXIT` may call `f` through `g`), and -// nothing is blind. +// trap (which runs at any time) or an alias (bash also expands an alias after one whose text ends with a blank, +// which the walk does not follow, so the scan cannot tell which uses the walk sees: `trap g EXIT` may call `f` +// through `g`), and nothing is blind. // -// The walk runs the static strings of `eval` and of traps (4.13), and this scan reads the same trees (`StringTrees`: -// the model's), as foreign text (a trap's action runs at any time; what a string assigns is not trusted). So the -// facts of statements (`arithNames`, `mayFail`, `associative`) are of the script's statements and of its strings' -// alike (`Trust.reads`); only the tree of an alias's text with the rest of its command, which the walk parses as it -// expands it, has a scan of its own (`treeFacts`). A string the walk does not read (an expansion in it, a glob, too -// deep) makes it walk again with nothing trusted (`Walker.found`), as does a trap's action that may leave what a -// signal interrupts. +// The walk runs the static strings of `eval` and of traps (4.13), and the aliases it expands (an alias's text with the +// rest of the command), and this scan reads the same trees (`StringTrees`: the model's), as foreign text (a trap's +// action runs at any time; what a string assigns is not trusted; the rest of a command is read where the command is). +// So the facts of statements (`arithNames`, `mayFail`, `associative`) are of every statement the walk runs. A string +// the walk does not read (an expansion in it, a glob, too deep) makes it walk again with nothing trusted +// (`Walker.found`), as does a trap's action that may leave what a signal interrupts. // Another shell's model (`bash -c`) trusts no more than the model of the script that starts it (`withinTrust`). import type { CommandSpec } from '../argv/spec'; -import { argWord, prefixOf } from '../argv/words'; +import { prefixOf } from '../argv/words'; import { builtins } from '../dict/builtins'; import { staticWordValue, substitutionNames } from '../staticword'; import { field, fields, type SyntaxNode } from '../syntax/tree'; -import { builtinParse, callbacks, commandName, DECLARATIONS, setOptionWrites, variableWrites, wordName, wrapped } from './effects'; +import { aliasName, aliasOf, builtinParse, callbacks, commandName, DECLARATIONS, setOptionWrites, variableWrites, wordName, wrapped } from './effects'; import { STATEMENTS } from './env'; import { alwaysNumeric, ARITH_ASSIGNED, assignsArithmetic, changedUnseen, optionRoles, readDeclaration, readDeclarationWords, type Declaration } from './vars'; @@ -60,8 +59,6 @@ export interface Trust { * a plain number (`a[i]=x`, `${s:i}`). An error there discards the rest of the line. */ mayFail(n: SyntaxNode): boolean; - /** Whether the facts of the statements of the tree `root` are here: the script's, and those of the strings it runs (`StringTrees`). */ - reads(root: SyntaxNode): boolean; } const NAME = /^[A-Za-z_]\w*$/; @@ -89,9 +86,17 @@ interface Write { /** * Where a node is: the function whose body holds it, whether it is in the text of an `eval`, a trap or an alias * (`foreign`), and whether the walk runs it where it stands (`now`: the script, or an `eval`'s string there, whose - * calls are calls the walk sees; not a trap's action or an alias's text). + * calls are calls the walk sees; not a trap's action or an alias's text). In the tree of an alias expanded at a + * command (`StringTrees.expansion`), what starts at `rest.at` is the rest of the command, where the command is. */ -interface Where { fn: string | undefined; foreign: boolean; now: boolean } +interface Where { fn: string | undefined; foreign: boolean; now: boolean; rest?: { at: number; where: Where } } + +/** + * How many trees of aliases expanded the scan reads from one command of the script (each alias in an alias's text + * may double them there); the commands of the script are not counted together. + */ +const ALIAS_EXPANSIONS = 1000; +const NO_NAMES: ReadonlySet = new Set(); const identifiers = (text: string): string[] => text.match(IDENTIFIERS) ?? []; @@ -119,13 +124,14 @@ function indirect(n: SyntaxNode): boolean { const joined = (values: Value[]): Value => (values.every((v): v is Extract => typeof v === 'object') ? { names: values.flatMap((v) => v.names), text: '' } : 'unknown'); +/** An alias an `alias` command defines: its name and text, the word that gives them, and where the command is. */ +interface AliasDef { name: string; value: string; word: SyntaxNode; where: Where } + /** Thrown when the scan finds that nothing is trusted (see `Scan.blindFor`). */ class StopScan {} class Scan { readonly writes = new Map(); - /** The trees read: the script's, and those of the strings it runs (see `foreign`); the facts of their statements are here. */ - readonly roots = new Set(); /** Names read by arithmetic, and the statements that read them where the walk does not run the arithmetic. */ readonly arith = new Set(); readonly pending = new Map>(); @@ -155,9 +161,13 @@ class Scan { blind: string | undefined; /** The variables whose values are command names (`$debug`), and whether they are quoted (not split). */ private readonly commandVariables: { name: string; quoted: boolean }[] = []; + /** The aliases the `alias` commands define (a name and its text), and the commands that may run one (see `aliases`). */ + private readonly aliasDefs: AliasDef[] = []; + private readonly aliasCommands: { c: SyntaxNode; name: string; where: Where }[] = []; + /** The aliases each tree of an alias expanded at a command comes from (bash does not expand an alias in itself). */ + private readonly expanding = new Map>(); - constructor(root: SyntaxNode, private readonly strings: StringTrees, private readonly stop: boolean, private readonly host: HostCommands) { - this.roots.add(root); + constructor(root: SyntaxNode, private readonly strings: StringTrees, private readonly host: HostCommands) { const defs = (n: SyntaxNode): void => { if (n.type === 'function_definition') this.functions.add(field(n, 'name')!.text); if (n.type === 'command' && ['alias', 'enable'].includes(commandName(n) ?? '')) this.renames = true; @@ -166,6 +176,7 @@ class Scan { defs(root); try { this.visit(root, { fn: undefined, foreign: false, now: true }); + this.aliases(); this.finish(); } catch (e) { /* c8 ignore next -- only `StopScan` is thrown on purpose */ @@ -173,18 +184,10 @@ class Scan { } } - /** Whether the scan stopped at a reason to trust nothing. */ - get stopped(): boolean { - return this.stop && this.blind !== undefined; - } - - /** - * Nothing is trusted, for `why`: the scan stops there (`StopScan`), since nothing it finds after would count, unless - * it is read for the facts of its statements (`stop` off). - */ - private blindFor(why: string): void { - this.blind ??= why; - if (this.stop) throw new StopScan(); + /** Nothing is trusted, for `why`: the scan stops there (`StopScan`), since nothing it finds after would count. */ + private blindFor(why: string): never { + this.blind = why; + throw new StopScan(); } private write(name: string, w: Where, plain: boolean, value: Value, nameref = false): void { @@ -278,6 +281,7 @@ class Scan { } private visit(n: SyntaxNode, w: Where): void { + if (w.rest && n.startIndex >= w.rest.at) w = w.rest.where; switch (n.type) { case 'function_definition': { this.changed(w); @@ -506,6 +510,8 @@ class Scan { } private command(c: SyntaxNode, w: Where): void { + const alias = aliasName(c); + if (alias !== undefined) this.aliasCommands.push({ c, name: alias, where: w }); const { name, args, lookup, direct } = this.nameOf(c); if (name === undefined) { // `$debug`, `"$run"`: a variable the script sets to command names (see `finish`). @@ -534,7 +540,6 @@ class Scan { const unread = spec.shellEffect?.runsUnread; if (unread) { this.blindFor(unread === 'file' ? 'a file is sourced' : 'a command line the model does not read is run'); - return; } const parse = builtinParse(name, args, spec); // The variables it sets, and the functions it runs later, as its spec says (src/semantics/effects.ts). @@ -563,9 +568,12 @@ class Scan { } case 'alias': for (const a of args) { + // (Read where commands may run it: see `aliases`. One the walk does not take, by its name: its text alone.) const v = staticWordValue(a); + const def = v === undefined ? undefined : aliasOf(a); const m = v === undefined ? null : /^[^=]+=([\s\S]*)$/.exec(v); - if (m) this.foreign(this.strings.alias(a, m[1]), 'an alias of text not known', w); + if (def) this.aliasDefs.push({ ...def, word: a, where: w }); + else if (m) this.foreign(this.strings.alias(a, m[1]), 'an alias of text not known', w); else if (v === undefined) this.blindFor('an alias of text not known'); } return; @@ -604,14 +612,57 @@ class Scan { * elsewhere (`now`: where it stands, an `eval`'s). None: its text is not known, for `why`. */ private foreign(root: SyntaxNode | undefined, why: string, w: Where, now = false): void { - if (!root) { - this.blindFor(why); - return; - } - this.roots.add(root); + if (!root) this.blindFor(why); this.visit(root, { fn: w.fn, foreign: true, now: now && w.now }); } + /** + * The aliases, read where the walk may run them: at each command named like an alias the script defines, the tree of + * each text the alias may have there with the rest of the command (the walk's: `StringTrees.expansion`), with the + * commands in it in turn, but not in an alias it comes from. Its text is foreign (bash also expands an alias after + * one whose text ends with a blank, which the walk does not follow), and the rest of the command is read where the + * command is, now as part of that tree: the facts of its statements are the walk's. An alias no command names is + * read alone, as foreign text. + */ + private aliases(): void { + const read = new Set(); + const done = new Map>(); + // (The command of the script each tree comes from, and how many trees each such command has.) + const origin = new Map(); + const counts = new Map(); + for (let more = true; more;) { + more = false; + // (The trees read add commands and definitions to both lists.) + for (let i = 0; i < this.aliasCommands.length; i++) { + const { c, name, where } = this.aliasCommands[i]; + let root = c; + while (root.parent) root = root.parent; + const from = this.expanding.get(root) ?? NO_NAMES; + if (from.has(name)) continue; + const top = origin.get(root) ?? c; + const texts = done.get(c) ?? new Set(); + done.set(c, texts); + for (const d of this.aliasDefs) { + if (d.name !== name || texts.has(d.value)) continue; + const count = (counts.get(top) ?? 0) + 1; + if (count > ALIAS_EXPANSIONS) this.blindFor('too many aliases expanded in aliases'); + counts.set(top, count); + texts.add(d.value); + read.add(d); + more = true; + const tree = this.strings.expansion(c, d.value); + if (this.strings.capped) this.blindFor('too many aliases expanded'); + this.expanding.set(tree, new Set([...from, name])); + origin.set(tree, top); + // (A command in an alias's text is foreign: so is the rest of it, in the tree of an alias expanded there.) + const { rest: _, ...outer } = where; + this.visit(tree, { fn: where.fn, foreign: true, now: false, rest: { at: d.value.length, where: outer } }); + } + } + } + for (const d of this.aliasDefs) if (!read.has(d)) this.foreign(this.strings.alias(d.word, d.value), 'an alias of text not known', d.where); + } + /** What is left once the whole script is read: subscripts, integers, the values read as arithmetic, callbacks. */ private finish(): void { for (const s of this.subscripts) { @@ -639,7 +690,6 @@ class Scan { const writes = this.writes.get(name); if (!writes) { this.blindFor(`arithmetic reads \`${name}\`, which the script never sets`); - continue; } if (this.refUnknown.has(name)) this.blindFor(`arithmetic reads \`${name}\`, a nameref whose target is not known`); for (const w of writes) { @@ -692,13 +742,20 @@ export const NO_HOST: HostCommands = {}; /** * The trees of the strings the scan reads as part of the script: those of the model (`Scripts` in * src/semantics/state.ts), whose walk runs them. So each string is parsed once, and the facts of its statements are - * the scan's (`Trust.reads`). + * the scan's. */ export interface StringTrees { /** The root of the string that `words` make (`eval a b`, a trap's action); undefined when its text is not known. */ script(words: readonly SyntaxNode[]): SyntaxNode | undefined; - /** The root of `text`, the value the word `word` of `alias` gives (the walk reads it with the rest of each command it starts: a tree of its own). */ + /** The root of `text`, the value the word `word` of `alias` gives, read alone (for an alias no command names). */ alias(word: SyntaxNode, text: string): SyntaxNode; + /** + * The root of the alias text `text` with the rest of the simple command `command`: the tree the walk runs where it + * expands the alias there. Past the trees a model makes one per command (`ALIAS_TREES`), a tree shared by the + * commands of the same text, and `capped` is set. + */ + expansion(command: SyntaxNode, text: string): SyntaxNode; + readonly capped: boolean; } /** @@ -707,35 +764,15 @@ export interface StringTrees { * the file's host defines (a bats file: `run`, `load`…), read by their specs. */ export function scriptTrust(root: SyntaxNode, strings: StringTrees, host = NO_HOST): Trust { - return readTrust(new Scan(root, strings, true, host)); -} - -/** The facts of statements read, per host and tree (`treeFacts`). */ -const facts = new WeakMap>(); - -/** - * The facts of the statements of a tree the script's scan did not read (`Trust.reads`): an alias's text with the rest - * of the command it starts, which the walk parses as it expands it. Read to the end, also where it finds a reason to - * trust nothing; cached per host and tree. - */ -export function treeFacts(root: SyntaxNode, strings: StringTrees, host = NO_HOST): Trust { - let cache = facts.get(host); - if (!cache) facts.set(host, (cache = new WeakMap())); - let trust = cache.get(root); - if (!trust) cache.set(root, (trust = readTrust(new Scan(root, strings, false, host)))); - return trust; -} - -/** The trust a scan gives. */ -function readTrust(scan: Scan): Trust { - if (scan.stopped) return { ...BLIND, blind: scan.blind! }; + const scan = new Scan(root, strings, host); + if (scan.blind) return { ...BLIND, blind: scan.blind }; const unseen = new Set(); for (const [name, ws] of scan.writes) if (ws.some((w) => w.foreign || (w.fn !== undefined && scan.callbacks.has(w.fn)))) unseen.add(name); - const trust: Trust = { - blind: scan.blind, - variable: (name) => !scan.blind && NAME.test(name) && !changedUnseen(name) && !scan.arith.has(name) && !unseen.has(name) + return { + blind: undefined, + variable: (name) => NAME.test(name) && !changedUnseen(name) && !scan.arith.has(name) && !unseen.has(name) && !scan.refTargets.has(name) && (scan.writes.get(name) ?? []).every((w) => w.plain), - closed: (name) => !scan.blind && !scan.exportAll && !scan.exported.has(name) && !scan.callbacks.has(name), + closed: (name) => !scan.exportAll && !scan.exported.has(name) && !scan.callbacks.has(name), arithNames: (n) => { const names = scan.pending.get(n); return names && [...names]; @@ -745,13 +782,11 @@ function readTrust(scan: Scan): Trust { return names && [...names]; }, mayFail: (n) => scan.fails.has(n), - reads: (root) => scan.roots.has(root), }; - return trust; } /** Nothing trusted: for a walk that found, as it went, arithmetic reading text not known (see `Walker`). */ -export const BLIND: Trust = { blind: 'arithmetic reads text not known', variable: () => false, closed: () => false, arithNames: () => undefined, associative: () => undefined, mayFail: () => false, reads: () => false }; +export const BLIND: Trust = { blind: 'arithmetic reads text not known', variable: () => false, closed: () => false, arithNames: () => undefined, associative: () => undefined, mayFail: () => false }; /** * What another shell's model trusts (`bash -c S`): its own script's trust, and no more than the outer model's, whose diff --git a/src/semantics/walk/aliases.ts b/src/semantics/walk/aliases.ts index b6765fee..f5fd3983 100644 --- a/src/semantics/walk/aliases.ts +++ b/src/semantics/walk/aliases.ts @@ -1,25 +1,11 @@ // The aliases of the walk of the shell state (src/semantics/walk/walker.ts): the `alias` commands of the script, and -// a command whose name is an alias, which runs the alias's text with the rest of the command, read again. -import { staticWordValue } from '../../staticword'; -import { descendants, field, fields, type SyntaxNode, type SyntaxTree } from '../../syntax/tree'; -import { commandName } from '../effects'; +// a command whose name is an alias, which runs the alias's text with the rest of the command, read again. The trees of +// those texts are the model's (`Scripts.expansion`): the trust scan reads the same ones (src/semantics/trust.ts). +import { descendants, fields, type SyntaxNode } from '../../syntax/tree'; +import { aliasName, aliasOf, commandName } from '../effects'; import { type AliasState, merge, option, type ShellState, withStatus } from '../shellstate'; -import { parser } from './shared'; import type { Walker } from './walker'; -/** Trees of alias texts, least recently used first; bounded, as a long-lived process (LSP, web) sees many texts. */ -const aliasTrees = new Map(); -const ALIAS_TREES = 256; - -/** `alias name=value` (or `'name=value'`): the name and what it stands for (static, or the text without its outer quotes). */ -export function aliasOf(a: SyntaxNode): { name: string; value: string } | undefined { - const v = staticWordValue(a); - const text = v ?? a.text; - const m = /^([\w.:@%+-]+)=(['"]?)([\s\S]*)\2$/.exec(text); - if (!m) return undefined; - return { name: m[1], value: v !== undefined ? v.slice(v.indexOf('=') + 1) : m[3] }; -} - /** The aliases the walk expands, and the `alias` commands that define them. */ export class WalkAliases { constructor(private readonly w: Walker) {} @@ -55,35 +41,23 @@ export class WalkAliases { } /** - * The state after the simple command `n` named `name`, run from `st` once its words are expanded, when its name is - * an alias that may be expanded (in the state its aliases come from, `Walker.expand`); undefined otherwise. + * The state after the simple command `n`, run from `st` once its words are expanded, when its name is an alias that + * may be expanded (in the state its aliases come from, `Walker.expand`): the alias's text, then the rest of the + * command, read and run instead. Undefined otherwise. */ - expand(n: SyntaxNode, name: string | undefined, st: ShellState): ShellState | undefined { - const nameNode = field(n, 'name')!; - const word = nameNode.namedChildren[0]; + expand(n: SyntaxNode, st: ShellState): ShellState | undefined { + const name = aliasName(n); const from = this.w.expand; - const alias = name !== undefined && word.type === 'word' && word.text === name && !this.expanding.has(name) ? from.aliases.get(name) : undefined; + const alias = name !== undefined && !this.expanding.has(name) ? from.aliases.get(name) : undefined; const expands = !this.w.bash ? 'on' : option(from, 'shopt', 'expand_aliases'); if (!alias || alias.state === 'off' || !(expands === 'on' || expands === 'unknown')) return undefined; - // The alias's text, then the rest of the command, read and run instead. - const tree = this.parseText(alias.value + n.text.slice(nameNode.endIndex - n.startIndex)); + const root = this.w.scripts.expansion(n, alias.value); this.expanding.add(name!); const host = this.w.aliasHost; this.w.aliasHost ??= n; - const after = this.w.sequence(tree.root.children, st); + const after = this.w.sequence(root.children, st); this.w.aliasHost = host; this.expanding.delete(name!); return alias.state === 'on' && expands === 'on' ? after : merge([withStatus(st, undefined), after], this.w.posix); } - - private parseText(text: string): SyntaxTree { - let tree = aliasTrees.get(text); - if (tree) aliasTrees.delete(text); - else { - tree = parser().parse(text); - if (aliasTrees.size >= ALIAS_TREES) aliasTrees.delete(aliasTrees.keys().next().value!); - } - aliasTrees.set(text, tree); - return tree; - } } diff --git a/src/semantics/walk/arith.ts b/src/semantics/walk/arith.ts index 34b45689..b14ba4ae 100644 --- a/src/semantics/walk/arith.ts +++ b/src/semantics/walk/arith.ts @@ -23,9 +23,13 @@ const ARITH_NESTING = 1000; const ARITH_MEMO = 8; /** What may be pasted into an arithmetic expression; bash expands it first. (A single-quoted text is not: it is an error.) */ const PASTES = ['simple_expansion', 'expansion', 'arithmetic_expansion', 'command_substitution', 'process_substitution', 'string', 'ansi_c_string', 'translated_string']; -/** The key of `set -u` among the reads of a value (no variable has this name). */ -const NOUNSET = '-u'; -/** What a value read as an expression has read so far: the variables (and `set -u`), whether it can be remembered, how deep it went. */ +/** + * The key, among the reads of a value, of what a read depends on besides the variables (no variable has this name): + * `set -u`, and whether the path ran what the walk does not read (`ShellState.opaque`), as `context` gives them. + */ +const CONTEXT = '-u'; +const context = (st: ShellState): string => `${option(st, 'set', 'nounset')}${st.opaque ? ' opaque' : ''}`; +/** What a value read as an expression has read so far: the variables (and `CONTEXT`), whether it can be remembered, how deep it went. */ interface ArithReads { deps: Map; pure: boolean; depth: number; unsure: boolean; assumed: boolean } /** * A value read as an expression, remembered with what it read and how deep it went: its value; or `unknown` (`deep`: @@ -319,7 +323,7 @@ export class WalkArith { } /** What the values reached from the variable `name` seem to assign, each remembered with the variables it read. */ - assignsMemo = new Map; targets: ReadonlySet }>(); + private readonly assignsMemo = new Map; targets: ReadonlySet }>(); /** * What the value of the variable `name` seems to assign, read as an expression, and the values of the variables it @@ -328,7 +332,7 @@ export class WalkArith { private valueAssigns(st: ShellState, name: string): ReadonlySet { const hit = this.assignsMemo.get(name); if (hit && this.sameReads(st, hit.deps)) return hit.targets; - const deps = new Map([[NOUNSET, option(st, 'set', 'nounset')]]); + const deps = new Map([[CONTEXT, context(st)]]); const targets = new Set(); const texts: string[] = []; const seen = new Set(); @@ -352,9 +356,9 @@ export class WalkArith { return targets; } - /** Whether the variables (and `set -u`) in `deps` are the same in `st`. */ + /** Whether the variables (and `CONTEXT`) in `deps` are the same in `st`. */ private sameReads(st: ShellState, deps: ReadonlyMap): boolean { - for (const [k, v] of deps) if ((k === NOUNSET ? option(st, 'set', 'nounset') : st.variables.get(k)) !== v) return false; + for (const [k, v] of deps) if ((k === CONTEXT ? context(st) : st.variables.get(k)) !== v) return false; return true; } @@ -529,15 +533,15 @@ export class WalkArith { /** What each value being read as an expression has read so far (see `arithValue`). */ readonly arithReads: ArithReads[] = []; /** The values of variables read as expressions, by name, with what each read (`ARITH_MEMO`, most recent first). */ - arithMemo = new Map(); + private readonly arithMemo = new Map(); - /** Notes that the value being read as an expression (if any) depends on the variable `name` and on `set -u`. */ + /** Notes that the value being read as an expression (if any) depends on the variable `name` and on `CONTEXT`. */ noteRead(st: ShellState, name: string): void { const top = this.arithReads[this.arithReads.length - 1]; if (!top) return; const v = st.variables.get(name); top.deps.set(name, v); - top.deps.set(NOUNSET, option(st, 'set', 'nounset')); + top.deps.set(CONTEXT, context(st)); // (A nameref reads other variables: not remembered.) if (isNameref(v)) top.pure = false; } diff --git a/src/semantics/walk/builtins.ts b/src/semantics/walk/builtins.ts index 15e8f723..3932b7f3 100644 --- a/src/semantics/walk/builtins.ts +++ b/src/semantics/walk/builtins.ts @@ -8,10 +8,9 @@ import { argWord, prefixOf } from '../../argv/words'; import { BASH_ONLY_BUILTINS, builtins } from '../../dict/builtins'; import { staticWordValue } from '../../staticword'; import { descendants, field, type SyntaxNode } from '../../syntax/tree'; -import { builtinParse, DECLARATIONS, setOptionWrites, variableWrites, wrapped } from '../effects'; +import { aliasOf, builtinParse, DECLARATIONS, setOptionWrites, variableWrites, wrapped } from '../effects'; import { attribute, type Attribute, merge, option, POSIX_ON, type ShellState, type Tri, withoutErrexitWhy } from '../shellstate'; import { optionRoles, readDeclarationWords, type Declaration } from '../vars'; -import { aliasOf } from './aliases'; import { ExpansionError, NAME, type Operand } from './shared'; import type { Walker } from './walker'; @@ -56,14 +55,13 @@ export class WalkBuiltins { const words = name === 'eval' ? args.slice(at.word) : [args[at.word]]; const script = this.w.script(words, name === 'eval' ? 'current' : 'later', n, parse, args); if ('why' in script) { - this.w.turnOpaque(); - if (!this.w.dead && !this.w.blind) this.w.walkAgain(`\`${name}\` of a string the walk does not read`); - return st; + if (!this.w.mode.dead && !this.w.blind) this.w.walkAgain(`\`${name}\` of a string the walk does not read`); + return this.w.opaque(st); } if (name === 'eval') return this.w.evalScript(script, n, st); const signals = parse.roles.filter((r) => r.role.role === 'operand').map((r) => staticWordValue(args[r.word]) ?? ''); const exit = (x: string) => /^(SIG)?EXIT$|^0$/i.test(x); - if (!this.w.silent) this.w.traps.push({ script, host: this.w.aliasHost ?? n, st, since: this.w.changeCount, exit: signals.some(exit), exitOnly: signals.every(exit), dead: this.w.dead > 0 }); + if (!this.w.silent) this.w.traps.push({ script, host: this.w.aliasHost ?? n, st, since: this.w.changeCount, exit: signals.some(exit), exitOnly: signals.every(exit), dead: this.w.mode.dead > 0 }); // (What it changes, it may change after any later statement; an EXIT trap runs at the end only.) if (!signals.every(exit)) { const keys = new Set(); @@ -165,7 +163,7 @@ export class WalkBuiltins { if (spec.shellEffect?.runsUnread && !hosted) { // (The file is not read: what it defines is not known here.) if (!this.w.silent) this.w.sources.push(this.w.aliasHost ?? n); - this.w.turnOpaque(); + st = this.w.opaque(st); } return st; } diff --git a/src/semantics/walk/conditions.ts b/src/semantics/walk/conditions.ts index 706d797a..41c34c55 100644 --- a/src/semantics/walk/conditions.ts +++ b/src/semantics/walk/conditions.ts @@ -46,7 +46,7 @@ export class WalkConditions { /** Records what the condition of `n` decided on this walk (undefined: nothing), with what it read. */ decide(n: SyntaxNode, outcome: string | undefined, reads: ReadonlyMap): void { - if (this.w.dead) return; + if (this.w.mode.dead) return; let d = this.decisions.get(n); if (!d) this.decisions.set(n, (d = { outcomes: new Set(), reads: new Map(), mixed: false })); d.outcomes.add(outcome ?? '?'); @@ -263,7 +263,7 @@ export class WalkConditions { /** `$?` in the statement being run, from `st`; not known in a statement with a substitution, which changes it. */ lastStatus(st: ShellState): string | undefined { if (st.status === undefined || descendants(this.w.node!, SUBSTITUTIONS).length) return undefined; - if (!this.w.dead) for (const r of this.reading) r.set('?', String(st.status)); + if (!this.w.mode.dead) for (const r of this.reading) r.set('?', String(st.status)); return String(st.status); } diff --git a/src/semantics/walk/walker.ts b/src/semantics/walk/walker.ts index 85a31f2c..76d441bc 100644 --- a/src/semantics/walk/walker.ts +++ b/src/semantics/walk/walker.ts @@ -15,7 +15,7 @@ import { POSIX_ON, reached, type ShellState, sameState, tri, type Tri, unreachable, type VariableState, withoutErrexitWhy, withStatus, } from '../shellstate'; import type { Scripts } from '../state'; -import { type HostCommands, NO_HOST, treeFacts, type Trust } from '../trust'; +import { type HostCommands, NO_HOST, type Trust } from '../trust'; import { alwaysNumeric, changedByBash, readDeclaration } from '../vars'; import { WalkAliases } from './aliases'; import { WalkArith } from './arith'; @@ -29,6 +29,8 @@ const POSIX_SHELLS = new Set(['sh', 'dash', 'ash', 'yash', 'ksh', 'mksh'] const SIMPLE = ['command', 'declaration_command', 'unset_command', 'variable_assignment', 'variable_assignments']; /** Where an expression is arithmetic: an arithmetic expansion in it is run with the expression. */ const ARITH_HOSTS = ['arithmetic_expansion', 'arithmetic_command', 'c_style_for_statement']; +/** The change (see `Walker.put`) of a path that ran what the walk does not read (`Walker.opaque`). */ +export const OPAQUE = 'opaque'; /** The statements whose failure stops the shell under `set -e` (a compound command stops in the command that fails). */ const ERREXIT_STOPS = new Set(['command', 'test_command', 'extended_test', 'arithmetic_command', 'subshell', 'pipeline']); @@ -121,7 +123,7 @@ export class Walker { /** The command whose alias, `eval` string or trap action is being run (its nodes are in their own tree). */ aliasHost: SyntaxNode | undefined; /** The walks of each function body: the state it was called with, and the state after it. */ - private readonly walks = new Map(); + private readonly walks = new Map }[]>(); /** What the walks of each function body changed. */ private readonly changes = new Map>(); /** The state each body being walked started with, and the one to walk it again from (see `recurse`). */ @@ -142,8 +144,13 @@ export class Walker { expand: ShellState = EMPTY; readonly bash: boolean; - /** How deep the walk is in code that does not run here (a branch whose condition is decided): see `deadRun`. */ - dead = 0; + /** + * The modes of the walk, each as how deep it is in it: `dead`, in code that does not run here (a branch whose + * condition is decided: see `deadRun`), where it records no decision; `quiet`, where `set -e` does not stop the shell + * (a condition, the left side of `&&`, `!`). A walk of a function body made in some modes is reused only in the same + * (`reuseKey`). + */ + readonly mode = { dead: 0, quiet: 0 }; /** The nodes run on some walk that is not in dead code. */ readonly live = new Set(); @@ -156,8 +163,6 @@ export class Walker { assumed = false; /** Where jumps go (see `Target`). */ private readonly targets: Target[] = []; - /** How deep the walk is in a place where `set -e` does not stop the shell (a condition, the left side of `&&`, `!`). */ - private quiet = 0; /** Nothing is trusted (src/semantics/trust.ts): no condition is decided, and every jump may be made or not. */ readonly blind: boolean; /** @@ -201,7 +206,7 @@ export class Walker { record(n: SyntaxNode, st: ShellState): void { if (this.silent) return; - if (!this.dead) this.live.add(n); + if (!this.mode.dead) this.live.add(n); const states = this.runStates.get(n); if (!states) this.runStates.set(n, [st]); else if (!states.includes(st)) states.push(st); @@ -283,7 +288,7 @@ export class Walker { const bashEnv = st.variables.get('BASH_ENV'); if (this.bash && bashEnv && bashEnv.set !== 'off' && bashEnv.value !== '') startup = true; if (startup) { - this.turnOpaque(); + st = this.opaque(st); // (What they run is not read: nothing is trusted.) if (!this.blind) this.walkAgain('another shell runs its startup files first'); const vars = new Map(); @@ -305,15 +310,6 @@ export class Walker { this.aliases.collect(root); } - /** - * What the trust scan found of the statement `n`: the script's scan, which reads the trees of the strings the walk - * runs too (`StringTrees`); for the tree of an alias's text with the rest of its command, a scan of that tree. - */ - private facts(n: SyntaxNode): Trust { - const root = rootOf(n); - return this.blind || this.trust.reads(root) ? this.trust : treeFacts(root, this.scripts, this.host); - } - program(root: SyntaxNode, start: ShellState): ShellState { this.collectDefinitions(root); const unset = new Set([...descendants(root, ['unset_command']).flatMap((u) => u.namedChildren), @@ -388,7 +384,12 @@ export class Walker { */ private widen(st: ShellState, keys: Set): ShellState { const changes = { set: new Map(), shopt: new Map(), variables: new Map(), functions: new Map(), aliases: new Map(), disabled: new Map(), hashed: new Map() } as { [K in Kind]: Map> }; + let opaque = st.opaque; for (const key of keys) { + if (key === OPAQUE) { + opaque = true; + continue; + } const kind = key.slice(0, key.indexOf(' ')) as Kind; const name = key.slice(kind.length + 1); if (kind === 'set' || kind === 'shopt') { @@ -416,6 +417,7 @@ export class Walker { // (The status, and whether the path gets here, stay.) return { ...st, + ...(opaque ? { opaque } : {}), set: st.set.withAll(changes.set), shopt: st.shopt.withAll(changes.shopt), variables: st.variables.withAll(changes.variables), functions: st.functions.withAll(changes.functions), aliases: st.aliases.withAll(changes.aliases), disabled: st.disabled.withAll(changes.disabled), hashed: st.hashed.withAll(changes.hashed), @@ -432,7 +434,7 @@ export class Walker { for (let i = 0; i < statements.length;) { let j = i + 1; while (j < statements.length && statements[j - 1].endPosition.row === statements[j].startPosition.row) j++; - const unit: Target = { kind: 'unit', dead: this.dead, aborts: [], maybes: [] }; + const unit: Target = { kind: 'unit', dead: this.mode.dead, aborts: [], maybes: [] }; this.targets.push(unit); const first = statements[i]; for (const t of statements.slice(i, j)) { @@ -477,16 +479,16 @@ export class Walker { if (st.status === undefined) return false; const errexit = option(st, 'set', 'errexit'); // (Where the status is tested, `set -e` does not stop the shell.) - return st.status === 0 || ((errexit === 'off' || errexit === 'default' || this.quiet > 0) && !this.opaque); + return st.status === 0 || ((errexit === 'off' || errexit === 'default' || this.mode.quiet > 0) && !st.opaque); } /** Runs `nodes` from `st` as code that does not run here: for the states of their nodes; what they change is dropped, and their conditions are not recorded. */ private deadRun(nodes: readonly SyntaxNode[], st: ShellState): void { - this.dead++; + this.mode.dead++; try { this.sequence(nodes, st); } finally { - this.dead--; + this.mode.dead--; } } @@ -500,7 +502,7 @@ export class Walker { jump(kind: 'break' | 'continue' | 'return' | 'exit', count: number | undefined, st: ShellState, sure = true): ShellState { st = withStatus(st, undefined); const collect = (t: { dead: number }, to: ShellState[]): void => { - if (t.dead === this.dead) to.push(st); + if (t.dead === this.mode.dead) to.push(st); }; let maybe = !sure || this.blind; if (kind === 'break' || kind === 'continue') { @@ -574,7 +576,7 @@ export class Walker { return st; } if (t.kind === 'unit') { - if (t.dead === this.dead) (ends ? t.aborts : t.maybes).push(st); + if (t.dead === this.mode.dead) (ends ? t.aborts : t.maybes).push(st); if (ends) return unreachable(st); } } @@ -594,11 +596,11 @@ export class Walker { /** Runs `run` where `set -e` does not stop the shell (a condition, the left side of `&&` or `||`, after `!`). */ private quietly(run: () => T): T { - this.quiet++; + this.mode.quiet++; try { return run(); } finally { - this.quiet--; + this.mode.quiet--; } } @@ -637,22 +639,22 @@ export class Walker { run(n: SyntaxNode, st: ShellState): ShellState { // (After a jump, the rest is dead code: walked from the state there, for the states of its nodes.) if (st.unreachable) { - this.dead++; + this.mode.dead++; try { this.run(n, reached(st)); } finally { - this.dead--; + this.mode.dead--; } return st; } this.record(n, st); // (Arithmetic the walk does not run: a subscript, an offset, `[[ x -eq y ]]`; a key of an array that must be - // associative here, else it is arithmetic too. One that may fail may discard the rest of the line: see `abort`.) - const facts = this.facts(n); - for (const name of facts.arithNames(n) ?? []) this.unsetRead(st, name); - if (facts.mayFail(n)) this.abort(st, false); - for (const name of facts.associative(n) ?? []) { - if (!this.dead && !this.blind && attribute(st, name, 'A') !== 'on') this.walkAgain(`\`${name}\` may not be associative where its subscript is read`); + // associative here, else it is arithmetic too. One that may fail may discard the rest of the line: see `abort`. + // The trust scan read the trees of the strings and of the aliases the walk runs too: `StringTrees`.) + for (const name of this.trust.arithNames(n) ?? []) this.unsetRead(st, name); + if (this.trust.mayFail(n)) this.abort(st, false); + for (const name of this.trust.associative(n) ?? []) { + if (!this.mode.dead && !this.blind && attribute(st, name, 'A') !== 'on') this.walkAgain(`\`${name}\` may not be associative where its subscript is read`); } const outer = this.node; this.node = n; @@ -681,7 +683,7 @@ export class Walker { * surely fails, where its status is not tested, ends the shell (the subshell). */ private errexit(n: SyntaxNode, st: ShellState, after: ShellState): ShellState { - if (this.quiet || !after.status || !ERREXIT_STOPS.has(n.type) || option(st, 'set', 'errexit') !== 'on') return after; + if (this.mode.quiet || !after.status || !ERREXIT_STOPS.has(n.type) || option(st, 'set', 'errexit') !== 'on') return after; // (The last stage of a pipeline: the pipeline's status counts.) if (n.parent?.type === 'pipeline') return after; return this.jump('exit', undefined, after); @@ -1008,7 +1010,7 @@ export class Walker { // (A name the script has not set may come from the environment: by custom, those are in capitals. Once a file // was sourced, a string run with `eval` or a trap set, it may come from there.) if (!v) { - if (/^[A-Z_][A-Z0-9_]*$/.test(target) || this.opaque) return undefined; + if (/^[A-Z_][A-Z0-9_]*$/.test(target) || st.opaque) return undefined; // (Taken as unset here, but not for deciding conditions: see `markAssumed`.) this.markAssumed(); return unset; @@ -1018,7 +1020,7 @@ export class Walker { // (Decisions read only the variables the script cannot change unseen: see src/semantics/trust.ts. A read in dead // code is no reason for a decision.) if (!this.trust.variable(name) || !this.trust.variable(target)) this.markAssumed(); - else if (text !== undefined && !this.dead) for (const r of this.conditions.reading) r.set(name, text); + else if (text !== undefined && !this.mode.dead) for (const r of this.conditions.reading) r.set(name, text); return text; } @@ -1027,7 +1029,7 @@ export class Walker { * environment, as text bash runs as an expression, which may assign any variable: nothing is trusted (see `found`). */ unsetRead(st: ShellState, name: string): void { - if (this.dead || this.blind) return; + if (this.mode.dead || this.blind) return; const target = this.builtins.resolve(st, name) ?? name; if (!st.variables.get(target) && !alwaysNumeric(target)) this.walkAgain(`arithmetic reads \`${target}\`, which the script may not have set`); } @@ -1043,15 +1045,14 @@ export class Walker { if (top) top.assumed = true; } - /** Whether a file was sourced, a string run with `eval` or a trap set: a name the script has not set may be set. */ - private opaque = false; - - /** `opaque` from now on: what was remembered of values read as expressions may be otherwise. */ - turnOpaque(): void { - if (this.opaque) return; - this.opaque = true; - this.arith.arithMemo = new Map(); - this.arith.assignsMemo = new Map(); + /** + * `st` on a path that ran what the walk does not read (`ShellState.opaque`). It counts as a change (`OPAQUE`): what + * may run after it, a body no call reached, a trap's action, a function walked again, may run on such a path too. + */ + opaque(st: ShellState): ShellState { + this.lastChange.set(OPAQUE, ++this.changeCount); + for (const c of this.collecting) c.add(OPAQUE); + return st.opaque ? st : { ...st, opaque: true }; } private pipeline(n: SyntaxNode, st: ShellState): ShellState { @@ -1129,6 +1130,7 @@ export class Walker { // (With the status the passes started from, `$?`, which the body may change too.) const was = { ...st, status: before.entry.status }; for (const key of before.keys) { + if (key === OPAQUE) continue; const kind = key.slice(0, key.indexOf(' ')) as Kind; const name = key.slice(kind.length + 1); (was as Record>)[kind] = (was[kind] as NameMap).with(name, before.entry[kind].get(name)); @@ -1136,7 +1138,7 @@ export class Walker { entry = merge([st, was], this.posix); } // `break` leaves the loop (the states there are merged into the one after it); `continue` ends the pass. - const target: Extract = { kind: 'loop', dead: this.dead, breaks: [], continues: [] }; + const target: Extract = { kind: 'loop', dead: this.mode.dead, breaks: [], continues: [] }; // A condition whose status is known (`holds`) is recorded at each test; one that stops the loop at its first test // makes the body dead. const test = (s: ShellState): { st: ShellState; holds?: boolean } => { @@ -1159,11 +1161,11 @@ export class Walker { for (let i = 0; i < 5; i++) { const c = test(entry); if (i === 0 && c.holds === false) { - this.dead++; + this.mode.dead++; try { pass(c.st); } finally { - this.dead--; + this.mode.dead--; } stopped = c.st; settled = true; @@ -1260,16 +1262,19 @@ export class Walker { } else { const walks = this.walks.get(def) ?? []; if (!this.silent) this.walks.set(def, walks); - // (A walk is reused only as it was made: in dead code or not, where `set -e` stops the shell or not, where an - // error ends a line or a subshell.) - const how = `${!!this.dead} ${!!this.quiet} ${!!this.unitTarget()}`; + const how = this.reuseKey(); const seen = walks.find((w) => w.how === how && (w.entry === st || sameState(w.entry, st))); - // (A walk reused gives again the states where an error in it ended the line: see `abort`.) + // (A walk reused gives again the states where an error in it ended the line, see `abort`, and makes its changes + // again here, for what collects them: a loop, the body that calls it, a trap's action.) const unit = this.unitTarget(); if (seen) { exit = seen.exit; - if (unit && unit.dead === this.dead) unit.aborts.push(...seen.aborts); + if (unit && unit.dead === this.mode.dead) unit.aborts.push(...seen.aborts); for (const m of seen.maybes) this.abort(m, false); + for (const key of seen.changed) { + this.lastChange.set(key, ++this.changeCount); + for (const c of this.collecting) c.add(key); + } } else if (walks.length >= WALKS_PER_FUNCTION) { exit = this.widen(st, this.changes.get(def)!); this.unwalked.add(def); @@ -1277,13 +1282,29 @@ export class Walker { } else { const from = unit?.aborts.length ?? 0; const maybesFrom = unit?.maybes.length ?? 0; - exit = this.walkBody(def, st, { def, saved: new Map() }); - if (!this.silent) walks.push({ entry: st, exit, how, aborts: unit?.aborts.slice(from) ?? [], maybes: unit?.maybes.slice(maybesFrom) ?? [] }); + const changed = new Set(); + this.collecting.push(changed); + try { + exit = this.walkBody(def, st, { def, saved: new Map() }); + } finally { + this.collecting.pop(); + } + if (!this.silent) walks.push({ entry: st, exit, how, aborts: unit?.aborts.slice(from) ?? [], maybes: unit?.maybes.slice(maybesFrom) ?? [], changed }); } } return fn.state === 'on' ? exit : merge([st, exit], this.posix); } + /** + * What a walk of a function body depends on besides its entry state: each mode of the walk (`mode`: in it or not), + * and whether an error ends a line there or a subshell (see `abort`). A walk is reused where these are the same + * (`call`). (A trap's action walked only for what it changes, `silent`, records nothing, and changes nothing a walk + * gives: it reuses the others.) + */ + private reuseKey(): string { + return [...Object.values(this.mode).map((depth) => depth > 0), !!this.unitTarget()].join(' '); + } + /** * A recursive call of `def` with `st`: when errexit there (or why it has its value) is not what the walk of the body * started with, the entry with errexit as on both paths, for `walkBody` to walk again. @@ -1339,7 +1360,7 @@ export class Walker { this.changes.set(def, changes); this.collecting.push(changes); // (`return` ends the body: the states there are merged into the one at its end.) - const target: Target = { kind: 'function', dead: this.dead, returns: [] }; + const target: Target = { kind: 'function', dead: this.mode.dead, returns: [] }; this.targets.push(target); let end: ShellState; try { @@ -1385,7 +1406,7 @@ export class Walker { st = this.redirections(n, this.expandAll(assignments, st), this.runsProgram(n, st)); // A prefix assignment to PATH empties the table of remembered paths too. if (assignments.some((a) => field(a, 'name')!.text === 'PATH')) st = this.builtins.forgetHashed(st); - const expanded = this.aliases.expand(n, name, st); + const expanded = this.aliases.expand(n, st); if (expanded) return expanded; // In POSIX mode, the assignments before a special builtin stay after it, exported. const posix = option(before, 'set', 'posix'); @@ -1439,12 +1460,12 @@ export class Walker { trapAction(script: NestedScript, host: SyntaxNode, st: ShellState, exitOnly: boolean, dead = false): void { this.targets.push({ kind: 'trap', exitOnly }); // (A trap set in code that never runs is never set: its action is walked as dead code.) - if (dead) this.dead++; + if (dead) this.mode.dead++; try { this.runScript(script, host, st); } finally { this.targets.pop(); - if (dead) this.dead--; + if (dead) this.mode.dead--; } } @@ -1453,7 +1474,7 @@ export class Walker { * (the walk is made again, blind). An EXIT trap's action runs as the shell ends: nothing is left to run. */ private escape(t: Extract): void { - if (!t.exitOnly && !this.dead && !this.blind) this.walkAgain("a trap's action may end what a signal interrupts (an error, `return`, `break` or `continue`)"); + if (!t.exitOnly && !this.mode.dead && !this.blind) this.walkAgain("a trap's action may end what a signal interrupts (an error, `return`, `break` or `continue`)"); } /** diff --git a/test/conditions.test.ts b/test/conditions.test.ts index 41787f54..ec7a29fb 100644 --- a/test/conditions.test.ts +++ b/test/conditions.test.ts @@ -157,7 +157,7 @@ describe('the status after a statement against bash (src/semantics/state.ts)', ( // What the model does not follow, or must not decide. ['set -o pipefail; [ -f /nonexistent ] | true', false], ['set -o pipefail; true | true', true], ['if true; then false; elif true; then true; fi', true], ['[() { return 3; }; [ a ]', false], ['set +e; [[ -o errexit ]]', true], ['set -o posix; [ a \\< b ]', false], ['[ -f /nonexistent ] && set -o posix; [ a \\< b ]', false], - ['[[ 1 -eq $nope ]]', false], ["[ $'\\xff' = a ]", false], ["[ $'a\\tb' = a ]", true], ['unset x; [ -z "$x" ]', false], ['x=; [ -z "$x" ]', true], ['x=1 ""', false], + ['[[ 1 -eq $nope ]]', false], ['[[ ~+ == /x ]]', false], ["[ $'\\xff' = a ]", false], ["[ $'a\\tb' = a ]", true], ['unset x; [ -z "$x" ]', false], ['x=; [ -z "$x" ]', true], ['x=1 ""', false], ['x=1 $(:)', false], ['x=1 "" >/dev/null', false], ['v="u + 1"; (( v )); (( v ))', false], ['x=1; eval :; [ "$x" = 1 ]', true], ['x=1; eval "$c"; [ "$x" = 1 ]', false], ['x=1; exec {x}>/dev/null; [ "$x" = 1 ]', false], ['[ "$(echo a)" = a ]', false], ['[ "${#x}" = 0 ]', false], ['[ "a\\b" = "a\\b" ]', true], ['shopt -s nocasematch; [[ A =~ a ]]', false], ["x=$'1\\r'; [[ $x -eq 1 ]]", false], ['x=$\'1\\n\'; [[ $x -eq 1 ]]', false], ['[[ $\'1\\n\' -eq 1 ]]', true], ['[ @(a) = a ]', false], diff --git a/test/fixtures/snippets.json b/test/fixtures/snippets.json index a729bb76..d9c99f38 100644 --- a/test/fixtures/snippets.json +++ b/test/fixtures/snippets.json @@ -1057,6 +1057,7 @@ "x=1 command bash -c ':'", "x=1; command \"$c\" read x; [ \"$x\" = 1 ] && :", "x=1; eval $'y=2\n(\nz=3'", + "x=1; f() { [ \"$x\" = 1 ] && :; }; if false; then f; fi; f", "x=1; y=2; if [ \"$x\" = 1 ] || [ \"$y\" = 2 ]; then :; fi", "x=5; trap \"trap 'y=\\$x' USR1\" INT\ntrap 'z=2' USR1; z=3 && echo \"$z\"", "x=`eval 'a\\$b'`", diff --git a/test/state.test.ts b/test/state.test.ts index 87662255..4463af93 100644 --- a/test/state.test.ts +++ b/test/state.test.ts @@ -288,6 +288,16 @@ describe('shell state: function bodies run at their calls', () => { const model = shellState(parser.parse('g() { :; }; f() { :; }; f; export -fn g; f')); assert.deepStrictEqual([model.end.functions.get('g')?.exported, model.end.functions.get('g')?.readonly], ['off', 'off']); }); + it('a walk reused makes its changes again: a trap that calls the function may change what it sets after the `trap`', () => { + // (The trap's action calls `f` from the state the first call had: it reuses that walk. Before, it collected nothing, + // and `x` was 1 after the `trap`, where bash has 2 once the signal came.) + const src = 'f() { x=2; }; x=1; f; x=1; trap f USR1; kill -USR1 $$; echo "$x"'; + assert.strictEqual(spawnSync('bash', ['-c', src], { encoding: 'utf8', env: { PATH: '/usr/bin:/bin' }, timeout: 5000 }).stdout, '2\n'); + assert.deepStrictEqual([variable(end(src), 'x')?.set, variable(end(src), 'x')?.value], ['unknown', undefined]); + // (A walk is reused in the same modes only: here in dead code, where it records no decision, then where it runs.) + const tree = parser.parse('x=1; f() { [ "$x" = 1 ] && :; }; if false; then f; fi; f'); + assert.strictEqual(shellState(tree).condition(descendants(tree.root, ['list']).find((l) => l.text.startsWith('['))!)?.outcome, 'true'); + }); it('a body has the state of its call: functions defined and options set after the definition count', () => { const src = 'main() { setup; ls; }\nsetup() { shopt -s nullglob; }\nmain'; @@ -503,6 +513,23 @@ describe('shell state: arithmetic', () => { // A value read before is not taken again for one that may have changed. assert.deepStrictEqual(after('x=\'foo + 1\'; (( a = x )); source ./c.sh; (( b = x ))', 'a', 'b'), ['set=1', 'unknown']); }); + it('says so on the paths that ran what the walk does not read, and after they join', () => { + // (bash: c.sh sets foo; the branch that does not source it reads foo unset; after the `if`, either may have run.) + const src = 'if [ "$1" ]; then source ./c.sh; else (( a = foo + 1 )); : "$a"; fi; (( b = foo + 1 )); echo "$a $b"'; + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'opaque-')); + try { + fs.writeFileSync(path.join(dir, 'c.sh'), 'foo=3\n'); + fs.writeFileSync(path.join(dir, 's.sh'), src); + const bash = (arg: string): string => spawnSync('bash', ['s.sh', arg], { cwd: dir, encoding: 'utf8', env: { PATH: '/usr/bin:/bin' }, timeout: 5000 }).stdout; + assert.deepStrictEqual([bash(''), bash('x')], ['1 1\n', ' 4\n']); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + assert.strictEqual(variable(before(src, ': "$a"'), 'a')?.value, '1'); + assert.deepStrictEqual(after(src, 'b'), ['unknown']); + // (A loop walked again in another loop, whose pass sources a file: the path is so from its start.) + assert.deepStrictEqual(after('for i in 1 2; do for j in 1 2; do (( b = foo + 1 )); . ./c.sh; done; done', 'b'), ['unknown']); + }); it('reads ++ and -- as bash does', () => { // After a name, `x++`; before one, `++x`; elsewhere, a sign (bash: y=8 z=8 w=8 v=4, 4 and x=3). assert.deepStrictEqual(after('x=-3; (( y = 5-$x ))', 'y'), ['set=8']); diff --git a/test/trust.test.ts b/test/trust.test.ts index ef99f0dd..8f0c866f 100644 --- a/test/trust.test.ts +++ b/test/trust.test.ts @@ -45,6 +45,45 @@ describe('the trust of the decisions (src/semantics/trust.ts)', () => { assert.strictEqual(decided(`${body}; eval f`), 'true'); assert.strictEqual(decided(`${body}; eval 'eval f'`), 'true'); for (const more of ['trap f INT', "eval 'trap f INT'", "trap 'eval f' INT", "shopt -s expand_aliases; alias g='f'"]) assert.strictEqual(decided(`${body}; ${more}`), undefined, more); + // (An alias named with a character the walk does not take: bash expands it, `g,h` runs `f`; its text is read alone.) + assert.strictEqual(bash(`${body}; f() { echo F; }; shopt -s expand_aliases; alias 'g,h=f'\ng,h`), 'F\n'); + assert.strictEqual(decided(`${body}; shopt -s expand_aliases; alias 'g,h=f'`), undefined); + }); + + it("reads the aliases the walk expands from the walk's own trees: the alias's text with the rest of the command", function () { + this.timeout(60000); + // (The key of an associative array is not arithmetic, also in an alias's text: `k` is not read. Before, that tree + // was read on its own, without `declare -A m`, and nothing was trusted.) + const src = "declare -A m; shopt -s expand_aliases; alias put='m[k]=1'\nput"; + assert.strictEqual(bash(`k='x=5'; x=1; ${src}\necho "\${!m[@]} $x"`), 'k 1\n'); + assert.strictEqual(decided(`${src}\nx=1; [ "$x" = 1 ] && :`), 'true'); + // (Each alias whose text names another doubles the trees to read: past a bound, nothing is trusted.) + const many = ['shopt -s expand_aliases', ...Array.from({ length: 11 }, (_, i) => `alias a${i}='a${i + 1}; a${i + 1}'`), 'a0', 'x=1; [ "$x" = 1 ] && :'].join('\n'); + assert.strictEqual(shellState(parser.parse(many)).blind, 'too many aliases expanded in aliases'); + // (The bound is per command of the script: many plain uses of aliases, even of one that names another, are read. + // Before, every use counted, and such a script trusted nothing.) + const defs = "alias ls='ls -F'; alias ll='ls -l'"; + const uses = (on: boolean, lines: string[]): string => [...(on ? ['shopt -s expand_aliases'] : []), defs, ...lines, 'x=1; [ "$x" = 1 ] && :'].join('\n'); + assert.strictEqual(bash(`x=1; ${uses(true, Array(3).fill('ll / >/dev/null'))}\necho $x`), '1\n'); + for (const on of [false, true]) { + assert.strictEqual(decided(uses(on, Array(1200).fill('ll /'))), 'true', `1,200 uses, expand_aliases ${on}`); + const forty = Array.from({ length: 40 }, (_, i) => `alias a${i}='echo ${i}'`); + assert.strictEqual(decided(uses(on, [...forty, ...Array.from({ length: 1200 }, (_, i) => `a${i % 40}`)])), 'true', `40 aliases, 30 uses each, expand_aliases ${on}`); + } + // (Nor do the trees of all the commands together grow without bound: 100 commands of a chain that doubles 8 times + // ask for 51,100. Before, 500 such commands of 9 made V8 run out of memory. Past the bound the walk still runs what + // the aliases do, in trees shared by text, and nothing is trusted.) + const chain = ['shopt -s expand_aliases', ...Array.from({ length: 8 }, (_, i) => `alias a${i}='a${i + 1}; a${i + 1}'`), "alias a8='set -f; y=2'", 'y=1', ...Array(100).fill('a0'), 'x=1; [ "$x" = 1 ] && :'].join('\n'); + assert.strictEqual(bash(`${chain}\necho "$y \${-//[^f]/}"`), '2 f\n'); + const heap = process.memoryUsage().heapUsed; + const t = Date.now(); + const tree = parser.parse(chain); + const model = shellState(tree); + assert.strictEqual(model.blind, 'too many aliases expanded'); + assert.strictEqual(model.condition(descendants(tree.root, ['list']).pop()!), undefined); + assert.deepStrictEqual([model.end.variables.get('y')?.value, model.end.set.get('noglob')], ['2', 'on']); + assert.ok(Date.now() - t < 30000, `took ${Date.now() - t} ms`); + assert.ok(process.memoryUsage().heapUsed - heap < 1e9, `${process.memoryUsage().heapUsed - heap} bytes`); }); it("reads bash's own variables from the catalog (src/semantics/vars.ts), as bash treats them", () => { // Arithmetic that reads a variable whose value is always a number runs no code: decisions stay.